Diagnosing and Fixing Systemd Service Failures from Unit File Misconfigurations

Recognizable Symptoms

A service that fails to start often shows the same quick‑fire pattern:

• systemctl status myservice reports Active: failed (Result: exit-code) with ExitCode=1 or 255.
• The status output contains a short message like Unit file … is missing or invalid or Failed to start …: ExecStart=… failed with exit code 1.
• Running journalctl -u myservice reveals lines such as ConditionFileNotEmpty=…: false or ConditionPathExists=…: false.

Common Causes

Cause	Typical Indicator
Condition directives evaluate to false	ConditionFileNotEmpty, ConditionPathExists, etc. show "false" in journal
ExecStart path is wrong or the binary is missing	"Failed to start …: ExecStart=… failed with exit code 1" or "no such file or directory"
Executable lacks execute permission or wrong owner	Permission denied errors in journal; ls‑l shows no x bit
Unit file permissions prevent systemd from reading it	systemd reports unit file not readable, or status shows "Invalid unit file"
Security options block required files (e.g., PrivateTmp, ProtectSystem)	Permission errors for files the service needs

Diagnostic Checklist

1. Check the status
   

   systemctl status myservice

   Verify Active: and ExitCode.
2. View recent logs
   

   journalctl -u myservice --since today

   Look for Condition failures, ExecStart errors, or permission messages.
3. Inspect the unit file
   

   systemctl cat myservice

   Confirm ExecStart path, Condition lines, and security options.
4. Verify file existence and permissions
   

   ls -l /etc/systemd/system/myservice.service
   ls -l /usr/bin/myservice
   

   Ensure the unit file is readable by root and the executable is executable.
5. Test condition paths manually
   If ConditionPathExists=/var/run/myservice.sock, check:

   test -e /var/run/myservice.sock && echo exists || echo missing

6. Check security context
   If ProtectSystem=full is set, the service cannot write to /etc. Verify required files are in allowed directories.

Fixes Tied to Findings

• False Condition – Edit the unit file to correct the path or remove the directive if unnecessary. Example:

  [Unit]
  ConditionPathExists=/etc/myapp/config.yaml
  

  If the file is missing, create it or adjust the condition.
• Bad ExecStart – Ensure the path points to an existing, executable binary. Add the full path or use $PATH carefully.

  [Service]
  ExecStart=/usr/bin/myapp --config /etc/myapp/config.yaml
  

• Permission issues – Set correct ownership and mode:

  chown root:root /usr/bin/myapp
  chmod 755 /usr/bin/myapp
  

  For the unit file, chmod 644 /etc/systemd/system/myservice.service.
• Security options blocking access – Adjust PrivateTmp=yes, ProtectSystem=full, or ReadOnlyDirectories=/etc to allow the service to touch required files. Alternatively, move necessary files into /var/tmp or a directory listed in ReadWriteDirectories=.
• Unit file cache – After changes, run:

  systemctl daemon-reload
  systemctl restart myservice
  

  Then re‑check status.

Escalation Criteria

If after applying the fixes the service still fails:

• Check for hidden dependencies: Requires= or After= directives that might be pulling in other units that fail.
• Inspect SELinux/AppArmor logs if enabled; a policy denial can masquerade as a unit failure.
• Verify that the system has sufficient resources (memory, disk, inodes) and that no cgroup limits are preventing start.
• If the unit uses templating (e.g., myservice@.service), confirm the instance name matches the expected configuration.

Practical Example

Suppose myapp.service fails with ExitCode=1. The unit file contains:

[Unit]
Description=My App
ConditionFileNotEmpty=/etc/myapp/config.yaml

[Service]
ExecStart=/usr/local/bin/myapp
User=nobody

[Install]
WantedBy=multi-user.target

1. Run systemctl status myapp → shows ConditionFileNotEmpty=…: false.
2. Check the file: ls -l /etc/myapp/config.yaml → file missing.
3. Create the file or remove the condition if the app can run without it.
4. Reload and restart: systemctl daemon-reload && systemctl restart myapp.
5. Verify with systemctl status myapp → now Active: active (running).

Verification Checklist

• Confirm systemctl status myservice shows Active: active (running) and ExitCode=0.
• Review journalctl -u myservice --since today for no error lines.
• Run systemctl show myservice to ensure the active unit file path matches the edited one.
• If the service writes logs to a file, check that the file exists and is writable.

Limitations and Caveats

• Systemd can cache unit files; stale configurations may persist until daemon-reload is executed.
• Changing security options can expose the system to new attack vectors; apply the minimal necessary permissions.
• When using templated units, instance names must match the file name pattern; mismatches silently result in a stopped unit.
• Some failures may be caused by external factors (e.g., disk space, memory limits) that are not evident in the unit file alone.

Practical Check

After making a change, run:

systemctl status myservice
journalctl -u myservice --since now | tail -n 20

These commands provide a quick snapshot of the current state and recent events.