Diagnosing Common NixOS Systemd Service Failures: A Practical Guide

Recognizable Symptom

A NixOS service that never starts and shows failed in systemctl status is a frequent pain point. Typical journal entries look like:

systemd[1]: Failed to start myservice.
systemd[1]: ExecStart=/run/current-system/sw/bin/myservice: No such file or directory

or:

systemd[1]: myservice.service: Failed with result 'exit-code'.
systemd[1]: myservice.service: Main process exited, code=exited, status=1

These lines hint at a mis‑configured unit file, missing binary, permission issue, or missing environment variable.

Cause & Diagnostic Table

Potential Cause	Diagnostic Indicator
Missing or incorrect ExecStart	Journal shows “No such file or directory” or non‑zero exit code.
File system permissions too restrictive	Journal logs “Permission denied” or “Operation not permitted”.
Required environment variables unset	Service crashes with a “undefined variable” error or similar stack trace.
Wrong ServiceType for daemon	systemd reports “service dead” immediately after start.
RuntimeDir or socket missing / mis‑owned	Start fails with “No such file or directory” or “Permission denied” on socket paths.

Ordered Checks & Fixes

1. Verify service status and logs

   # systemctl status myservice.service
   # journalctl -u myservice.service
   

   Look for the exact error message that matches one of the table entries.

2. Validate unit syntax and attributes

   # systemd-analyze verify /etc/systemd/system/myservice.service
   

   Any syntax error or unsupported option will be reported here.

3. Check ExecStart path and permissions

   # ls -l /run/current-system/sw/bin/myservice
   

   Ensure the binary exists and is readable/executable by the service’s user. On NixOS, binaries live under the current system store; avoid manual permission changes – instead, set runAs or serviceConfig.User in configuration.nix.

4. Confirm required environment variables

   # systemctl show -p Environment myservice.service
   

   If a variable is missing, add it to the unit file or to configuration.nix:

   services.myservice = {
     enable = true;
     environment = {
       MY_VAR = "value";
     };
   };
   

   After editing configuration.nix, rebuild:

   # nixos-rebuild switch
   

5. Adjust ServiceType for forking daemons

   Many daemons start a child and exit immediately. If ServiceType=simple is used, systemd will think the service died. Change to forking or notify as appropriate.

   ExecStart=/run/current-system/sw/bin/mydaemon
   ServiceType=forking
   

6. Verify RuntimeDir and socket ownership

   # ls -ld /var/run/myservice
   # stat -c "%U %G" /var/run/myservice
   

   Runtime directories should be owned by the service user. If missing, create and set ownership:

   # mkdir -p /var/run/myservice
   # chown myservice:myservice /var/run/myservice
   

   Again, prefer declarative creation via services.myservice.runtimeDir = "/var/run/myservice"; in configuration.nix.

7. Re‑enable and restart the service

   # systemctl daemon-reload
   # systemctl restart myservice.service
   # systemctl status myservice.service
   

   Check that the status is now active (running) and that no errors appear in the journal.

Escalation Criteria

• If the service still fails after all checks, verify that the binary itself is functional outside systemd: run it manually as the service user.
• Check for SELinux or AppArmor denials that may appear in /var/log/audit/audit.log or /var/log/kern.log.
• Consult the service’s upstream documentation for any additional runtime requirements (e.g., network sockets, external APIs).
• If the issue persists, consider opening an issue on the NixOS GitHub or seeking help in the NixOS IRC/Matrix channels with the exact error logs.

Practical Verification Checklist

# systemctl status myservice.service | grep -E 'active|failed'
# journalctl -u myservice.service --since '10 minutes ago'
# systemd-analyze verify /etc/systemd/system/myservice.service
# nixos-rebuild test   # to see if changes would apply

Use nixos-rebuild test to preview the effect of configuration.nix changes without rebooting.