Despite my best efforts, I end up logging into Linux servers and checking how they’re doing way too often. In recent projects I’ve also been responsible for adding way too many .service files and making sure they work. All of that was based on a vague understanding of how systemd works, without ever really looking into the finer details and the constant feeling that there are a lot of features I was missing. With this post, I want to explore some of these aspects and figure out how to write better daemons. See it as a set of notes on the things I want to at in more depth.
Proper status updates
Services can let systemd know what their internal state is, and to have it restart them when they don’t respond.
As far as I can tell,
by default, systemd services are of type
and are assumed to be ready immediately after they are started.
This might not be the case for a program that requires some time to start up,
or that waits for a connection to be initialized.
To allow the service to notify systemd of its state,
you need to set the type to
To tell systemd that the service is ready, you trigger a notification. The easiest to test with is:
In a real service,
you’d call sd_notify (or rust-notify in Rust)
with a string starting with
This string can contain various newline delimited values,
as described in the Description section here.
You can, for example, append
STATUS=Good to go!.
will tell systemd that the service is reloading or exiting respectively.
The same notification system can also be used to
let systemd make sure your service is doing fine.
Specifically, by adding something like
systemd will expect you to send
less then every 5 seconds.
You can define the sockets your services will consume and let systemd manage them for you. The advantages are:
- Your socket will stay alive even through service restarts (if I understand correctly)
- You can set your service to only start once there is traffic on the socket (“socket activation”, also used by macOS’ launchd for example)
By default, your services run in an environment similar to just executing them with bash as the correct user. That is convenient to get stuff running, but might be a bit much if you’re security conscious. They are, however, a bunch of neat things you can set to limit what your process can do. Here’s a few examples:
[Service] PrivateTmp=yes InaccessibleDirectories=/home ReadOnlyDirectories=/var CapabilityBoundingSet=~CAP_SYS_PTRACE DeviceAllow=/dev/null rw TemporaryFileSystem=/var:ro BindReadOnlyPaths=/var/foo/data