Systemd

Todo/To read:

Shutdown and Reboot

With systemd:

sudo systemctl poweroff
sudo systemctl reboot

Without systemd:

poweroff
reboot
shutdown -r 2  # schedule reboot at 2 minutes

Daemon Management

SysV init style scripts:

ls /etc/init.d/

sudo systemctl [enable|disable] name.service
sudo systemctl [start|stop|restart|reload|reload-or-restart] name.service
systemctl status [-l|--full] [name.service|pid]
systemctl list-unit-files --type service
systemctl [is-active|is-enabled|is-failed] name.service
# Masked service is unable to start - even as a dependency (somewhat dangerous)
sudo systemctl [mask|unmask] name.service
# Show documentation
systemctl help name.service
# Enable and start immediately
sudo systemctl enable --now unit

Reset failed statuses:

sudo systemctl reset-failed

Time and Locales

Time

Sources:

Set timezone:

timedatectl  # show current setup
timedatectl list-timezones
sudo timedatectl set-timezone Europe/Moscow

Setup NTP servers (sudo vim /etc/systemd/timesyncd.conf):

[Time]
NTP=0.ru.pool.ntp.org 1.ru.pool.ntp.org 2.ru.pool.ntp.org 3.ru.pool.ntp.org
FallbackNTP=ntp.ubuntu.com time.cloudflare.com

Enable NTP:

sudo timedatectl set-ntp true
sudo systemctl restart systemd-timesyncd.service
timedatectl status  # --> System clock synchronized: yes

Locales

localectl list-locales
localectl status
localectl set-locale LANG=en_US.UTF-8

Targets and Runlevels

Sysv runlevels vs systemd targets:

Targets Units                        Runlevel Description
runlevel0.target, poweroff.target    Shut down and power off the system.
runlevel1.target, rescue.target      Set up a rescue shell.
runlevel2.target, multi-user.target  Set up a non-graphical multi-user system.
runlevel3.target, multi-user.target  Set up a non-graphical multi-user system.
runlevel4.target, multi-user.target  Set up a non-graphical multi-user system.
runlevel5.target, graphical.target   Set up a graphical multi-user system.
runlevel6.target, reboot.target      Shut down and reboot the system.

Default runlevel:

systemctl get-default  # --> graphical.target
sudo systemctl set-default multi-user.target

Another option is adding systemd.unit=multi-user.target or similar to kernel parameters of the bootloader.

Switch to certain runlevel:

sudo systemctl isolate multi-user.target  # go to text mode

Unit files

Edit Systemd Files

# Change file in-place
sudo vim /etc/systemd/system/ufw.service
# Put only section and options that should be changed
sudo systemctl edit ufw.service
# Pull a new copy of entire file
sudo systemctl edit --full ufw.service

Drop changes:

sudo systemctl revert ufw.service

Show all files related to service:

systemctl cat ufw.service

Example: changing ExecStart:

[Service]
ExecStart=
ExecStart=new cmd

Note: all commands that take multiple commands need to be cleaned up as shown.

Unit Files

List paths being looked up:

systemctl show --property=UnitPath

main ones being:

  • /usr/lib/systemd/system/: units by packages

  • /etc/systemd/system/: units by system administrator

Unit file example:

[Unit]
Description=OpenBSD Secure Shell server
# Documentation=man:sshd(8)  # or https:://...
# Before=
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
# User=
# Environment=VAR1=1 VAR2=2 VAR3=3
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755
# PrivateNetwork=yes
# PrivateTmp=yes

[Install]
WantedBy=multi-user.target
# RequiredBy=  # if service fails, target would not boot
Alias=sshd.service

After applying changes or creating new unit:

sudo systemctl daemon-reload

Instances

If syncthing@.service then it is not callable by itself, but you should call it as, for example, syncthing@lain.service. Then %i in the unit file will be substituted with lain:

...
[Service]
User=%i
ExecStart=/usr/bin/syncthing -no-browser -no-restart -logflags=0
...

Dependencies

systemctl list-dependencies sshd.service
systemctl list-dependencies sshd.service --all  # list recursively
systemctl list-dependencies sshd.service --reverse  # show who depends on it

Unit Properties

Show list of properties:

systemctl show sshd.service

Display a single property:

systemctl show sshd.service -p Conflicts

Static files (the ones that a launched dynamically)

Example of micro-httpd:

/lib/systemd/system/micro-httpd@.service
[Unit]
Description=micro-httpd
Documentation=man:micro-httpd(8)

[Service]
User=nobody
Group=www-data
ExecStart=-/usr/sbin/micro-httpd /var/www/html
StandardInput=socket
/lib/systemd/system/micro-httpd.socket
[Unit]
Description=micro-httpd
Documentation=man:micro-httpd(8)

[Socket]
ListenStream=0.0.0.0:80
Accept=true

[Install]
WantedBy=sockets.target

Two types of socket activation:

  1. Accept=yes

    • a single instance of the service is started for each connection

    • “wait” under inetd/xinetd

  2. Accept=no

    • a single instance of the service is started for each connection

    • “nowait” under inetd/xinetd

To disable: edit --full and Accept=false and daemon-reload.

Temporary file units

Source: Managing temporary files with systemd-tmpfiles on Red Hat Enterprise Linux 7

Explore with:

systemctl status systemd-tmpfiles-setup.service
cat /lib/systemd/system/systemd-tmpfiles-setup.service
systemctl status systemd-tmpfiles-clean.timer
man 5 tmpfiles.d
systemd-tmpfiles [command]

systemd-tmpfiles directories (highest to lowest):

  • /etc/tmpfiles.d/*.conf

  • /run/tmpfiles.d/*.conf

  • /usr/lib/tmpfiles.d/*.conf

Configuration file syntax of systemd-tmpfiles:

Type, Path, Mode, UID, GID, Age, and Arguments

Example: samba created /run/samba/ for it’s daemon with /usr/lib/tmpfiles.d/samba.conf setting:

D /run/samba 0755 root root

Another directory creation example (cat /usr/lib/tmpfiles.d/httpd.conf):

d /run/httpd 710 root apache
d /run/httpd/htcacheclean 700 apache apache

File removal example (/usr/lib/tmpfiles.d/rpm.conf):

r /var/lib/rpm/__db.*

Symlink creation and recursive file copy (/usr/lib/tmpfiles.d/etc.conf):

L /etc/os-release - - - - ../usr/lib/os-release
L /etc/localtime - - - - ../usr/share/zoneinfo/UTC
L+ /etc/mtab - - - - ../proc/self/mounts
C /etc/nsswitch.conf - - - -
C /etc/pam.d - - - -

The L+ creates link forcefully.

C means copy file/directory from /usr/share/factory/.

Time option usage (/usr/lib/tmpfiles.d/cups.conf):

d /run/cups 0755 root lp -
d /run/cups/certs 0511 lp sys -

d /var/spool/cups/tmp - - - 30d

Files older (both atime, mtime and ctime) than 30 days on /var/spool/cups/tmp will be deleted.

Timer units

Source: Arch wiki systemd/Timers

Check with:

systemctl list-timers [--all]
man 5 systemd.timer

Timer units:

  • Realtime: much as cron; activate on calendar event (OnCalendar=)

  • Monotonic: activate after a time span relative to a starting point (have format On[Type]Sec=)

A timer which will start 15 minutes after boot and again every week while the system is running (/etc/systemd/system/foo.timer):

[Unit]
Description=Run foo weekly and on boot

[Timer]
OnBootSec=15min
OnUnitActiveSec=1w

[Install]
WantedBy=timers.target

Starts once a week (at 12:00am on Monday). When activated, it triggers the service immediately if it missed the last start time (option Persistent=true), for example due to the system being powered off:

...
[Timer]
OnCalendar=weekly
Persistent=true
...

OnCalendar uses the format:

DayOfWeek Year-Month-Day Hour:Minute:Second

Ex.:

OnCalendar=Mon,Tue *-*-01..04 12:00:00
# or
OnCalendar=Mon..Fri 22:30
OnCalendar=Sat,Sun 20:00

Run systemd-analyze calendar to check expression validity:

systemd-analyze calendar weekly  # ``daily``, ``weekly`` etc refer to specific time
systemd-analyze calendar "Mon,Tue *-*-01..04 12:00:00"

Use RandomizedDelaySec in [Timer] section to avoid multiple (for example, weekly) services start simultaneously.

Use AccuracySec=1us to the [Timer]`` section, to avoid the inaccuracy of the 1m by default.

Transient .timer units:

# Touch file after 30sec
sudo systemd-run --on-active=30 /bin/touch /tmp/foo
# Can also be combined with existing unit
sudo systemd-run --on-active="12h 30m" --unit someunit.service

Networking

By default Ubuntu uses netplan /etc/netplan/*.yaml files for configuration, which uses systemd-networkd as a backend (NetworkManager is another option).

Hostaname

Show:

hostname  # --> u-navi
hostnamectl  # show names and some server info

Set:

echo u-navi | sudo tee /etc/hostname  # Would not last after reboot
sudo hostnamectl set-hostname u-navi  # The right way

Substitute netplan with plain networkd

Source: Disable netplan.io and use native systemd-networkd on ubuntu 18.04.

Remove netplan and enable networkd:

sudo apt remove netplan
sudo systemctl enable systemd-networkd

Create /etc/systemd/network/99-wildcard.network:

[Match]
Name=en*

[Network]
DHCP=ipv4
#DHCP=yes
#IPv6AcceptRA=yes

Edit systemd-networkd with the following to enable debug logging:

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

Reload the service and watch it.

Reboot.

Check:

networkctl  # --> ... ens3 ether routable configured ...
networkctl status ens3

Wait for enp0s3 only

Source: [Задержка при загрузке Ubuntu Server](https://tokmakov.msk.ru/blog/item/494)

Create an override:

sudo systemctl edit systemd-networkd-wait-online.service

with the folllowing content:

[Service]
ExecStart=
ExecStart=/lib/systemd/systemd-networkd-wait-online --interface=enp0s3

Reload systemd:

sudo systemctl daemon-reload

Journaling

Sources:

  • Journald stores data in /var/log/journal/ in binary form (structured logging)

  • The service just writes everything to stdout/stderr and it is being logged by systemd

  • Unlike syslog, journald stores timezones

  • Indexing data (faster lookup than syslog)

  • Access control

  • Automatic log rotation

Settings

  • Edit /etc/systemd/journald.conf

  • sudo systemctl reload systemd-journald.service

Previous sessions preview

Needs to be activated (it is not - by default - on some distros):

/etc/systemd/journald.conf
[Journal]
Storage=persistent

(or Storage=auto and create /var/run/journal directory).

List previous boots (sessions):

journalctl --list-boots

Show log for certain boot:

journalctl -b 0
# or...
journalctl -b 323000d9802642a098d47ae1b7384424

CLI

Basic examples:

journalctl     # all logs
journalctl -b  # all logs since last boot
journalctl -r  # list entries in reverse order

Filtering examples:

# Filter by date
journalctl --since "2015-07-20 17:15:00"
journalctl ---since yesterday
journalctl --since 09:00 --until now
journalctl --since 10:00 --until "1 hour ago"
# Filter by service
journalctl _SYSTEMD_UNIT=nginx.service
journalctl -u nginx.service  # shorter version
journalctl -u nginx.service --since yesterday
journalctl -u nginx.service -u php-fpm.service --since today
# Filter by process, user and group
journalctl _PID=381
journalctl _UID=33
# FIlter by executable's path
journalctl _EXE=/usr/bin/sudo
journalctl -F _UID  # List all possible values for field _UID
journalctl -F _GUID
# Other possible fields
man systemd.journal-fields
# Filter by path
journalctl /usr/bin/docker
# Kernel messages
journalctl -k
journalctl --dmesg
journalctl -k -b -2  # Kernel messages for certain previous boot
# Recent events
journalctl -n     # Latest 10 events
journalctl -n 20  # Latest 20 events
# Grep through logs
journalctl --grep 'Network Name Lookups'
# Filter by error level
journalctl --priority err

Dumping everything to terminal:

journalctl --no-pager
# or...
SYSTEMD_PAGER=cat journalctl

Follow mode:

journalctl -f

Formatting/exporting/outputtig

Possible output formats:

  • cat – only messages

  • export – binary format (for export or backup);

  • short – syslog format;

  • short-iso – syslog format with ISO 8601 timestamps;

  • short-monotonic – syslog format with monotonic timestamps;

  • short-precise – syslog format with precise (milliseconds) timestamps;

  • verbose – most verbose format (with all fields).

Example:

journalctl -u nginx.service -o json-pretty

Log rotation

# Show disk space used  by logs
journalctl --disk-usage
# Rotate
sudo journalctl --vacuum-size=1G
sudo journalctl --vacuum-time=1years

Set rotation through /еtc/systemd/journald.conf file:

  • SystemMaxUse= max logs total size;

  • SystemKeepFree= keep at least this much free space;

  • SystemMaxFileSize= max size of log file;

  • RuntimeMaxUse= max logs total size in /run;

  • RuntimeKeepFree= keep at least this much free space in /run;

  • RuntimeMaxFileSize= max size of log file in /run.

Containers

Docker

docker run --rm --name my-pg \
           -e POSTGRES_PASSWORD=123 \
           --log-driver=journald \
           -d postgres

journalctl CONTAINER_NAME=my-pg --all

To log by default:

/etc/docker/daemon.json
{
 "log-driver": "journald"
}

sudo systemctl restart docker

Systemd-nspawn

journalctl --machine $MACHINE_NAME

Centralized logs storage

Systemd uses following components for this task (not present on some distros):

  • systemd-journal-remote

  • systemd-journal-upload

  • systemd-journal-gatewayd

Snoopy

Used:

Install with sudo apt install snoopy and choose to insert line into /etc/ld.so.preload. This setting can be changed with sudo dpkg-reconfigure snoopy.

Note: it made my browsers (chromium, firefox) to misbehave:

/etc/ld.so.preload
/lib/x86_64-linux-gnu/libsnoopy.so

Settings are stored at /etc/snoopy.ini. Drop excessive information from logs by configuring snoopy with following configuration:

[snoopy]
; Log line format
message_format = "[login: %{login}, %{tty_username} as %{eusername}]: %{cmdline}"

; Events filter
filter_chain = "only_tty"

Look log with journalctl -xe -t snoopy.

Systemd-nspawn

Create debian machine file structure:

sudo debootstrap --arch=amd64 jessie /var/lib/machines/container1/

Enter the chroot and set root password:

sudo systemd-nspawn -D /var/lib/machines/container1/ --machine test_container
passwd

Leave with ^[[[.

sudo machinectl start container1

Capabilities

Sources:

Run command with systemd and show it’s output with journalctl:

sudo systemd-run whoami  # --> Running as unit: run-xxxx.service
journalctl -u run-xxxx.service  # --> ... whoami[28327]: root ...

Simpler one: run with tty connected:

sudo systemd-run -t whoami  # --> ... ... root

Systemd creates temporary unit file on the fly.

Specify user:

sudo systemd-run -t --uid=nobody whoami
sudo systemd-run -t -p User=nobody whoami
sudo systemd-run -t -p DynamicUser=yes whoami  # create temporary user on the fly

Let’s try ping localhost interactively:

sudo systemd-run -t -p IPAddressDeny=127.0.0.1/8 /bin/bash
# ping 127.0.0.1
# -->
# PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
# ping: sendmsg: Operation not permitted

Open shell:

sudo systemd-run -p ProtectHome=yes --shell

Options:

ProtectHome=yes|read-only|no  # /home inaccessible or read-only
    # yes: /home and /root are gone
ProtectSystem=yes|full|strict|no  # full: [/usr, /boot, /etc] are read-only
 #   (use ReadWritePaths to punch holes)
 # yes: [/usr, /boot] read-only
 # strict: all read-only except [/dev, /proc, /sys]
PrivateTmp=yes|no  # tied to lifecycle of service (/tmp and /var/tmp)
    # debuggable through /tmp/systemd-private-<random-suffix>-<name.service>-<random>/tmp
PrivateNetwork=yes|no  # no network access (only lo device)
PrivateDevices=yes|no  # isolate /dev except some commons (only includes pseudo-devices)
    # useful when service doesn't need access to raw devices

JoinsNamespaceOf=  # join namespaces with other service (ex.: they will share same /tmp)
    # use case: MySQL (and many other) puts it's listening socket to /tmp
CapabilityBoundingSet=  # makes use of unix capabilities (root privileges slitted to bits)
    # limits capabilities the process is allowed to obtain. It doesn't grant any
    # Example: CapabilityBoundingSet=CAP_SYS_TIME for NTP client
    # (won't be able to do nothing privileged except set the time)
AmbientCapabilities=  # grants capabilities that the process normally wouldn't have started with
MountFlags=slave  # no able to change mount table

InaccessiblePaths=
ReadOnlyPaths=
ReadWritePaths=
BindPaths=
ReadOnlyBindPaths=

StateDirectory=          # /run/foo/
CacheDirectory=          # /var/lib/foo/
LogsDirectory=           # /var/cache/foo/
RuntimeDirectory=        # /var/log/foo/
ConfigurationDirectory=  # /etc/foo/

User=
Group=
SupplementaryGroups=
LimitNPROC=  # for fork() protection
RootDirectory=  # a bit like chroot (advanced topic)
LimitFSIZE=0  # limit file size the service can create (here: can't create any files)
DeviceAllow=/dev/sda5 rwm  # get access to no devices but sda5
    # DeviceAllow=char-alsa rw  # can access only audio
TasksMax=  # against fork bomb (advices); tasks=processes+threads
    # default: 512

# Low level stuff
MemoryDenyWriteExecute=yes  # disable ability to create executable-writable mappings
NoNewPrivileges=yes  # no further uid and gid changes
RestrictAddressFamilies=AF_UNIX|AF_INET|AF_INET6|AF_CAN|AF_APPLETALK
RestrictSUIDSGID=yes
ProtectKernelTunables=yes
ProtectHostname=yes
ProtectKernelLogs=yes
SystemCallArchitecture=native|x86_64|i386|...
SystemCallFilter=
LockPersonality=yes

SyscallFilter=@obsolete  # syscall1 | syscall2 | @group
                         # group ex.: @basic-io
                         # systemd-analyze syscall-filter @obsolete

IPAdressAllow=10.20.30.0/24 1.2.3.4
IPAdressDeny=*
IP{Ingress,Egress}FilterPath=

CPUQuota, MemoryMax, TasksMax, RuntimeMaxSec.
RootDyrectory, RootImage, MountAPIVFS
PrivateUsers

Examples:

sudo systemd-run -t -p User=lain -p RuntimeDirectory=foo ls -ld /run/foo
ls -ld /run/foo  # --> does not exist (cleaned up)

sudo systemd-run -t -p DynamicUser=true whoami  # --> ... run-u99

echo -e 'asdf\nasdf' |\
sudo systemd-run --pipe -p DynamicUser=1 bash -c 'grep .; whoami' |\
sudo systemd-run --pipe -p DynamicUser=1 bash -c 'grep .; whoami' |\
sudo systemd-run --pipe -p DynamicUser=1 bash -c 'grep .; whoami'
# --->
# Running as unit: run-u101.service
# Running as unit: run-u103.service
# Running as unit: run-u102.service
# asdf
# asdf
# run-u101
# run-u103
# run-u102

PrivateNetwork=yes is a recommended way to run network services. A daemon does not open a socket itself, it receives a socket from manager.

Grade service security:

systemd-analyze security systemd-resolved.service

Perfomance

Show how much time each service takes to start:

systemd-analyze blame

Plot for more information:

systemd-analyze plot > systemd-perf.svg
firefox systemd-perf.svg

Other

IPMI was failing for whatever reason, so I disabled it:

sudo systemctl disable openipmi.service