[systemd][admin] systemd notes

Some goodies:
http://0pointer.net/blog/

Design:
http://www.0pointer.de/blog/projects/systemd.html

Source code [systemd design] for how SD_LISTEN_FDS_START can be used in a project:
https://github.com/systemd/systemd/blob/37ec0fdd3443a77a5120cf55002fedcb5b1dd069/src/activate/activate.c#L188

systemd for Developers:
http://0pointer.de/blog/projects/socket-activation.html

reference:
https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files

Admin Hats on

Basic atom :

Unit /  k8s' 'Kind' idea, probably copy from systemd.

A 'unit' refers to any resource that the system knows how to operate on and manage.

Units can be of many types, the most common type is a "service"

(indicated by a unit file ending in .service).

Units in some ways can be said to similar to services or jobs in other init systems. 

A 'Unit' can be used to abstract services, network resources, devices, filesystem mounts,

and isolated resource pools.

INI file location ref:

https://www.freedesktop.org/software/systemd/man/systemd.unit.html

Default location:

/lib/systemd/system

Run-time unit definitions: 

(priority between /etc/systemd/system and /lib/systemd/system, 

systemd daemon uses this location for dynamically created unit files created at runtime, all changes made in this directory will be lost when the server is rebooted.)

/run/systemd/system

User's location:

/etc/systemd/system

Add snippets to the existing system unit:

e.g

Service file:

example.service

Snippet directory:
https://wiki.archlinux.org/index.php/systemd#Drop-in_files

example.service.d/

    .conf    <= file end with .conf

Some features that units are able implement easily are:

socket-based activation: Sockets associated with a service are best broken out of the daemon itself in order to be handled separately. This provides a number of advantages, such as delaying the start of a service until the associated socket is first accessed. This also allows the system to create all sockets early in the boot process, making it possible to boot the associated services in parallel.

bus-based activation: Units can also be activated on the bus interface provided by D-Bus. A unit can be started when an associated bus is published.

path-based activation: A unit can be started based on activity on or the availability of certain filesystem paths. This utilizes inotify.

device-based activation: Units can also be started at the first availability of associated hardware by leveraging udev events.

implicit dependency mapping: Most of the dependency tree for units can be built by systemd itself. You can still add dependency and ordering information, but most of the heavy lifting is taken care of for you.

instances and templates: Template unit files can be used to create multiple instances of the same general unit. This allows for slight variations or sibling units that all provide the same general function.

easy security hardening: Units can implement some fairly good security features by adding simple directives. For example, you can specify no or read-only access to part of the filesystem, limit kernel capabilities, and assign private /tmp and network access.

drop-ins and snippets: Units can easily be extended by providing snippets that will override parts of the system's unit file. This makes it easy to switch between vanilla and customized unit implementations.

Type of Units ref:

https://www.freedesktop.org/software/systemd/man/systemd.unit.html
https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files

To differentiate what type of Unit, the suffix of a file provides the information:

.service: A service unit describes how to manage a service or application on the server. This will include how to start or stop the service, under which circumstances it should be automatically started, and the dependency and ordering information for related software.

.socket: A socket unit file describes a network or IPC socket, or a FIFO buffer that systemd uses for socket-based activation. These always have an associated .service file that will be started when activity is seen on the socket that this unit defines.

.netdev:  Virtual Network Device configuration

.network:  Network configuration

.device: A unit that describes a device that has been designated as needing systemd management by udev or the sysfs filesystem. Not all devices will have .device files. Some scenarios where .device units may be necessary are for ordering, mounting, and accessing the devices.

.mount: This unit defines a mountpoint on the system to be managed by systemd. These are named after the mount path, with slashes changed to dashes. Entries within /etc/fstab can have units created automatically.

.automount: An .automount unit configures a mountpoint that will be automatically mounted. These must be named after the mount point they refer to and must have a matching .mount unit to define the specifics of the mount.

.swap: This unit describes swap space on the system. The name of these units must reflect the device or file path of the space.

.target: A target unit is used to provide synchronization points for other units when booting up or changing states. They also can be used to bring the system to a new state. Other units specify their relation to targets to become tied to the target's operations.
Reference: https://www.freedesktop.org/software/systemd/man/systemd.target.html

[Unit] 
Description=Emergency Mode with Networking
Requires=emergency.target systemd-networkd.service
After=emergency.target systemd-networkd.service
AllowIsolate=yes

When adding dependencies to other units, it's important to check if they set "DefaultDependencies="
Service units, unless they set DefaultDependencies=no, automatically get a dependency on sysinit.target
In above example, both emergency.target and systemd-networkd.service have DefaultDependencies=no, so they are suitable for use in this target, and do not pull in sysinit.target

.path: This unit defines a path that can be used for path-based activation. By default, a .service unit of the same base name will be started when the path reaches the specified state. This uses inotify to monitor the path for changes.

.timer: A .timer unit defines a timer that will be managed by systemd, similar to a cron job for delayed or scheduled activation. A matching unit will be started when the timer is reached.

.snapshot: A .snapshot unit is created automatically by the systemctl snapshot command. It allows you to reconstruct the current state of the system after making changes. Snapshots do not survive across sessions and are used to roll back temporary states.

.slice: A .slice unit is associated with Linux Control Group nodes, allowing resources to be restricted or assigned to any processes associated with the slice. The name reflects its hierarchical position within the cgroup tree. Units are placed in certain slices by default depending on their type.

.scope: Scope units are created automatically by systemd from information received from its bus interfaces. These are used to manage sets of system processes that are created externally.

INI format ref:

List of all directives(即config in SystemD.

https://www.freedesktop.org/software/systemd/man/systemd.directives.html

https://www.freedesktop.org/software/systemd/man/systemd.syntax.html

kubelet.service INI (Case sensitive):

1. Adding sections which are not standard systemd entries, use [X-WHATEVER]
2. When reset a value from snippets, simply assign empty string:
   e.g
   Wants=
3. boolean variable: 0, off, no, false / 1 true, on, yes (kinda like Cmake)
4. device, target, snapshot, and scope unit types have no unit-specific directives.

# metadata
[Unit]
Description=kubelet: The Kubernetes Node Agent
# systemctl status shows this information. 'man' is also allowed.
Documentation=https://kubernetes.io/docs

# This Unit dependencies.
# reference: https://serverfault.com/questions/812584/in-systemd-whats-the-difference-between-after-and-requires
Requires=

# Like Requires but less restricted. If listing Units not being activated, this Unit still runs.
# Most services use this for parallelization.
Wants=docker.service crio.service

# Like Requires but in runtime mode. If listing Units stop, this Unit stops.
BindsTo=

# Listing Units won't be started unless this Unit starts. Doesn't mean dependencies.
# Must use with previous mentioned actions.
Before=

# This Unit only starts after the Listing Units. Doesn't mean dependencies.
# Must use with previous mentioned actions.
After=

# Exclusive Units. This Unit starts will cause listing Units to stop.
Conflicts=

# Test current system conditions, if not met, systemd will skip this Unit.
Condition...=

# Like Condition..., but has negative effect.
Assert...=

# only applicable for services.
[Service]
# Categorizes services by their process and daemonizing behavior.
# simple: Default if not mentioned Type= or Busname= , but the ExecStart= is set.
    # Any communication should be handled outside of the unit through a second unit of the appropriate type
    # (like through a .socket unit if this unit must communicate using sockets).
# forking: This service type is used when the service forks a child process, exiting the parent process almost immediately.
    #This tells systemd that the process is still running even though the parent exited.
# oneshot:  This type indicates that the process will be short-lived and that systemd should wait for the process to exit
    # before continuing on with other units. This is the default Type= and ExecStart= are not set. It is used for one-off tasks.
# dbus: This indicates that unit will take a name on the D-Bus bus. When this happens, systemd will continue to process the next unit.
# notify: This indicates that the service will issue a notification when it has finished starting up.
    #The systemd process will wait for this to happen before proceeding to other units.
# idle: This indicates that the service will not be run until all jobs are dispatched.
Type=

# The directive is commonly used with the oneshot type.
# It indicates that the service should be considered active even after the process exits.
RemainAfterExit=

# If the service type is marked as "forking", this directive is used to set the path of the file that
#should contain the process ID number of the main child that should be monitored.
PIDFile=

# The directive should be set to the D-Bus bus name that the service will attempt to acquire when using the "dbus" service type.
BusName=

# Specifies access to the socket that should be used to listen for notifications when the "notify" service type is selected.
# Can be "none", "main", or "all. 
# The default, "none", ignores all status messages. 
# The "main" option will listen to messages from the main process and 
# the "all" option will cause all members of the service's control group to be processed.
NotifyAccess=

# This specifies the full path and the arguments of the command to be executed to start the process.
# This may only be specified once (except for "oneshot" services). 
# If the path to the command is preceded by a dash "-" character, non-zero exit statuses will be accepted without marking the unit activation as failed.
ExecStart=/usr/bin/kubelet

# This indicates the circumstances under which systemd will attempt to automatically restart the service.
# "always", "on-success", "on-failure", "on-abnormal", "on-abort", or "on-watchdog". 
# These will trigger a restart according to the way that the service was stopped.
Restart=always
StartLimitInterval=0
RestartSec=10

# Only Unit can be 'enabled' could have this section.
[Install]
# Creates a directory under /etc/systemd/system/multi-user.target.wants
# which contains a sym-link to this Unit INI, thus created a dependency graph.
WantedBy=multi-user.target

# Creates a directory under /etc/systemd/system/XXX.target.requires
# which contains a sym-link to this Unit INI, thus created a dependency graph.
# More restricted , Like [Unit]Requires=
RequiredBy=

# This directive allows the unit to be enabled under another name as well. 
# Among other use cases, this allows multiple providers of a function to be available, 
# so that related units can look for any provider of the common aliased name.
Alias=

# This directive allows units to be enabled or disabled as a set.
# Supporting units that should always be available when this unit is active can be listed here.
# They will be managed as a group for installation tasks.
Also=

# For template units which can produce unit instances with unpredictable names,
# this can be used as a fallback value for the name if an appropriate name is not provided.
DefaultInstance=

Template Specifiers:

https://www.freedesktop.org/software/systemd/man/systemd.unit.html

%n: Anywhere where this appears in a template file, the full resulting unit name will be inserted.

%N: This is the same as the above, but any escaping, such as those present in file path patterns, will be reversed.

%p: This references the unit name prefix. This is the portion of the unit name that comes before the @ symbol.

%P: This is the same as above, but with any escaping reversed.

%i: This references the instance name, which is the identifier following the @ in the instance unit. This is one of the most commonly used specifiers because it will be guaranteed to be dynamic. The use of this identifier encourages the use of configuration significant identifiers. For example, the port that the service will be run at can be used as the instance identifier and the template can use this specifier to set up the port specification.

%I: This specifier is the same as the above, but with any escaping reversed.

%f: This will be replaced with the unescaped instance name or the prefix name, prepended with a /.

%c: This will indicate the control group of the unit, with the standard parent hierarchy of /sys/fs/cgroup/ssytemd/ removed.

%u: The name of the user configured to run the unit.

%U: The same as above, but as a numeric UID instead of name.

%H: The host name of the system that is running the unit.

%%: This is used to insert a literal percentage sign.

CMD:

$ systemctl

$ systemctl start nginx.service

$ systemctl stop nginx.service

$ systemctl restart nginx.service

$ systemctl reload nginx.service

Enable at boot:

$ systemctl enable nginx.service

$ systemctl disable nginx.service

List state:

$ systemctl list-units

$ systemctl list-units --all

$ systemctl list-unit-files

Log:

$ journalctl

edit the /etc/systemd/journald.conf file and set the Storage= option to "persistent", or create the persistent directory by typing sudo mkdir -p /var/log/journal

current boot log:

$ journalctl -b

kernel log:

$ journalctl -k

$ journalctl -k -b

Unit status:

$ systemctl status nginx.service

$ journalctl -u nginx.service

$ journalctl -b -u nginx.service

Read a unit file:

$ systemctl cat nginx.service

$ systemctl show nginx.service

List the dependency tree of a unit (which units systemd will attempt to activate when starting the unit), type:

$ systemctl list-dependencies nginx.service

$ systemctl list-dependencies --all nginx.service

Modify Unit Files:

$ systemctl edit nginx.service

$ systemctl edit --full nginx.service

then reload:

$ systemctl daemon-reload

Use Targets (Runlevels 0 ~ 6 as traditional):

Service and other unit files can be tied to a target and multiple targets can be active at the same time.

$ systemctl list-unit-files --type=target

$ systemctl get-default

set default boot target:

$ systemctl set-default multi-user.target

list target dependencies:

$ systemctl list-dependencies multi-user.target

We can modify the system state to transition between targets with the isolate option. 

This will stop any units that are not tied to the specified target.

Be sure that the target you are isolating does not stop any essential services:

$ systemctl isolate multi-user.target

Poweroff:

$ systemctl poweroff

Reboot:

$ systemctl reboot

$ systemctl rescue

Minus sign before variable definition:
- in front of the path means "ignore errors".