Linux Services

Linux Service Management with systemd

Infrastructure engineer managing Linux services with systemd for reliable operation, dependency ordering, resource isolation, and automated recovery in cloud and on-prem environments.

TL;DR Checklist

• Write proper systemd unit files with correct type, restart policy, and resource limits
• Configure service dependencies with After=, Wants=, and Requires= directives
• Set up socket activation for services that don't need to be running constantly
• Configure journal logging with appropriate retention and forwarding
• Set up systemd timers for scheduled tasks instead of cron where possible
• Verify service health with health checks and OnFailure handlers
• Configure resource limits in unit files (CPU, memory, I/O)
• Test service failure and recovery with manual stop/start cycles

When to Use

Use this skill when:

• Creating new systemd services — You're deploying an application that needs to run as a managed system service
• Fixing service dependency issues — Services are starting in the wrong order or failing due to missing dependencies
• Implementing socket activation — You want to reduce startup time and resource usage by activating services on demand
• Configuring scheduled tasks — You prefer systemd timers over cron for system-integrated scheduling
• Managing service lifecycle — You need reliable restart policies, resource limits, and automatic recovery for production workloads
• Troubleshooting service failures — A service is crashing, not starting, or failing health checks

When NOT to Use

Avoid this skill for:

• Container-based deployments — Use container orchestration (Docker Compose, Kubernetes) instead of systemd for container management
• One-off script execution — Use cron or at for simple scheduled tasks, not systemd timers
• Long-running daemons managed by supervisor — If you already use supervisord or similar process managers, don't migrate to systemd without clear benefit
• Batch jobs with complex error handling — Use dedicated job schedulers (Celery, Airflow) instead of systemd for complex workflows

Use resource-management when you need deeper resource isolation beyond systemd's built-in cgroup support. Use observability for setting up monitoring and alerting for service health.

Core Workflow

1. Write Systemd Unit Files

Create properly structured unit files for service management.

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application Service
Documentation=https://example.com/docs
After=network.target postgresql.service
Wants=network-online.target
Requires=postgresql.service

[Service]
Type=notify
User=myapp
Group=myapp

# Execution
ExecStart=/opt/myapp/bin/server --config /etc/myapp/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/opt/myapp/bin/server --shutdown

# Restart and recovery
Restart=on-failure
RestartSec=5s
StartLimitIntervalSec=300
StartLimitBurst=5

# Environment
Environment=APP_ENV=production
EnvironmentFile=/etc/myapp/env

# Resource limits (cgroups)
CPUQuota=200%
MemoryMax=4G
MemoryHigh=3G
IOWeight=500

# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/myapp /var/log/myapp
PrivateTmp=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
RemoveIPC=yes
SystemCallFilter=@system-service
SystemCallArchitectures=native

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

# File descriptor limit
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Checkpoint: Unit file is created with all required directives, reloaded with systemctl daemon-reload, and the service starts successfully.

2. Configure Dependency Ordering

Set up correct service start ordering with dependency directives.

Dependency Type    │ Directive       │ Behavior
─────────────────────────────────────────────────────────────
Must start after   │ After=X         │ Ordering only; service X doesn't need to be up
Must require       │ Requires=X      │ If X fails, this service stops too
Soft dependency    │ Wants=X         │ Try to start X, but don't fail if X is unavailable
Condition          │ ConditionPath   │ Only start if path exists/doesn't exist
                   │ Exists=...      │
                   │ Condition       │
                   │ FileExists=...  │

# /etc/systemd/system/webapp.service
[Unit]
Description=Web Application
# Strong dependencies — app cannot function without these
After=network.target postgresql.service redis.service
Requires=postgresql.service

# Soft dependency — app works without cache but prefers it
Wants=redis.service

# Conditional: only start if config exists
ConditionPathExists=/etc/webapp/config.yaml

[Service]
# ... service configuration ...

Checkpoint: Service starts in correct order. All hard dependencies are satisfied before the service starts. systemctl list-dependencies myapp.service shows correct tree.

3. Configure Socket Activation

Use socket activation for services that don't need to run constantly.

# /etc/systemd/system/myapp.socket
[Unit]
Description=My Application Socket

[Socket]
ListenStream=8443
Accept=no

# Security
SocketUser=myapp
SocketGroup=myapp
SocketMode=0660

# Backlog
Backlog=128

# Free resources when idle
IdleExitTimeout=60

[Install]
WantedBy=sockets.target

# /etc/systemd/system/myapp@.service
# Note the @ — template unit for accepting connections
[Unit]
Description=My Application Service Instance
Requires=myapp.socket
After=myapp.socket

[Service]
Type=notify
User=myapp
Group=myapp
ExecStart=/opt/myapp/bin/server --config /etc/myapp/config.yaml --socket $LISTEN_FDS

# Inherit socket from parent
ExecStartPre=-/usr/bin/mkdir -p /run/myapp
StandardInput=socket

[Install]
WantedBy=multi-user.target

Checkpoint: Socket is listening on the configured port. Service starts on first connection and stops after idle timeout. systemctl status myapp.socket shows active (listening).

4. Configure Journal and Logging

Set up journal logging with proper retention and forwarding.

# Journal configuration
cat > /etc/systemd/journald.conf << 'JOURNAL'
[Journal]
# Persistent storage
Storage=persistent

# Size limits
SystemMaxUse=500M
SystemKeepFree=1G
RuntimeMaxUse=100M

# Retention
MaxFileSec=1week
MaxForwardWatchdog=30sec

# Rate limiting
RateLimitInterval=30sec
RateLimitBurst=1000

# Compression
Compress=yes

# Forward to syslog (for legacy tools)
ForwardToSyslog=yes

# Syslog identification
SyslogIdentifier=myapp
JOURNAL

sudo systemctl restart systemd-journald

# Query journal for a specific service
journalctl -u myapp.service --since "1 hour ago" --no-pager
journalctl -u myapp.service --follow  # Live tail

# Filter by priority (emergency=0 to debug=7)
journalctl -u myapp.service -p warning --since today

# View with resources
journalctl -u myapp.service --output=json-pretty | jq '.[] | {timestamp: .__REALTIME_TIMESTAMP, message: .MESSAGE}'

Checkpoint: Journal storage is persistent, size limits prevent disk exhaustion, and log queries return expected results.

5. Configure Systemd Timers

Replace cron with systemd timers for better integration.

# /etc/systemd/system/myapp-cleanup.timer
[Unit]
Description=Run MyApp cleanup daily
Requires=myapp-cleanup.service

[Timer]
# Run daily at 2:30 AM 02:30:00
# Also run 5 minutes after boot (one-time)

# Accuracy: allow 1-minute window for power saving
AccuracySec=1min

# Persistent: catch up on missed runs after downtime
Persistent=true

# Randomized delay to prevent thundering herd
RandomizedDelaySec=5min

[Install]
WantedBy=timers.target

# /etc/systemd/system/myapp-cleanup.service
[Unit]
Description=MyApp data cleanup task
After=network.target

[Service]
Type=oneshot
User=myapp
Group=myapp
ExecStart=/opt/myapp/bin/cleanup --full
TimeoutStartSec=3600

# Resource limits
MemoryMax=2G
CPUQuota=100%

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp-cleanup

Checkpoint: Timer is enabled and scheduled. systemctl list-timers --all shows the timer with next trigger time. Manual trigger works: systemctl start myapp-cleanup.service.

6. Configure Health Checks and Recovery

Set up automatic recovery for service failures.

# /etc/systemd/system/myapp.service (with health checks)
[Unit]
Description=My Application Service
After=network.target postgresql.service

[Service]
Type=notify
User=myapp
ExecStart=/opt/myapp/bin/server

# Restart policy
Restart=on-failure
RestartSec=5s

# Health check with Watchdog
WatchdogSec=30
# Application must call sd_notify("WATCHDOG=1") every 30 seconds

# Failure handlers

# Environment for health check
Environment=HEALTH_CHECK_PATH=/health
Environment=HEALTH_CHECK_PORT=8080

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/report-success.service
[Unit]
Description=Report successful start
After=myapp.service

[Service]
Type=oneshot
ExecStart=/bin/bash -c 'curl -sf http://localhost:8080/health | jq . && logger "myapp started successfully"'

Checkpoint: Service restarts automatically on failure. Watchdog detects hung processes. OnFailure handler triggers for cascade prevention.

Implementation Patterns

Pattern 1: Service Types (BAD vs. GOOD)

BAD — Using wrong service type

# ❌ BAD: Using Type=simple for a service that notifies systemd
[Unit]
Description=My Application

[Service]
Type=simple
ExecStart=/opt/myapp/bin/server

# Problem: systemd considers the process "started" as soon as it forks.
# If the server takes 10 seconds to bind a port and report ready,
# dependent services will start before the app is actually ready.
# systemd has no way to know the service is not yet accepting connections.

# ❌ BAD: Using Type=forking for a modern service
# Problem: forking requires the process to explicitly fork() and exit
# the parent. Modern frameworks (Python uvicorn, Node, Go) don't fork.
# Using forking with a non-forking process causes systemd to think
# the service exited immediately.

GOOD — Match service type to application behavior

# ✅ GOOD: Type=notify for applications using sd_notify()
[Unit]
Description=My Application

[Service]
Type=notify
# Application calls sd_notify("READY=1") when ready
# Calls sd_notify("WATCHDOG=1") periodically
ExecStart=/opt/myapp/bin/server
WatchdogSec=30
Restart=on-failure
RestartSec=5s

# ✅ GOOD: Type=simple for processes that are the main process
[Unit]
Description=Simple Script Service

[Service]
Type=simple
# The ExecStart process IS the main process — systemd tracks it directly
ExecStart=/opt/scripts/monitor.sh
Restart=always
RestartSec=10s

# ✅ GOOD: Type=exec for wrapper scripts
[Unit]
Description=Wrapper Service

[Service]
Type=exec
# Starts a process group; systemd tracks the last process
ExecStartPre=/opt/scripts/prepare.sh
ExecStart=/opt/scripts/run.sh

Pattern 2: Template Services for Multi-Instance Deployments

# /etc/systemd/system/myapp@.service
# Template unit — use myapp@instance1.service, myapp@instance2.service
[Unit]
Description=My Application Instance %i
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=notify
User=myapp
Group=myapp

# %i is the template instance name (e.g., "instance1")
Environment=APP_INSTANCE=%i
ExecStart=/opt/myapp/bin/server \
    --instance %i \
    --config /etc/myapp/%i/config.yaml \
    --port $((8443 + %i))

Restart=on-failure
RestartSec=5s
CPUQuota=100%
MemoryMax=2G

[Install]
WantedBy=multi-user.target

# Enable specific instances
sudo systemctl enable --now myapp@1.service
sudo systemctl enable --now myapp@2.service

# List all instances
systemctl list-units 'myapp@*'

# Stop all instances
systemctl stop 'myapp@*'

Pattern 3: Cloud-Ready Service with Metadata Integration

Bash — Generate cloud-ready systemd service

#!/bin/bash
# Generate cloud-ready systemd service with cloud metadata awareness
# Usage: ./generate-cloud-service.sh <service-name> [exec-command] [instance-id] [region]

set -euo pipefail

SERVICE_NAME="${1:-webapp}"
EXEC_CMD="${2:-/opt/${SERVICE_NAME}/bin/server}"
INSTANCE_ID="${3:-$(curl -s -m 2 http://169.254.169.254/latest/meta-data/instance-id 2>/dev/null || echo 'unknown')}"
REGION="${4:-$(curl -s -m 2 http://169.254.169.254/latest/meta-data/placement/availability-zone 2>/dev/null | sed 's/[a-z]$//' || echo 'us-east-1')}"

SERVICE_FILE="/etc/systemd/system/${SERVICE_NAME}.service"

# Detect cloud provider
DETECT_PROVIDER() {
    if curl -s -m 2 http://169.254.169.254/latest/meta-data/instance-id 2>/dev/null | grep -q '^i-'; then
        echo "aws"
    elif curl -s -m 2 -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/name 2>/dev/null | grep -q .; then
        echo "gcp"
    elif curl -s -m 2 -H "Metadata-Severity: Interactive" "http://169.254.169.254/metadata/instance?api-version=2021-02-01" 2>/dev/null | grep -q '"compute"'; then
        echo "azure"
    else
        echo "bare-metal"
    fi
}

PROVIDER=$(DETECT_PROVIDER)
echo "Detected cloud provider: $PROVIDER"

# Generate systemd service file
cat > "$SERVICE_FILE" << EOF
[Unit]
Description=${SERVICE_NAME} on ${INSTANCE_ID} (${PROVIDER})
After=network.target
Documentation=https://example.com/docs/${SERVICE_NAME}
Wants=network-online.target

[Service]
Type=notify
User=${SERVICE_NAME}
Group=${SERVICE_NAME}

ExecStart=${EXEC_CMD}
Restart=on-failure
RestartSec=5s
WatchdogSec=30

# Resource limits
CPUQuota=200%
MemoryMax=4G
IOWeight=300

# Security
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/${SERVICE_NAME} /var/log/${SERVICE_NAME}
PrivateTmp=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
RemoveIPC=yes
SystemCallFilter=@system-service
SystemCallArchitectures=native

# Environment
Environment=AWS_REGION=${REGION}
Environment=INSTANCE_ID=${INSTANCE_ID}
Environment=CLOUD_PROVIDER=${PROVIDER}
Environment=APP_ENV=production
Environment=LOG_LEVEL=info
Environment=METRICS_ENABLED=true

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=${SERVICE_NAME}

LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

# Generate journal configuration
JOURNAL_CONF="/etc/systemd/journald.d/${SERVICE_NAME}.conf"
mkdir -p /etc/systemd/journald.d
cat > "$JOURNAL_CONF" << EOF
[Journal]
Storage=persistent
SystemMaxUse=500M
MaxFileSec=1week
Compress=yes
ForwardToSyslog=yes
EOF

echo "Generated ${SERVICE_FILE}"
echo "Generated ${JOURNAL_CONF}"
echo ""
echo "Run: systemctl daemon-reload && systemctl enable --now ${SERVICE_NAME}"

Constraints

MUST DO

• MUST use Type=notify for applications that support sd_notify() — provides accurate readiness signaling to systemd
• MUST set Restart=on-failure for all production services — automatic recovery is essential for reliability
• MUST configure resource limits (CPU, memory, I/O) in unit files — prevents runaway processes from affecting the host
• MUST use dependency directives correctly: After= for ordering, Requires= for hard dependencies, Wants= for soft dependencies
• MUST set up journal logging with size limits — prevent disk exhaustion from unbounded log growth
• MUST use systemd timers over cron for system-integrated scheduling — better failure handling and dependency management
• MUST enable services with systemctl enable to persist across reboots

MUST NOT DO

• MUST NOT use systemctl disable without a documented reason — every enabled service should have a documented purpose
• MUST NOT set Restart=always for services that should only restart on failure — always restarts even on explicit stop
• MUST NOT put sensitive data in Environment directives in unit files — use EnvironmentFile with restricted permissions instead
• MUST NOT use Type=forking for modern non-forking applications — it causes startup failures
• MUST NOT run services as root — create dedicated service accounts with minimum required privileges
• MUST NOT override journal default retention without justification — unbounded journal storage causes disk exhaustion
• MUST NOT use OnFailure=halt.target or OnFailure=poweroff.target in production — cascading failures kill the entire system

Related Skills