Server Performance

Server Performance Tuning — Production Linux

Performance Tuning Checklist

Before tuning, always establish a baseline:

# Baseline snapshot
ss -s                              # Socket summary
cat /proc/sys/net/core/somaxconn   # Current connection backlog
ulimit -n                          # File descriptor limit (per-process)
cat /proc/sys/fs/file-max          # Global fd limit
free -h                            # Memory/swap
swapon --show                      # Active swap
cat /sys/block/sda/queue/scheduler # I/O scheduler

Kernel / Network Tuning via sysctl

/etc/sysctl.d/99-production.conf:

# ── Connection Backlog ────────────────────────────────────────────────────────
# Maximum number of connections that can be queued for acceptance
# Default: 4096 (Ubuntu 22.04). Match this to Nginx worker_connections.
net.core.somaxconn = 65535

# Maximum number of packets in the kernel's receive queue per NIC
net.core.netdev_max_backlog = 65535

# Maximum number of SYN requests in the half-open connection queue
net.ipv4.tcp_max_syn_backlog = 65535

# ── TIME_WAIT Tuning ──────────────────────────────────────────────────────────
# Reduce TIME_WAIT timeout from 60s to 30s
# Note: Cannot go below ~15s safely (RFC recommends 2*MSL)
net.ipv4.tcp_fin_timeout = 30

# Allow reuse of TIME_WAIT sockets for new outgoing connections
# Safe for servers that are not load balancers
net.ipv4.tcp_tw_reuse = 1

# ── Ephemeral Ports ───────────────────────────────────────────────────────────
# Default range: 32768-60999. Expand for high-outbound-connection servers.
net.ipv4.ip_local_port_range = 1024 65535

# ── TCP Keepalive ─────────────────────────────────────────────────────────────
# Start keepalive probes after 60s idle (default: 7200s = 2 hours)
net.ipv4.tcp_keepalive_time = 60

# Interval between keepalive probes
net.ipv4.tcp_keepalive_intvl = 10

# Number of failed probes before declaring connection dead
net.ipv4.tcp_keepalive_probes = 6

# ── Memory ───────────────────────────────────────────────────────────────────
# Swappiness: 10 is appropriate for SSD-based servers with sufficient RAM.
# Set to 1 for Redis/memory-intensive services, NOT to 0 (may cause OOM issues).
vm.swappiness = 10

# dirty_ratio: % of RAM that can be dirty before processes are forced to write
# Reduce for databases to avoid sudden I/O stalls (default: 20)
vm.dirty_ratio = 10

# dirty_background_ratio: % at which background writeback starts (default: 5)
vm.dirty_background_ratio = 3

# ── File Descriptors ──────────────────────────────────────────────────────────
# Global kernel limit for open file handles
fs.file-max = 2097152

# ── Network Receive Buffers ───────────────────────────────────────────────────
# Increase receive/send buffer maximums for high-throughput servers
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728

# ── Connection Tracking (if firewall/NAT in use) ──────────────────────────────
# Increase conntrack table size to avoid "nf_conntrack: table full" errors
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 300

Apply immediately:

sudo sysctl -p /etc/sysctl.d/99-production.conf
# Verify
sysctl net.core.somaxconn net.ipv4.tcp_fin_timeout vm.swappiness

File Descriptor (ulimit) Configuration

System-wide: /etc/security/limits.conf

# Syntax: <domain> <type> <item> <value>
# * applies to all users except root
*               soft    nofile          65535
*               hard    nofile          65535
root            soft    nofile          65535
root            hard    nofile          65535
www-data        soft    nofile          1048576
www-data        hard    nofile          1048576
postgres        soft    nofile          65535
postgres        hard    nofile          65535

Requires PAM pam_limits.so to be active (enabled by default on Ubuntu). Verify:

sudo -u www-data bash -c 'ulimit -n'

Per-Service Systemd Override

/etc/systemd/system/nginx.service.d/limits.conf:

[Service]
LimitNOFILE=1048576

/etc/systemd/system/postgresql.service.d/limits.conf:

[Service]
LimitNOFILE=65535
LimitNPROC=65535

sudo systemctl daemon-reload
sudo systemctl restart nginx
# Verify nginx sees the new limit
cat /proc/$(pgrep -o nginx)/limits | grep 'open files'

Nginx Performance Configuration

/etc/nginx/nginx.conf (worker and global section):

# Match CPU core count; "auto" sets it automatically
worker_processes auto;

# Increase from default 1024. Match fs.file-max / worker_processes.
worker_rlimit_nofile 65535;

events {
    # Number of simultaneous connections per worker process
    # Total capacity: worker_processes * worker_connections
    worker_connections 16384;

    # Use epoll on Linux for efficient I/O multiplexing
    use epoll;

    # Accept multiple connections per epoll event (reduces syscall overhead)
    multi_accept on;
}

http {
    # Send file data directly from kernel buffer (zero-copy)
    sendfile on;

    # Bundle response headers with first data packet (requires sendfile)
    tcp_nopush on;

    # Reduce latency on small packets by disabling Nagle's algorithm
    tcp_nodelay on;

    # Keep connections open for 30 requests or 65 seconds, whichever comes first
    keepalive_timeout 65;
    keepalive_requests 1000;

    # Cache open file descriptors (avoids repeated open()/stat() calls)
    open_file_cache max=10000 inactive=30s;
    open_file_cache_valid 60s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;

    # Compress text responses (substantial bandwidth saving)
    gzip on;
    gzip_comp_level 2;           # Level 2 is sweet spot for CPU vs ratio
    gzip_min_length 1024;
    gzip_types text/plain text/css text/javascript application/json
               application/javascript text/xml application/xml;

    # Upstream keepalive (for proxy_pass to app servers)
    upstream app_backend {
        server 127.0.0.1:3000;
        server 127.0.0.1:3001;
        keepalive 64;            # Persist 64 idle connections to backend
    }

    server {
        location / {
            proxy_pass http://app_backend;
            proxy_http_version 1.1;   # Required for keepalive to backend
            proxy_set_header Connection "";
        }
    }
}

Test and reload:

sudo nginx -t && sudo systemctl reload nginx

PgBouncer: PostgreSQL Connection Pooling

Postgres creates a new OS process per connection (~5–10 MB RAM). At 500 connections that is 2.5–5 GB overhead before a single query runs. PgBouncer multiplexes client connections onto a small pool of real server connections.

Pool Modes

/etc/pgbouncer/pgbouncer.ini:

[databases]
# Format: <pgbouncer_db_name> = host=<pg_host> port=<pg_port> dbname=<real_db>
myapp = host=127.0.0.1 port=5432 dbname=myapp_production

[pgbouncer]
listen_addr = 127.0.0.1
listen_port = 6432

auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt

pool_mode = transaction

# Maximum client connections PgBouncer will accept
max_client_conn = 1000

# Number of real Postgres connections per database/user pair
default_pool_size = 20

# Reserve pool for emergencies (used when default pool exhausted)
reserve_pool_size = 5

# After this many seconds, reserve pool kicks in
reserve_pool_timeout = 3

# Kill idle server connections after N seconds of inactivity
server_idle_timeout = 600

# Maximum age of server connection before forced recycling (detects schema changes)
server_lifetime = 3600

# Maximum time to wait for a connection from pool before error
pool_mode_timeout = 30

# Log connections
log_connections = 0        # Set to 1 during debugging only
log_disconnections = 0
log_pooler_errors = 1

# Admin interface
admin_users = postgres
stats_users = monitoring_user

# PID file and socket
pidfile = /var/run/pgbouncer/pgbouncer.pid
unix_socket_dir = /var/run/pgbouncer

/etc/pgbouncer/userlist.txt (md5 hash of password):

"myapp_user" "md5HASH_OF_PASSWORD"

Generate: echo -n 'PASSWORDmyapp_user' | md5sum | awk '{print "md5"$1}'

sudo systemctl enable --now pgbouncer
# Connect through PgBouncer (port 6432 instead of 5432)
psql -h 127.0.0.1 -p 6432 -U myapp_user myapp
# Check pool stats
psql -h 127.0.0.1 -p 6432 -U postgres pgbouncer -c "SHOW POOLS;"
psql -h 127.0.0.1 -p 6432 -U postgres pgbouncer -c "SHOW STATS;"

Memory: Transparent Huge Pages

THP causes latency spikes in databases (Redis, MongoDB, PostgreSQL) due to compaction pauses.

# Check current status
cat /sys/kernel/mm/transparent_hugepage/enabled

# Disable for current session
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

# Persist across reboots — add to /etc/rc.local or a systemd oneshot unit

/etc/systemd/system/disable-thp.service:

[Unit]
Description=Disable Transparent Huge Pages
DefaultDependencies=no
After=sysinit.target local-fs.target
Before=basic.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'
ExecStart=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/defrag'
RemainAfterExit=yes

[Install]
WantedBy=basic.target

OOM Killer Tuning

# Check OOM score of a process (higher = more likely to be killed)
cat /proc/$(pgrep postgres)/oom_score

# Protect critical processes from being OOM-killed (-1000 = never kill)
echo -1000 | sudo tee /proc/$(pgrep postgres)/oom_score_adj

# Set in systemd unit to persist across restarts:
# [Service]
# OOMScoreAdjust = -900    # Range: -1000 (never kill) to +1000 (kill first)

NUMA (on multi-socket servers):

# Check NUMA topology
numactl --hardware

# Bind process to NUMA node 0 and its local memory
numactl --cpunodebind=0 --membind=0 -- /usr/bin/postgres -D /var/lib/postgresql/16/main

CPU: Scheduling and Affinity

# Change I/O priority of a process (ionice class: 1=realtime, 2=best-effort, 3=idle)
sudo ionice -c 2 -n 0 -p $(pgrep backup-script)

# Change CPU scheduling priority (nice: -20=highest, 19=lowest)
sudo renice -n -5 -p $(pgrep postgres)

# Pin process to specific CPU cores (reduce cache invalidation)
sudo taskset -cp 0,1 $(pgrep -o nginx)

# Set CPU frequency governor to "performance" (disable power-saving for latency-sensitive)
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Disk I/O Tuning

I/O Scheduler

# Check available schedulers
cat /sys/block/sda/queue/scheduler
# [none] mq-deadline kyber bfq  ← brackets show active

# For NVMe SSDs: "none" is best (hardware queue handles ordering)
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler

# For SATA SSD: "mq-deadline" gives good latency
echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler

# For HDDs with mixed workloads: "bfq" (Budget Fair Queuing)
echo bfq | sudo tee /sys/block/sda/queue/scheduler

# Persist via udev rule (/etc/udev/rules.d/60-io-scheduler.rules):
# ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"
# ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"

Mount Options

/etc/fstab — add noatime for read-heavy filesystems (eliminates access time writes):

UUID=xxxx  /var/lib/postgresql  ext4  defaults,noatime  0 2
UUID=xxxx  /var/www             ext4  defaults,noatime  0 2

# Remount without rebooting
sudo mount -o remount,noatime /var/lib/postgresql

# Enable periodic TRIM for SSDs (runs weekly by default on Ubuntu)
sudo systemctl enable fstrim.timer
sudo systemctl start fstrim.timer
# Manual trim
sudo fstrim -v /

Performance Profiling Tools

# perf top — live flame of kernel + userspace (requires linux-tools)
sudo perf top -g

# Record 30s of CPU activity for process
sudo perf record -g -p $(pgrep -o myapp) -o /tmp/perf.data -- sleep 30
sudo perf report -i /tmp/perf.data

# strace — trace syscalls (high overhead, use only on dev/staging or briefly in prod)
sudo strace -p $(pgrep -o myapp) -e trace=network,file -T 2>&1 | head -100

# ltrace — trace library calls
sudo ltrace -p $(pgrep -o myapp) -c 2>&1    # summary mode, lower overhead

# lsof — list open files and sockets for a process
sudo lsof -p $(pgrep -o myapp) | wc -l      # Count open fds
sudo lsof -p $(pgrep -o myapp) | grep TCP   # TCP connections

Before/After Comparison: Connection Handling

Anti-Patterns

Troubleshooting