# Server Performance > Linux server and application-layer performance tuning for production workloads, covering kernel networking parameters, file descriptor limits, Nginx worker config, database connection pooling (PgBouncer), I/O scheduler tuning, and profiling tools. USE WHEN: - Tuning kernel sysctl parameters for high-concurrency web servers - Increasing file descriptor limits for Nginx, Node.js, or Postgres - Configuring Nginx worker settings for optimal throughput - Setting up PgBouncer as a Postgres connection pool - Diagnosing high TIME_WAIT, OOM kills, or disk I/O bottlenecks - Tuning vm.swappiness, dirty ratios, and I/O schedulers for SSD servers - Profiling with perf, strace, or flamegraphs DO NOT USE FOR: - Application-level code profiling (use language-specific profiling tools) - Kubernetes resource requests/limits (use the kubernetes skill) - Network interface bonding or VLAN configuration - Windows Server performance tuning - Skill: `claude-dev-suite/server-performance` (Agent Skill) - Install (CLI): `npx skillmds@latest add claude-dev-suite/server-performance` - Raw SKILL.md: https://api.skillmd.com/api/skills/claude-dev-suite/server-performance/raw - Safety review: pending (external: skill-scanner WARNING, skillspector WARNING) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: claude-dev-suite (https://skillmd.com/u/claude-dev-suite) - Updated: 2026-09-08 - Page: https://skillmd.com/skills/claude-dev-suite/server-performance --- # Server Performance Tuning — Production Linux ## Performance Tuning Checklist Before tuning, always establish a baseline: ```bash # Baseline snapshot ss -s # Socket summary cat /proc/sys/net/core/somaxconn # Current connection backlog ulimit -n # File descriptor limit (per-process) cat /proc/sys/fs/file-max # Global fd limit free -h # Memory/swap swapon --show # Active swap cat /sys/block/sda/queue/scheduler # I/O scheduler ``` --- ## Kernel / Network Tuning via sysctl `/etc/sysctl.d/99-production.conf`: ```ini # ── Connection Backlog ──────────────────────────────────────────────────────── # Maximum number of connections that can be queued for acceptance # Default: 4096 (Ubuntu 22.04). Match this to Nginx worker_connections. net.core.somaxconn = 65535 # Maximum number of packets in the kernel's receive queue per NIC net.core.netdev_max_backlog = 65535 # Maximum number of SYN requests in the half-open connection queue net.ipv4.tcp_max_syn_backlog = 65535 # ── TIME_WAIT Tuning ────────────────────────────────────────────────────────── # Reduce TIME_WAIT timeout from 60s to 30s # Note: Cannot go below ~15s safely (RFC recommends 2*MSL) net.ipv4.tcp_fin_timeout = 30 # Allow reuse of TIME_WAIT sockets for new outgoing connections # Safe for servers that are not load balancers net.ipv4.tcp_tw_reuse = 1 # ── Ephemeral Ports ─────────────────────────────────────────────────────────── # Default range: 32768-60999. Expand for high-outbound-connection servers. net.ipv4.ip_local_port_range = 1024 65535 # ── TCP Keepalive ───────────────────────────────────────────────────────────── # Start keepalive probes after 60s idle (default: 7200s = 2 hours) net.ipv4.tcp_keepalive_time = 60 # Interval between keepalive probes net.ipv4.tcp_keepalive_intvl = 10 # Number of failed probes before declaring connection dead net.ipv4.tcp_keepalive_probes = 6 # ── Memory ─────────────────────────────────────────────────────────────────── # Swappiness: 10 is appropriate for SSD-based servers with sufficient RAM. # Set to 1 for Redis/memory-intensive services, NOT to 0 (may cause OOM issues). vm.swappiness = 10 # dirty_ratio: % of RAM that can be dirty before processes are forced to write # Reduce for databases to avoid sudden I/O stalls (default: 20) vm.dirty_ratio = 10 # dirty_background_ratio: % at which background writeback starts (default: 5) vm.dirty_background_ratio = 3 # ── File Descriptors ────────────────────────────────────────────────────────── # Global kernel limit for open file handles fs.file-max = 2097152 # ── Network Receive Buffers ─────────────────────────────────────────────────── # Increase receive/send buffer maximums for high-throughput servers net.core.rmem_max = 134217728 net.core.wmem_max = 134217728 net.ipv4.tcp_rmem = 4096 87380 134217728 net.ipv4.tcp_wmem = 4096 65536 134217728 # ── Connection Tracking (if firewall/NAT in use) ────────────────────────────── # Increase conntrack table size to avoid "nf_conntrack: table full" errors net.netfilter.nf_conntrack_max = 1048576 net.netfilter.nf_conntrack_tcp_timeout_established = 300 ``` Apply immediately: ```bash sudo sysctl -p /etc/sysctl.d/99-production.conf # Verify sysctl net.core.somaxconn net.ipv4.tcp_fin_timeout vm.swappiness ``` --- ## File Descriptor (ulimit) Configuration ### System-wide: `/etc/security/limits.conf` ``` # Syntax: # * applies to all users except root * soft nofile 65535 * hard nofile 65535 root soft nofile 65535 root hard nofile 65535 www-data soft nofile 1048576 www-data hard nofile 1048576 postgres soft nofile 65535 postgres hard nofile 65535 ``` Requires PAM `pam_limits.so` to be active (enabled by default on Ubuntu). Verify: ```bash sudo -u www-data bash -c 'ulimit -n' ``` ### Per-Service Systemd Override `/etc/systemd/system/nginx.service.d/limits.conf`: ```ini [Service] LimitNOFILE=1048576 ``` `/etc/systemd/system/postgresql.service.d/limits.conf`: ```ini [Service] LimitNOFILE=65535 LimitNPROC=65535 ``` ```bash sudo systemctl daemon-reload sudo systemctl restart nginx # Verify nginx sees the new limit cat /proc/$(pgrep -o nginx)/limits | grep 'open files' ``` --- ## Nginx Performance Configuration `/etc/nginx/nginx.conf` (worker and global section): ```nginx # Match CPU core count; "auto" sets it automatically worker_processes auto; # Increase from default 1024. Match fs.file-max / worker_processes. worker_rlimit_nofile 65535; events { # Number of simultaneous connections per worker process # Total capacity: worker_processes * worker_connections worker_connections 16384; # Use epoll on Linux for efficient I/O multiplexing use epoll; # Accept multiple connections per epoll event (reduces syscall overhead) multi_accept on; } http { # Send file data directly from kernel buffer (zero-copy) sendfile on; # Bundle response headers with first data packet (requires sendfile) tcp_nopush on; # Reduce latency on small packets by disabling Nagle's algorithm tcp_nodelay on; # Keep connections open for 30 requests or 65 seconds, whichever comes first keepalive_timeout 65; keepalive_requests 1000; # Cache open file descriptors (avoids repeated open()/stat() calls) open_file_cache max=10000 inactive=30s; open_file_cache_valid 60s; open_file_cache_min_uses 2; open_file_cache_errors on; # Compress text responses (substantial bandwidth saving) gzip on; gzip_comp_level 2; # Level 2 is sweet spot for CPU vs ratio gzip_min_length 1024; gzip_types text/plain text/css text/javascript application/json application/javascript text/xml application/xml; # Upstream keepalive (for proxy_pass to app servers) upstream app_backend { server 127.0.0.1:3000; server 127.0.0.1:3001; keepalive 64; # Persist 64 idle connections to backend } server { location / { proxy_pass http://app_backend; proxy_http_version 1.1; # Required for keepalive to backend proxy_set_header Connection ""; } } } ``` Test and reload: ```bash sudo nginx -t && sudo systemctl reload nginx ``` --- ## PgBouncer: PostgreSQL Connection Pooling Postgres creates a new OS process per connection (~5–10 MB RAM). At 500 connections that is 2.5–5 GB overhead before a single query runs. PgBouncer multiplexes client connections onto a small pool of real server connections. ### Pool Modes | Mode | Description | Best For | |------|-------------|----------| | `session` | Server connection held for lifetime of client session | Legacy apps that use session-level state | | `transaction` | Server connection returned to pool after each transaction | **Recommended** for most stateless web apps | | `statement` | Server connection returned after each statement | Only when no multi-statement transactions used | `/etc/pgbouncer/pgbouncer.ini`: ```ini [databases] # Format: = host= port= dbname= myapp = host=127.0.0.1 port=5432 dbname=myapp_production [pgbouncer] listen_addr = 127.0.0.1 listen_port = 6432 auth_type = md5 auth_file = /etc/pgbouncer/userlist.txt pool_mode = transaction # Maximum client connections PgBouncer will accept max_client_conn = 1000 # Number of real Postgres connections per database/user pair default_pool_size = 20 # Reserve pool for emergencies (used when default pool exhausted) reserve_pool_size = 5 # After this many seconds, reserve pool kicks in reserve_pool_timeout = 3 # Kill idle server connections after N seconds of inactivity server_idle_timeout = 600 # Maximum age of server connection before forced recycling (detects schema changes) server_lifetime = 3600 # Maximum time to wait for a connection from pool before error pool_mode_timeout = 30 # Log connections log_connections = 0 # Set to 1 during debugging only log_disconnections = 0 log_pooler_errors = 1 # Admin interface admin_users = postgres stats_users = monitoring_user # PID file and socket pidfile = /var/run/pgbouncer/pgbouncer.pid unix_socket_dir = /var/run/pgbouncer ``` `/etc/pgbouncer/userlist.txt` (md5 hash of password): ``` "myapp_user" "md5HASH_OF_PASSWORD" ``` Generate: `echo -n 'PASSWORDmyapp_user' | md5sum | awk '{print "md5"$1}'` ```bash sudo systemctl enable --now pgbouncer # Connect through PgBouncer (port 6432 instead of 5432) psql -h 127.0.0.1 -p 6432 -U myapp_user myapp # Check pool stats psql -h 127.0.0.1 -p 6432 -U postgres pgbouncer -c "SHOW POOLS;" psql -h 127.0.0.1 -p 6432 -U postgres pgbouncer -c "SHOW STATS;" ``` --- ## Memory: Transparent Huge Pages THP causes latency spikes in databases (Redis, MongoDB, PostgreSQL) due to compaction pauses. ```bash # Check current status cat /sys/kernel/mm/transparent_hugepage/enabled # Disable for current session echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag # Persist across reboots — add to /etc/rc.local or a systemd oneshot unit ``` `/etc/systemd/system/disable-thp.service`: ```ini [Unit] Description=Disable Transparent Huge Pages DefaultDependencies=no After=sysinit.target local-fs.target Before=basic.target [Service] Type=oneshot ExecStart=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' ExecStart=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' RemainAfterExit=yes [Install] WantedBy=basic.target ``` --- ## OOM Killer Tuning ```bash # Check OOM score of a process (higher = more likely to be killed) cat /proc/$(pgrep postgres)/oom_score # Protect critical processes from being OOM-killed (-1000 = never kill) echo -1000 | sudo tee /proc/$(pgrep postgres)/oom_score_adj # Set in systemd unit to persist across restarts: # [Service] # OOMScoreAdjust = -900 # Range: -1000 (never kill) to +1000 (kill first) ``` NUMA (on multi-socket servers): ```bash # Check NUMA topology numactl --hardware # Bind process to NUMA node 0 and its local memory numactl --cpunodebind=0 --membind=0 -- /usr/bin/postgres -D /var/lib/postgresql/16/main ``` --- ## CPU: Scheduling and Affinity ```bash # Change I/O priority of a process (ionice class: 1=realtime, 2=best-effort, 3=idle) sudo ionice -c 2 -n 0 -p $(pgrep backup-script) # Change CPU scheduling priority (nice: -20=highest, 19=lowest) sudo renice -n -5 -p $(pgrep postgres) # Pin process to specific CPU cores (reduce cache invalidation) sudo taskset -cp 0,1 $(pgrep -o nginx) # Set CPU frequency governor to "performance" (disable power-saving for latency-sensitive) echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor ``` --- ## Disk I/O Tuning ### I/O Scheduler ```bash # Check available schedulers cat /sys/block/sda/queue/scheduler # [none] mq-deadline kyber bfq ← brackets show active # For NVMe SSDs: "none" is best (hardware queue handles ordering) echo none | sudo tee /sys/block/nvme0n1/queue/scheduler # For SATA SSD: "mq-deadline" gives good latency echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler # For HDDs with mixed workloads: "bfq" (Budget Fair Queuing) echo bfq | sudo tee /sys/block/sda/queue/scheduler # Persist via udev rule (/etc/udev/rules.d/60-io-scheduler.rules): # ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none" # ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline" ``` ### Mount Options `/etc/fstab` — add `noatime` for read-heavy filesystems (eliminates access time writes): ```fstab UUID=xxxx /var/lib/postgresql ext4 defaults,noatime 0 2 UUID=xxxx /var/www ext4 defaults,noatime 0 2 ``` ```bash # Remount without rebooting sudo mount -o remount,noatime /var/lib/postgresql # Enable periodic TRIM for SSDs (runs weekly by default on Ubuntu) sudo systemctl enable fstrim.timer sudo systemctl start fstrim.timer # Manual trim sudo fstrim -v / ``` --- ## Performance Profiling Tools ```bash # perf top — live flame of kernel + userspace (requires linux-tools) sudo perf top -g # Record 30s of CPU activity for process sudo perf record -g -p $(pgrep -o myapp) -o /tmp/perf.data -- sleep 30 sudo perf report -i /tmp/perf.data # strace — trace syscalls (high overhead, use only on dev/staging or briefly in prod) sudo strace -p $(pgrep -o myapp) -e trace=network,file -T 2>&1 | head -100 # ltrace — trace library calls sudo ltrace -p $(pgrep -o myapp) -c 2>&1 # summary mode, lower overhead # lsof — list open files and sockets for a process sudo lsof -p $(pgrep -o myapp) | wc -l # Count open fds sudo lsof -p $(pgrep -o myapp) | grep TCP # TCP connections ``` --- ## Before/After Comparison: Connection Handling | Scenario | Default Config | Tuned Config | |----------|---------------|--------------| | `net.core.somaxconn` | 4096 | 65535 | | Max Nginx connections (4 workers) | 4 × 1024 = 4096 | 4 × 16384 = 65536 | | Postgres connections (per process, no pooling) | 100 connections, ~500 MB overhead | PgBouncer pool_size=20, ~100 MB, 1000 clients | | TIME_WAIT socket clearing | 60 seconds | 30 seconds | | Ephemeral port range | 32768–60999 (28,231 ports) | 1024–65535 (64,511 ports) | | Per-process file descriptors (Nginx) | 1024 (default ulimit) | 65535 (limits.conf) | --- ## Anti-Patterns | Anti-Pattern | Problem | Fix | |--------------|---------|-----| | `vm.swappiness=60` on SSD servers | OS aggressively swaps RAM-resident hot data to SSD, adding latency | Set `vm.swappiness=10` for general servers, `1` for databases | | No connection pooling for Postgres | Each client holds a Postgres process (~5–10 MB RAM + context switches) | Deploy PgBouncer in transaction mode in front of Postgres | | `worker_processes 1` in Nginx | Single Nginx worker can't saturate multi-core CPU | Set `worker_processes auto` | | Transparent Huge Pages enabled for Redis/MongoDB | THP compaction causes 100ms+ latency spikes | Disable THP via systemd oneshot unit | | `open_file_cache` disabled | Nginx does open()/stat() on every request for static files | Enable `open_file_cache max=10000 inactive=30s` | | Default I/O scheduler (CFQ) on NVMe | CFQ adds unnecessary seek optimization for drives that have no seek latency | Switch to `none` for NVMe, `mq-deadline` for SATA SSD | | `dirty_ratio=20` on database servers | Large dirty page bursts cause sudden I/O stalls during writeback | Reduce to `dirty_ratio=10 dirty_background_ratio=3` | | `net.ipv4.tcp_tw_recycle=1` | Deprecated in Linux 4.12+, causes issues with NAT | Remove entirely; use `tcp_tw_reuse=1` instead | | Not applying ulimits via systemd | Limits in `/etc/security/limits.conf` don't apply to systemd-started services | Set `LimitNOFILE` in the `[Service]` section | | `pool_mode=session` in PgBouncer | Session mode ties up server connection for entire client session, negating pooling benefit | Use `transaction` mode for stateless web apps | --- ## Troubleshooting | Symptom | Likely Cause | Diagnostic & Fix | |---------|--------------|------------------| | Many TIME_WAIT sockets | Clients opening/closing connections rapidly | `ss -o state time-wait | wc -l` — tune `tcp_fin_timeout=30`, `tcp_tw_reuse=1` | | `connection refused` under load | `somaxconn` or `tcp_max_syn_backlog` too low | `netstat -s | grep overflow` — increase both to 65535 | | OOM kills in journal | Process exceeds available RAM | `journalctl -k | grep oom-killer` — add swap, increase RAM, or tune `OOMScoreAdjust` | | Disk I/O saturated under moderate load | I/O scheduler not optimal, or THP writeback storms | `iostat -xz 1` — check `%util`; disable THP; reduce `dirty_ratio` | | Nginx 502 under high load | Backend connection pool exhausted | `ss -s` on backend — increase PgBouncer `max_client_conn`; scale backend | | PgBouncer `no more connections allowed` | `max_client_conn` hit | `SHOW CLIENTS` in pgbouncer console; increase `max_client_conn` | | `too many open files` error in Nginx logs | `worker_rlimit_nofile` not applied | Check `cat /proc/$(pgrep -o nginx)/limits`; verify systemd override loaded | | High `wa` (I/O wait) in `vmstat` / `top` | Slow disk or `dirty_ratio` causing flush stall | `iostat -xz 1 sda`; enable `noatime`; check I/O scheduler | | perf data shows kernel function dominating | System call overhead (e.g., excessive `stat()` calls) | Enable `open_file_cache` in Nginx; audit app for repeated file checks | | NUMA imbalance on multi-socket server | Processes accessing remote NUMA memory (+2x latency) | `numastat -m` — bind DB process to single NUMA node with `numactl` |