SysOps Script Library
Purpose
Deliver production-hardened shell script templates for enterprise Linux infrastructure management including automated deployments, system monitoring, credential handling, service orchestration, and operational automation. These patterns serve as foundational components for large-scale infrastructure operations.
Prerequisites
Required Environment
- Linux/Unix production system (bash 4.0+)
- Appropriate administrative permissions
- Standard utilities (rsync, openssl, curl, etc.)
Required Knowledge
- Advanced bash scripting techniques
- Linux system administration
- Production deployment workflows
- Infrastructure security principles
Outputs and Deliverables
- Deployment Automation - Application and service deployment scripts
- Infrastructure Monitoring - System health and performance tracking
- Credential Management - Secure handling of authentication data
- Service Orchestration - Multi-service coordination and restart automation
Core Workflow
Phase 1: Deployment Scripts
Application Deployment
#!/bin/bash
app_name="web-application"
deploy_path="/opt/applications/$app_name"
config_path="/etc/$app_name"
# Deploy application to production environment
mkdir -p "$deploy_path"
tar -xzf "$app_name-latest.tar.gz" -C "$deploy_path"
chown -R appuser:appgroup "$deploy_path"
echo "Application deployed to $deploy_path"
Configuration Deployment
#!/bin/bash
config_file="$1"
target_server="$2"
# Deploy configuration files to remote servers
if [ -z "$config_file" ] || [ -z "$target_server" ]; then
echo "Usage: $0 <config_file> <target_server>"
exit 1
fi
scp "$config_file" "$target_server:/etc/app/config/"
ssh "$target_server" "systemctl reload application-service"
echo "Configuration deployed and service reloaded"
Multi-Server Deployment
#!/bin/bash
servers=("prod-web-01" "prod-web-02" "prod-web-03")
artifact="application-v1.2.3.tar.gz"
# Deploy artifact across multiple production servers
for server in "${servers[@]}"; do
echo "Deploying to $server..."
scp "$artifact" "deploy@$server:/tmp/"
ssh "deploy@$server" "cd /opt/apps && tar -xzf /tmp/$artifact"
echo "Deployment to $server completed"
done
echo "Multi-server deployment finished"
Phase 2: Infrastructure Monitoring
Service Health Monitor
#!/bin/bash
services=("nginx" "postgresql" "redis-server")
alert_email="ops-team@company.com"
# Monitor critical services and alert on failures
for service in "${services[@]}"; do
if ! systemctl is-active --quiet "$service"; then
echo "CRITICAL: $service is down" | mail -s "Service Alert" "$alert_email"
systemctl restart "$service"
echo "Attempted restart of $service"
fi
done
Resource Usage Reporter
#!/bin/bash
output_file="/var/log/resource-report-$(date +%Y%m%d-%H%M).log"
# Generate comprehensive resource usage report
{
echo "Resource Usage Report - $(date)"
echo "================================"
echo ""
echo "CPU Usage:"
mpstat 1 5 | tail -1
echo ""
echo "Memory Usage:"
free -h
echo ""
echo "Disk I/O:"
iostat -x 1 3
echo ""
echo "Network Traffic:"
ifstat 1 3
} > "$output_file"
echo "Report saved to $output_file"
Performance Baseline Collector
#!/bin/bash
baseline_dir="/var/baseline-data"
timestamp=$(date +%Y%m%d_%H%M%S)
mkdir -p "$baseline_dir"
# Collect system performance baseline metrics
vmstat 1 10 > "$baseline_dir/vmstat_$timestamp.log"
sar -u 1 10 > "$baseline_dir/cpu_$timestamp.log"
sar -r 1 10 > "$baseline_dir/memory_$timestamp.log"
echo "Baseline data collected in $baseline_dir"
Phase 3: Database Operations
Database Backup with Rotation
#!/bin/bash
db_name="production_db"
db_user="backup_user"
backup_dir="/backup/databases"
retention_days=7
# Create timestamped database backup
backup_file="$backup_dir/${db_name}_$(date +%Y%m%d_%H%M%S).sql.gz"
mysqldump -u "$db_user" "$db_name" | gzip > "$backup_file"
# Remove backups older than retention period
find "$backup_dir" -name "${db_name}_*.sql.gz" -mtime +"$retention_days" -delete
echo "Database backup completed: $backup_file"
Database Replication Monitor
#!/bin/bash
master_host="db-master.internal"
replica_host="db-replica.internal"
# Check replication lag on database replica
replication_status=$(mysql -h "$replica_host" -e "SHOW SLAVE STATUS\G" | grep "Seconds_Behind_Master" | awk '{print $2}')
if [ "$replication_status" != "0" ]; then
echo "WARNING: Replication lag detected: $replication_status seconds"
fi
Phase 4: Service Orchestration
Graceful Service Restart
#!/bin/bash
service_name="${1:-nginx}"
max_wait=30
# Perform graceful service restart with health check
echo "Initiating graceful restart of $service_name"
systemctl reload "$service_name"
# Wait for service to become healthy
counter=0
while [ $counter -lt $max_wait ]; do
if systemctl is-active --quiet "$service_name"; then
echo "Service $service_name is healthy"
exit 0
fi
sleep 1
((counter++))
done
echo "ERROR: Service $service_name failed to become healthy"
exit 1
Rolling Restart Orchestration
#!/bin/bash
nodes=("app-node-01" "app-node-02" "app-node-03")
service="web-application"
# Perform rolling restart across cluster nodes
for node in "${nodes[@]}"; do
echo "Restarting $service on $node"
ssh "$node" "systemctl restart $service"
# Wait for node to become healthy before proceeding
sleep 10
health_status=$(ssh "$node" "systemctl is-active $service")
if [ "$health_status" == "active" ]; then
echo "$node is healthy, proceeding to next node"
else
echo "ERROR: $node failed health check"
exit 1
fi
done
echo "Rolling restart completed successfully"
Phase 5: Log Management
Log Rotation Script
#!/bin/bash
log_dir="/var/log/application"
archive_dir="/var/log/archive"
max_age_days=30
mkdir -p "$archive_dir"
# Rotate and compress application logs
for logfile in "$log_dir"/*.log; do
if [ -f "$logfile" ]; then
filename=$(basename "$logfile")
gzip -c "$logfile" > "$archive_dir/${filename}_$(date +%Y%m%d).gz"
> "$logfile" # Truncate original log
fi
done
# Remove archived logs older than retention period
find "$archive_dir" -name "*.gz" -mtime +"$max_age_days" -delete
echo "Log rotation completed"
Critical Event Extractor
#!/bin/bash
source_log="${1:-/var/log/syslog}"
output_file="critical_events_$(date +%Y%m%d_%H%M).txt"
# Extract critical and error events from system logs
grep -iE "critical|error|fatal|panic|segfault" "$source_log" \
| grep -v "informational" \
> "$output_file"
event_count=$(wc -l < "$output_file")
echo "Found $event_count critical events, saved to $output_file"
if [ "$event_count" -gt 100 ]; then
echo "WARNING: High volume of critical events detected"
fi
Phase 6: Network Operations
Port Availability Checker
#!/bin/bash
host="${1:-localhost}"
ports=(80 443 22 3306 5432)
echo "Port Availability Check for $host"
echo "=================================="
for port in "${ports[@]}"; do
if timeout 2 bash -c "echo >/dev/tcp/$host/$port" 2>/dev/null; then
echo "Port $port: OPEN"
else
echo "Port $port: CLOSED"
fi
done
API Endpoint Health Monitor
#!/bin/bash
endpoints=(
"https://api.company.com/health"
"https://api.company.com/status"
"https://internal-api.company.local/ready"
)
log_file="/var/log/api-health.log"
timestamp=$(date '+%Y-%m-%d %H:%M:%S')
for endpoint in "${endpoints[@]}"; do
http_code=$(curl -o /dev/null -s -w "%{http_code}" --max-time 5 "$endpoint")
if [ "$http_code" == "200" ]; then
echo "[$timestamp] $endpoint: HEALTHY ($http_code)" >> "$log_file"
else
echo "[$timestamp] $endpoint: UNHEALTHY ($http_code)" >> "$log_file"
# Trigger alert logic here
fi
done
Phase 7: Backup Operations
Incremental Backup Manager
#!/bin/bash
source_dir="/data/critical"
backup_base="/backup/incremental"
snapshot_file="$backup_base/snapshot.txt"
mkdir -p "$backup_base"
# Perform incremental backup using rsync
rsync -av --delete \
--backup --backup-dir="$backup_base/$(date +%Y%m%d_%H%M%S)" \
"$source_dir/" "$backup_base/current/"
# Update snapshot timestamp
date > "$snapshot_file"
echo "Incremental backup completed"
Disaster Recovery Validator
#!/bin/bash
backup_location="/backup/disaster-recovery"
test_restore_path="/tmp/restore-test"
# Validate disaster recovery backup integrity
latest_backup=$(ls -t "$backup_location"/*.tar.gz | head -1)
mkdir -p "$test_restore_path"
tar -xzf "$latest_backup" -C "$test_restore_path" 2>/dev/null
if [ $? -eq 0 ]; then
echo "Backup integrity validated: $latest_backup"
rm -rf "$test_restore_path"
else
echo "ERROR: Backup validation failed for $latest_backup"
exit 1
fi
Quick Reference
Production Script Best Practices
| Practice | Description |
|---|---|
| Error handling | Use set -euo pipefail for strict error checking |
| Logging | Redirect output to timestamped log files |
| Idempotency | Design scripts to be safely re-runnable |
| Atomicity | Use temporary files and atomic moves |
| Validation | Check prerequisites before executing critical operations |
Common Deployment Patterns
| Pattern | Use Case |
|---|---|
| Blue-green | Zero-downtime deployments |
| Canary | Gradual rollout with monitoring |
| Rolling | Sequential node updates |
| Immutable | Complete infrastructure replacement |
Monitoring Thresholds
| Metric | Warning | Critical |
|---|---|---|
| CPU Usage | 70% | 90% |
| Memory Usage | 80% | 95% |
| Disk Space | 75% | 90% |
| Load Average | 2.0 | 4.0 |
Constraints and Limitations
- Always test scripts in non-production environments first
- Use absolute paths to avoid ambiguity in automated execution
- Quote all variables to handle whitespace and special characters properly
- Many operations require elevated privileges (root or sudo)
- Implement proper error handling and logging for production use
- Consider idempotency when designing automation scripts
- Use
bash -x script.shfor debugging and troubleshooting - Validate input parameters before executing critical operations
- Implement appropriate timeout mechanisms for network operations
- Document all external dependencies and required system packages