agenticdevops Log Analysis
Techniques for analyzing logs across different platforms and formats.
When to Use This Skill
Use this skill when:
- Investigating errors or issues
- Searching for patterns in logs
- Correlating events across systems
- Building log queries
Universal Patterns
Search Basics
# Simple search
grep "ERROR" app.log
# Case insensitive
grep -i "error" app.log
# With line numbers
grep -n "ERROR" app.log
# With context (3 lines before/after)
grep -C 3 "ERROR" app.log
grep -B 3 -A 3 "ERROR" app.log
# Count occurrences
grep -c "ERROR" app.log
Multiple Patterns
# OR - match any
grep -E "ERROR|WARN|FATAL" app.log
# AND - match all (same line)
grep "ERROR" app.log | grep "database"
# NOT - exclude pattern
grep "ERROR" app.log | grep -v "expected"
Time-Based Filtering
# Last hour (if timestamp is in log)
grep "$(date '+%Y-%m-%d %H')" app.log
# Date range
awk '/2024-01-15 10:00/,/2024-01-15 11:00/' app.log
# Recent entries (tail)
tail -1000 app.log | grep "ERROR"
Kubernetes Logs
Pod Logs
# Current logs
kubectl logs <pod>
# Previous container (after crash)
kubectl logs <pod> --previous
# Follow live
kubectl logs -f <pod>
# Last N lines
kubectl logs --tail=100 <pod>
# Since time
kubectl logs --since=1h <pod>
# All containers in pod
kubectl logs <pod> --all-containers
# By label
kubectl logs -l app=nginx --all-containers
Multi-Pod Logs
# All pods with label
kubectl logs -l app=myapp --all-containers --prefix
# Stern (better multi-pod tailing)
stern myapp -n namespace
# With regex
stern "myapp-.*" --since 1h
Search in Logs
# Grep in kubectl logs
kubectl logs <pod> | grep -i error
# With timestamps
kubectl logs --timestamps <pod> | grep "ERROR"
# Recent errors
kubectl logs --since=1h <pod> | grep -E "ERROR|Exception"
Docker Logs
# Basic logs
docker logs <container>
# Follow
docker logs -f <container>
# Tail
docker logs --tail 100 <container>
# Since time
docker logs --since 1h <container>
# With timestamps
docker logs -t <container>
# Search
docker logs <container> 2>&1 | grep "ERROR"
JSON Logs (jq)
Basic Parsing
# Pretty print
cat log.json | jq .
# Extract field
cat log.json | jq '.message'
# Multiple fields
cat log.json | jq '{time: .timestamp, msg: .message}'
Filtering
# Filter by field value
cat log.json | jq 'select(.level == "error")'
# Contains string
cat log.json | jq 'select(.message | contains("database"))'
# Multiple conditions
cat log.json | jq 'select(.level == "error" and .service == "api")'
JSONL (JSON Lines)
# Each line is JSON
cat logs.jsonl | jq -c 'select(.level == "error")'
# Extract field from each line
cat logs.jsonl | jq -r '.message'
# Count by level
cat logs.jsonl | jq -r '.level' | sort | uniq -c | sort -rn
CloudWatch Logs
# Tail logs
aws logs tail /aws/lambda/function-name --follow
# Since time
aws logs tail /aws/lambda/function-name --since 1h
# Filter pattern
aws logs tail /aws/lambda/function-name --filter-pattern "ERROR"
CloudWatch Insights
# Start query
aws logs start-query \
--log-group-name /aws/lambda/function-name \
--start-time $(date -d '1 hour ago' +%s) \
--end-time $(date +%s) \
--query-string '
fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 50
'
# Get results
aws logs get-query-results --query-id <query-id>
Common Patterns
Error Aggregation
# Top error messages
grep "ERROR" app.log | sort | uniq -c | sort -rn | head -20
# Errors per hour
grep "ERROR" app.log | awk '{print $1, $2}' | cut -d: -f1 | uniq -c
Response Time Analysis
# Extract response times (assuming format: "response_time=123ms")
grep -oP 'response_time=\K\d+' app.log | \
awk '{sum+=$1; count++} END {print "avg:", sum/count, "count:", count}'
# Slow requests (>1000ms)
grep -P 'response_time=\d{4,}' app.log
Status Code Analysis
# Count by status code
grep -oP 'status=\K\d+' app.log | sort | uniq -c | sort -rn
# 5xx errors
grep -P 'status=5\d\d' app.log
IP/User Analysis
# Top IPs
grep -oP '\d+\.\d+\.\d+\.\d+' access.log | sort | uniq -c | sort -rn | head -10
# Requests per user
grep -oP 'user=\K\S+' app.log | sort | uniq -c | sort -rn
Correlation Techniques
By Request ID
# Find all logs for a request
grep "request_id=abc123" *.log
# Across pods
kubectl logs -l app=myapp --all-containers | grep "request_id=abc123"
By Timestamp
# Events around a specific time
awk '/2024-01-15 10:30:4/,/2024-01-15 10:30:5/' app.log
Across Services
# Find related events
for service in api worker database; do
echo "=== $service ==="
grep "order_id=12345" /var/log/$service.log
done
Log Formats
Apache/Nginx Access Logs
# Status codes
awk '{print $9}' access.log | sort | uniq -c | sort -rn
# Response times (if configured)
awk '{print $NF}' access.log | sort -n | tail -20
# Top URLs
awk '{print $7}' access.log | sort | uniq -c | sort -rn | head -10
Syslog
# By service
grep "sshd" /var/log/syslog
# Failed logins
grep "Failed password" /var/log/auth.log
# By severity
grep -E "(error|crit|alert|emerg)" /var/log/syslog
Quick Reference
# Find errors in last hour
grep "$(date '+%Y-%m-%d %H')" app.log | grep -i error
# Top 10 error messages
grep -i error app.log | sort | uniq -c | sort -rn | head -10
# JSON logs: filter and format
cat logs.jsonl | jq 'select(.level=="error") | "\(.timestamp) \(.message)"'
# Kubernetes: errors across all pods
kubectl logs -l app=myapp --all-containers --since=1h | grep -i error
# AWS CloudWatch: recent errors
aws logs tail /aws/lambda/func --since 1h --filter-pattern "ERROR"
Related Skills
- k8s-debug: For Kubernetes-specific log analysis
- docker-ops: For Docker log management
- incident-response: For correlating logs during incidents