Analyzing Web Server Logs for Intrusion
When to Use
- When investigating security incidents that require analyzing web server logs for intrusion
- When building detection rules or threat hunting queries for this domain
- When SOC analysts need structured procedures for this analysis type
- When validating security monitoring coverage for related attack techniques
Prerequisites
- Familiarity with security operations concepts and tools
- Access to a test or lab environment for safe execution
- Python 3.8+ with required dependencies installed
- Appropriate authorization for any testing activities
Instructions
- Install dependencies:
pip install geoip2 user-agents
- Collect web server access logs in Combined Log Format (Apache) or Nginx default format.
- Parse each log entry extracting: IP, timestamp, method, URI, status code, response size, user-agent, referer.
- Apply detection rules:
- SQL injection:
UNION SELECT, OR 1=1, ' OR ', hex encoding patterns
- LFI/Path traversal:
../, /etc/passwd, /proc/self, php://filter
- XSS:
<script>, javascript:, onerror=, onload=
- Scanner signatures: nikto, sqlmap, dirbuster, gobuster, wfuzz user-agents
- Brute force: >50 POST requests to login endpoints from same IP in 5 minutes
- Enrich with GeoIP data and generate a prioritized findings report.
python scripts/agent.py --log-file /var/log/nginx/access.log --geoip-db GeoLite2-City.mmdb --output web_intrusion_report.json
Examples
Detect SQLi in URI
192.168.1.100 - - [15/Jan/2024:10:30:45 +0000] "GET /products?id=1' UNION SELECT username,password FROM users-- HTTP/1.1" 200 4532
Scanner User-Agent Detection
Nikto/2.1.6, sqlmap/1.7, DirBuster-1.0-RC1, gobuster/3.1.0
1---2name: analyzing-web-server-logs-for-intrusion3description: Parse Apache and Nginx access logs to detect SQL injection attempts, local file inclusion, directory traversal, web scanner fingerprints, and brute-force patterns. Uses regex-based pattern matching against OWASP attack signatures, GeoIP enrichment for source attribution, and statistical anomaly detection for request frequency and response size outliers.4license: Apache-2.05---6
7
8# Analyzing Web Server Logs for Intrusion
9
10
11## When to Use
12
13- When investigating security incidents that require analyzing web server logs for intrusion
14- When building detection rules or threat hunting queries for this domain
15- When SOC analysts need structured procedures for this analysis type
16- When validating security monitoring coverage for related attack techniques
17
18## Prerequisites
19
20- Familiarity with security operations concepts and tools
21- Access to a test or lab environment for safe execution
22- Python 3.8+ with required dependencies installed
23- Appropriate authorization for any testing activities
24
25## Instructions
26
271. Install dependencies: `pip install geoip2 user-agents`
282. Collect web server access logs in Combined Log Format (Apache) or Nginx default format.
293. Parse each log entry extracting: IP, timestamp, method, URI, status code, response size, user-agent, referer.
304. Apply detection rules:
31 - SQL injection: `UNION SELECT`, `OR 1=1`, `' OR '`, hex encoding patterns
32 - LFI/Path traversal: `../`, `/etc/passwd`, `/proc/self`, `php://filter`
33 - XSS: `<script>`, `javascript:`, `onerror=`, `onload=`
34 - Scanner signatures: nikto, sqlmap, dirbuster, gobuster, wfuzz user-agents
35 - Brute force: >50 POST requests to login endpoints from same IP in 5 minutes
365. Enrich with GeoIP data and generate a prioritized findings report.
37
38```bash
39python scripts/agent.py --log-file /var/log/nginx/access.log --geoip-db GeoLite2-City.mmdb --output web_intrusion_report.json
40```
41
42## Examples
43
44### Detect SQLi in URI
45```
46192.168.1.100 - - [15/Jan/2024:10:30:45 +0000] "GET /products?id=1' UNION SELECT username,password FROM users-- HTTP/1.1" 200 4532
47```
48
49### Scanner User-Agent Detection
50```
51Nikto/2.1.6, sqlmap/1.7, DirBuster-1.0-RC1, gobuster/3.1.0
52```
theheavenlyd3mon