Implementing Network Traffic Baselining
Overview
Network traffic baselining establishes normal communication patterns by analyzing historical NetFlow/IPFIX data to create statistical profiles of expected behavior. This skill uses Python pandas to compute hourly and daily traffic distributions, per-host byte/packet counts, protocol ratios, and top-N talker profiles. Anomalies are detected using z-score thresholds and IQR (interquartile range) outlier methods, enabling SOC analysts to identify deviations such as data exfiltration spikes, beaconing patterns, and unusual port usage.
When to Use
Trigger phrases:
"implementing network traffic baselining"
"Build network traffic baselines from NetFlow/IPFIX data using Python pandas for "
When deploying or configuring implementing network traffic baselining capabilities in your environment
When establishing security controls aligned to compliance requirements
When building or improving security architecture for this domain
When conducting security assessments that require this implementation
Prerequisites
- NetFlow v5/v9 or IPFIX flow data exported as CSV or JSON
- Python 3.8+ with pandas and numpy libraries
- Historical flow data (minimum 7 days recommended for baseline)
Steps
# Example: IOC detection
import re
IOC_PATTERNS = {
"ip": r"\b(?:\d{1,3}\.){3}\d{1,3}\b",
"domain": r"\b[a-z0-9-]+\.[a-z]{2,}\b",
"hash_md5": r"\b[a-f0-9]{32}\b",
"hash_sha256": r"\b[a-f0-9]{64}\b",
}
def extract_iocs(text: str) -> dict:
return {k: re.findall(v, text) for k, v in IOC_PATTERNS.items()}
- Ingest NetFlow/IPFIX records from CSV or JSON exports
- Compute hourly and daily traffic volume distributions (bytes, packets, flows)
- Build per-source-IP baseline profiles with mean, median, standard deviation
- Calculate protocol and port distribution baselines
- Apply z-score anomaly detection to identify statistical outliers
- Flag flows exceeding IQR-based thresholds as potential anomalies
- Generate baseline report with anomaly alerts
Expected Output
JSON report containing traffic baselines (hourly/daily profiles), per-host statistics, detected anomalies with z-scores, and top talker rankings with deviation indicators.
When NOT to Use
- You need to test the implementation (use performing-* skills)
- Task is about configuring existing tools (use configuring-* skills)
- You need to analyze security events (use analyzing-* skills)
- Task is about building detection rules (use building-* skills)
- You don't have access to the target environment
- Task requires vendor-specific expertise (consult vendor docs)
Red Flags
- Performing actions without explicit written authorization from the asset owner
- Testing against production systems without a defined scope and rules of engagement
- Capturing traffic on networks without authorization or privacy considerations
- Leaving packet captures containing sensitive data unencrypted on disk
- Deploying inline blocking rules without testing for false positives first
Process
- Prepare — Gather requirements, verify prerequisites, set up environment
- Execute — Run implementing network traffic baselining workflow with configured parameters
- Verify — Validate output meets requirements, document results
Verification
- All steps executed successfully against a test environment before production use
- Output documented with screenshots or logs demonstrating expected behavior
- Captures verified as complete with no dropped packets
- Detection rules tested against known-benign traffic for false positive rate
- Alert thresholds validated and tuned to reduce noise
Anti-Rationalization Table
| Rationalization |
Reality |
| "We are too small to be targeted" |
Automated attacks target everyone. Size does not matter. |
| "Security slows us down" |
A breach slows you down 100x more. Build security in from the start. |
| "We will fix it after launch" |
Vulnerabilities in production are exploited within hours. Fix before deploy. |
1---2name: implementing-network-traffic-baselining3description: Use when build network traffic baselines from NetFlow/IPFIX data using4 Python pandas for statistical analysis, z-score anomaly detection, and hourly/daily5 traffic pattern profiling. Use when building network traffic baselines from netflow/ipfix6 data using python pandas.7domain: cybersecurity8subdomain: network-security9tags:10- netflow11- ipfix12- traffic-analysis13- baselining14- anomaly-detection15- pandas16- network-monitoring17version: '1.0'18author: oyi7719license: Apache-2.020nist_csf:21- PR.IR-0122- DE.CM-0123- ID.AM-0324- PR.DS-0225category: cybersecurity26metadata:27 source: https://skillmd.com/skills/oyi77/implementing-network-traffic-baselining28---293031# Implementing Network Traffic Baselining3233## Overview3435Network traffic baselining establishes normal communication patterns by analyzing historical NetFlow/IPFIX data to create statistical profiles of expected behavior. This skill uses Python pandas to compute hourly and daily traffic distributions, per-host byte/packet counts, protocol ratios, and top-N talker profiles. Anomalies are detected using z-score thresholds and IQR (interquartile range) outlier methods, enabling SOC analysts to identify deviations such as data exfiltration spikes, beaconing patterns, and unusual port usage.363738## When to Use39**Trigger phrases:**40- "implementing network traffic baselining"41- "Build network traffic baselines from NetFlow/IPFIX data using Python pandas for "424344- When deploying or configuring implementing network traffic baselining capabilities in your environment45- When establishing security controls aligned to compliance requirements46- When building or improving security architecture for this domain47- When conducting security assessments that require this implementation4849## Prerequisites5051- NetFlow v5/v9 or IPFIX flow data exported as CSV or JSON52- Python 3.8+ with pandas and numpy libraries53- Historical flow data (minimum 7 days recommended for baseline)5455## Steps5657```python58# Example: IOC detection59import re6061IOC_PATTERNS = {62 "ip": r"\b(?:\d{1,3}\.){3}\d{1,3}\b",63 "domain": r"\b[a-z0-9-]+\.[a-z]{2,}\b",64 "hash_md5": r"\b[a-f0-9]{32}\b",65 "hash_sha256": r"\b[a-f0-9]{64}\b",66}6768def extract_iocs(text: str) -> dict:69 return {k: re.findall(v, text) for k, v in IOC_PATTERNS.items()}70```71721. Ingest NetFlow/IPFIX records from CSV or JSON exports732. Compute hourly and daily traffic volume distributions (bytes, packets, flows)743. Build per-source-IP baseline profiles with mean, median, standard deviation754. Calculate protocol and port distribution baselines765. Apply z-score anomaly detection to identify statistical outliers776. Flag flows exceeding IQR-based thresholds as potential anomalies787. Generate baseline report with anomaly alerts7980## Expected Output8182JSON report containing traffic baselines (hourly/daily profiles), per-host statistics, detected anomalies with z-scores, and top talker rankings with deviation indicators.83## When NOT to Use8485- You need to test the implementation (use performing-* skills)86- Task is about configuring existing tools (use configuring-* skills)87- You need to analyze security events (use analyzing-* skills)88- Task is about building detection rules (use building-* skills)89- You don't have access to the target environment90- Task requires vendor-specific expertise (consult vendor docs)919293## Red Flags9495- Performing actions without explicit written authorization from the asset owner96- Testing against production systems without a defined scope and rules of engagement97- Capturing traffic on networks without authorization or privacy considerations98- Leaving packet captures containing sensitive data unencrypted on disk99- Deploying inline blocking rules without testing for false positives first100101## Process1021031. **Prepare** — Gather requirements, verify prerequisites, set up environment1041. **Execute** — Run implementing network traffic baselining workflow with configured parameters1051. **Verify** — Validate output meets requirements, document results106107## Verification108109- All steps executed successfully against a test environment before production use110- Output documented with screenshots or logs demonstrating expected behavior111- Captures verified as complete with no dropped packets112- Detection rules tested against known-benign traffic for false positive rate113- Alert thresholds validated and tuned to reduce noise114115## Anti-Rationalization Table116117| Rationalization | Reality |118|---|---|119| "We are too small to be targeted" | Automated attacks target everyone. Size does not matter. |120| "Security slows us down" | A breach slows you down 100x more. Build security in from the start. |121| "We will fix it after launch" | Vulnerabilities in production are exploited within hours. Fix before deploy. |