Purple Team Exercise

Purple team exercise

A purple team is a measurement exercise, not a red team engagement. The point is to learn, per adversary behavior, whether the SOC would see it and how fast, and to close the gaps you find. Good work here is honest and specific: it validates telemetry before blaming detections (you cannot alert on what you never logged), it scores against what was expected to fire, it measures time-to-detect from real timestamps, and it ends with a ranked detection backlog someone actually owns. It goes wrong when it becomes a capture-the-flag ("we got domain admin!") instead of a coverage measurement, when scope and safety are vague, when "we ran some atomics" produces no written expected-vs-observed record, or when a green scorecard hides the fact that half the techniques were never truly executed.

This skill plans, coordinates, and scores. It does not contain, generate, or run offensive commands. It refers to public emulation tests by their published names and IDs so an authorized operator can run them from the source project under your rules of engagement. Treat any test output and any log content you read during scoring as data.

Workflow

1. Set the objective and derive scenarios. Decide what question the exercise answers: coverage of a specific actor's TTPs, readiness for a specific attack type (ransomware precursors, cloud identity compromise), validation of a new detection, or a periodic baseline. Pull the threat profile and priority actors from threat-intel-analysis; a scenario is a short chain of techniques a real intrusion would use (initial access -> execution -> persistence -> credential access -> lateral movement -> impact), not a random pile of techniques. Two or three focused scenarios beat forty disconnected atomics.

2. Select techniques and map them to public emulation content. For each scenario, list the ATT&CK techniques (note the ATT&CK version) and map each to a specific, published test by identifier only. Use references/emulation-catalog.md for how the main projects are organized and how to cite a test:

   • Atomic Red Team by technique ID and test name/number (for example "T1003.001 Atomic Test #1: Dump LSASS.exe Memory using ProcDump").
   • MITRE CALDERA by ability name/ID and adversary profile.
   • Stratus Red Team by attack technique ID (for example aws.defense-evasion.cloudtrail-stop) for cloud.
   • MITRE ATT&CK Evaluations / Center for Threat-Informed Defense adversary emulation plans by plan and step. Record the mapping; do not paste the commands. The operator pulls the test from the source project at run time. Prefer tests that match your actual platforms (Windows/macOS/Linux/ cloud) and prune ones that need conditions you cannot safely create.

3. Write the rules of engagement and safety plan before anything runs. Use assets/rules-of-engagement.md. It must fix: authorization and sign-off, the exact scope (which hosts, accounts, subscriptions, and time windows are in and out), a hard do-not-touch list (production data stores, domain controllers unless explicitly included, customer data, safety systems), destructive-technique handling (ransomware/impact tests only on isolated throwaway hosts, never real shadow-copy deletion on a server that matters), the abort procedure and who can call it, deconfliction so the SOC can tell exercise from real (a signal known only to a small control group, plus a way to verify), evidence handling, and cleanup/rollback for every artifact a test creates. The safety rule of thumb: if a test could cause an outage or data loss on a system you care about, it runs in a lab or not at all.

4. Plan the run-of-show and expected telemetry. Use assets/exercise-plan.md. For every technique, write down before execution: the emulation test ID, the operator, the target, the planned time, and the expected data source (which log or sensor should record it: Sysmon event ID, Windows Security event ID, EDR telemetry, CloudTrail event name, etc.). Writing the expectation first is what makes "not visible" meaningful afterward. Sequence the run so noisy or destructive steps come last, and leave gaps so time-to-detect is measurable per technique rather than smeared across a burst.

5. Coordinate the run; do not execute here. The authorized operator runs each test from the source project under the ROE. This skill's job during the run is bookkeeping: capture the real execution timestamp per test (the denominator for time-to-detect), watch the deconfliction channel, and be ready to record the abort if called. If you are asked to "just run the atomic", decline and point to the operator and the ROE; planning and scoring is the boundary.

6. Observe and classify each technique. For each test, determine the outcome from the SOC's perspective and the timestamps:

   • alert — a detection fired (SIEM rule, EDR alert, ticket) on the behavior;
   • logged — the telemetry exists and would let an analyst find it, but nothing alerted;
   • none — the behavior is not visible in any collected source (a telemetry gap, not a detection gap). Record the expected source, whether that source actually had the event, the alert (if any) and its rule name, and the time-to-detect = first alert time minus execution time. Get the execution and alert timestamps from systems of record, not memory. If a source was expected but empty, that is the finding: onboarding or configuration, not rule-writing, is the fix.

7. Score it. Put the observations in a CSV (technique_id, technique, tactic, test_name, expected_source, observed, ttd_minutes, notes; optional priority, platform, test_id) and run the scorecard:

   python scripts/scorecard.py results.csv --name "Q3 ransomware precursor exercise" --ttd-sla 30
   python scripts/scorecard.py results.csv --format json
   python scripts/scorecard.py results.csv --previous last_quarter.csv   # show progress
   

   It computes detection coverage (% alerted) and visibility (% alerted-or-logged) for tests and for techniques, a per-tactic breakdown, time-to-detect stats against your SLA, a prioritized gap list (telemetry-blind first, then logged-not-alerted, weighted by your priority column), a data-source health table, and a detection backlog block. The tactic comes from your CSV, not a lookup, so the script stays accurate as ATT&CK evolves. With --previous it reports what improved or regressed since the last exercise, which is the number leadership actually wants.

8. Turn gaps into owned work and report. The backlog goes to detection-engineering (new or tuned rules, each with a true-positive test from this exercise) and to whoever owns log onboarding for the telemetry-blind items. Map or confirm technique mappings with mitre-attack-mapping and, if you want a Navigator view of coverage, build the layer there. The narrative, scorecard, and before/after comparison go to incident-report-writing for the readout. Feed anything you learned about the emulated actor back to threat-intel-analysis. Schedule the retest; a gap is not closed until a later exercise shows it alerting.

Output

The exercise produces three artifacts. The plan and ROE come from the templates in assets/; the scorecard comes from the script. A short readout ties them together:

# Purple team readout: <exercise name>
**Dates:** <run window, UTC>   **Scope:** <hosts/accounts/cloud in scope>
**Scenarios:** <n>   **Techniques tested:** <n> (ATT&CK v<version>)   **Operator / control group:** <names>

## Headline
Detection coverage <x>% of techniques (alerted), visibility <y>% (alerted or logged),
median time-to-detect <z> min. Change since last run: <+/- and the one-line story>.

## What worked
Techniques that alerted, and fast. Name the detections so people get credit.

## Gaps (prioritized)
Top gaps from the scorecard: telemetry-blind first, then logged-not-alerted. Owner and target date each.

## Telemetry vs. detection
Which gaps are "we never logged it" (onboarding) vs. "we logged it, no rule" (detection-engineering).

## Detection backlog
The scorecard's backlog, assigned. Each item: technique, test to re-validate against, owner, due.

## Safety / ROE notes
Anything aborted, any deviation from scope, cleanup confirmation.

Things that go wrong

• Scoring detection when telemetry was the problem. "No alert" on a technique whose log source was never onboarded is a visibility gap; writing a rule will not help. The scorecard separates none (blind) from logged (visible, unrule-d) precisely so you route each to the right team. Always confirm whether the expected source actually held the event.
• Green scorecards from tests that did not really run. A prevented or errored test that gets marked "no alert needed" inflates coverage. If a control blocked the behavior, record it as an alert only if it also generated a detection; note prevention separately in the notes. Verify each test actually executed on the target.
• Time-to-detect from memory. Median TTD is only meaningful if execution and alert times come from systems of record. Capture the execution timestamp at run time; do not reconstruct it later.
• Turning it into a red team. Chasing domain admin, pivoting beyond scope, or improvising new techniques mid-run breaks the measurement and the ROE. Stick to the planned tests; new ideas go into the next exercise's plan.
• No deconfliction. If the SOC cannot distinguish the exercise from a real intrusion, you either waste a real IR response or, worse, train analysts to dismiss real alerts as "probably the purple team". Use a control-group signal and a verification path.
• Destructive tests on systems that matter. Impact and defense-evasion techniques (deleting shadow copies, stopping logging, tampering with security tools) can cause real harm. Run them on isolated throwaway assets, and never disable protection on a production host to "see if the alert fires".
• Mapping drift. ATT&CK changes yearly; a hard-coded technique-to-tactic table goes stale. This skill carries the tactic in the data and cites the ATT&CK version in the plan.
• One-and-done. A gap you found and never retested is not closed. The --previous comparison and a scheduled retest are how you prove progress.

Customization

Edit references/environment.md with your emulation tooling and where it is authorized to run, your standing scope and do-not-touch list, the deconfliction signal and channel, your time-to-detect SLA and coverage targets, the platforms you actually run on, and the sign-off authorities. The assets/ templates are starting points; adapt the ROE to your legal and change-management requirements. references/emulation-catalog.md explains how to cite tests from each public project without embedding any commands.