Gds Detective

GDS Detective

A GDS detective applies proven detection recipes to surface anomaly types that basic find_anomalies misses: joint deviations in event patterns, Simpson's paradox in composites, temporal bursts, neighbor contamination, trajectory shapes, collective drift, and cross-pattern discrepancies.

Each recipe below is a tested sequence of tool calls. Adapt parameters to the sphere at hand — the shapes are universal, the thresholds are starting points.

For concrete output examples of each recipe, see references/examples.md.

FDR control and diverse selection

Detection recipes that start with find_anomalies benefit from two parameters: fdr_alpha applies Benjamini-Hochberg FDR control so that the candidate list has a bounded false discovery rate before expensive downstream recipe steps run, and select="diverse" uses submodular facility location to return candidates spanning different anomaly signatures rather than redundant near-duplicates. Set fdr_alpha=0.05 when the recipe feeds into manual verification or cross-pattern confirmation; use select="diverse" when the goal is to discover the range of anomaly types present, not just the most extreme instances. Both parameters also work on attract_boundary, find_hubs, and find_drifting_entities.

Storey adaptive FDR — fdr_method="storey" with p_value_method="chi2" expands the candidate list by 10–15% on patterns whose delta distribution has a real null mass (moderate super-anomaly regime). Both params must be set together; Storey without chi2 p-values is a no-op. On compressed or saturated distributions the effect vanishes — detective recipes should default to fdr_method="bh" and only opt into Storey when the pattern's median(delta_norm) ≈ sqrt(df) with a clean tail.

FDR axis selection — fdr_axis selects what FDR is controlled over. The default "entity" controls the false-discovery rate over per-entity anomaly tests on ||delta||² (one p-value per entity, df = dimensionality) — use it when the question is "which entities are jointly anomalous across the whole shape vector". The "per_dim" axis runs an independent BH/Storey correction per dimension on chi²(1) univariate p-values for each cell of the delta matrix and keeps an entity iff any dim's q-value clears alpha — use it when the question is "which dimension is anomalous for this entity", because a single dim driving many discoveries no longer inflates the threshold for unrelated dims (so a rare-but-real single-dim signal that would be lost in a joint norm test surfaces). The "both" axis requires the entity to survive both tests — strictest, lowest false-positive rate, smallest candidate list. Per-dim mode attaches q_values_per_dim (one q per dim), min_q_per_dim (the smallest), and dominant_q_dim_idx (the index of that smallest dim) to every returned polygon — investigator drilldown can read which dim drove each survivor straight from the result.

Ranking by per-dim significance — rank_by="min_q_per_dim" re-ranks the post-FDR survivors by smallest per-dim q-value ascending (most extreme single-dim signal first), instead of the default rank_by="delta_norm" which sorts by joint norm descending. Use when an entity with one knife-edge dimension matters more to the investigation than an entity with diffuse moderate elevation across many dims. Requires fdr_alpha set and fdr_axis ∈ {"per_dim", "both"} (no q-values otherwise); incompatible with select="diverse" (which has its own ordering). Note: chi²(1) is direction-agnostic — both extreme-positive and extreme-negative deviations produce small p-values — so on patterns where some dims are anti-signal (high |delta| correlated with non-target class, e.g. legitimate-traffic outliers in fraud detection), per-dim FDR will flag both wings of those dims. Mitigation when labels exist: use engine.dim_audit.compute_per_dim_label_auroc to identify anti-signal dims and silence them via dimension_weights={dim: 0.0} before ranking.

Multi-resolution FDR: spatial × temporal hierarchies

When the pattern declares fdr_hierarchy: (spatial) and/or fdr_temporal_hierarchy: (temporal) in sphere.yaml, find_anomalies accepts two extra params:

• fdr_resolution: "<spatial_level_name>" — gates returned anomalies to entities whose cell at this level cleared per-level BH/Storey FDR at fdr_alpha.
• fdr_temporal_resolution: "<temporal_level_name>" — same, for the temporal axis.

Both can be combined: intersection-FDR — an entity survives iff its cell cleared every named level on BOTH axes.

Cell test stat

Per-cell evidence is a Fisher exact 2×2 upper-tail p-value comparing the cell's anomaly rate against the rest of the population. Reads is_anomaly boolean column on geometry. Non-parametric, exact, copes with low-count cells (cells with few entities behave correctly — a z-test would break here).

Per-level Tippett aggregation

A coarse-level cell's evidence is the minimum p-value among its child cells (Tippett combination). This is the standard hierarchical-test composition rule and is conservative: coarse levels reject only when at least one child level fires very strongly. Per-level BH/Storey then applies to the coarse-cell p-value set.

When to use

• fdr_resolution — population-level anomaly rate jumped in a sub-group (per-bank, per-region, per-community). Surfaces "this bank is suspicious" rather than "these accounts are suspicious".
• fdr_temporal_resolution — epoch-level shifts the entity-axis test cannot see (a time window has elevated anomaly density even though individual entities pass entity-FDR).
• Both — strictest filter, useful for finding the cells where risk concentrates at the latest tag.

What this filter does and does NOT do

Multi-resolution FDR is a per-cell filter on the geographic / temporal axis. It localises anomalies to a hot-spot region and provides cell_q_spatial / cell_q_temporal / cell_path as pre-computed statistical evidence for SAR rationale. It does NOT tighten the entity-level ranking inside surviving cells. Survivor count and per-K precision depend on how the cells partition the population, not directly on alpha.

Two failure modes to watch for:

• Single-cell levels collapse to no contrast. A hierarchy level that resolves to one unique value across the whole sphere has p=1.0 by construction (single cell == whole population), so it rejects nothing. Check temporal_bucket granularity against the data's actual time range: a 90-day bucket on a 50-day sphere is one cell.
• Intersection returns zero survivors when one axis is degenerate. Drop to single-axis mode (fdr_resolution only, or fdr_temporal_resolution only) before raising alpha.

Entity-axis FDR auto-upgrades when fdr_resolution is set

When fdr_resolution or fdr_temporal_resolution is set on the entity axis, the navigator silently switches p_value_method from "rank" to "chi2" and fdr_method from "bh" to "storey" — the documented defaults are degenerate in combination with cell-level gating (rank-based p-values are uniform, BH rejects nothing, entity-level FDR collapses to a no-op and survivors order by delta_norm alone). The upgrade restores actionable ranking inside surviving cells. Explicit non-default values pass through unchanged; fdr_axis="per_dim" / "both" use chi²(1) per-dim regardless of p_value_method, so no upgrade happens there.

The user-visible surface is therefore: setting fdr_resolution=<level> is enough to get both the cell-level gate AND the workload-correct entity ranking. No manual combo needed.

Anti-signal direction

Like fdr_axis="per_dim", this surface is direction-agnostic: an anomaly-rate DROP in a cell will not flag (upper-tail test), but an anomaly-rate SPIKE will. This is the intended direction for fraud / outlier investigation.

Example invocation

# Spatial gate — entity-axis FDR auto-upgrades to chi2+Storey under
# the hood when fdr_resolution is set
find_anomalies(pattern_id="account_pattern", top_n=20, fdr_alpha=0.05,
               fdr_resolution="bank")

# Intersection — both axes named; same auto-upgrade applies. Only
# meaningful when both hierarchies are non-degenerate.
find_anomalies(pattern_id="account_pattern", top_n=20, fdr_alpha=0.05,
               fdr_resolution="bank", fdr_temporal_resolution="quarter")

# Power-user override — pass explicit values to bypass the auto-upgrade
find_anomalies(pattern_id="account_pattern", top_n=20, fdr_alpha=0.05,
               fdr_resolution="bank",
               p_value_method="chi2", fdr_method="bh")

Choosing a recipe

Event anomaly rate

Catches multi-dimensional joint deviations where each column individually looks normal. When explain_anomaly is available on event patterns, use bregman_contribution + kind to identify whether the anomaly is driven by count structure (poisson), magnitude (gaussian), or binary flags (bernoulli) — this directs which recipe to run next (burst detection for poisson, threshold analysis for gaussian, flag analysis for bernoulli).

Run both calls:

aggregate(event_pattern, group_by_line=anchor_line,
          geometry_filters={"is_anomaly": true}, limit=50)
aggregate(event_pattern, group_by_line=anchor_line, limit=50)

For each entity in the first result, compute rate = anomalous_count / total_count.

• Entities with >15% rate are suspect (baseline is typically around 5%).
• event_rate_divergence_alerts only covers entities below theta — this approach catches ALL high-rate entities.
• List ALL entities above 15% rate in the report with exact rate.
• For entity lines <50K, sampling is unnecessary.

Composite subgroup analysis (Simpson's paradox)

For each composite pattern, run with dual ranking:

aggregate_anomalies(composite_pattern, group_by="parent_key_col", limit=50)
  -> ranked by count of anomalous composites (default)
find_anomalies(composite_pattern, rank_by_property="avg_price_dim", limit=50)
  -> ranked by value metric (catches subgroup price inflation)

Parents from either ranking are suspect. Threshold: >=2 anomalous composites (not >=5 — subgroup inflation typically affects only 2-3 categories). Run both calls on each composite pattern independently.

Key insight: an entity can have NORMAL aggregate stats but inflate ONLY its top few subgroup categories. The composite pattern captures per-subgroup deviation that aggregate analysis misses.

Temporal burst detection

When sphere_overview(detail="full") returns event_rate_divergence_alerts, temporal windowed analysis helps localize the burst.

Get the temporal range from get_sphere_info or dive_solid timestamps.
You need min/max dates to pick yearly windows.

Split into yearly windows. Run one aggregate per year:
aggregate(event_pattern, group_by_line=anchor_line, metric="count",
          time_from="YYYY-01-01", time_to="YYYY+1-01-01", limit=50)
At least 3 consecutive years provide a reliable baseline.

Compare per-entity counts across years:
-> 2x+ count spike in one year vs adjacent = burst/splitting
-> Cross-reference with event_rate_divergence_alerts entity keys
-> Entities in BOTH alert list AND spike year = high confidence

If a spike year is found, drill down into quarters:
aggregate(event_pattern, group_by_line=anchor_line, metric="count",
          time_from="YYYY-01-01", time_to="YYYY-04-01", limit=50)
-> pinpoints burst to specific quarter

Yearly windows work better than halves — halves dilute single-year bursts.

Windowed aggregate for temporal burst localization: use aggregate(event_pattern_id, group_by_line, time_from=window_start, time_to=window_end) to compare event counts across time windows. Faster than dive_solid for initial burst detection — dive_solid is entity-level temporal history while windowed aggregate gives per-entity counts across the full population in one call.

If edge table available (edge_stats returns has_edge_table: true), degree_velocity(key, pattern_id) corroborates burst timing — accelerating degree alongside event spike = strong behavioral change. find_graph_geometry_tension(key, anchor_pattern, line_id=event_pattern) cross-tabs behavioural k-NN against graph adjacency: high n_suspicious_total (out-of-peer-group counterparties) corroborates burst — entity transacting with parties outside its behavioural cohort is a fraud-typical signature on AML-class data.

Neighbor contamination

A normal entity whose geometric neighbors are systematically anomalous. The entity itself is NOT flagged — the signal is in the neighborhood.

Prefer detect_neighbor_contamination if available — it uses inverted search (starts from anomalies, finds contaminated normals) which is more effective than the manual recipes below. If edge table available, contagion_score(key, pattern_id) or contagion_score_batch(keys, pattern_id) gives the exact anomalous/total neighbor ratio directly. Use manual recipes only as fallback when neither tool is available.

As-of reconstruction: contagion_score, contagion_score_batch, and degree_velocity (as well as the other edge-table graph primitives) accept an optional timestamp_cutoff parameter (Unix seconds). When set, only edges with timestamp <= cutoff are considered — use this to reconstruct neighbor contamination or burst velocity as they looked on the day of a known incident.

From anomalous outward (cheaper, try first):

find_anomalies(pattern, top_n=50) -> get top anomalous entities
For 10-15 anomalous entities:
  find_similar_entities(entity_key, pattern, top_n=10)
Check each neighbor: is it NORMAL (is_anomaly=false)?
A normal entity surrounded by anomalous entities = contamination target.
For each normal neighbor found:
  find_similar_entities(normal_key, pattern, top_n=10)
  -> check_anomaly_batch on ITS neighbors
  -> if >50% of its neighbors are anomalous: confirmed target

From population sample (broader, use if the first approach finds nothing):

Sample 50 normal entities (delta_rank_pct 30-70, is_anomaly=false)
For each: find_similar_entities(key, pattern, top_n=10)
check_anomaly_batch(neighbor_keys, pattern)
Normal entity with >50% anomalous neighbors = contamination target

This detects entities that are individually normal but positioned in anomalous geometric neighborhoods — invisible to any single-entity scan. The target is the NORMAL entity, not the anomalous neighbors.

Trajectory shape analysis

Prefer detect_trajectory_anomaly if available — it performs a full temporal scan and ranks by wasted_motion, directly targeting non-linear trajectories. Use the manual recipe below only as fallback.

When find_drifting_entities shows low displacement but dive_solid reveals a non-monotonic temporal shape (arch, V-shape, spike-recovery).

find_drifting_entities(pattern, window=365)
-> check BOTH ends: high displacement = linear drift,
   low displacement does not mean no signal
-> entities with many temporal slices but near-zero displacement may have
   non-linear trajectories (arch, V-shape) that cancel out over time

Pick 3-5 entities with moderate displacement (rank 20-40, not top-10):
dive_solid(entity_key, pattern)
-> inspect delta values across temporal slices
-> arch shape (up then down) or V-shape (down then up) = trajectory anomaly

If arch/V-shape found:
find_drifting_similar(entity_key, pattern, top_n=20)
-> finds entities with the SAME trajectory shape
-> returns a cohort sharing the deformation pattern

Report the full cohort with trajectory description.

Linear drift has high displacement. Non-linear trajectories (arch, V) have near-zero all-time displacement but distinctive shape. Only dive_solid + find_drifting_similar catches these.

Population segment shift

When find_regime_changes detects a changepoint, identify WHICH population segment shifted — not just that a shift happened.

find_regime_changes(pattern)
-> changepoint at date X with magnitude Y

Run segment analysis — check each property dimension:
find_anomalies(pattern, property_filters={"nation": "<value>"}, limit=50)
-> repeat for each major segment value (nation, category, region)
-> which segment has the most anomalies post-changepoint?

contrast_populations(pattern, pre_window, post_window)
-> which dimensions shifted? Does it match the segment hypothesis?

Report: "segment S shifted by Z% from date X — N entities affected"

Individual entities in the shifted segment may each be only mildly anomalous (below theta). The anomaly is at the GROUP level — no single entity crosses the threshold but the segment centroid moves significantly.

Collective drift

find_drifting_entities(pricing_pattern, window=365)
-> high displacement + consistent direction across ALL windows = drift

dive_solid(entity_key, pricing_pattern)
-> monotonically increasing delta across years = collective drift

find_regime_changes(pricing_pattern)
-> absent/weak changepoint = gradual shift, not step change

compare_entities(drift_entities, normal_entities, pricing_pattern)
-> which dimension drives the drift?

Use window=365 (not shorter) to amplify gradual signal. signal_quality="mixed" in sphere_overview indicates the pattern is sensitive to drift.

Multi-pattern investigation

When an entity is NOT anomalous in the expected pattern:

cross_pattern_profile(key, line_id)
-> source_count >= 2? Another pattern catches what the first missed.
-> >30% anomalous events = strong signal even if anchor says "normal"

Check related composite patterns:
Many anomalous composites from one entity = systematic issue.

composite_risk(key, line_id)
-> Wilson harmonic-mean p-value (HMP) across patterns —
   robust under positive dependence between p-values
   (the regime where multiple patterns fire on the same entity).
-> combined_p < 0.05 = significant even if no single pattern flags it.

combine_anomaly_pvalues([(pattern_id, p), ...], method="hmp")
-> Lower-level composition primitive. Pass arbitrary (pattern_id, p_value) pairs —
   works on detectors outside cross_pattern_profile (e.g. an external rules engine
   p-value, a typology recipe score converted to a p, a chain-coherent run score).
-> method="hmp" (default; robust under positive dependence) or "fisher"
   (classical independence assumption).
-> Returns combined p + per-pattern reliability_flags pass-through.

classify_detector_consensus([(pattern_id, p), ...])
-> Labels the set of detector p-values as one of: unanimous_anomaly /
   majority_anomaly / split / unanimous_normal. Use as a triage filter
   when an entity has 4+ detectors and you need an at-a-glance verdict
   before drilling. Disagreement among detectors ("split") is itself
   a signal — the entity may sit at the boundary of two different
   anomaly typologies.

passive_scan for confirmation

Use for confirmation, not discovery.

find_anomalies per pattern -> top suspects
passive_scan(line_id, threshold=1) -> all single+ source
passive_scan(line_id, threshold=2) -> confirmed multi-source only

Optional: add confidence filter for stable-anomaly confirmation
  find_anomalies(pattern_id, top_n=50, min_confidence=0.8)
  -> entities where bootstrap confidence >= 0.8 (populations <= 50K only)
  -> use as high-priority confirmation list: if an entity appears here
     AND in passive_scan(threshold=2), it is the strongest confirmation tier

Cross-line bridging: passive_scan("<anchor_line>") auto-discovers sibling lines (same source_id). Pre-0.1.x spheres need explicit sources.

Confidence-threshold guidance for confirmation tiers:

• min_confidence >= 0.8 + threshold=2 → highest-confidence tier, escalate immediately
• min_confidence 0.5-0.8 + threshold=2 → confirmed multi-source but borderline stability
• threshold=2 only (no confidence filter) → standard multi-source confirmation
• threshold=1 only → broad suspect list, expect higher FP rate

Exhaustive enumeration

NEVER report "top 3 examples" — enumerate ALL entities that match the criterion with limit=50. Incomplete entity lists are the #1 reason for missed recall in benchmarks.

• limit=50 (not default 20) on ALL investigation queries
• having={"gt": threshold} to extract full cohort above metric value
• aggregate_anomalies(group_by=...) for large anomaly populations
• If the tool returns exactly limit rows, increase limit or note truncation

"There are ~50 like this, here are 5 examples" is incomplete — list them all.

Re-ranking by property

find_anomalies ranks by delta_norm. But the most geometrically extreme entity may not be the most business-relevant.

find_anomalies(pattern, rank_by_property="<dim_from_profiling_alert>")

Run for every profiling_alert dimension to surface property-specific extremes.

Temporal artifact check

At session open, compare get_sphere_info() temporal date range against the dataset's described time window. If the temporal span is implausibly wide relative to the actual data (e.g., decades for a dataset covering months), the timestamps are a sphere construction artifact — not data corruption.

get_sphere_info() -> check temporal_range min/max
If span > 10x the expected dataset window:
  -> flag as sphere construction artifact in report (INFO, not CRITICAL)
  -> use slice indices and relative ordering for temporal analysis
  -> do NOT report absolute timestamps as evidence of data corruption

This prevents misclassifying synthetic temporal axes as critical findings. Temporal tools (dive_solid, find_drifting_entities, find_regime_changes) still work correctly on the delta structure — the relative ordering of slices is reliable even when absolute timestamps are synthetic.

Physical bound validation

For event patterns with movement-related dimensions (speed, distance, duration, velocity, or similar), validate extremes against physical plausibility before interpreting them as behavioral anomalies.

rank_by_property(event_pattern, "<speed_dim>", direction="desc", limit=50)
rank_by_property(event_pattern, "<distance_dim>", direction="desc", limit=50)
rank_by_property(event_pattern, "<duration_dim>", direction="asc", limit=50)

Check the extremes against domain-appropriate physical bounds:

• Speed exceeding plausible limits for the transport mode
• Distance exceeding plausible range for the service area
• Duration near zero with non-zero distance (teleportation artifact)
• Duration extremely high with near-zero distance (meter/GPS stuck)

Entities violating physical bounds are data quality findings — GPS artifacts, meter errors, or system clock issues — not behavioral anomalies. Report them as data quality with the specific bound violated and the entity count affected.

Payment type segmentation

For event patterns with tip, gratuity, or rate dimensions, ALWAYS segment by payment method before drawing conclusions about anomaly distribution. Different payment methods have structurally different tipping behavior that is definitional, not anomalous.

aggregate(event_pattern, group_by="<payment_type_dim>",
          geometry_filters={"is_anomaly": true}, limit=50)
aggregate(event_pattern, group_by="<payment_type_dim>", limit=50)

Compare anomaly rates per payment type. If one payment type has a dramatically higher anomaly rate AND the driving dimension is tip-related, the signal is likely payment-type structural:

• Cash/non-electronic payments typically have zero tips by definition (no electronic tip capture mechanism)
• Mixing payment types inflates zero-tip anomaly counts

Report the structural explanation and recommend payment-type-stratified analysis rather than flagging the entire zero-tip population as anomalous.

Unregistered entity check

For entity patterns with an ID, vendor, source, or provider dimension, verify that all distinct values map to known entities in the domain.

get_line_profile(line_id, "<id_dim>") -> enumerate distinct values
Cross-reference against known entities in the domain context.

Unregistered or unexpected IDs are a geometric signal — they form isolated clusters because their behavior doesn't match any registered population. For each unregistered value:

search_entities(line_id, "<id_dim>", "<unknown_value>") -> get entity keys
find_anomalies(pattern, property_filters={"<id_dim>": "<unknown_value>"})

Characterize: is this a test entity, a data migration artifact, a decommissioned operator, or an undocumented participant? The geometric isolation alone is a finding — combine with business context to assess severity.

When things don't work

• Tool returns empty results — try different parameters, wider sample, or a different pattern. Some patterns may not have the data shape you expect.
• Tool errors — check pattern_id and version, verify sphere is open. If the error persists, report it rather than retrying in a loop.
• No anomalies found — not every sphere has every anomaly type. Report "no signal detected" as a valid finding.
• Aggregate returns 0 rows — the event pattern may lack events in the requested time window. Widen the window or check the temporal range with get_sphere_info.
• find_similar_entities returns only the query entity — the pattern population may be too small or the entity is an extreme outlier with no geometric neighbors. Try a different pattern or increase top_n.

Skill delegation

Full detection examples: references/examples.md