Context Graph Mining

Context Graph Mining

This skill mines a procurement-agent Arize project for decision patterns, builds a context graph view of how the organization actually decides, and proposes concrete updates to the agent's rules and prompts.

When to use

• After a batch of human overrides has been applied (e.g. Vera Fye reviewing 100+ requests via procurement-reviewer/).
• When you want to surface rule-reality gaps — where the agent's encoded rules disagree with how the org actually decides.
• When you want concrete diff proposals against data/policies.json, data/vendors.json, or the evaluator system prompt.

What you have access to

• The procurement-agent Arize project — export traces, spans, and annotations via the arize-trace skill or the arize.exporter.ArizeExportClient Python SDK.
• The procurement-agent/ codebase — src/agent/evaluator.py, src/agent/tools.py, data/policies.json, data/vendors.json.
• The procurement-reviewer/src/reviewer.py system prompt — Vera Fye's institutional knowledge as currently encoded.

Workflow

Phase 1 — Discovery

Build the trace inventory.

For each session.id (one per purchase request):

• The process.run root span — agent's recommendation, confidence, reasoning, and tool calls (check_policy, lookup_vendor, check_budget).
• The override.run root span — the agent re-run with Vera's decision injected.
• Annotations on the override root: Reviewer Decision, Reviewer Confidence, Reviewer Name labels and the annotation.notes block (reasoning + precedent + conditions).

Use the arize-trace skill for shell-level exports. For programmatic export of annotations, use ArizeExportClient.export_model_to_df and read the annotation.<name>.label and annotation.notes columns.

Sanity check: confirm len(sessions) matches the expected request count and that every override span has a non-null Reviewer Decision label.

Phase 2 — Pattern discovery

Three priority queries surface the actionable patterns:

1. Rule-reality gap — every session where agent.recommendation != reviewer.decision. Group by (vendor, department, urgency, amount-band). Clusters with N >= 5 examples are candidates.
2. Precedent reuse — extract the precedent field from annotation.notes; cluster by similarity. Frequently-cited precedents are tribal knowledge that should become explicit rules.
3. Confidence-vs-override correlation — when agent.confidence == "high" but Vera overrides anyway, the agent has confidently wrong heuristics. Highest-priority targets.

Each pattern produces a structured finding:

"For requests matching X, the agent recommends Y but reviewers override to Z in N/M cases. Cited reason: '...'."

Include 2-3 example session IDs as evidence.

Phase 3 — Proposals

For each finding, read the relevant source and propose a concrete diff:

Output proposals as unified diffs with rationale, not as freeform prose.

Phase 4 — Validation handoff (out of scope here)

Validation runs as Arize experiments — out of scope for this skill. Just describe the experiment that would validate each proposal: input dataset (the captured override traces), metric (agreement-with-reviewer rate), regression check.

Output

Produce a single markdown report with this structure:

1. Summary — sessions analyzed, override rate, top three patterns.
2. Rule-reality gaps — table of clusters; for each: evidence (linked Arize sessions), proposed fix, expected effect.
3. Precedent extraction — top tribal knowledge found in annotation.notes.
4. Proposed diffs — unified diffs against the codebase, each with rationale and validation plan.
5. Open questions — patterns that need more data or are ambiguous.

Save the report to .context-graph-mining/report-<timestamp>.md (gitignored) and return its content as your final response.

Constraints

• Read-only — this skill does not apply diffs. Output is a markdown report; humans or downstream tooling apply changes.
• Cite evidence — every proposed change must reference at least 3 distinct Arize sessions. If a cluster has fewer, mark it as "weak signal" rather than a proposal.
• No fabricated patterns — if the data does not support a finding, say so.
• Domain-agnostic in spirit, procurement-specific in this run — the patterns and entities (vendor, department, policy) are derived from trace structure, not hardcoded.

Related skills

• arize-trace — export traces, spans, sessions.
• arize-annotation — read annotation configs and labels.
• arize-dataset / arize-experiment — for the validation handoff step described in Phase 4.
• arize-link — generate clickable links to evidence sessions for the report.