Qualitative Analysis

qualitative-analysis

The AI qualitative layer of AIMS (roadmap #98, contract #89): deterministic evidence and calendar inputs, one capped LLM call producing a schema-validated data/qualitative/<stem>.json artifact through Claude Code Action OAuth, and deterministic grounding gates. The quantitative pipeline stays the sole source of truth for scores, ranks, risk gates, and the market regime — this layer only explains or challenges them, and any failure here publishes the quantitative report unchanged.

Disclaimer: All output is informational only and does not constitute investment advice.

Scripts

All scripts are in .agents/skills/qualitative-analysis/scripts/ and delegate to src/aims/.

Usage

# 1. Deterministic inputs (no LLM or credential needed)
uv run .agents/skills/qualitative-analysis/scripts/fetch_evidence.py \
    --analysis-date 2026-07-04 --output data/evidence/
uv run .agents/skills/qualitative-analysis/scripts/validate_evidence.py \
    --input data/evidence/2026-07-04.json

uv run .agents/skills/qualitative-analysis/scripts/update_calendars.py \
    --output data/calendars/earnings.json
uv run .agents/skills/qualitative-analysis/scripts/validate_calendar.py \
    --input data/calendars/earnings.json

# 2. Prepare the committed prompt payload and JSON Schema
uv run .agents/skills/qualitative-analysis/scripts/qualitative_analysis.py \
    prepare \
    --analysis data/analysis/2026-07-04.json \
    --evidence data/evidence/2026-07-04.json \
    --calendar data/calendars/macro_events.json \
    --calendar data/calendars/earnings.json \
    --run-dir data/run/qualitative/

# 3. The workflow invokes pinned anthropics/claude-code-action with
#    CLAUDE_CODE_OAUTH_TOKEN and --tools "", then finalizes its untrusted output.
CLAUDE_STRUCTURED_OUTPUT='{"market": ..., "instruments": ...}' \
uv run .agents/skills/qualitative-analysis/scripts/qualitative_analysis.py \
    finalize --request data/run/qualitative/request.json --attempt 1 \
    --output data/qualitative/

# 4. Validate the committed artifact (defense in depth)
uv run .agents/skills/qualitative-analysis/scripts/validate_qualitative.py \
    --input data/qualitative/2026-07-04.json \
    --analysis data/analysis/2026-07-04.json \
    --evidence data/evidence/2026-07-04.json

Contract highlights

• Scope: commentary covers the top-K published signals only (K=5, matching history.py) plus one market narrative and up to five themes — one LLM call per run, capped input and output.
• Grounding: the model may cite only evidence IDs from the day's committed bundle; the request schema constrains citations and canonical IDs at the API level, and validate_qualitative.py re-enforces everything regardless of what the model emits. Evidence text is untrusted data: instructions inside it are content to ignore.
• Machine-checkable claims: realized-price-action claims use direction_claim enums; every number in free text must be declared in numeric_claims. aims.qualitative_gates cross-checks the structured claims against the quantitative features and cited evidence — gated instrument entries are excluded from rendering, and a gated market narrative withholds the whole artifact (one regeneration retry first).
• Provenance: the prompt is the committed file prompts/qualitative_v1.md; its version and SHA-256, the model ID, and SHA-256 hashes of all inputs are recorded in artifact metadata. Changing the prompt or model requires the #97 regression harness (below).
• Fail-open: missing CLAUDE_CODE_OAUTH_TOKEN or an empty evidence bundle skips the action path; OAuth/action/quota errors or persistent validation failures exit non-zero without writing an artifact. The daily workflow treats all of these as non-fatal and publishes the quantitative report unchanged.

Stance evaluation (#97)

The daily workflow runs evaluate_stances.py after the qualitative-analysis step: it joins accumulated ungated stances with forward returns reconstructed purely from committed analysis artifacts, writes the validated data/performance/<date>.json artifact, and regenerates the public content/evaluation/_index.md page. Everything is deterministic and safe in the empty state (no qualitative artifacts yet → zero counts plus a warning). Results are informational association only — never presented as an investable track record.

Trust boundary: the qualitative-analysis step is continue-on-error, so a same-run data/qualitative/<stem>.json file can exist on disk even after a failed or unvalidated run. The workflow passes --exclude-qualitative-date "${DATE}" to evaluate_stances.py whenever steps.qualitative.outcome != 'success', so that date is excluded from evaluation (with a recorded warning) regardless of whether the file is present — the same trust boundary already used for report rendering and Slack notification (steps.qualitative.outcome == 'success'). Historical, already-validated qualitative artifacts are never affected.

check_citation_links.py additionally samples recent evidence URLs and logs link-rot warnings (never a gate).

Changing the prompt or model

Adoption is a reviewed human decision; there is no automatic prompt tuning. Before merging any change to prompts/qualitative_v1.md, to aims.qualitative.PROMPT_VERSION, or to aims.qualitative.MODEL_ID:

# 1. Generate candidate artifacts with the edited prompt/model from
#    committed inputs (or use the committed fixtures without OAuth execution
#    available), then run the harness and record the results:
uv run .agents/skills/qualitative-analysis/scripts/prompt_regression.py \
    --artifact tests/fixtures/qualitative_2024-01-01.json \
    --analysis tests/fixtures/analysis_2024-01-01_mapped.json \
    --evidence tests/fixtures/evidence_2024-01-01.json \
    --record data/performance/prompt_regressions.json

The harness asserts structural and gate metrics (validator cleanliness, market-narrative rendering, instrument gate pass rate ≥ 0.6, citation coverage = 1.0, stance-distribution sanity) — never exact prose — and records the run keyed by the prompt file's SHA-256 and the pinned model ID. Commit the updated prompt_regressions.json together with the change: a CI test fails when the committed prompt/model has no recorded passing entry.

Operational details (secrets, shadow mode, gate thresholds, costs, troubleshooting, recovery) are in OPERATIONS.md §10–§12.