Aeo Baseline

AEO Baseline

Source: github.com/psyduckler/aeo-skills Part of: AEO Skills Suite Methodology: aeo-v1 Output schema: aeo-evidence-v1

The atomic measurement primitive for AI visibility tracking. One command, one run, one evidence file with every signal you can extract from 20 grounded Gemini responses per prompt.

The Retrieval Framework

You cannot edit a model's training data. But for search-grounded models like Gemini, you can influence what enters the candidate set the model selects from when it grounds an answer. That candidate set — the recurring retrieval set for a prompt — is what this skill measures.

Every metric in the output traces back to a saved raw response. If you disagree with the score, recompute it from the evidence file — components are stored separately.

Requirements

• Gemini API key (free from aistudio.google.com) — set as GEMINI_API_KEY
• Python 3.9+ — stdlib only, no pip install needed
• A workspace config — either an existing aeo.config.json (produced by aeo-init) or a one-off via --prompt

Usage

# Default: read aeo.config.json, run 20 samples per prompt, write to aeo-data/
GEMINI_API_KEY="$GEMINI_API_KEY" python3 scripts/baseline.py

# Verify the API key works without running a measurement
python3 scripts/baseline.py --doctor

# Print projected cost without making any API calls
python3 scripts/baseline.py --estimate-cost

# Run a single ad-hoc prompt (skips most of the config)
GEMINI_API_KEY="$GEMINI_API_KEY" python3 scripts/baseline.py --prompt "best CRM for small business"

# Override sample count for higher-confidence measurements
GEMINI_API_KEY="$GEMINI_API_KEY" python3 scripts/baseline.py --runs 50

# Skip the cost confirmation prompt (for cron / CI use)
GEMINI_API_KEY="$GEMINI_API_KEY" python3 scripts/baseline.py --yes

Run from the skill directory. Resolve scripts/baseline.py relative to this SKILL.md.

Options

Signals Extracted

Each of the 20 samples per prompt contributes to:

• Brand mention — whole-word match for the brand name and any aliases
• Brand citation — domain match in grounding chunks, with 1-based position
• Competitor mention — text + grounding match for each tracked competitor
• Query fan-out — the literal Google Search queries Gemini fired
• Entities — proper-noun extraction (CamelCase + multi-word capitalized)
• Sentiment — ±50-char window scan for positive/negative keywords near each brand mention
• Recommendation — brand mention in the trailing 25% of the response

All extraction is deterministic and rule-based — same response in, same numbers out. Sentiment will move to LLM-based in v1.2.

Output

A single file: <output-dir>/run_<UTC-timestamp>.json matching the public aeo-evidence-v1 schema.

The file is append-only by design — each run produces a new timestamped file. Trend analysis is done by aeo-report reading the whole directory.

Top-level layout:

{
  "schema_version": "aeo-evidence-v1",
  "workspace": {...},
  "run": {"run_id", "timestamp", "provider", "model", "samples", "methodology_version", "estimated_cost_usd", "actual_cost_usd", "latency_ms_p50"},
  "prompts": [
    {
      "prompt_id", "prompt_text", "intent",
      "successful_samples", "failed_samples",
      "samples": [/* each sample with raw text, queries, citations, mentions, entities */],
      "aggregates": {"mention_rate", "mention_rate_ci", "citation_rate", ..., "query_fanout", "entity_universe", "competitor_share", "sentiment_distribution"},
      "visibility_score": {"value", "methodology_version", "components": {/* mention, citation, position, recommendation, sentiment */}}
    }
  ]
}

Cost Management

This skill respects two limits from aeo.config.json:

• limits.confirm_over_usd — if the projected cost exceeds this, the user is prompted to confirm before any API calls (default $5)
• limits.max_daily_cost_usd — informational only in v1; honored by aeo-track for scheduled runs (default $25)

Override the per-sample cost estimate via env var:

export GEMINI_COST_PER_SAMPLE_USD=0.0005

Use --estimate-cost to dry-run.

Scoring (aeo-v1)

The composite visibility score is a transparent weighted average, scaled 0–100:

visibility_score = 100 × (
    0.30 × mention_rate
  + 0.25 × citation_rate
  + 0.20 × position_score
  + 0.15 × recommendation_rate
  + 0.10 × sentiment_score
)

Every component is stored in the evidence file. To re-score with different weights, modify them and recompute from the raw samples — no new API calls needed.

See METHODOLOGY.md for the full definitions.

Pairs With

• aeo-init (coming) — creates aeo.config.json
• aeo-track (coming) — installs a daily cron to re-run this skill
• aeo-report (coming) — reads accumulated evidence files and generates Markdown + SVG trend reports
• aeo-optimize (coming) — reads the latest evidence file and recommends concrete content actions

Notes

• Each sample's raw response text is stored in the evidence file. If you want to reduce file size, you can post-process the JSON to drop samples[].raw_response_text after extraction.
• Failed samples (HTTP errors, timeouts) are recorded with error rather than dropped silently. successful_samples and failed_samples are explicit at the prompt level.
• Wilson 95% CI is reported alongside every proportion metric — at N=20, expect a width of ~30 percentage points for mid-range rates. See METHODOLOGY.md §2 for the full table.
• Gemini grounding occasionally rate-limits at high concurrency. _shared.call_gemini retries with exponential backoff up to 5 attempts.