T3 Hardware Scoring

MantaBase T3 Hardware Audit System — v2.2

Objectives

• Collect product information → Brand Blind → Triple-Auditor independent scoring → Eagle Eye Validation → Final Judge synthesis
• Output: objective Tool / Toy / Trash classification with Eagle Eye safety veto
• Core principles: evidence-first, parallel audits, zero hallucination, transparent math

Architecture Overview

Main Agent                             Subagents
───────────────────────────────────    ──────────────────────────────
Step 1: Data Collection (web tools)
Step 2: Organize + Brand Blind
  ↓ write 02-brand-blinded.md
                                       Step 3: ×3 parallel (Tool/Toy/Trash)
                                         · read file paths, print JSON to stdout
                                         · main agent writes 03-*.json files
  ↓ receive JSON, write files
  ↓ Step 3.5: validate + merge (script)
                                       Step 4: ×1 Eagle Eye Validator (conditional)
                                         · AI subagent, not a Python script
                                         · outputs adjustments[] diff
  ↓ apply diffs, re-merge
Step 5: synthesize_results.py
Step 6: 99-audit-report.md

Key v2.2 changes:

• Step 1 uses the agent's own web tools; the bundled crawler is gone
• Step 3.5 is now a script (validate_auditor_json.py), not prose instructions
• Every auditor report is validated before it can reach the Final Judge
• Missing litmus_gate / bad totals are hard errors instead of silent defaults
• final_label stays parseable; display_label carries the annotations
• The skill ships a golden worked example and a test suite (tests/)

No Python dependencies. Both scripts are stdlib-only.

Data Collection Strategy (Step 1-2)

Use whatever web capability the host agent has — WebFetch / WebSearch, a browser tool, or an Exa/Firecrawl MCP if one is connected. Fetch the product page directly and search for independent coverage. Do not shell out to a bundled crawler: the host's fetcher already handles redirects, JS-rendered pages, and rate limits far better than a bare requests.get, and it reads the page into context in one step.

If no web capability is available, ask the user to paste the product page text and any reviews, then continue from Step 2 unchanged. Everything downstream operates on 02-brand-blinded.md, not on how the text was obtained.

Sources to gather (in order)

1. Official product page (specs, pricing, marketing claims — quote these verbatim)
2. Third-party hands-on reviews
3. Community discussion (Reddit, forums) for real-user pain points
4. Investigation/news coverage (privacy issues, lawsuits, controversies)

Collect at least 3 independent sources. Save raw extracts to 01-level0-extracts.md with source tags [S1], [S2], etc.

Minimum data threshold

Proceed only when you have:

• At least one official source with tech specs + price
• At least one third-party review with hands-on findings
• Marketing claims verbatim (needed for the Eagle Eye Honest check)

Pre-launch data flag

If the product meets ALL of the following, append (Pre-launch) to the case_id and skip Step 4:

• Zero independent hands-on reviews
• ≥3 critical specs undisclosed (battery, weight, connectivity, regulatory)
• Pre-order only / not shipping

Procedure

Step 1: Collect Raw Data

• Fetch the official product URL with the agent's web tools
• Search for independent reviews and community threads
• Save raw extracts to 01-level0-extracts.md with source tags

Step 2: Organize + Brand Blind (combined step)

• Read references/organize-guide.md
• Read references/brand-blinding-guide.md
• Merge all sources into one structured document
• Apply Brand Blinding inline (replace brand names with [BRAND], [PRODUCT], [FEATURE])
• Preserve ALL adjectives and marketing claims verbatim — needed by downstream auditors
• Brand-blinded text should be ≤ 1200 words — deduplicate aggressively but keep all unique claims
• Save to 02-brand-blinded.md

A complete example of a finished Brand-Blinded document is at tests/fixtures/golden-ai-pendant/02-brand-blinded.md.

Step 3: Triple Auditor Scoring (independent, parallel)

Each Auditor sees only the Brand-Blinded text from Step 2.

Subagent execution rules (mandatory):

• Pass the file path to 02-brand-blinded.md — do NOT embed full text in the prompt
• Each subagent reads the file, reads their rubric file, and prints JSON to stdout
• Main agent writes files — subagents never call write (prevents timeout file loss)
• verbatim_evidence limits:
  • score 0–1: max 1 quote (or [] for score 0)
  • score 2–3: max 2 quotes
• Recommended timeouts: Tool=150s, Toy=150s, Trash=180s

🟢 Tool Auditor

• Reads references/tool-auditor.md
• Template: references/tool-auditor-template.md
• Score 11 items (0-3 scale), max 33 points
• Output litmus_gate: "Yes"/"No" at top level

🟡 Toy Auditor

• Reads references/toy-auditor.md
• Template: references/toy-auditor-template.md
• Score 11 items (0-3 scale), max 33 points
• Output litmus_gate: "Yes"/"No" at top level

🔴 Trash Auditor

• Reads references/trash-auditor.md
• Template: references/trash-auditor-template.md
• Reads references/trash-red-flags.md (required for Eagle Eye)
• Score 14 items (0-3 scale), max 42 points
• Eagle Eye triggers → auto-set score to 3 for flagged items
• Output litmus_gate: "Yes"/"No" and critical_issues: [...] at top level

Key principles:

• Each auditor works from Brand-Blinded text only
• No cross-talk between auditors during scoring
• Evidence FIRST, then score — no score without a verbatim quote
• If a feature isn't in the text, it scores 0

Worked examples of all three finished reports are in tests/fixtures/golden-ai-pendant/.

Step 3.5: Validate and merge (mandatory)

python3 scripts/validate_auditor_json.py merge \
  --tool  03-tool-auditor.json \
  --toy   03-toy-auditor.json \
  --trash 03-trash-auditor.json \
  --output auditor_reports.json

This validates all three reports and refuses to merge if any check fails:

• every rubric item present (11 / 11 / 14) and scored within 0–3
• total_score equals the sum of its items; section subtotals match
• evidence-first: no score above 0 without a verbatim quote
• top-level litmus_gate present and consistent with the nested answer
• one critical_issues entry per "Triggered:" item

🚨 Never hand-write auditor_reports.json from memory or a summary. Every value must be copied from a 03-*.json file. That is what this command does — if it reports an error, re-run the affected auditor rather than editing scores.

Individual reports can be checked as they arrive:

python3 scripts/validate_auditor_json.py validate --role trash 03-trash-auditor.json

Step 4: Eagle Eye Validator (conditional, AI-driven)

Read references/eagle-eye-validator.md

Trigger conditions — run Step 4 if ANY is true:

• Trash report critical_issues is non-empty
• Any auditor has any item scored 3
• Composite score is in the Gray Zone (-10 to +10)

Otherwise: skip directly to Step 5.

Execution:

• Spawn 1 subagent (timeout: 120s)
• Subagent reads 02-brand-blinded.md + only the flagged items from 03-*.json
• Performs three checks:
  1. Eagle Eye trigger validity — do both conflicting quotes exist verbatim?
  2. Architectural Plausibility — are on-device/local claims technically feasible for the form factor?
  3. Score=3 evidence check — does the evidence actually support the 3-level rubric?
• Outputs an adjustments[] diff (not a full re-score)
• Main agent applies the diffs to the 03-*.json files, then re-runs the Step 3.5 merge so the totals stay derived rather than patched

Step 5: Final Judge Synthesis

python3 scripts/synthesize_results.py --input auditor_reports.json --text

Run the script — do not do this arithmetic by hand. The rules interact (four primary conditions, two secondary tiers, a Gray Zone gate override, and the veto), and both worked examples in references/t3-classification.md previously disagreed with the code. The script is the reference implementation; the tests pin it to the spec.

Steps applied, per references/t3-classification.md:

1. Normalize: Tool=(raw/33)×100, Toy=(raw/33)×100, Trash=(raw/42)×100
2. Composite = max(NormTool, NormToy) − NormTrash
3. Primary: the category meeting 2+ of its 4 conditions
4. Secondary: threshold tiers, plus the Gray Zone negative-composite rule
5. Eagle Eye Veto: critical_issues non-empty → Trash forced into the label, confidence forced to Review Required

Step 6: Generate Audit Report

Output file: 99-audit-report.md Directory: tmp/reports/t3-{YYYY-MM-DD}-{slug}/

Follow the YAML schema in references/report-schema.md. Use final_label for the YAML metadata and display_label in the prose.

Output Format for Messaging Surfaces (WhatsApp / Telegram)

When delivering results in a chat interface, do not paste the full file. Use this condensed format instead:

🏷️ T3 Audit — {Product Name}
Price: ${price} | Date: {YYYY-MM-DD}

📊 Scores
🟢 Tool:  {raw}/33 → {norm}/100
🟡 Toy:   {raw}/33 → {norm}/100
🔴 Trash: {raw}/42 → {norm}/100
Composite: {composite:+.1f}

🏆 Classification: {display_label}
Confidence: {confidence}

{If Eagle Eye:}
🚨 Eagle Eye: {trigger name}
"{verbatim quote A}"
vs.
"{verbatim quote B}"

🟢 Tool highlights (3 bullets)
🟡 Toy highlights (3 bullets)
🔴 Trash issues (3 bullets)

💬 One-line verdict

Rules for messaging output:

• No markdown tables (WhatsApp doesn't render them)
• Use bold for emphasis, bullet lists for details
• Keep the total message under 600 words
• If Eagle Eye triggered, always show the conflicting quotes

Important Constraints

🚨 Information Isolation (critical)

• Brand-Blinded text is the ONLY input for auditors
• Auditors must never see original brand names, marketing copy, or each other's reports during scoring
• The Final Judge reads only the three auditor JSON outputs, not the original product text

📊 Objective data completeness

• Collect specs, performance, reliability, market, and cost data
• Brand Blinding must retain all specs and numbers
• Every score must link to extracted verbatim evidence

Eagle Eye enforcement

• The Trash Auditor MUST check every item against references/trash-red-flags.md
• Eagle Eye trigger → automatic score 3 → populate critical_issues[]
• The Eagle Eye Validator (Step 4) re-checks triggers including Architectural Plausibility
• The Final Judge must respect the Eagle Eye Veto regardless of composite math

synthesize_results.py input format

{
  "tool":  { "total_score": <int 0-33>, "litmus_gate": "Yes|No", "extract_for_report": {...} },
  "toy":   { "total_score": <int 0-33>, "litmus_gate": "Yes|No", "extract_for_report": {...} },
  "trash": { "total_score": <int 0-42>, "litmus_gate": "Yes|No",
             "critical_issues": ["<eagle eye trigger text>", ...],
             "extract_for_report": {...} }
}

litmus_gate is required on all three. It is one of only four conditions that decide the primary classification, so a missing gate is a wrong verdict, not a smaller one. The script errors rather than defaulting it.

Testing

tests/run_tests.sh

Runs the unit tests, the end-to-end golden case, and a CLI smoke test. Stdlib Python only — no install step.

tests/fixtures/golden-ai-pendant/ is a complete worked audit of a fictional product: Brand-Blinded source text, all three auditor reports, and the classification they synthesize to. It doubles as the reference for what a finished, internally consistent auditor report looks like — the rubric guides show only fragments.

One test worth knowing about: test_every_quote_exists_in_the_source_text checks that every string in every verbatim_evidence array appears in 02-brand-blinded.md. The skill's central claim is zero hallucination, and that makes it mechanically checkable rather than merely asserted.

tests/test_trigger_consistency.py pins the Eagle Eye trigger set. Four files list the triggers, and an auditor subagent reads only one of them — so the tables are deliberately duplicated, and pinned instead of de-duplicated. The canonical index lives in references/trash-red-flags.md; add a pattern there first, then propagate. The test fails if any copy — or the pattern table in validate_auditor_json.py — drifts.

tests/check_links.py verifies that every file the skill tells the agent to read actually exists — a dead reference mid-run means the agent improvises the rubric.

Resource Index

Scripts

• scripts/validate_auditor_json.py — validate one auditor report, or validate + merge all three (Step 3.5)
• scripts/synthesize_results.py — Final Judge math (normalization + Eagle Eye + Litmus Gate)

References