Semantic Depth Scorer
Run any AI product announcement, demo recording, or vendor pitch through a 10-dimension semantic-depth rubric. Distinguishes access-only products (agents can click but cannot understand) from meaning-rich products (agents can act with structured intent).
Trigger
Use when the user provides an AI product description, demo link, announcement post, or vendor pitch and wants to know if it is production-ready for autonomous agent use. Also trigger on "/semantic-score", "semantic depth test", "is this agent-ready", "demo theater check".
Phase 1: Input Capture
Ask the user for:
- What is the product? Name, vendor, and a 1-paragraph description or paste of the announcement
- What action type does it perform? (data retrieval / write action / permission management / payment / scheduling / communication / other)
- Context of evaluation (optional): purchase decision, integration planning, advisory review, internal audit
If the user provides a URL, fetch its content and extract the relevant product description before scoring.
Phase 2: 10-Dimension Scoring
Score each dimension 1-5. Apply the rubric strictly — do not round up based on vendor claims, only on observable evidence from the product description.
| # |
Dimension |
1 (Access Only) |
3 (Partial) |
5 (Meaning Rich) |
| 1 |
Action Vocabulary |
Free-form / natural language only; agents must guess valid actions |
Named actions exist but undocumented or unstable |
Typed, versioned action schema; agents can enumerate valid actions |
| 2 |
Permission Encoding |
Permissions granted via UI checkboxes or ambient session state |
Scoped tokens exist but not machine-introspectable |
Structured permission tokens: agents can read scope, expiry, and what is permitted |
| 3 |
Validation Paths |
No programmatic validation; agent must infer success from output text |
Partial validation: some errors are structured |
Deterministic validation: every action has a machine-checkable success/failure contract |
| 4 |
State Observability |
Agent cannot read current state without screen-scraping |
State readable via API but schema is opaque |
Full state observable: typed schema, versioned, predictable |
| 5 |
Error Semantics |
Errors are human-readable strings only |
Errors have codes but no remediation hints |
Errors are structured: code + affected field + suggested fix |
| 6 |
Commitment Primitives |
All actions are instant-commit with no rollback |
Some actions support dry-run or preview |
Actions are staged: agents can preview, validate, then commit or abort |
| 7 |
Intent Capture |
API encodes WHAT happened only |
Some context fields available |
API encodes WHY: purpose, authorizing policy, downstream effects |
| 8 |
Notification Model |
Side effects propagate only through human-visible channels (email, UI notification) |
Webhooks exist but payload is opaque |
Machine-readable event stream: side effects are structured and subscribable by agents |
| 9 |
Schema Richness |
No schema; undocumented or changes without notice |
OpenAPI or equivalent exists but incomplete |
Fully typed, versioned, stable schema with changelog |
| 10 |
Agent Feedback Loops |
No outcome reporting; agent cannot verify the action took effect |
Basic status endpoint |
Agent receives structured outcome: confirmation, side effects, affected resources |
For each dimension, state:
- Score (1-5)
- One sentence of evidence from the product description
- What would need to change to reach a 5
Phase 3: Score Calculation
- Sum all 10 dimension scores (max 50).
- Express as a percentage:
(sum / 50) * 100.
- Classify:
| Score |
Label |
Meaning |
| 0-30 |
Access Only |
Agents can perform actions but cannot understand them; supervision required for every step |
| 31-50 |
Partial Depth |
Some semantic primitives present; usable for low-stakes automation with guardrails |
| 51-70 |
Emerging |
Most dimensions covered; a few gaps will cause incidents in production |
| 71-90 |
Meaning Rich |
Production-ready for most agent use cases; specific gaps noted |
| 91-100 |
Semantic Native |
Purpose-built for agent use; all primitives present |
Phase 4: Report Output
# Semantic Depth Score: [Product Name]
Date: [date]
Evaluator: [context from Phase 1]
## Score: XX/100 — [Label]
## Dimension Breakdown
| Dimension | Score | Evidence |
|-----------|-------|----------|
| Action Vocabulary | X/5 | ... |
| Permission Encoding | X/5 | ... |
| Validation Paths | X/5 | ... |
| State Observability | X/5 | ... |
| Error Semantics | X/5 | ... |
| Commitment Primitives | X/5 | ... |
| Intent Capture | X/5 | ... |
| Notification Model | X/5 | ... |
| Schema Richness | X/5 | ... |
| Agent Feedback Loops | X/5 | ... |
## Weakest Dimensions (upgrade path)
1. [lowest-scoring dimension] — [what needs to change]
2. ...
## Verdict
[2-3 sentences: is this product suitable for autonomous agent use? What is the primary risk?]
## Recommendation
[ ] Purchase / integrate as-is
[ ] Integrate with a semantic wrapper layer (see /tool-audit)
[ ] Wait — revisit in [N months] when [specific gap] is addressed
[ ] Reject — access-only; will require sustained human supervision
Phase 5: Verification
Source
Reconstructed from Nate Jones newsletter (2026-05-06): "The next AI platform winner won't have the best model. They'll own something most companies don't even see yet." Rubric derived from Nate's ten-dimension semantic-depth test for AI product evaluation.
1---2name: semantic-score3description: Score any AI product, vendor pitch, or announcement across 10 semantic-depth dimensions to distinguish "access-only" demos from "meaning-rich" products ready for production agent use. Use when evaluating AI tooling, reviewing a vendor pitch, auditing an integration, or the user says "/semantic-score", "semantic depth test", "score this AI product", "is this demo theater", or "how ready is this for agents".4---56# Semantic Depth Scorer78Run any AI product announcement, demo recording, or vendor pitch through a 10-dimension semantic-depth rubric. Distinguishes access-only products (agents can click but cannot understand) from meaning-rich products (agents can act with structured intent).910## Trigger1112Use when the user provides an AI product description, demo link, announcement post, or vendor pitch and wants to know if it is production-ready for autonomous agent use. Also trigger on "/semantic-score", "semantic depth test", "is this agent-ready", "demo theater check".1314## Phase 1: Input Capture1516Ask the user for:171. **What is the product?** Name, vendor, and a 1-paragraph description or paste of the announcement182. **What action type does it perform?** (data retrieval / write action / permission management / payment / scheduling / communication / other)193. **Context of evaluation** (optional): purchase decision, integration planning, advisory review, internal audit2021If the user provides a URL, fetch its content and extract the relevant product description before scoring.2223## Phase 2: 10-Dimension Scoring2425Score each dimension 1-5. Apply the rubric strictly — do not round up based on vendor claims, only on observable evidence from the product description.2627| # | Dimension | 1 (Access Only) | 3 (Partial) | 5 (Meaning Rich) |28|---|-----------|-----------------|-------------|------------------|29| 1 | **Action Vocabulary** | Free-form / natural language only; agents must guess valid actions | Named actions exist but undocumented or unstable | Typed, versioned action schema; agents can enumerate valid actions |30| 2 | **Permission Encoding** | Permissions granted via UI checkboxes or ambient session state | Scoped tokens exist but not machine-introspectable | Structured permission tokens: agents can read scope, expiry, and what is permitted |31| 3 | **Validation Paths** | No programmatic validation; agent must infer success from output text | Partial validation: some errors are structured | Deterministic validation: every action has a machine-checkable success/failure contract |32| 4 | **State Observability** | Agent cannot read current state without screen-scraping | State readable via API but schema is opaque | Full state observable: typed schema, versioned, predictable |33| 5 | **Error Semantics** | Errors are human-readable strings only | Errors have codes but no remediation hints | Errors are structured: code + affected field + suggested fix |34| 6 | **Commitment Primitives** | All actions are instant-commit with no rollback | Some actions support dry-run or preview | Actions are staged: agents can preview, validate, then commit or abort |35| 7 | **Intent Capture** | API encodes WHAT happened only | Some context fields available | API encodes WHY: purpose, authorizing policy, downstream effects |36| 8 | **Notification Model** | Side effects propagate only through human-visible channels (email, UI notification) | Webhooks exist but payload is opaque | Machine-readable event stream: side effects are structured and subscribable by agents |37| 9 | **Schema Richness** | No schema; undocumented or changes without notice | OpenAPI or equivalent exists but incomplete | Fully typed, versioned, stable schema with changelog |38| 10 | **Agent Feedback Loops** | No outcome reporting; agent cannot verify the action took effect | Basic status endpoint | Agent receives structured outcome: confirmation, side effects, affected resources |3940For each dimension, state:41- Score (1-5)42- One sentence of evidence from the product description43- What would need to change to reach a 54445## Phase 3: Score Calculation46471. Sum all 10 dimension scores (max 50).482. Express as a percentage: `(sum / 50) * 100`.493. Classify:5051| Score | Label | Meaning |52|-------|-------|---------|53| 0-30 | Access Only | Agents can perform actions but cannot understand them; supervision required for every step |54| 31-50 | Partial Depth | Some semantic primitives present; usable for low-stakes automation with guardrails |55| 51-70 | Emerging | Most dimensions covered; a few gaps will cause incidents in production |56| 71-90 | Meaning Rich | Production-ready for most agent use cases; specific gaps noted |57| 91-100 | Semantic Native | Purpose-built for agent use; all primitives present |5859## Phase 4: Report Output6061```62# Semantic Depth Score: [Product Name]63Date: [date]64Evaluator: [context from Phase 1]6566## Score: XX/100 — [Label]6768## Dimension Breakdown69| Dimension | Score | Evidence |70|-----------|-------|----------|71| Action Vocabulary | X/5 | ... |72| Permission Encoding | X/5 | ... |73| Validation Paths | X/5 | ... |74| State Observability | X/5 | ... |75| Error Semantics | X/5 | ... |76| Commitment Primitives | X/5 | ... |77| Intent Capture | X/5 | ... |78| Notification Model | X/5 | ... |79| Schema Richness | X/5 | ... |80| Agent Feedback Loops | X/5 | ... |8182## Weakest Dimensions (upgrade path)831. [lowest-scoring dimension] — [what needs to change]842. ...8586## Verdict87[2-3 sentences: is this product suitable for autonomous agent use? What is the primary risk?]8889## Recommendation90[ ] Purchase / integrate as-is91[ ] Integrate with a semantic wrapper layer (see /tool-audit)92[ ] Wait — revisit in [N months] when [specific gap] is addressed93[ ] Reject — access-only; will require sustained human supervision94```9596## Phase 5: Verification9798- [ ] All 10 dimensions scored with evidence from the product description, not vendor marketing claims99- [ ] Score classification matches the numeric total100- [ ] Recommendation is one of the four options above101- [ ] Any score of 1-2 on Permission Encoding or Validation Paths triggers an explicit risk callout102103## Source104105Reconstructed from Nate Jones newsletter (2026-05-06): "The next AI platform winner won't have the best model. They'll own something most companies don't even see yet." Rubric derived from Nate's ten-dimension semantic-depth test for AI product evaluation.
m2ai-portfolio