Enterprise Agentic-Readiness Scorer
Overview
Ch1 opens with a promise and a trap. The promise: an agent that pursues
goals instead of answering questions. The trap: "you fire up your favorite
LLM, add it to your agentic framework of choice, and connect it to a
vector-based RAG system. Should be easy, right? Wrong." A naive
vector-only approach creates five fatal flaws that are not bugs but an
architectural failure preventing the system from becoming truly agentic.
This skill turns that diagnosis into a score. It checks four things the
chapter argues are required for enterprise agency:
- The five fatal flaws are cured. Each flaw is cured only by a
specific graph capability (context amnesia by evolving memory,
relationship blindness by entity relationships, temporal ignorance by
temporal evolution, reasoning paralysis by multi-hop reasoning, tool
chaos by tool orchestration).
- The three agency dimensions are calibrated. Autonomy, action, and
authority are sliding scales, not binary — and Ch1's point is
calibration, not maximization (a real-estate agent has high autonomy
but deliberately low pricing authority).
- The four emergent capabilities are present. Autonomous
decision-making, contextual understanding, strategic tool utilization,
memory persistence.
- The decision-trace test passes. Per Marple's test in the Enterprise
Context Graphs section: can the system tell you not just what happened,
but what alternatives were considered and rejected?
When to Use
- Before greenlighting an enterprise agent for production deployment
- Reviewing a vendor's "context graph" claim against the rejected-alternatives test
- Comparing a naive-vector prototype to a graph-augmented redesign
- Architecture review where someone proposes "just add a bigger vector store"
Phrases: "is this agent production-ready", "enterprise agentic readiness",
"score my RAG architecture", "are we naive vector RAG", "context graph vs
search index".
When NOT to Use
- Ranking LLMs by quality — Ch1 is explicit that the flaws are
architectural, not model-quality ("cannot be addressed by expanding
context windows or refining embedding techniques")
- Consumer FAQ / support bots where queries map to a text snippet — Ch1
names these as a great fit for plain vector RAG
- Single-turn Q&A with no actions, state, or temporal evolution
Process
| Step |
Input |
Action |
Output |
Verification |
| 1 |
profile JSON (graph_capabilities map) |
lib.score_flaws(caps) |
(points, cured, open_flaws) |
each flaw cured only by its mapped capability |
| 2 |
profile JSON (agency map) |
lib.score_agency(agency) |
(points, missing dims) |
scores coverage/calibration, not magnitude |
| 3 |
profile JSON (capabilities map) |
lib.score_capabilities(caps) |
(points, missing) |
proportional to capabilities present |
| 4 |
captures_rejected_alternatives bool |
lib.decision_trace_test(b) |
(15 or 0, note) |
binary test per Marple |
| 5 |
full profile |
lib.assess(profile) |
score + band + recommendations |
score bounded 0-100; band matches thresholds |
Rationalizations
| Agent rationalization |
Documented rebuttal |
| "Our LLM is frontier-grade, so we are production-ready." |
Ch1: the five flaws "aren't bugs. They represent an architectural failure" and "cannot be addressed by optimizing search or tweaking embedding models." Model quality does not cure an architecture gap. |
| "We have a vector store, that covers retrieval." |
A vector store cures none of the five flaws by itself. Score it: all five stay open, band is NAIVE-VECTOR. The flaws are cured by graph capabilities, not by a vector index. |
| "We logged everything, so we have a context graph." |
Marple's test (Ch1, The Context Graph): can it tell you what alternatives were rejected? Logging final states is read-time data. The decision-trace test is worth 15 points precisely to catch relabeled search indexes. |
| "Max out autonomy and authority for a powerful agent." |
Ch1: agency dimensions are sliding scales and must be calibrated, not maximized. The real-estate agent has high autonomy, near-zero pricing authority. This scorer rewards calibration coverage, not magnitude. |
Red Flags
- Score is high but decision_trace is 0. The agent may pass the
capability checklist while recording only outcomes; it will become "a
high-fidelity log of failure" if it ever hallucinates (Ch1 counter-thesis).
- All five flaws open but band is not NAIVE-VECTOR. Scoring bug —
open flaws should dominate; recheck the FLAW_CURE mapping.
- Agency magnitude drives the score. Misreads Ch1: a deliberately
low-authority agent is correct design, not a deficiency.
Non-Negotiable Verification
- Run the benchmark battery.
python cli.py benchmark must report:
- all-open flaws score 0; all-cured score the full 40
- each flaw cured only by its mapped graph capability
- decision-trace test is binary 15/0
- a perfect profile is exactly 100 and PRODUCTION-READY; empty is NAIVE-VECTOR
- agency scores calibration coverage, not magnitude
- Verify CLI help.
python cli.py --help exits 0 and prints the SKILL.md description.
Security Posture
- Prompt injection. The profile JSON is untrusted input - a vendor can
self-report flattering capability booleans to inflate the score. The scorer
only reads fixed keys against a fixed rubric; adversarial field values can
skew the score but never execute, so verify claimed capabilities (especially
captures_rejected_alternatives) against evidence before trusting a band.
- Data exfiltration. No network calls, no file writes. Architecture profiles
may describe internal systems; they stay in-process and surface only in the
stdout report the caller owns.
- Privilege escalation. No shell invocation, no eval, no dynamic import. A
PRODUCTION-READY band is advisory - it authorizes nothing; the deployment
gate that consumes the score owns the actual go/no-go decision.
Source Attribution
Distilled from Agentic GraphRAG (O'Reilly) by Anthony Alcaraz and Sam Julien
— Ch1: The Crisis of Agentic AI, specifically the five-fatal-flaws opening
(the naive-vector failure), the "Defining Agentic AI" agency dimensions and
capabilities, and the "Enterprise Context Graphs" decision-trace test.
Supporting references: Singhal 2012 (strings to things), Lilian Weng 2023
(LLM-as-brain), Arvind Jain / Glean context-data-platform, Kirk Marple /
Graphlit rejected-alternatives test.
1---2name: enterprise-readiness-scorer3description: Score a proposed or deployed enterprise agent against the architectural requirements Ch1 argues are non-negotiable: absence of the five fatal flaws of naive vector RAG (context amnesia / relationship blindness / temporal ignorance / reasoning paralysis / tool chaos), calibration of the three agency dimensions (autonomy / action / authority), presence of the four emergent capabilities, and the decision-trace test that separates a real context graph from a relabeled search index. Produces a 0-100 score, a band (PRODUCTION-READY / PILOT-READY / PROTOTYPE / NAIVE-VECTOR), and a gap-closing recommendation per open flaw. Use before greenlighting an enterprise agent for production. NOT for ranking models (Ch1 says the flaws are architectural, not model-quality), NOT for consumer FAQ bots where vector RAG is a fine fit.4---56# Enterprise Agentic-Readiness Scorer78## Overview910Ch1 opens with a promise and a trap. The promise: an agent that pursues11goals instead of answering questions. The trap: "you fire up your favorite12LLM, add it to your agentic framework of choice, and connect it to a13vector-based RAG system. Should be easy, right? Wrong." A naive14vector-only approach creates **five fatal flaws** that are not bugs but an15architectural failure preventing the system from becoming truly agentic.1617This skill turns that diagnosis into a score. It checks four things the18chapter argues are required for enterprise agency:19201. **The five fatal flaws are cured.** Each flaw is cured only by a21 specific graph capability (context amnesia by evolving memory,22 relationship blindness by entity relationships, temporal ignorance by23 temporal evolution, reasoning paralysis by multi-hop reasoning, tool24 chaos by tool orchestration).252. **The three agency dimensions are calibrated.** Autonomy, action, and26 authority are sliding scales, not binary — and Ch1's point is27 *calibration*, not maximization (a real-estate agent has high autonomy28 but deliberately low pricing authority).293. **The four emergent capabilities are present.** Autonomous30 decision-making, contextual understanding, strategic tool utilization,31 memory persistence.324. **The decision-trace test passes.** Per Marple's test in the Enterprise33 Context Graphs section: can the system tell you not just what happened,34 but what alternatives were considered and rejected?3536## When to Use3738- Before greenlighting an enterprise agent for production deployment39- Reviewing a vendor's "context graph" claim against the rejected-alternatives test40- Comparing a naive-vector prototype to a graph-augmented redesign41- Architecture review where someone proposes "just add a bigger vector store"4243Phrases: "is this agent production-ready", "enterprise agentic readiness",44"score my RAG architecture", "are we naive vector RAG", "context graph vs45search index".4647## When NOT to Use4849- Ranking LLMs by quality — Ch1 is explicit that the flaws are50 architectural, not model-quality ("cannot be addressed by expanding51 context windows or refining embedding techniques")52- Consumer FAQ / support bots where queries map to a text snippet — Ch153 names these as a great fit for plain vector RAG54- Single-turn Q&A with no actions, state, or temporal evolution5556## Process5758| Step | Input | Action | Output | Verification |59|------|-------|--------|--------|--------------|60| 1 | profile JSON (graph_capabilities map) | `lib.score_flaws(caps)` | (points, cured, open_flaws) | each flaw cured only by its mapped capability |61| 2 | profile JSON (agency map) | `lib.score_agency(agency)` | (points, missing dims) | scores coverage/calibration, not magnitude |62| 3 | profile JSON (capabilities map) | `lib.score_capabilities(caps)` | (points, missing) | proportional to capabilities present |63| 4 | captures_rejected_alternatives bool | `lib.decision_trace_test(b)` | (15 or 0, note) | binary test per Marple |64| 5 | full profile | `lib.assess(profile)` | score + band + recommendations | score bounded 0-100; band matches thresholds |6566## Rationalizations6768| Agent rationalization | Documented rebuttal |69|------------------------|--------------------|70| "Our LLM is frontier-grade, so we are production-ready." | Ch1: the five flaws "aren't bugs. They represent an architectural failure" and "cannot be addressed by optimizing search or tweaking embedding models." Model quality does not cure an architecture gap. |71| "We have a vector store, that covers retrieval." | A vector store cures none of the five flaws by itself. Score it: all five stay open, band is NAIVE-VECTOR. The flaws are cured by graph capabilities, not by a vector index. |72| "We logged everything, so we have a context graph." | Marple's test (Ch1, The Context Graph): can it tell you what alternatives were *rejected*? Logging final states is read-time data. The decision-trace test is worth 15 points precisely to catch relabeled search indexes. |73| "Max out autonomy and authority for a powerful agent." | Ch1: agency dimensions are sliding scales and must be *calibrated*, not maximized. The real-estate agent has high autonomy, near-zero pricing authority. This scorer rewards calibration coverage, not magnitude. |7475## Red Flags7677- **Score is high but decision_trace is 0.** The agent may pass the78 capability checklist while recording only outcomes; it will become "a79 high-fidelity log of failure" if it ever hallucinates (Ch1 counter-thesis).80- **All five flaws open but band is not NAIVE-VECTOR.** Scoring bug —81 open flaws should dominate; recheck the FLAW_CURE mapping.82- **Agency magnitude drives the score.** Misreads Ch1: a deliberately83 low-authority agent is correct design, not a deficiency.8485## Non-Negotiable Verification86871. **Run the benchmark battery.** `python cli.py benchmark` must report:88 - all-open flaws score 0; all-cured score the full 4089 - each flaw cured only by its mapped graph capability90 - decision-trace test is binary 15/091 - a perfect profile is exactly 100 and PRODUCTION-READY; empty is NAIVE-VECTOR92 - agency scores calibration coverage, not magnitude932. **Verify CLI help.** `python cli.py --help` exits 0 and prints the SKILL.md description.9495## Security Posture9697- **Prompt injection.** The profile JSON is untrusted input - a vendor can98 self-report flattering capability booleans to inflate the score. The scorer99 only reads fixed keys against a fixed rubric; adversarial field values can100 skew the score but never execute, so verify claimed capabilities (especially101 `captures_rejected_alternatives`) against evidence before trusting a band.102- **Data exfiltration.** No network calls, no file writes. Architecture profiles103 may describe internal systems; they stay in-process and surface only in the104 stdout report the caller owns.105- **Privilege escalation.** No shell invocation, no eval, no dynamic import. A106 PRODUCTION-READY band is advisory - it authorizes nothing; the deployment107 gate that consumes the score owns the actual go/no-go decision.108109## Source Attribution110111Distilled from *Agentic GraphRAG* (O'Reilly) by Anthony Alcaraz and Sam Julien112— Ch1: The Crisis of Agentic AI, specifically the five-fatal-flaws opening113(the naive-vector failure), the "Defining Agentic AI" agency dimensions and114capabilities, and the "Enterprise Context Graphs" decision-trace test.115Supporting references: Singhal 2012 (strings to things), Lilian Weng 2023116(LLM-as-brain), Arvind Jain / Glean context-data-platform, Kirk Marple /117Graphlit rejected-alternatives test.
AnthonyAlcaraz