# Agent Concierge Probe

> Agent Concierge Probe

- Skill: `the-utopia-studio/agent-concierge-probe` (Agent Skill, multi-file: 13 files)
- Install (CLI): `npx skillmds@latest add the-utopia-studio/agent-concierge-probe`
- Raw SKILL.md: https://api.skillmd.com/api/skills/the-utopia-studio/agent-concierge-probe/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: The-Utopia-Studio (https://skillmd.com/u/the-utopia-studio)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/the-utopia-studio/agent-concierge-probe

---


# Agent Concierge Probe

## What it does
You hand a real, already-mapped task to an agent, let it run the task end to end across several outcomes, and instrument every run. Output is two things: the cost-per-outcome priced to the cent, and a frontier map showing which steps the agent does unaided and which a human has to take over. It compresses "is this automatable, and what does one outcome cost" into measured numbers instead of a hunch.

## The Icarus reframe
A generic "can an agent do it?" demo shows one happy-path run and a vibe. This runbook refuses a cost claim without its parts: the read-out must show input tokens, output tokens, tool calls, and human-fix minutes separately, each priced, before it states a cost-per-outcome — a bare "$2/task" is not a result. And it treats the red steps (where a human had to intervene) as the actual finding: those steps *are* the automation frontier, the line the product cannot yet cross. The probe is honest about cost-per-outcome and the frontier. It lies about self-serve UX and about trust/adoption, and it says so in every read-out.

## When to use / When NOT
Use it when the workflow is already mapped (ideally by a concierge run, so you know the exceptions) and you need to know what an agent can do unaided and what one outcome costs.

Do not use it when:
- You have not chosen a probe — that is `probe-matrix`.
- You want to test willingness to pay, or you (the founder) will do the task by hand — that is `concierge-probe`.
- You want to know whether people would *use* or *trust* the thing — this probe lies about both. Adoption and trust need a concierge or field probe.
- You want to fake an interface to test would-they-use-it — that is `wizard-of-oz-probe`.
- You want to test whether a workflow reads / is legible on paper before anyone builds it — that is `paper-sketch-probe`.

## Method
Follow the runbook. Fill `template.md` as you go.

**Step 1 — Take a mapped workflow.** Break the task into discrete steps. If a concierge run already exposed the exceptions, carry them in — the agent will hit them.

**Step 2 — Instrument before you run.** Set the meters and the rates *first*:

| Meter | Unit | Rate (set now) |
|---|---|---|
| Input tokens | per 1M | $__ |
| Output tokens | per 1M | $__ |
| Tool / API calls | per call | $__ |
| Human-fix time | per minute | $__ (loaded rate) |

If you cannot capture human-fix minutes, you cannot run this probe — the human-fix column is where the cost hides.

**Step 3 — Run N outcomes.** N ≥ **5**. One run is anecdote; cost-per-outcome needs a denominator. Log every run, including the ones that failed (a failed run still spent tokens).

**Step 4 — Price it to the cent.** Show the breakdown, then the number.

```
CPO = [ (in_tokens × rate_in) + (out_tokens × rate_out)
        + (tool_calls × rate_call)
        + (human_fix_min × human_rate) ] ÷ successful_outcomes
```

Reporting a cost-per-outcome without the token / tool / human-fix breakdown is the kill line. Expected band from the honesty contract is ~$0.50–5 / task; if your CPO exceeds what a concierge customer paid, flag the inverted economics.

**Step 5 — Map the frontier.** Colour every step:

| Colour | Meaning |
|---|---|
| Green | agent did it unaided, every run |
| Amber | agent did it but needed a retry, reprompt, or verification |
| Red | a human had to take over |

Red steps are the automation frontier. If a red step sits on the critical path, the task is not automatable yet — that is a finding, not a failure.

**Step 6 — Read out against the honesty contract.**

| Honest about | Must NOT claim |
|---|---|
| Cost-per-outcome, to the cent, with breakdown | Self-serve UX — you drove the agent |
| The automation frontier (green / amber / red) | Trust / adoption — not tested here |

## Evidence standard
Done means: ≥5 logged runs, a cost-per-outcome with all four cost components shown and priced, and a frontier map with every step coloured. The measured CPO is `[Fact]`; the claim that it will fall with prompt tuning is `[Hypothesis]`; a green step observed over 5 runs is `[Fact]` for those runs, `[Assumption]` at higher volume. Behaviour beats opinion: the agent actually completing a step (0.7, behaviour observed) outranks a belief that it "should be able to." Trust and adoption are untested and must be tagged `[Hypothesis, not tested]`, never asserted.

## Gotchas
- **The invisible human.** The founder quietly re-runs, edits the output, or fixes a tool call and does not log the minutes. That erases the true cost and paints a red step green. Log every intervention.
- **N=1 costing.** Pricing off one lucky clean run. Failed and retried runs spend real tokens; exclude them and the CPO is fiction.
- **Bare-number costing.** "About $2 a task" with no breakdown is the kill line — it hides where the cost lives and cannot be acted on.
- **Frontier drift into adoption.** "The agent handled 8/10 steps, so operators will trust it." This probe says nothing about trust. Report the frontier; send trust to a concierge or field probe.

## Examples
See `examples/sample.md` — Mentix runs an agent to triage plant maintenance work orders end to end, prices each triaged order to the cent with the full token/tool/human-fix breakdown, and maps the frontier where ambiguous fault codes force a human takeover.

## Related skills
- `../probe-matrix` — the router that sends you here; run it first if unsure.
- `../concierge-probe` — run this first: it gives you the real workflow (and its exceptions) plus the price a customer actually paid, which you compare the CPO against.
- `../wizard-of-oz-probe` — use instead when the question is would-they-use-it-if-it-worked, not what-does-it-cost.
- `../../02-customer-discovery/evidence-ladder` — the rungs the read-out scores against.
- Supersedes nothing. It absorbs the concierge idea and beats the "can an agent do it?" demo by forcing a priced, itemised cost-per-outcome and a frontier map instead of one happy-path run.