# Aidd Riteway AI

> Teaches agents how to write correct riteway ai prompt evals (.sudo files) for multi-step flows that involve tool calls. Use when writing prompt evals, creating .sudo test files, or testing agent skills that use tools such as gh, GraphQL, or external APIs.

- Skill: `paralleldrive/aidd-riteway-ai` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add paralleldrive/aidd-riteway-ai`
- Raw SKILL.md: https://api.skillmd.com/api/skills/paralleldrive/aidd-riteway-ai/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: paralleldrive (https://skillmd.com/u/paralleldrive)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/paralleldrive/aidd-riteway-ai

---


# 🧪 aidd-riteway-ai

Act as a top-tier AI test engineer to write correct `riteway ai` prompt evals
for multi-step agent skills that involve tool calls.

Refer to `/aidd-tdd` for assertion style (given/should/actual/expected) and
test isolation principles.

Refer to `/aidd-requirements` for the **"Given X, should Y"** format
when writing assertions inside `.sudo` eval files.

---

## Process

1. Read the skill under test and its functional requirements
2. Identify the discrete steps in the skill's flow
3. Create one `.sudo` eval file per step (Rule 1), placed in `ai-evals/<skill-name>/`
4. For each file, write the `userPrompt` — include mock tool preambles for unit evals (Rule 2), assert tool calls for step 1 (Rule 3), supply previous step output for step N > 1 (Rule 4)
5. Write assertions derived strictly from functional requirements in `Given X, should Y` format (Rule 7)
6. Create small, single-condition fixture files as needed (Rule 6)
7. Verify against the Eval Authoring Checklist below

---

## Eval File Structure

A `.sudo` eval file has three sections:

```
import 'ai/skills/<skill-name>/SKILL.md'

userPrompt = """
<prompt sent to the agent under test>
"""

- Given <condition>, should <observable behavior>
- Given <condition>, should <observable behavior>
```

Assertions are bullet points written after the `userPrompt` block.
Each assertion tests one distinct observable behavior derived from the
functional requirements of the skill under test.

---

## Rule 1 — One eval file per step

Given a multi-step flow under test, write **one `.sudo` eval file per step**
rather than combining all steps into a single overloaded `userPrompt`.

Naming convention:

```
ai-evals/<skill-name>/step-1-<description>-test.sudo
ai-evals/<skill-name>/step-2-<description>-test.sudo
```

Do not collapse multiple steps into one file. Each file tests exactly one
discrete agent action.

---

## Rule 2 — Unit evals: tell the agent it is in a test environment

Given a unit eval for a step that involves tool calls (gh, GraphQL, REST API),
include a preamble in the `userPrompt` that:

1. Tells the prompted agent it is operating in a test environment.
2. Provides mock tools with stub return values.
3. Instructs the agent to use the mock tools instead of calling real APIs.

Example preamble:

```
You have the following mock tools available. Use them instead of real gh or GraphQL calls:

mock gh pr view => returns:
  title: My PR
  branch: feature/foo
  base: main

mock gh api (list review threads) => returns:
  [{ id: "T_01", resolved: false, body: "..." }]
```

---

## Rule 3 — Step 1: assert tool calls, do not pre-supply answers

Given a unit eval for **step 1** of a tool-calling flow, assert that the agent
makes the correct tool calls. Do **not** pre-supply the answers those calls
would return — that defeats the purpose of the eval.

Correct pattern for step 1:

```
userPrompt = """
You have mock tools available. Use them instead of real API calls.
Run step 1 of your skill under test: fetch the PR details and review threads.
"""

- Given mock gh tools, should call gh pr view to retrieve the PR branch name
- Given mock gh tools, should call gh api to list the open review threads
- Given the review threads, should present them before taking any action
```

Wrong pattern (pre-supplying answers in step 1):

```
# ❌ Do not do this — it removes the assertion value
userPrompt = """
The PR branch is feature/foo.
The review threads are: [...]
Now generate delegation prompts.
"""
```

---

## Rule 4 — Step N > 1: supply previous step output as context

Given a unit eval for **step N > 1**, include the output of the previous step
as context inside the `userPrompt`. This makes each eval independently
executable without running the prior steps live.

Example for step 2:

```
userPrompt = """
You have mock tools available. Use them instead of real calls.

Triage is complete. The following issues remain unresolved:

Issue 1 (thread ID: T_01):
  File: src/utils.js, line 5
  "add() subtracts instead of adding"

Generate delegation prompts for the remaining issues.
"""
```

---

## Rule 5 — E2E evals: use real tools, follow -e2e.test.sudo naming

Given an e2e eval, use real tools (no mock preamble) and follow the
`-e2e.test.sudo` naming convention to mirror the project's existing unit/e2e
split:

```
ai-evals/<skill-name>/step-1-<description>-e2e.test.sudo
```

E2E evals run against live APIs. Only run them when the environment is
configured with the necessary credentials.

---

## Rule 6 — Fixture files: small, one condition per file

Given fixture files needed by an eval, keep them small (< 20 lines) with
**one clear bug or condition per file**. Fixtures live in:

```
ai-evals/<skill-name>/fixtures/<filename>
```

Example fixture (`add.js`):

```js
export const add = (a, b) => a - b; // bug: subtracts instead of adds
```

Do not combine multiple bugs in one fixture file. Each fixture must make the
assertion conditions unambiguous.

---

## Rule 7 — Assertions: derived from functional requirements only

Given assertions in a `.sudo` eval, derive them strictly from the functional
requirements of the skill under test using the `/aidd-requirements`
format:

```
- Given <condition>, should <observable behavior>
```

Include only assertions that test **distinct observable behaviors**. Do not:

- Assert implementation details (e.g. internal variable names)
- Repeat the same observable behavior with different wording
- Assert things that are implied by another assertion already in the file

---

## Eval Authoring Checklist

Before saving a `.sudo` eval file, verify:

- [ ] One step per file (Rule 1)
- [ ] Unit evals include mock tool preamble (Rule 2)
- [ ] Step 1 asserts tool calls, not pre-supplied answers (Rule 3)
- [ ] Step N > 1 includes previous step output as context (Rule 4)
- [ ] E2E evals use `-e2e.test.sudo` suffix (Rule 5)
- [ ] Fixture files are small, one condition each (Rule 6)
- [ ] Assertions derived from functional requirements, no duplicates (Rule 7)

---

Commands {
  🧪 /aidd-riteway-ai - write correct riteway ai prompt evals for multi-step tool-calling flows
}