Quality

Quality

Quality answers:

What must this project deliver, which methods verify each promise, and what does the current evidence justify believing?

It maintains a quality project graph and asks the deterministic quality-tools engine to compute a four-score quality index from that graph. It never invents a score or treats a score as the goal.

Before any command, read independence. A system cannot establish trust by grading its own claims. This principle governs every workflow, especially evidence classification and human validation.

Use one formal vocabulary

Use these terms consistently:

Do not introduce a second grouping such as release area. Use a view when a reusable subset of features must be assessed together. Use evidence gap in prose; proof_gap is only the legacy literal field name in the current quality-map contract.

The quality project graph

Explain this model before asking a new user to choose an artifact or command:

repository (storage)
└── quality project
    └── features
        └── quality checks: what must hold
            └── verification methods: how each check is evaluated
                └── runtime observations: what happened when executable methods ran?

view ─────────────── selects project features
observation set ──── selects runtime-result sources

The graph is represented by checked-in files and the evidence artifacts they reference:

IDs, paths, and optional test-case names are the graph's edges. Generated recommendations and fix-prompts.md are engine output, not graph source.

Example:

repository: monots
project: Shiplight developer tooling
features:
├── 001-web-agent-engine
├── 002-shiplightai-cli
└── 003-shiplightai-mcp-server

view shiplightai = [001-web-agent-engine, 002-shiplightai-cli]
view mcp-server = [001-web-agent-engine, 003-shiplightai-mcp-server]

CLI and MCP are independently releasable scopes. A particular CLI tarball or MCP package version is a release candidate only when an assessment identifies that concrete revision/artifact.

Shared contracts:

• Independence and human gates
• Graph layout and edit ownership
• Authored and runtime vocabularies

The workflow

Guide users through the lifecycle; do not make them infer an order from the artifact names:

1. Start or inventory. Determine whether the repository is spec-driven, brownfield, or already partially mapped.
2. Map the project. Construct project → feature structure and choose the highest-priority feature.
3. Map each feature. Define its quality checks and connect existing verification methods.
4. Connect runtime. Arrange for evidence producers to publish canonical quality-observations.json, within the producer edit boundary below, then configure transport-only sources and sets that locate those files.
5. Assess. Run quality-tools and explain the scores together. Coverage, evidence confidence, and structure confidence are available from step 3 onward; Quality also needs step 4.
6. Improve. Diagnose the weak score, improve the underlying structure, evidence, implementation, or runtime wiring, then assess again.
7. Repeat. Expand feature by feature, highest priority and risk first.

/quality start orchestrates steps 1–5 for a new or unmapped repository. The more specific commands let an experienced user enter at any stage.

Producer edit boundary

Quality never authors tests, edits producer-owned reports, or changes the logic that determines a result. When explicitly authorized, Quality may add only mechanical workflow glue that serializes and uploads an already-determined result as canonical quality-observations.json. It must not manufacture or reinterpret a status or change test commands, gates, retries, or failure semantics.

Without explicit authorization, propose the exact emit/upload change and record the observation gap without editing the producer. Follow independence for the complete boundary.

Canonical join-key invariant

evidence.test_case is a runtime join key, not a readable sub-artifact label. Before retaining or adding one, locate an actual canonical observation record (or a freshly converted native report) containing the same path and test_case. Cite that record during the audit. Source test names, checklist headings, workflow jobs/steps, and scenario titles do not satisfy this precondition, even when copied verbatim. If no record exists, remove or omit the pin, keep the readable pointer in notes or command, and record the runtime emission gap. The only exception is improve wiring the producer to emit that exact key in the same change.

Tool version gate

This skill's canonical-observation contract requires @shiplightai/quality-tools 0.3.0. Before invoking quality-tools or authoring observation configuration, verify that interface:

npx --yes @shiplightai/quality-tools@^0.3.0 observations --help

Do not fall back to an unversioned package or 0.2.x. If the pinned package is unavailable, do not write transport-only observation config; report the package publication blocker. When working inside the quality-tools source checkout, use its repository-local CLI instead:

pnpm exec tsx packages/quality-tools/src/cli.ts observations --help

Commands

The optional [scope] argument is a natural-language focus, not a required project name, filesystem path, or saved assessment view. For start, use it to tell the agent which capability or risk area to map first (for example, billing or checkout); omit it when the agent should choose from the repository's priorities. A saved assessment scope is a separate named view in .quality/config/views.yaml.

Bare /quality means status, followed by the command menu.

Load the supporting project-map contract or brownfield reconstruction only when start or map-project requires it.

Routing contract

1. Match the exact command first, then its natural-language outcome. There are no legacy command aliases.
2. Pass unconsumed text as scope or target: /quality map-feature checkout → map-feature with target checkout.
3. Read the command reference and every shared/reference file it requires.
4. If the request names no command:
   • use start when the repository has no usable quality graph and the user asked to establish one;
   • use status for a read-only question;
   • use assess when the user asks to refresh or compute scores;
   • use improve when the user asks to raise quality posture or close gaps.
5. State what will be read, generated, or edited before acting.
6. Stop at human gates. An agent may propose structure, priorities, reviewed checks, or accepted risk, but must never validate intent for the owner.

The four scores and the honest improvement lever

Always report these side by side. Never blend them into a single agent-authored verdict.

Quality is the only one of the four that needs runtime observations. When no observation set is configured or its results cannot be acquired, report the other three as measured and say why Quality is missing.

Treat the score as a diagnostic, not a target. Never remove scope, weaken a check, misclassify evidence, accept risk, or promote provenance to make a number rise.

Reporting integrity

• Copy scores, counts, and resolution totals from engine or parsed artifact output. When no structured total exists, enumerate first and calculate the count mechanically; verify that headings, totals, and lists agree.
• Group every cause under the score it actually affects. Missing or policy-insufficient verification methods affect coverage; method type and gate strength affect evidence confidence; runtime outcomes/acquisition/resolution affect Quality; feature/check/priority provenance and review affect structure confidence. Do not cite a structure gate as an evidence-confidence cause.
• Interpret policy fields independently using engine semantics. require_gate means any engine-recognized gate context; required_contexts carries exact context requirements such as release-ci. Prefer engine diagnostics over a narrower natural-language reinterpretation.
• Record a human decision without adding a name, email, account, or other attribution unless the user explicitly supplied that identity for the record or an authoritative artifact already contains it. Never infer attribution from Git config, environment metadata, or the operating-system account.

Relationship to evidence producers

Quality maps and judges existing evidence. It may hand a concrete evidence gap to a producer such as /shiplight cover, /shiplight create-yaml-tests, or /shiplight create-agent-verification. Producers create or run tests and other verification methods; Quality confirms the resulting artifact, connects it to the graph, and remeasures. The only producer-side exception is the explicitly authorized mechanical workflow glue defined above.