feedback — learn from corrections, honestly
Follow the shared contract in ../README.md. "Learning" here means enriching the local knowledge base and example store (README's honest positioning) — nothing else. Promotion is always human-validated.
Prerequisite
A generated test book to compare against (.qaia/testbooks/<US-ID>/, from testbook-generate). If none exists — no checkpoint, no test book — say so and offer us-ingest to start the journey instead of asking for corrections that have nothing to be diffed against.
Steps
Collect. Ask what the user changed or rejected in the test book (or diff the edited .feature files against the generated version if both exist). For each correction, capture: scenario ID, what was wrong, the corrected form, and why — the why is the valuable part.
Classify. Each correction is one of:
business-rule — the generation contradicted or missed a domain rule → candidate for knowledge/;style — wording, structure, granularity preference → candidate for a project convention entry;one-off — specific to this US, not generalizable → example only.
Store examples. Write each correction to .qaia/feedback/examples/<US-ID>-<n>.md with its classification and provenance.
Propose promotions: only when the same pattern appears in ≥ 2 stored examples, or the user explicitly asks for immediate promotion (single-criterion — the "states a reusable rule" shortcut promoted everything and filtered nothing). Rules get stable IDs BR-KB-nnn (counter persisted in rules.md frontmatter); examples get <US-ID>-<nnn> with the counter in examples/. When a promoted rule shapes a generated scenario, the scenario carries a # rule: BR-KB-nnn comment and the coverage matrix lists applied rules — flagging sibling scenarios of the same AC for regeneration. ⚠ VALIDATION: present the proposed promotion with this callout, verbatim; on approval, hand the rule to rag-build (which handles contradiction checks and the index); record the promotion in feedback/rules.md with links to its source examples.
If you own the product rather than the tests, read this. You do not need the rest of
this page.
- What you're being asked: a correction is being proposed as a permanent rule — either
because it has come up more than once, or because someone asked for it directly. Which of
the two is stated above this box. It
looks like a standing rule of your business rather than a one-off fix. You are being
asked to confirm that it is — in the wording below.
- Why it matters: a confirmed rule is reused on every future story, automatically, by
everyone on the team. That is the payoff. It is also the risk: a rule that is almost
right, or true only for one product line, quietly propagates into stories nobody
connected it to. Read the wording, not just the idea — especially any "always" or
"never".
- If you don't answer: nothing is promoted. The corrections stay as isolated examples
attached to their own story, and future generations may repeat the same mistake — which
is the safe outcome, not the harmful one.
Confirming a rule here changes future tests, never past ones, and it can be withdrawn later — by deleting its entry from feedback/rules.md and the corresponding line in knowledge/index.md, which is a two-line edit a human makes; there is no automated revoke, and saying otherwise would be a promise nothing keeps.
- Prune. When promoting, mark source examples
promoted; offer to archive examples older than ~6 months that never recurred — the store must not grow unbounded, or retrieval degrades and the signal drowns.
- Close the loop. Tell the user which promoted rules will affect future generations, and remind them the effect is measured — not guaranteed — via the gold set: reapplication of a raw stored example is probabilistic, a promoted rule is the reliable path, and promising more than that would be the dishonest version of "learning".
Guardrails
- Never promote without explicit validation, even for "obvious" corrections.
- Contradiction between a new correction and an existing rule → surface it (via
rag-build's arbitration), never store both silently — two contradictory rules in the base make every later generation a coin toss.
- Feedback content follows knowledge rules: no secrets, no personal data, provenance mandatory.
1---2name: feedback3description: Capture the tester's corrections on a generated test book, store them as examples, and propose validated promotion of recurring corrections into knowledge-base rules so future generations improve. Use after someone has reviewed or reworked generated tests, when the same correction keeps coming back run after run, or when asked to make QAIA learn a project's own conventions instead of repeating the same mistake. Final step of the QAIA journey.4---56# feedback — learn from corrections, honestly78Follow the shared contract in `../README.md`. "Learning" here means enriching the local knowledge base and example store (README's honest positioning) — nothing else. Promotion is **always human-validated**.910## Prerequisite1112A generated test book to compare against (`.qaia/testbooks/<US-ID>/`, from `testbook-generate`). If none exists — no checkpoint, no test book — say so and offer `us-ingest` to start the journey instead of asking for corrections that have nothing to be diffed against.1314## Steps15161. **Collect.** Ask what the user changed or rejected in the test book (or diff the edited `.feature` files against the generated version if both exist). For each correction, capture: scenario ID, what was wrong, the corrected form, and **why** — the why is the valuable part.172. **Classify.** Each correction is one of:18 - `business-rule` — the generation contradicted or missed a domain rule → candidate for `knowledge/`;19 - `style` — wording, structure, granularity preference → candidate for a project convention entry;20 - `one-off` — specific to this US, not generalizable → example only.213. **Store examples.** Write each correction to `.qaia/feedback/examples/<US-ID>-<n>.md` with its classification and provenance.224. **Propose promotions**: only when the same pattern appears in **≥ 2** stored examples, or the user explicitly asks for immediate promotion (single-criterion — the "states a reusable rule" shortcut promoted everything and filtered nothing). Rules get stable IDs `BR-KB-nnn` (counter persisted in `rules.md` frontmatter); examples get `<US-ID>-<nnn>` with the counter in `examples/`. When a promoted rule shapes a generated scenario, the scenario carries a `# rule: BR-KB-nnn` comment and the coverage matrix lists applied rules — flagging sibling scenarios of the same AC for regeneration. ⚠ VALIDATION: present the proposed promotion with this callout, verbatim; on approval, hand the rule to `rag-build` (which handles contradiction checks and the index); record the promotion in `feedback/rules.md` with links to its source examples.2324 > **If you own the product rather than the tests, read this. You do not need the rest of25 > this page.**26 >27 > - **What you're being asked:** a correction is being proposed as a permanent rule — either28> because it has come up more than once, or because someone asked for it directly. Which of29> the two is stated above this box. It30 > looks like a standing rule of your business rather than a one-off fix. You are being31 > asked to confirm that it is — in the wording below.32 > - **Why it matters:** a confirmed rule is reused on every future story, automatically, by33 > everyone on the team. That is the payoff. It is also the risk: a rule that is *almost*34 > right, or true only for one product line, quietly propagates into stories nobody35 > connected it to. Read the wording, not just the idea — especially any "always" or36 > "never".37 > - **If you don't answer:** nothing is promoted. The corrections stay as isolated examples38 > attached to their own story, and future generations may repeat the same mistake — which39 > is the safe outcome, not the harmful one.40 >41 > Confirming a rule here changes future tests, never past ones, and it can be withdrawn later — by deleting its entry from `feedback/rules.md` and the corresponding line in `knowledge/index.md`, which is a two-line edit a human makes; there is no automated revoke, and saying otherwise would be a promise nothing keeps.425. **Prune.** When promoting, mark source examples `promoted`; offer to archive examples older than ~6 months that never recurred — the store must not grow unbounded, or retrieval degrades and the signal drowns.436. **Close the loop.** Tell the user which promoted rules will affect future generations, and remind them the effect is measured — not guaranteed — via the gold set: reapplication of a raw stored example is probabilistic, a promoted rule is the reliable path, and promising more than that would be the dishonest version of "learning".4445## Guardrails4647- Never promote without explicit validation, even for "obvious" corrections.48- Contradiction between a new correction and an existing rule → surface it (via `rag-build`'s arbitration), never store both silently — two contradictory rules in the base make every later generation a coin toss.49- Feedback content follows knowledge rules: no secrets, no personal data, provenance mandatory.
QAIA-Project