Close Conformance Gaps

Close Conformance Gaps

Use this skill when an automated run or a maintainer asks you to raise this repo's Durable Streams conformance pass rate, implement a gated/skipped conformance section, or fix a regression that turned a passing conformance test red.

The goal is to make the server implementation satisfy more of the vendored conformance suite without regressing any test that currently passes. The conformance suite is the protocol's source of truth; the server is what must adapt to it.

Core Requirements

• The source of truth is the vendored suite at packages/server-conformance-tests/src/index.ts. Read the test that defines the behavior before changing the server. Never edit the conformance suite to make a test pass.
• Validate against the Node host (bun run conformance:node), which boots a server and runs the full black-box suite. CI cannot rely on the Durable Object host (it needs wrangler dev); note DO parity as follow-up rather than blocking on it.
• Prefer implementing protocol behavior once in the core (packages/effect-durable-streams-server-core/src) so every host benefits. Only touch a host's substrate Layer when the behavior is genuinely platform-specific.
• Never leave the default suite red. bun run conformance:node must still pass every test it passed before (currently 326/332). If you implement only a slice of a gated section, keep that section gated in conformance/run.test.ts so the default run stays green, and document the remaining work in your response.
• Land one reviewable unit per run — a coherent slice of a section, or a single regression fix — not a sprawling multi-feature change.

Background: where the gaps are

The suite passes 326/332 on both hosts. The 6 non-passing tests are the Reserved subscription APIs section (describe.runIf(options.subscriptions) near the bottom of packages/server-conformance-tests/src/index.ts), which exercises the reserved __ds/subscriptions/{id} and __ds/subscriptions/{id}/callback webhook / pull-wake endpoints. That section is gated off by subscriptions: false in conformance/run.test.ts.

Two kinds of gap qualify as targets:

1. A gated/unimplemented section — implement a reviewable slice of it.
2. A regression — a test that used to pass now fails. Fixing it always takes priority over new feature work.

Workflow

1. Establish the baseline

Run the full suite against the Node host and record the pass/fail/skip counts:

bun run conformance:node

If any test that should pass is failing, that regression is your target for this run — skip ahead to step 4. Otherwise, the target is a gated section (the subscription APIs).

Completion criterion: you know the current pass count and whether there is a regression or you are working a gated section.

2. Read the conformance spec for your target

Open packages/server-conformance-tests/src/index.ts and read the exact tests for your target section. Note every endpoint, header, status code, and body shape they assert. These assertions are the contract.

For the subscription section, read the Reserved subscription APIs describe block in full before writing any code.

Completion criterion: you can list the specific HTTP behaviors the failing tests require.

3. Make the gated section runnable (first time only)

The section is gated by a literal subscriptions: false in conformance/run.test.ts. To validate slices without forcing the whole gated section green at once, make the gate env-driven (a small, reviewable change):

runConformanceTests({
  baseUrl,
  longPollTimeoutMs,
  subscriptions: process.env.CONFORMANCE_SUBSCRIPTIONS === "true",
})

This keeps the default bun run conformance:node behavior identical (section stays gated) while letting you opt the section in during development. Only flip the default to enabled once the entire section passes.

Completion criterion: you can run the target section on demand without changing the default suite's behavior.

4. Implement the smallest correct slice in the core

Add or fix behavior in packages/effect-durable-streams-server-core/src:

• New routes belong in src/http/router.ts; engine logic in src/stream-service.ts; pure protocol helpers in src/domain/*.
• If the feature needs a new capability the host must provide (e.g. delivering a webhook), model it as a substrate Service tag under src/services/ and add an in-memory implementation under src/memory/, then wire the Node host (packages/node-durable-streams-server/src) so CI can validate it. Note the Durable Object substrate as a follow-up if you don't implement it this run.
• Follow the existing Effect v4 idioms: typed StreamError channel mapped at the edge, Effects for fallible operations, no non-null assertions or unguarded JSON.parse.

Keep the slice small enough to review in one sitting.

Completion criterion: the slice compiles and targets the specific assertions you listed in step 2.

5. Validate

Run, in order, the narrowest relevant check first:

# the target section (env-gated) against the Node host
CONFORMANCE_SUBSCRIPTIONS=true bun run conformance:node -t "Reserved subscription APIs"

# the full default suite — MUST stay green (no regressions)
bun run conformance:node

# typecheck every package and run core unit tests
bun run check
bun run test

If you only implemented a slice, the target section may still have failures — that is acceptable as long as the default suite (step's second command) is fully green and you report the remaining work. If the full section now passes, flip the default subscriptions to enabled in conformance/run.test.ts and re-run bun run conformance:node to confirm the new tests count as passing by default.

Completion criterion: the default suite passes with no regressions, check and test pass, and you can state the exact new pass count or the precise blocker.

6. Format the response

When running as a CI agent, format your final response according to references/response-template.md. This response becomes the PR body.

Completion criterion: the response follows the template and reports baseline → new pass counts, what was implemented, and what remains.

Review Checklist

• The conformance suite was read before the server was changed, and the suite itself was not edited to force a pass.
• Protocol behavior lives in the core unless it is genuinely platform-specific.
• bun run conformance:node still passes every test it passed before (no regression below the baseline).
• A partially-implemented gated section is left gated by default so the default run stays green; remaining work is documented.
• bun run check and bun run test pass.
• The diff is one reviewable slice or one regression fix, not a sprawl.
• The final response follows the response template with baseline → new counts.

Anti-Patterns to Avoid

• Editing packages/server-conformance-tests/src/index.ts to make a test pass.
• Flipping subscriptions: true by default while the section still has failures (this turns the default suite red).
• Implementing a feature in a host substrate when it belongs in the core.
• Reintroducing non-null assertions, unguarded JSON.parse, or untyped error throws to take a shortcut.
• Bundling unrelated cleanup or refactors into a conformance PR.