Adding an ingest stage / table to ax
Hard-won checklist - each step below has a CI gate that fails if you skip it.
New SurrealDB table
- Add the
DEFINE TABLE ... SCHEMAFULL+ fields topackages/schema/src/schema.surql(top-level fields explicit; nested objects → JSON-encodedstring; datetimes via JSDate). - Register it in
SCHEMA_TABLES(apps/axctl/src/queries/insights.ts) - a mirror test diffsDEFINE TABLEnames vsSCHEMA_TABLESand fails CI if missing. - In a worktree, run
bun installso@ax/schemaresolves to the worktree copy (not the main tree's symlink) before the mirror test will pass.
New derive-tagged ingest stage
- Co-locate the
StageDefat the bottom of the stage file (mirrorapps/axctl/src/ingest/derive-opportunities.ts):export const FooKey = Schema.Literal("foo")+fooStage. - Register in
apps/axctl/src/ingest/stage/registry.ts: add the import, addFooKeyto theIngestStageKeySchema.Union([...]), addfooStagetoALL_STAGES. - Update
apps/axctl/src/cli/effect-cli.test.ts: bump theresolveIngestStages: default runs every stage.toHaveLength(N)by 1, and add your key (sorted) to the--derive-onlylist. CI fails otherwise. - Isolate failures: wrap the stage body in
Effect.catchCausereturning a zero-row stat, so a stage error never aborts the surrounding ingest.
New CLI subcommand
Document it in BOTH cli-reference gates or CI fails: docs/cli.md (or README) + apps/site/public/llms.txt (scripts/check-cli-reference.ts), AND a card in apps/site/app/routes/docs/-cli-reference.data.ts (scripts/check-site-cli-reference.test.ts).
Effect v4 beta gotchas
- Multi-arg
Schema.Literal(...)collapses → useSchema.Literals([...]). Schema.Datedoesn't JSON round-trip → useSchema.DateFromString.check(Schema.isDateValid()).