Objectstack Upgrade

Upgrading an ObjectStack metadata project across a protocol major

preflight → mechanical chain → semantic residue → acceptance report

⛔ The boundary — read this before the first command

Never hand-write a rewrite the chain already applies. If a key was renamed, the conversion table knows the rename; running the chain attributes each rewrite to a hop and proves the result is schema-valid. A hand-edit does neither, and it silently diverges the moment the chain gains an entry.

Never add a tolerant read to make old metadata load. No ?? alias, no "accept both spellings" branch, no coercion in the project's own code. A key was retired because nothing enforced it or because exactly one spelling survives; re-admitting the old one at the consumer is how the defect the retirement closed comes back inside the customer's repo, where no gate can see it.

Never resolve a residue item by guessing the owner's intent. The residue exists precisely because it is a business decision. The rule for when you decide alone and when you ask is in Layer 2 — it is the most important paragraph in this skill.

Never report "upgraded" without the acceptance artifacts. validate green plus the report is the machine criterion. Absent either, the status is in progress, whatever the diff looks like.

One conversionId per commit, where the project's review culture allows it. The id is the commit's subject line and its justification.

Quickstart

# 0 · preflight — what major is this project on, what major is installed?
grep -rn "protocol" objectstack.config.ts package.json | head
node -p "require('@objectstack/spec/package.json').version"

# 1 · mechanical — replay the chain (reads the config, writes nothing but --out)
os validate > .upgrade/validate-before.txt 2>&1 || true   # the control, kept
os migrate meta --from 16 --step
os migrate meta --from 16 --json > .upgrade/migrate.json
os migrate meta --from 16 --out .upgrade/migrated.stack.json

# 2 · semantic residue — harvest the prescriptions, then work each item
node -e "console.log(require('fs').readFileSync(require.resolve('@objectstack/spec/package.json').replace('package.json','spec-changes.json'),'utf8'))" > .upgrade/spec-changes.json

# 3 · acceptance — all four, not three
os validate                      # green (compare against validate-before.txt)
tsc --noEmit                     # tombstones type the retired keys as `never`
os migrate meta --from 17        # must say "Nothing to migrate"
# → write .upgrade/REPORT.md (template in §3.4)

Everything below is the long form of those four steps.

0 · Preflight

Establish the FROM major — do not guess it

--from is the protocol major the metadata was authored against, not the one installed. Three sources, in order of authority:

1. manifest.engines.protocol in the stack config — the declared answer, checked by the boot handshake. It is a range, not a version: every scaffold and example writes the caret form (engines: { protocol: '^17' }), and an exact '16.0.0' is only a range of one. Take the major it targets — the floor: '^17' → --from 17, '>=15 <18' → --from 15, never the upper bound.
2. The last @objectstack/spec major the project ever installed — read the lockfile history (git log -p pnpm-lock.yaml | grep -m5 '@objectstack/spec') when the manifest is absent or stale.
3. Ask. A manifest that says 16 on a project last touched two years ago is a claim, not a measurement. If (1) and (2) disagree, settle it — ⛔ never default to the lower one: a default flip (3.3) stamps its constraint onto a source already past that major.

Arriving several majors late is the designed-for case. os migrate meta --from 10 replays every step in order; there is no penalty for lateness and no requirement to upgrade one major at a time.

Make the work reviewable before you change anything

git checkout -b upgrade/protocol-17
mkdir -p .upgrade         # every artifact this skill produces lands here

The .upgrade/ directory is the deliverable's workspace: the machine outputs (migrate.json, migrated.stack.json, spec-changes.json, retired-names.txt) and the human output (REPORT.md). Keeping them in the repo for the review, and deleting them on merge, is the usual arrangement — decide it with the owner.

1 · Mechanical layer — invoke the chain

What os migrate meta actually does

os migrate meta --from 16                       # replay 16 → current
os migrate meta --from 16 --step                # per-hop checkpoint (bisect a failure)
os migrate meta --from 16 --to 17               # stop at a specific major
os migrate meta --from 16 --json                # machine-readable result
os migrate meta --from 16 --out migrated.json   # write the canonicalized stack
os migrate meta --from 16 apps/crm/objectstack.config.ts   # pick the stack explicitly

It loads the stack config, normalizes it without applying the load-time conversion pass, replays each major's conversions as a chain hop, and then parses the result against the current schema to prove the output is valid. What it prints:

• Applied N mechanical change(s) — one line per rewritten site, as path: from → to (conversionId). This is the diff, already attributed.
• N manual change(s) require your judgment — the chain's semantic entries for the majors you crossed, each with a why and a verify line. These are Layer 2's input, not a warning to dismiss.
• Migrated stack is schema-valid — or the warning that it is not yet, which means a residue item is still blocking the parse.
• Pending data migrations, when the chain crosses into a major with per-deployment data gates — see below.

⚠ The one fact that surprises every operator

os migrate meta does not rewrite your source files. It rewrites the loaded stack in memory and reports the diff. The only file it writes is --out, a JSON snapshot.

Porting the printed edits into the project's own sources is yours. Work from that list, one conversionId at a time; use --out as the oracle you diff against, never as the file you ship.

os migrate meta --from 16 --out .upgrade/migrated.stack.json
# then, after porting the edits into the real sources:
os migrate meta --from 17 --out .upgrade/recheck.json   # should apply 0 changes

Stored rows: rehydration replays the same conversions

A deployment's sys_metadata rows are the other subject. They are handled for you at read time — the metadata loader and the ObjectQL plugin both pass each stored row through applyConversionsToStoredItem, which replays the conversion chain over a single item including entries retired from the load path. A row written under protocol 16 therefore rehydrates in its protocol-17 shape without anybody editing it.

Rehydration is a read-time projection: the rows on disk keep their old shape, and you never hand-edit sys_metadata. To make it durable, run the stored pass — read-only by default:

os migrate meta --stored                     # preview, writes nothing
os migrate meta --stored --type view --type object   # narrow the pass
os migrate meta --stored --apply --yes       # rewrite the rows

⛔ --yes is not optional for you. Without it --apply asks for confirmation, and in any non-TTY — every agent session — it refuses instead, exiting 1 with confirmation_required.

--stored takes no --from: a stored row carries its own history, so the pass replays the whole chain. The authored-source flags and the stored-only flags are mutually exclusive, and mixing them is refused rather than ignored.

Data migrations are not metadata migrations

When the chain crosses into a major carrying per-deployment data gates, the run ends by naming each one, what staying un-run costs, and that --apply is the only writing mode. Read that output rather than re-deriving it. What it cannot tell you is whether they have run: that happens against each deployment's own database, and nothing in the metadata upgrade observes it.

⛔ Carry every gate the run printed into the report as pending, by name. A gate nobody was told about is served by nobody.

2 · Semantic residue — the part that is yours

A conversion can rename a key, drop a dead one, or lift a value onto its declared block. It cannot make a decision. Everything it cannot do lands here.

2.1 Harvest the instruction sources — all of them ship

The prescriptions are not on a docs site you have to be online for. They ship inside the installed package. Measured against the published @objectstack/spec file list:

# every tombstone prescription the installed spec carries, deduped (`| wc -l` sizes it)
grep -rho '\[REMOVED\][^"]*' node_modules/@objectstack/spec/json-schema/ | sort -u

# protocol version, support floor, and this crossing's conversion / semantic counts
node -e "
  const p = require.resolve('@objectstack/spec/package.json');
  const j = require(p.replace('package.json','spec-changes.json'));
  const to = parseInt(j.protocolVersion, 10);
  const e = j.perMajor.find(x => x.to === to);
  console.log(j.protocolVersion, 'floor', j.supportFloor, '| →', to + ':',
              e.converted.length, 'converted,', e.migrated.length, 'semantic');
"

# the FROM → TO table for one retired key
grep -n -B4 -A20 'transform' node_modules/@objectstack/spec/CHANGELOG.md | less

⛔ Never carry a remembered count or a pinned table into the report. Every number here is a reading of this install; measure it at the start of the upgrade and again in the report.

Not reachable from a consumer project, so do not send anyone there: the conversion and migration registries (src/conversions/registry.ts, src/migrations/registry.ts) and the platform repo's generated upgrade guide are not in the published package — only src/**/*.zod.ts is. Their consumer-facing projection is spec-changes.json and the chain's own --json output, which is exactly what the table above points at.

2.2 The three residue classes

R1 · Intent choice — a key was retired with no single lossless target. The conversion drops the key (so the project parses) and, where it matters, emits a notice naming the site. What the key was for still has to go somewhere, and where is a business statement.

R2 · Custom code calling a retired API. The chain's semantic entries name these: a service slot that no longer exists, an engine method that was removed, a context field that was renamed. Metadata parses fine; the project's own TypeScript is what breaks — or worse, keeps compiling while reading undefined.

R3 · Prose that still teaches the old shape. READMEs, comments, ADRs, seed fixtures, and the project's own AI conventions file. Nothing fails, and the next agent to read the repo re-authors the retired shape from it.

2.3 A worked R1 — the retired field-mapping transform

The shape in a protocol-16 project:

{
  "connectors": [{
    "name": "sap_erp",
    "fieldMappings": [
      { "source": "order_value", "target": "order_total",
        "transform": { "type": "javascript", "expression": "value / 100" } }
    ]
  }]
}

The chain deletes the key (field-mapping-transform-removed) and the schema tombstones it, so the parse error is the prescription: the union had five members and no runtime ever executed any of them. The customer wrote it because they wanted a transformation, and that need is real regardless.

The prescription names one live target; the rest is the business decision:

That third row is frequently the truth: the member never executed, so the connector has been landing raw values for as long as it has been running. Whether the downstream data is wrong is a question only the owner can answer — exactly the kind of finding the report exists to surface.

2.4 Decide alone, or ask the owner

Decide it yourself when all three hold:

1. The prescription names exactly one target. The tombstone or conversion summary gives a single FROM → TO, with the value unchanged.
2. The evidence is in the project. A grep in the repo settles it — the skill that already owns the tool, the import that already exists, the field the predicate already references.
3. Being wrong fails a gate. A mistaken choice breaks tsc or validate rather than changing behaviour quietly.

Ask the owner when any one of these holds:

1. Two or more real targets, and the choice is a business statement — the transform case above.
2. The change is observable without a test failing — security posture (an authentication default), row visibility (an access predicate), retry counts, retention. A wrong call here ships silently and is discovered by an auditor.
3. Capability has to be re-declared somewhere new, so choosing wrong removes a capability instead of breaking a build. Agent tooling that has to move inside a specific skill is the canonical shape.
4. The source is dead or undocumented in their repo — nothing to decide from. Say so; do not invent a rationale.

How to ask. One message, per item, carrying: the site (file and path), the prescription verbatim, the options with what each costs, your recommendation and why, and what you will verify once they choose. Never a bare "how should I handle transform?" — that hands the reading work back to the person with the least context about the diff.

While you wait, do not stall the upgrade. Park the item in the report as AWAITING DECISION, keep the mechanical layer complete, and keep going. A project can be schema-valid with open residue items; it just is not done.

2.5 Working an R2 — retired APIs in the project's own code

The chain's semantic entries are the search list. For each one, the surface it names is a string you grep for in the project's own source:

# from the chain's own output — the surfaces it says it cannot fix for you
os migrate meta --from 16 --json | node -e "
  let s=''; process.stdin.on('data',d=>s+=d).on('end',()=>{
    for (const t of JSON.parse(s).todos) console.log(t.surface);
  })"
# then, for each surface, search the project (not node_modules)
grep -rn "<surface-token>" src/ app/ --include='*.ts' --include='*.tsx'

Two traps that have cost real upgrades a lap:

• A renamed context field keeps compiling. When a read moves from one key to another and the old key is simply absent afterwards, the code reads undefined and every branch quietly takes its false path. Verify against a real dispatch, not a fixture — invoke the path and assert the value observed under the canonical key.
• A rename is not always the fix. If the old read was itself wrong, renaming it migrates the defect rather than the code. Read the semantic entry's reason before applying its replacement.

2.6 Working an R3 — the prose sweep

Run it last, once the shapes are settled, and run it over the whole repo:

# every retired key name the installed spec knows, as a search list
grep -rho '\[REMOVED\] `[^`]*`' node_modules/@objectstack/spec/json-schema/ \
  | sed 's/.*`\(.*\)`.*/\1/' | sort -u > .upgrade/retired-names.txt

Then sweep the project's *.md, comments, seed fixtures, and its AI conventions file. The conventions file matters most: it is what the next agent loads before it writes anything, so a retired shape left there re-enters the codebase on the next feature, long after the upgrade closed.

3 · Acceptance — what "upgraded" means

Four artifacts. Three are machine-checked; the fourth is the one a human reads.

3.1 Typed

tsc --noEmit

A retired key is not merely absent from the schema — it is declared as a tombstone whose input type is never. Assigning anything to it fails to compile, at the authoring site, before anything runs. A green tsc is therefore positive evidence that no retired key survives in typed sources.

3.2 Parse-gated

The same tombstone rejects at parse time, and the rejection carries the prescription rather than a generic "unrecognized key". This is the channel that catches metadata tsc cannot see: JSON files, database rows, anything built at runtime. You do not have to do anything to enable it — but you do have to prove it is live for this project, because a schema that silently strips is indistinguishable from one that accepts. See the reverse check.

3.3 Validate

os validate            # green is the criterion
os validate --strict   # warnings become errors — agree with the owner whether this is the bar

os validate runs two passes: the protocol schema (where tombstones reject) and the author-time rule set. Read the two separately — a rule finding about a missing sharing model or an options-less choice field is a pre-existing project-quality issue, not upgrade residue. Establish which is which by running os validate once before you start, on the un-upgraded source, and keeping that output as the control. Fixing the project's standing lint debt may be a welcome side-effect, but it is not this upgrade, and it must not be reported as part of it.

⚠ A green validate does NOT mean the chain has nothing left to do

Some conversions are migration-chain-only: the loader deliberately does not apply them and no tombstone rejects the old shape, because the change is a default flip rather than a rename — auto-applying it would stamp a constraint onto sources that deliberately omit it. The 16 → 17 crossing has one: field-required-notnull-explicit, which writes the physical storage.notNull that required used to imply on its own.

A project carrying only that shape validates green while the chain still has work. So validate green is necessary and not sufficient, and the criterion that closes the gap is the replay:

os migrate meta --from <target-major>    # must report "Nothing to migrate"

Run both. A report that cites only validate cannot see this class at all.

3.4 The report — the human half

The upgrade is not finished by a passing command; it is finished by a document a maintainer can read in five minutes and a year from now. Write .upgrade/REPORT.md:

# Protocol 16 → 17 upgrade — <project>

**Status:** complete | complete with N open decisions
**Spec:** <installed @objectstack/spec version>  ·  **Chain:** 16 → 17
**Verified:** `os validate` green · `tsc --noEmit` green · replay-from-17 applies 0 changes

## 1 · Mechanical (applied by the chain)

| Site | Change | Conversion |
|:--|:--|:--|
| `objects[crm_lead].fields.name` | `required: true` → `+ storage.notNull: true` | `field-required-notnull-explicit` |
| … | | |

_N sites, M conversions. Ported into sources from `os migrate meta --out`._

## 2 · Semantic residue (decided)

### `connector.fieldMappings[].transform` — RESOLVED
- **Site:** `src/connectors/sap.ts:24`
- **Prescription:** <verbatim from the tombstone>
- **Options:** import-mapping `transform` · ETL step · delete
- **Decision:** delete — owner confirmed the values arrive pre-scaled.
  _Decided by: <who>, <date>._
- **Verified:** `os validate` green; connector sync run against staging, 200 rows, values unchanged.

## 3 · Open decisions

| Item | Site | Options | Recommendation | Blocking? |
|:--|:--|:--|:--|:--|
| `agent.tools` → which skill | `src/ai/support-bot.ts:12` | `case_management` · new skill | `case_management` | no — parses without it |

## 4 · Pending, per deployment

- [ ] `os migrate files-to-references` — media values only warn until it passes.
- [ ] `os migrate value-shapes` — stored reference/JSON values unchecked until it passes.
- [ ] `os migrate meta --stored --apply` — rows rehydrate correctly today; this makes it durable.

## 5 · Not changed, and why

- <retired surface the project never used>  — no occurrences.

3.5 Prove the gate is real, do not assume it

Make the parse gate fire once before you report it as evidence: take a value the chain would have converted, put it back after migrating, and parse it.

# a stack that still carries a retired key, fed straight to the schema
os validate .upgrade/residue-probe.config.mjs

Write the probe as a plain data literal — ⛔ no define* call. os validate loads the config without authored-source mode, and every define* helper is a Schema.parse(), so a probe written the way a real config is written throws inside the load and never reaches the gate you are trying to prove.

Predict the outcome before you run it, then record which one you actually got — "it passed" is a different fact in every row:

3.6 Stored rows: assert them, do not believe them

The four artifacts above cover the authored sources. For a deployment, the stored pass is itself the assertion — run it read-only and branch on the exit code:

os migrate meta --stored --json    # 0 = every row canonical, 1 = work left

That is what makes "this deployment is on protocol N" a CI check rather than a belief. Nothing gates on it having run; the read-time rehydration is still the guarantee.

The v17-canonical shapes, compiled

What the protocol-16 shapes in this skill's examples look like after the upgrade. This block is type-checked against the published spec, so it cannot rot into teaching a shape that no longer compiles:

import { ObjectSchema } from '@objectstack/spec/data';
import { defineAgent } from '@objectstack/spec/ai';

// `conditionalRequired` → `requiredWhen`; `required` now also states the
// physical constraint explicitly via `storage.notNull`.
export const Lead = ObjectSchema.create({
  name: 'crm_lead',
  sharingModel: 'public_read_write',
  label: 'Lead',
  fields: {
    name: { type: 'text', required: true, storage: { notNull: true } },
    status: { type: 'select', required: true, storage: { notNull: true } },
    due_date: { type: 'date', requiredWhen: 'record.stage == "closed"' },
    notes: { type: 'textarea' },
  },
});

// Agent capability is reached through skills — there is no inline tool list.
export const SupportBot = defineAgent({
  name: 'support_bot',
  label: 'Support Bot',
  role: 'Front-line support triage',
  instructions: 'Answer support questions and open cases when needed.',
  skills: ['case_management'],
});

Failure modes

Scripting the run instead of reading it? Every --json failure above carries a stable error code to branch on — stored_only_flag, missing_from_major, unsupported_from_major, confirmation_required, database_busy — and the prose row is the same fact for a human.

Cross-skill routing

• Authoring the corrected metadata — load the domain skill for the shape you are fixing (data, ui, automation, ai, api, i18n); each one routes on to formula for any CEL you rewrite.
• Runtime, plugin, and CLI questions the upgrade turns up — load platform.