# Shortcircuit Streaming

> Reference for ShortCircuit XT serialization and messaging (src/scxt-core/json/, src/scxt-core/messaging/). Covers taocpp/json trait patterns, SC_DESCRIBE metadata, c2s/s2c message pairs, the three-thread architecture, backward compatibility, and how to add a new parameter or message.

- Skill: `baconpaul/shortcircuit-streaming` (Agent Skill)
- Install (CLI): `npx skillmds@latest add baconpaul/shortcircuit-streaming`
- Raw SKILL.md: https://api.skillmd.com/api/skills/baconpaul/shortcircuit-streaming/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: baconpaul (https://skillmd.com/u/baconpaul)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/baconpaul/shortcircuit-streaming

---


# ShortCircuit XT Streaming and Messaging

**Repo:** `surge-synthesizer/shortcircuit-xt`. Paths are relative to the repo root, and
mostly to `src/scxt-core/`.

Companion skills: `shortcircuit-engine` (the data being streamed), `shortcircuit-ui` (the
client end), `sc-stream-enum` (procedure for streaming one enum).

---

## 1. Three threads

```
Client / UI  ──c2s──▶  Serialization  ──lambda──▶  Audio
     ▲                      │   ▲                    │
     └────────s2c───────────┘   └────ring buffer─────┘
```

**Audio** runs `Engine::processAudio()`. It drains `serializationToAudioQueue` and runs the
queued lambdas, and pushes state notifications onto `audioToSerializationQueue`. Both are
`sst::cpputils::SimpleRingBuffer`, lock-free. Nothing else here may allocate or block.

**Serialization** runs `MessageController::runSerialization()`. It wakes on a condition
variable or every ~50 ms, executes queued client messages under `modifyStructureMutex`,
drains the audio queue, and pushes `s2c` messages back to registered clients. This thread
may allocate, lock and do I/O — it is where sample loading and JSON streaming happen.

**Client** may be the JUCE editor, the console UI, or a test harness. It sends `c2s`, gets
`s2c` through a registered callback, and never touches engine memory. The single sanctioned
exception is `Engine::SharedUIMemoryState`, read directly for VU and voice display.

`MessageController::threadingChecker` asserts which thread you are on. If you are unsure
whether code runs on audio or serialization, assert rather than guess.

### Getting work onto the audio thread

```cpp
cont.scheduleAudioThreadCallback(
    [](engine::Engine &e) { /* runs on the audio thread */ },
    [](const engine::Engine &e) { /* then back on serialization, to reply to the client */ });
```

Three variants, and picking the wrong one causes rare, ugly bugs:

| Call | Use for |
|---|---|
| `scheduleAudioThreadCallback` | Value changes. Cheap, non-structural. |
| `scheduleAudioThreadCallbackUnderStructureLock` | Touching structure while the audio thread runs. |
| `stopAudioThreadThenRunOnSerial` | Work that must happen with no audio thread at all — anything attaching or detaching samples, or resurrecting groups and parts from JSON. |

The lambda is stored in a fixed-size ring-buffer slot. Capture small; do not capture
anything that allocates on destruction from the audio side.

### Wire format

`c2s` and `s2c` payloads serialize with **msgpack**, not JSON — see `PROCESS_AS_JSON` in
`messaging/client/detail/client_serial_impl.h`. Flip it to 1 locally when you want to read
messages during debugging. Patch files on disk are JSON; the message bus is not.

---

## 2. Streaming with taocpp/json

### SC_STREAMDEF

Trait specializations live in `json/`. The core macro is in `json/scxt_traits.h`:

```cpp
SC_STREAMDEF(MyType,
    SC_FROM({
        v = {{"field", t.field}, {"other", t.other}};      // t is the source object
    }),
    SC_TO({
        findIf(v, "field", result.field);                  // result is the target
        findIf(v, "other", result.other);
    }))
```

`SC_FROM` builds JSON from `t`. `SC_TO` populates `result` from `v`.

### Helpers

| Helper | Behaviour |
|---|---|
| `findIf(v, "key", r)` | Read if present; leave `r` alone if not. Returns bool. |
| `findIf(v, {"old", "new"}, r)` | Try several key names — how a renamed field stays loadable. |
| `findOrSet(v, "key", def, r)` | Read, else assign `def`. **This is the one to reach for on a new field.** |
| `findOrDefault(v, "key", r)` | Read, else value-initialize. |
| `findEnumIf(v, "key", r)` | Read an int and cast to enum. |
| `addUnlessDefault<T>(v, "key", def, val)` | Only write when non-default. Keeps files small; costs you nothing on read because the reader defaults. |

### Traits files

`ls src/scxt-core/json/` is the index. Roughly: `engine_traits.h` for
engine/part/group/zone/patch/bus, `modulation_traits.h` for modulators and matrices,
`sample_traits.h`, `dsp_traits.h`, `datamodel_traits.h` for `ParamMetaData`,
`selection_traits.h`, `missing_resolution_traits.h`, `utils_traits.h`, and `scxt_traits.h`
for the base template and the macros themselves. `extensions.h` and `stream.{h,cpp}` hold
the entry points.

If you cannot tell which file a type belongs in, find where its siblings are declared and
follow them.

### Enums

Enums need `STREAM_ENUM` plus a `toString`/`fromString` pair on the owning class. That is
its own three-file procedure — use the `sc-stream-enum` skill.

`STREAM_ENUM_WITH_DEFAULT` omits the key entirely when the value equals the given default.
Use it sparingly and only where the enum has one tightly controlled use, because it makes
the absence of a key meaningful.

### Conditional streaming

The current reason for streaming is a thread-local, set by
`Engine::StreamGuard(Engine::StreamReason::…)` with values `IN_PROCESS`, `FOR_MULTI`,
`FOR_PART`, `FOR_DAW`. Read it through the macros:

```cpp
SC_FROM({
    if (SC_STREAMING_FOR_DAW)
        v["dawOnlyThing"] = t.dawOnlyThing;
})
```

`SC_STREAMING_FOR_DAW`, `SC_STREAMING_FOR_MULTI`, `SC_STREAMING_FOR_DAW_OR_MULTI`.

### Backward compatibility

`configuration.h` holds `currentStreamingVersion` as a date-shaped hex constant. **Bump it
when you change the format in a way older readers or newer readers need to know about**,
and read its current value from the file — never quote one from memory.

While unstreaming a full engine, the version the file was written with is available:

```cpp
if (SC_UNSTREAMING_FROM_PRIOR_TO(0x2026'08'11))
    result.thing = legacyDefaultFor(result);
```

The rule that keeps old patches loading: **a new field must have a sensible value when the
key is absent.** `findOrSet` with a default is usually the whole job; a version check is
only needed when the correct default depends on other fields.

`tests/streaming.cpp` and `tests/extension_guarantee_tests.cpp` cover this. A format change
without a test there is a format change that will regress.

---

## 3. Parameter metadata (SC_DESCRIBE)

`datamodel/metadata_detail.h`. `SC_DESCRIBE` attaches `ParamMetaData` — range, default,
display format, UI hints — to a struct's fields. It describes; it never stores values.

```cpp
SC_DESCRIBE(MyDataType, {
    SC_FIELD(member, pmd().asPercent().withDefault(0.5f).withName("Depth"));
    SC_FIELD_ARRAY(members, N, pmd().asDecibel());
    SC_FIELD_ARRAY_MEMBER(items, field, N, pmd().asPan());
    SC_FIELD_COMPUTED(member, basePmd, dynamicFn);   // metadata depending on runtime state
})
```

`SC_FIELD_COMPUTED` is for parameters whose range or naming depends on the object's state —
a processor parameter whose meaning changes with the processor type, for instance.

The builder methods themselves (`asPercent`, `asEnvelopeTime`, `withUnorderedMapFormatting`,
`canTemposync`, …) belong to `sst-basic-blocks`. Those are documented in the
`sst-param-metadata` skill, which is the place to look for display scales, string
round-tripping and modulation readouts.

The UI binds widgets off this metadata, so a missing or wrong `SC_DESCRIBE` shows up as a
knob with the wrong range rather than a compile error.

---

## 4. Messages

Message structs live in `messaging/client/*_messages.h` — `ls` that directory for the
current set; the filenames say what they cover (zone, group, part, processor, mixer,
macro, browser, selection, structure, patch_io, interaction, missing_resolution,
enginestatus, debug).

### The macros

Defined in `messaging/client/client_macros.h`.

```cpp
// UI → serialization
CLIENT_TO_SERIAL(MyMessage, c2s_my_message, PayloadType, {
    /* body runs on the serialization thread; payload, engine, cont are in scope */
});

// UI → serialization, payload bound to a specific engine struct
CLIENT_TO_SERIAL_CONSTRAINED(MyMessage, c2s_my_message, PayloadType, engine::Zone::SomeStruct, {
    /* … */
});

// serialization → UI
SERIAL_TO_CLIENT(MyNotification, s2c_my_notification, PayloadType, onMyNotification);

// paired request/response
CLIENT_SERIAL_REQUEST_RESPONSE(MyThing, c2s_id, C2SPayload, s2c_id, S2CPayload,
                               executeSerialization, onClientMethod);
```

**The macro emits the `ClientToSerializationType<>` specialization itself.** You do not
hand-write dispatch registrations — older instructions saying to add one to
`client_serial_impl.h` are obsolete.

The `_CONSTRAINED` variant carries a `bound_t` naming the engine struct the payload edits.
That is what lets the generic offset-based update helpers in
`messaging/client/detail/message_helpers.h` find the field, and what the UI's begin-edit /
undo-gesture machinery keys off. **Prefer it for anything editing a field inside a known
struct** — you get undo grouping for free.

### Payloads

Anything msgpack-serializable: primitives, `std::string`, `std::vector`, `std::array`,
`std::tuple`, or a struct with an `SC_STREAMDEF`. Many messages use the shared
`detail::…DiffMsg_t` tuple shapes rather than bespoke structs — look there before
inventing one.

Payloads cross a fixed-size queue. Keep them small; do not put a whole patch in one where
an index would do.

---

## 5. Adding a parameter, end to end

1. **Field** on the engine struct (`engine/…`).
2. **Streaming** — a line in the relevant `SC_STREAMDEF`, using `findOrSet` with a default
   so old patches still load.
3. **Metadata** — an `SC_FIELD` in the struct's `SC_DESCRIBE`.
4. **Message** — usually none. If the struct already has a `CLIENT_TO_SERIAL_CONSTRAINED`
   update message covering it, the offset-based path handles a new field with no new
   message at all. Only add one for genuinely new behaviour.
5. **New message id**, if you did add one: append to the enum in `client_serial.h`, before
   the `num_…` sentinel. The ids must stay contiguous.
6. **Response handler** — see `shortcircuit-ui` §S2C.
7. **Widget** — see `shortcircuit-ui`.
8. **Test** — `tests/streaming.cpp` for round-tripping,
   `tests/message_bounds_tests.cpp` if you touched the id space.

Steps 1–3 are frequently the whole task. Reach for step 4 last.

---

## 6. Serialization loop

`MessageController::runSerialization()`:

1. Wait on the condition variable, up to ~50 ms.
2. Execute any client messages, under `modifyStructureMutex`.
3. Drain `audioToSerializationQueue` through `parseAudioMessageOnSerializationThread()` —
   structure refreshes, macro updates, processor refreshes, deferred deletes, sample purge
   requests.
4. Coalesce batched updates (macros especially) and send `s2c` to registered clients.

Audio-thread notifications are `a2s_*` ids in `messaging/audio/`. Note
`a2s_delete_this_pointer`: the audio thread cannot free, so it hands the pointer back for
the serialization thread to destroy. If you allocate something the audio thread stops
using, that is the mechanism.

---

## 7. Pitfalls

| Symptom | Cause |
|---|---|
| Compile error in a trait | Payload type has no `SC_STREAMDEF`. Add one in the matching `*_traits.h`. |
| Old patches load with zeros | Used `findIf` where the field needed `findOrSet` with a real default. |
| Preprocessor error in a message body | A `<A,B>` template argument list inside a macro body. Wrap the call in an extra set of parentheses. |
| Change applies but the UI never updates | Nothing sent `s2c`, or the handler did not fire the data-cache notification. |
| Intermittent crash on load | Sample attach done via `scheduleAudioThreadCallback` instead of `stopAudioThreadThenRunOnSerial`. |
| Structure change races | Mutated structure without `modifyStructureMutex`. |

---

## 8. Entry points

| Task | Where |
|---|---|
| Stream / unstream a patch | `json::streamEngineState`, `json::unstreamEngineState`, `json::unstreamPartState` (`json/stream.h`) |
| Message dispatch | `client::serializationThreadExecuteClientMessage()` |
| Client registration | `MessageController::registerClient()` |
| Coordinator | `MessageController` in `messaging/messaging.h` |
| Send from client | `messaging::client::clientSendToSerialization(Msg(payload), msgController)` |
| Send to client | `serializationSendToClient(s2c_id, payload, msgController)` |