AI Generator Selection
Pick the right AI building blocks for the app being built, before wiring
anything. The full per-generator table lives in
references/generator-matrix.md — read it when
you need exact platform support or generator properties.
Announce at start: "I'm using the ai-generators skill to pick the AI
components."
Decision procedure
- Name the feature, not the tech: "answers questions about the menu",
"transcribes speech live", "generates ambient visuals".
- List the target devices (ask if unknown — this decides everything):
phone/tablet, Android signage box, Apple TV, desktop, or web preview.
- Apply the offline rule. BRICKS apps are expected to keep working
offline. If the feature must work offline → local generators only. Cloud
generators (anthropicLlm / openaiLlm / openaiTts / GenerativeMedia) need a
degrade path: cached content, default media, or a hidden feature state.
- Pick per feature (details in the matrix):
- Text generation / chat →
llm (GGML, most platforms) as default
local choice; mlxLlm (iOS/tvOS 17+, also vision-language);
neuropilotLlm (MediaTek Genio) / qnnLlm (Qualcomm) when the device
has that silicon; appleLlm on iOS/tvOS 26+ (zero model download);
onnxLlm for transformers.js ONNX models; anthropicLlm / openaiLlm
for cloud or LAN-hosted OpenAI-compatible servers (ollama, llama.cpp
server).
- Speech-to-text →
speechInference (Whisper GGML) for utterances;
realtimeTranscription for continuous live captions (bundles VAD);
platformStt / appleStt when native platform recognition is enough
and its platform limits are acceptable.
- Text-to-speech →
tts (ONNX — the only local TTS that also works
on web) or ggmlTts (native GGML); appleTts for zero-setup Apple
voices; openaiTts cloud.
- Wake/turn detection →
vadInference.
- Semantic search / RAG →
vectorStore (+ reranker for quality).
- AI images/video on screen → the
GenerativeMedia brick (cloud
providers; always configure the offline fallbacks).
- One assistant vs hand-wiring: if the feature is a conversational agent
(voice or chat) that may also need tools or file search, prefer the single
assistant generator — it orchestrates LLM + STT + TTS + MCP tools + file
search with auto-summary, and its live policies release contexts when idle.
Hand-wire individual generators when you need a custom pipeline shape
(e.g. captions only, or search without chat).
- State the choice to the user in one short block: feature → generator(s)
→ why (platform + offline + quality), then continue with
ai-model-selection for the actual model and ai-recipes for wiring.
Cloud vs local — the four questions
| Question |
Points to local |
Points to cloud |
| Must it work offline? |
yes → local, hard requirement |
no |
| Is the content sensitive (customers, cameras, mics)? |
yes → on-device |
no |
| Fleet size × usage cost? |
many devices, constant use → local is free at runtime |
few devices, bursty |
| Quality bar? |
small models suffice (menus, FAQs, captions) |
needs frontier quality |
A LAN-hosted OpenAI-compatible server (ollama / llama.cpp server via
openaiLlm with a custom base URL) is the middle path: local-network privacy
and no per-token cost, with bigger models than the device could run.
Buttress (BRICKS remote inference) is the other middle path: the llm
(GGML), mlxLlm, and speechInference generators can offload transparently
to a workspace-bound GPU server on the LAN — same generator, same events, and
a use-local fallback that keeps the offline story intact when the server is
unreachable. Prefer it over the openaiLlm route when the app should keep its
on-device pipeline shape. Setup (server install, workspace binding, the
buttressConnectionSettings property) is covered by the built-in bricks-ctor
skill's references/buttress.md — read that before wiring Buttress.
Hard platform limits to check early
llm (GGML) wants >8 GB device RAM — most signage boxes fail this;
use small models on NPU paths (neuropilotLlm/qnnLlm), onnxLlm,
Buttress offload, or a LAN server instead.- TTS (GGML) does not work on web (no vocoder) — use
tts (ONNX) there.
- Web preview runs single-threaded — treat it as a functional check, not
a performance test.
- tvOS:
appleStt is unavailable; Scene3D is experimental; check the matrix
row before promising a feature.
When Not To Use
- Choosing or configuring the CTOR editor's own chat model — that's app
settings, not the app being built.
- The model/hardware sizing question ("which GGUF fits this box") — use
ai-model-selection.
- The wiring itself (events, state, verification) — use
ai-recipes.
1---2name: ai-generators3description: Use before adding any AI feature to a BRICKS app — chat, Q&A, voice interface, transcription, semantic search / RAG, or AI-generated media. Picks the right generator for the feature and target platform from the built-in catalog - local LLMs (llm/GGML, mlxLlm, neuropilotLlm, qnnLlm, onnxLlm, appleLlm), cloud LLMs (anthropicLlm, openaiLlm and compatible endpoints), speech-to-text (speechInference, realtimeTranscription, platformStt, appleStt, onnxStt), text-to-speech (ggmlTts, tts/ONNX, openaiTts, appleTts), vadInference, reranker, vectorStore, the assistant orchestrator, and the GenerativeMedia brick. Weighs offline requirements, privacy, latency, and per-device cost before wiring anything. Triggers on "add AI", "add a chatbot / voice assistant", "which generator should I use", "make it answer questions about…". Do NOT use for choosing the CTOR editor's own chat model (that is app settings, not the app being built).4license: MIT5---67# AI Generator Selection89Pick the right AI building blocks for the app being built, before wiring10anything. The full per-generator table lives in11[references/generator-matrix.md](references/generator-matrix.md) — read it when12you need exact platform support or generator properties.1314**Announce at start:** "I'm using the ai-generators skill to pick the AI15components."1617## Decision procedure18191. **Name the feature**, not the tech: "answers questions about the menu",20 "transcribes speech live", "generates ambient visuals".212. **List the target devices** (ask if unknown — this decides everything):22 phone/tablet, Android signage box, Apple TV, desktop, or web preview.233. **Apply the offline rule.** BRICKS apps are expected to keep working24 offline. If the feature must work offline → local generators only. Cloud25 generators (anthropicLlm / openaiLlm / openaiTts / GenerativeMedia) need a26 degrade path: cached content, default media, or a hidden feature state.274. **Pick per feature** (details in the matrix):28 - **Text generation / chat** → `llm` (GGML, most platforms) as default29 local choice; `mlxLlm` (iOS/tvOS 17+, also vision-language);30 `neuropilotLlm` (MediaTek Genio) / `qnnLlm` (Qualcomm) when the device31 has that silicon; `appleLlm` on iOS/tvOS 26+ (zero model download);32 `onnxLlm` for transformers.js ONNX models; `anthropicLlm` / `openaiLlm`33 for cloud or LAN-hosted OpenAI-compatible servers (ollama, llama.cpp34 server).35 - **Speech-to-text** → `speechInference` (Whisper GGML) for utterances;36 `realtimeTranscription` for continuous live captions (bundles VAD);37 `platformStt` / `appleStt` when native platform recognition is enough38 and its platform limits are acceptable.39 - **Text-to-speech** → `tts` (ONNX — the only local TTS that also works40 on web) or `ggmlTts` (native GGML); `appleTts` for zero-setup Apple41 voices; `openaiTts` cloud.42 - **Wake/turn detection** → `vadInference`.43 - **Semantic search / RAG** → `vectorStore` (+ `reranker` for quality).44 - **AI images/video on screen** → the `GenerativeMedia` brick (cloud45 providers; always configure the offline fallbacks).465. **One assistant vs hand-wiring:** if the feature is a conversational agent47 (voice or chat) that may also need tools or file search, prefer the single48 `assistant` generator — it orchestrates LLM + STT + TTS + MCP tools + file49 search with auto-summary, and its live policies release contexts when idle.50 Hand-wire individual generators when you need a custom pipeline shape51 (e.g. captions only, or search without chat).526. **State the choice** to the user in one short block: feature → generator(s)53 → why (platform + offline + quality), then continue with54 `ai-model-selection` for the actual model and `ai-recipes` for wiring.5556## Cloud vs local — the four questions5758| Question | Points to local | Points to cloud |59|---|---|---|60| Must it work offline? | yes → local, hard requirement | no |61| Is the content sensitive (customers, cameras, mics)? | yes → on-device | no |62| Fleet size × usage cost? | many devices, constant use → local is free at runtime | few devices, bursty |63| Quality bar? | small models suffice (menus, FAQs, captions) | needs frontier quality |6465A LAN-hosted OpenAI-compatible server (ollama / llama.cpp server via66`openaiLlm` with a custom base URL) is the middle path: local-network privacy67and no per-token cost, with bigger models than the device could run.6869**Buttress (BRICKS remote inference)** is the other middle path: the `llm`70(GGML), `mlxLlm`, and `speechInference` generators can offload transparently71to a workspace-bound GPU server on the LAN — same generator, same events, and72a `use-local` fallback that keeps the offline story intact when the server is73unreachable. Prefer it over the `openaiLlm` route when the app should keep its74on-device pipeline shape. Setup (server install, workspace binding, the75`buttressConnectionSettings` property) is covered by the built-in `bricks-ctor`76skill's `references/buttress.md` — read that before wiring Buttress.7778## Hard platform limits to check early7980- `llm` (GGML) wants **>8 GB device RAM** — most signage boxes fail this;81 use small models on NPU paths (`neuropilotLlm`/`qnnLlm`), `onnxLlm`,82 Buttress offload, or a LAN server instead.83- **TTS (GGML) does not work on web** (no vocoder) — use `tts` (ONNX) there.84- **Web preview runs single-threaded** — treat it as a functional check, not85 a performance test.86- tvOS: `appleStt` is unavailable; Scene3D is experimental; check the matrix87 row before promising a feature.8889## When Not To Use9091- Choosing or configuring the CTOR editor's own chat model — that's app92 settings, not the app being built.93- The model/hardware sizing question ("which GGUF fits this box") — use94 `ai-model-selection`.95- The wiring itself (events, state, verification) — use `ai-recipes`.
mybigday