# Pipelines As Code > Use this skill when working with Materialize's alloy log-processing or metric-processing pipelines — anything under `packages/alloy-pipelines/`. Triggers include: building, modifying, reviewing, or rendering pipelines; working with the embedded JSONSchemas under `packages/mzmon-lib/schemas/alloy/`; deciding between typed blocks and the `raw:` escape hatch; extending the schemas to add a new component, stage, or attribute; debugging `mz-monitoring-build gen-pipelines` output; writing Materialize-specific log-processing patterns (level normalization, structured metadata, label families, drop/limit conventions); writing or reviewing alloy stages, loki.process pipelines, discovery.kubernetes, loki.relabel rules, or sources like loki.source.file / loki.source.journal. Also use it whenever someone refers to "log processing", "metric processing", or anything alloy-shaped, even if they don't use the word "pipeline". - Skill: `materializeinc/pipelines-as-code` (Agent Skill) - Install (CLI): `npx skillmds@latest add materializeinc/pipelines-as-code` - Raw SKILL.md: https://api.skillmd.com/api/skills/materializeinc/pipelines-as-code/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Product & Planning - Author: materializeinc (https://skillmd.com/u/materializeinc) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/materializeinc/pipelines-as-code --- # Pipelines as Code This skill is the entry point for the Materialize pipelines-as-code project. **Stable conventions live in the repo docsite** under [`docs/content/reference/internal/pipelines/`](../../../docs/content/reference/internal/pipelines/) — this file is intentionally slim and links into the docsite at heading-level granularity. The non-link content below is the **state snapshot**: what currently exists, what's in flight, and what's queued. ## Audience reminder The **pipelines themselves** target alloy (the binary), so authoring decisions favor *contributors* rather than end users. There is no panel-description-voice equivalent — readability of the YAML and the rendered `.alloy` is the goal. The **docsite reference pages** target repo contributors (SRE, Field Engineering, CloudOps, Database Engineers) and AI agents reading this skill. ## Where to find what | Looking for… | Read | |---|---| | Pipeline model, strict-attributes policy, `raw:` escape rules, how to extend the schema | [Authoring](../../../docs/content/reference/internal/pipelines/authoring.md) | | Log pipeline conventions, label families, retention | [Logging](../../../docs/content/reference/internal/pipelines/logging.md) (stub) | | Metrics pipeline conventions | [Metrics](../../../docs/content/reference/internal/pipelines/metrics.md) (stub) | Frequently needed deep links into Authoring: - [The strict-attributes / raw-escape policy](../../../docs/content/reference/internal/pipelines/authoring.md#the-strict-attributes--raw-escape-policy) — what to do when an undocumented attribute is rejected - [Schema layout](../../../docs/content/reference/internal/pipelines/authoring.md#schema-layout) — file/`$id` map and how cross-file `$ref`s resolve - [Adding an attribute](../../../docs/content/reference/internal/pipelines/authoring.md#adding-an-attribute-to-an-existing-component) — minimal extension - [Adding a typed sub-block](../../../docs/content/reference/internal/pipelines/authoring.md#adding-a-typed-sub-block) — the `selectors`-style pattern - [Adding a typed component](../../../docs/content/reference/internal/pipelines/authoring.md#adding-a-typed-component) — Rust struct + schema + ComponentBlock variant - [Reference-valued attributes](../../../docs/content/reference/internal/pipelines/authoring.md#reference-valued-attributes) — why `forward_to` / `targets` need `Expression::ref_name`, not `String` - [Load-bearing invariants](../../../docs/content/reference/internal/pipelines/authoring.md#load-bearing-invariants) — AttributeValue order, alignment quirk, schemas-as-docs - [Sub-block recursion and the `raw` escape](../../../docs/content/reference/internal/pipelines/authoring.md#sub-block-recursion-and-the-raw-escape) - [How validation interacts with rendering](../../../docs/content/reference/internal/pipelines/authoring.md#how-validation-interacts-with-rendering) — YAML → `serde_json::Value` → schema validation → typed `Pipeline` → `config.alloy` ## Schema and code map ``` packages/alloy-pipelines/ ← YAML inputs ├── gateway.yaml – gateway processing pipeline ├── gateway-dest-stub.yaml – gateway default egress tail (swappable) └── agent.yaml packages/mzmon-lib/schemas/alloy/ ← validation schemas (embedded into the binary) ├── mzmon-alloy.schema.yaml – entry ├── top.schema.yaml – description / logging / livedebugging / blocks ├── loki.schema.yaml – loki.*, stage.* $defs, shared `rule` $def ├── discovery.schema.yaml – discovery.*, cross-ref'ing `rule` from loki ├── prometheus.schema.yaml – prometheus.*, cross-ref'ing `rule` from loki + `target` from discovery ├── otelcol.schema.yaml – otelcol.processor.{batch,memory_limiter,attributes,groupbyattrs,filter,transform} + output/action/conditions/statements sub-blocks └── common/ – shared fragments. $id tail MUST match the file path ├── raw.schema.yaml – {raw: } escape hatch + block primitives ├── attribute.schema.yaml – attributeValue (anyOf: literal | expression | …) └── expression.schema.yaml – sys.env / function / operator / ref expression (each fragment's `$id` + the `ID_*` const in validate.rs + the relative `$ref`s must agree) packages/mzmon-lib/src/alloy/ ← AST, render, validate, pipeline (Rust) ├── ast.rs – Block / AttributeValue / Expression / Expressable ├── render.rs – write_to + alloy-fmt-canonical formatting ├── validate.rs – embedded schemas + jsonschema validator + hints ├── pipeline.rs – Pipeline::from_yaml_str (YAML → Value → validate → typed) ├── test_support.rs – assert_renders (oracle: pipes through `alloy fmt`) └── components/ ← typed sugar; tests colocated with impl ├── top.rs – LoggingBlock, LiveDebuggingBlock ├── capsule.rs – LogsReceiver, MetricsReceiver, OtelcolConsumer, RelabelRules, TargetEntry (+ string_map, logs_receiver_list, metrics_receiver_list, otelcol_consumer_list, target_list helpers) ├── loki.rs – LokiEchoBlock, LokiSourceJournalBlock, ... ├── relabel.rs – RelabelRule + RelabelSubBlock (shared by *.relabel) ├── discovery.rs – DiscoveryKubernetesBlock, DiscoveryRelabelBlock, │ KubernetesSubBlock variants ├── prometheus.rs – Prometheus{Echo,Relabel,Scrape,ReceiveHttp,RemoteWrite, OperatorPodMonitors,OperatorServiceMonitors}Block (+ endpoint/http/clustering/basic_auth/tls_config/ selector/scrape sub-blocks) └── otelcol.rs – OtelcolProcessor{Batch,MemoryLimiter,Attributes,GroupByAttrs,Filter,Transform}Block (+ shared output block, attributes action block, Ottl type, Processor/Attributes/Filter/Transform sub-block enums) packages/mz-monitoring-build/ ← CLI: `mz-monitoring-build gen-pipelines` ``` Build/test: ``` make pipelines # render YAML → .alloy + run `alloy validate` per target cargo test -p mzmon-lib # unit + oracle tests (uses `alloy fmt` if present) cargo clippy -p mzmon-lib --all-targets # lints cargo fmt # the team runs this in pre-commit; run it before # committing any Rust changes ``` --- # Current Pipeline State This section captures the live state so the next session has something concrete to start from. **Update it when state changes meaningfully** (new pipeline, new typed component, schema gap closed, etc). ## Pipeline inventory | YAML | Rendered to | Status | |---|---|---| | `packages/alloy-pipelines/gateway.yaml` | `charts/.../pipelines/gateway.alloy` | **implemented** — carries logs AND metrics. Logs: port of the reference `staging-gateway.alloy` `inputProcessor` (level/timestamp/per-service field normalization, per-level rate limits, final label shaping) + ingress (`loki.source.api`, `loki.source.kubernetes_events`, `otelcol.receiver.otlp`→`otelcol.exporter.loki` bridge — all typed), sampled debug tap. Metrics: processed primarily in **otelcol**, converted to Prometheus only at the write. `prometheus.receive_http` + `prometheus.operator.{podmonitors,servicemonitors}` (clustering on, env-driven scrape defaults) → `otelcol.receiver.prometheus "inputBridge"` (prom→OTLP; the OTLP receiver's metrics join here too) → `otelcol.processor.filter "inputMetricProcessor"` (rule-free passthrough today; the metric filter/transform tiers land here) → `otelcol.processor.memory_limiter "outputMemoryLimiter"` (refuse at 75%) → `otelcol.processor.batch "outputBatch"` → `otelcol.processor.filter "egress"` seam → `otelcol.exporter.prometheus "outputBridge"` (OTLP→prom, `add_metric_suffixes=false`) → `prometheus.relabel "egress"` → `prometheus.remote_write`. NB: the metrics filter is `inputMetricProcessor`; `inputProcessor` is the *logs* `loki.process`. Both log and metric egress seams supplied downstream (see below) | | `packages/alloy-pipelines/gateway-dest-stub.yaml` | `charts/.../pipelines/gateway-dest-stub.alloy` | **build-time egress stand-in ONLY** — the egress seams (`loki.process "egress"` for logs, `otelcol.processor.filter "egress"` for metrics, plus the post-bridge `prometheus.relabel "egress"`) + default `loki.write`/`prometheus.remote_write "destination"`. NOT what deploys: at install the destination is rendered by `charts/.../templates/_alloy_helpers.tpl` (Helm, from `.Values.pipeline.{logging,metrics}.gateway.destination.*`), which **skips the JSONSchema** — only the pre-validate jobs (`alloy validate`) check it. The stub exists so `make pipelines` can `alloy validate` `gateway.alloy` jointly (it dangles the `egress` refs alone). Keep it in step with the helper, but the helper is source of truth. Metrics helper auth: `none`/`basicAuth`/`bearer`/`sigv4` (sigv4 = AMP + IRSA) | | `packages/alloy-pipelines/agent.yaml` | `charts/.../pipelines/agent.alloy` | **implemented & largely typed** — staging-agent parity (journal + node-local pod logs → gateway, sampled debug tap), plus **node-local cAdvisor metrics** on a separate OTLP egress: `prometheus.exporter.cadvisor "local"` → typed `prometheus.scrape "cadvisor"` → typed `prometheus.relabel "cadvisor"` (maps `container_label_io_kubernetes_*` onto `namespace`/`pod`/`container`, adds `node`, drops the pod sandbox) → `otelcol.receiver.prometheus "cadvisorBridge"` → `otelcol.exporter.otlp "gateway"` (`AGENT_OTLP_DEST`, default `alloy-gateway..svc:4317`). Logs still leave via `loki.write`; folding them into the OTLP exporter is tracked separately | Both pass `alloy validate`. `agent.yaml` is a faithful port of the reference `staging-agent.alloy` (`packages/ref-alloy-pipelines/`, not checked in). **There are ZERO `raw:` blocks left in any pipeline.** Every component in `agent.yaml`, `gateway.yaml`, `gateway-metrics.yaml`, and `gateway-dest-stub.yaml` is typed and schema-validated. `raw:` remains in the schemas as the escape hatch — the last `oneOf` branch of every sub-block list, and available for any component — but nothing currently uses it. Adding an untyped component should be a deliberate, temporary choice, not the default. The conversion was verified by rendering before and after: `agent.alloy`, `gateway-metrics.alloy`, and `gateway-dest-stub.alloy` came out **byte-identical**, and `gateway.alloy` differs only in `logs`/`metrics` ordering inside one `output` block (the typed struct's canonical order vs. YAML order — semantically irrelevant to alloy, see the attribute-order note below). Both configs were also load-tested under a real `alloy run`, since `alloy validate` cannot see capsule/type mismatches. **Agent log sidechannel** (`gateway.yaml`): the gateway tails the agent DaemonSet's own logs from the Kubernetes API — `discovery.kubernetes "agent_pods"` (typed, server-side `selectors.label = app.kubernetes.io/name=alloy-agent`) → `discovery.relabel "agent_pods"` (typed) → `loki.source.kubernetes "agent_logs"` → `loki.process.inputProcessor`. Points that are load-bearing rather than stylistic: - **The agent no longer tails its own logs** — `discovery.relabel.local_pods` in `agent.yaml` drops targets whose `app.kubernetes.io/name` is `alloy-agent`. The sidechannel is always on, so self-collection would deliver every line twice. The two changes are a pair. - **`clustering { enabled = true }` on the tail is required, not an optimization.** The gateway runs 2+ replicas and each would otherwise tail every agent pod. - **RBAC is already covered** by the alloy subchart's default `rbac.rules` (`pods`, `pods/log`, `namespaces`), which the chart does not override. - **The `{app="alloy"}` selector in `inputProcessor` was dead.** `app` comes from `app.kubernetes.io/name`, which the subchart aliases make `alloy-agent`/`alloy-gateway`, so the reference deployment's bare `alloy` never matched — meaning alloy's own logs were never logfmt-parsed and the `loki.echo` drop never ran. Now `{app=~"alloy(-agent|-gateway)?"}`. That drop is what stops the echo tap's re-emitted pod logs being ingested a second time via the sidechannel. - Verified end-to-end against a real API server (kind), not just `alloy validate`: discovery → relabel → tail delivered live log lines, confirming `targets = discovery.relabel.X.output` is the right capsule shape for `loki.source.kubernetes`. **Container metrics come from the kubelet, not from a cAdvisor we run.** `gateway.yaml`: `discovery.kubernetes` role=node → `discovery.relabel` (node-label allowlist) → `prometheus.scrape "kubelet_cadvisor"` (https, `/metrics/cadvisor`, SA bearer token, clustered) → the otelcol metrics pipeline. Everything below was measured on a live GKE cluster, not reasoned about: - **An in-agent `prometheus.exporter.cadvisor` cost ~750Mi per node** against a 200Mi logs-only envelope, and OOM-crashlooped the DaemonSet (180-224 restarts). A heap profile attributed ~34% to cAdvisor housekeeping and ~31% to the scrape loop reading it. The kubelet already computes those stats for eviction and the metrics API, so running our own computed them twice per node. - **The completeness argument for running our own does not survive measurement.** A GKE kubelet serves **69** distinct `container_*` metrics vs **70** from an in-process cAdvisor; the only real loss is `container_health_state`, which nothing queries. All 23 `container_*` metrics this repo's queries reference are present. Where GKE dashboards look sparse it is the *collector* scraping a subset, not the kubelet withholding data. - **The kubelet serves the `process`-collector family too** (`container_file_descriptors`, `container_sockets`, `container_threads`, `container_ulimits_soft`, `container_processes`). Those are why the agent briefly ran `hostPID: true`; that privilege — and the `/proc//environ` exposure it grants alongside uid 0 — is no longer needed. - **The kubelet labels series `namespace` / `pod` / `container` directly**, so none of the `container_label_io_kubernetes_*` remapping an in-process cAdvisor needs applies. - **TLS: the in-cluster CA verifies the kubelet certificate on GKE** (measured), so `ca_file` + the SA token is enough and `insecure_skip_verify` stays false. Other distributions may need it true, and the failure is silent — hence `GATEWAY_KUBELET_TLS_INSECURE`. - **`clustering { enabled = true }` on the scrape is required**, not an optimization: the gateway runs many replicas and each would otherwise scrape every node (~6.7k series/node). - **RBAC needs nothing added** — the alloy subchart's default `clusterRules` already grant `nodes`, `nodes/metrics`, `nodes/proxy`. - **`GOMEMLIMIT` is set on both roles** (~80% of the memory limit). Go's GC has no knowledge of a cgroup limit, so it grows the heap toward a ceiling the kernel enforces by killing the process; telling the runtime about it converts an OOM-kill into GC pressure. Keep it in step with the limit whenever either moves. **Top-level loki.* components**: `loki.echo`, `loki.process`, `loki.relabel`, `loki.source.journal`, `loki.source.file` (incl. the `file_match` sub-block), `loki.source.api` (+`http` server), `loki.source.kubernetes` (+`clustering`), `loki.source.kubernetes_events`, `loki.write` (+`endpoint`). **Top-level discovery.* components**: `discovery.kubernetes`, `discovery.relabel`. **Top-level prometheus.* components**: `prometheus.echo`, `prometheus.relabel`, `prometheus.scrape`, `prometheus.receive_http`, `prometheus.remote_write`, `prometheus.operator.podmonitors`, `prometheus.operator.servicemonitors`. Scoped to the stable, in-cluster surface — see [Metrics](../../../docs/content/reference/internal/pipelines/metrics.md) for the per-component `raw:`-deferred list (remote_write endpoint auth/TLS/queue tuning, operator `client`, scrape oauth2/authorization, selector `match_expression`, receive_http `tls`). **`prometheus.*` sub-blocks**: `clustering` (`enabled`, shared by scrape + operators), `basic_auth`/`tls_config` (on scrape), `http` (receive_http server), `endpoint` (remote_write — `url` + scalars + a raw-only nested `blocks` for auth/queue), `selector` (operator — `match_labels` + raw-only nested `blocks` for `match_expression`), `scrape` (operator defaults), and the shared `rule` (cross-file `$ref` to `loki.schema.yaml#/$defs/rule`, reusing the `RelabelRule` sugar). `prometheus.scrape`'s `scrape_interval`/`scrape_timeout` are `Expressable` via the existing `expressionOrDuration` `$def` (the same one `operatorScrape` uses), so the interval can be env-driven — it is the dominant cost lever on a per-node cAdvisor. That widened the struct enough that `ComponentBlock::PrometheusScrape` is now **`Box`ed** for `clippy::large_enum_variant`, like `memory_limiter`. **Top-level `prometheus.exporter.*` components**: `prometheus.exporter.cadvisor` (the only one so far). Its collector allowlists are [`ExpressableList`](#expressablelist) rather than `Expressable`, and its `to_block` emits attributes in *declaration* order rather than grouped by type, so `store_container_labels` renders next to the allowlist it gates. **Top-level otelcol.* components**: `otelcol.processor.{batch,memory_limiter,attributes,groupbyattrs,filter,transform}`. `filter` is now wired into `gateway.yaml` (the metrics pipeline is otelcol-primary — see the inventory row above); the rest are schema+sugar only so far. `memory_limiter`'s limit knobs (`check_interval`/`limit`/`spike_limit`/`limit_percentage`/`spike_limit_percentage`) are all `Expressable` for env-driven sizing (its `ComponentBlock` variant is `Box`ed — five `Expressable` fields made it the outsized variant, `clippy::large_enum_variant`). Receivers and exporters are now typed too: `otelcol.receiver.{otlp,prometheus}` and `otelcol.exporter.{otlp,loki,prometheus}`. Sub-blocks: `grpc`/`http` (one shared `server` struct — the *variant* names the block, so its `ToBlock` is hand-written rather than the dispatch macro; an empty `grpc: {}` is meaningful, since presence is what enables the listener), `client` + nested `tls` on the OTLP exporter. Still deferred to `raw:`: connectors, the per-processor `debug_metrics` block, exporter `sending_queue`/`retry_on_failure`, and (on filter) the deprecated `traces`/`metrics`/`logs` blocks. **OTTL** (`filter`/`transform`) is carried by a dedicated `Ottl` newtype (`ast.rs`, a sealed `LiteralScalar` so fields are `Vec>` — literal OTTL string OR an expression). We do NOT model OTTL's grammar/function library; statements render verbatim as normal (escaped) alloy strings. filter's `*_conditions` blocks and transform's `*_statements`/`statements` blocks share one Rust struct each (`FilterConditionsBlock`, `TransformStatementsBlock`), so their `ToBlock` is hand-written (not the macro) to pass each block name. **GOTCHA — otelcol needs alloy >= 1.17 and a label on every component.** Confirmed by a real `alloy validate` (the schema/renderer agreeing is NOT proof — these aren't wired into a pipeline, so `make pipelines` doesn't validate them): (1) filter's `*_conditions` blocks are 1.17+ only — older alloy has just the deprecated `traces`/`metrics`/`logs` and errors "unrecognized block name"; we pin 1.17.1. (2) EVERY otelcol component must have a label or alloy load fails "must have a label" — so `label` is a **required** field on the otelcol structs and schema (`required: [label]`), deliberately diverging from the loki/prometheus families which leave `label` optional. **`otelcol.*` sub-blocks**: `output` (shared — `metrics`/`logs`/`traces` lists of the `OtelcolConsumer` capsule; bare refs), `action` (attributes), `*_conditions` (filter: `context` + `conditions`), `*_statements` (transform: `context`/`statements`/`conditions`/`error_mode`), `statements` (transform inferred: `trace`/`metric`/`log`). Sub-block enums: `ProcessorSubBlock`, `AttributesSubBlock`, `FilterSubBlock`, `TransformSubBlock`. **`loki.process` stages** (and `stage.match` body, recursively): `stage.match`, `stage.drop`, `stage.limit`, `stage.regex`, `stage.replace`, `stage.template`, `stage.logfmt`, `stage.json`, `stage.timestamp`, `stage.labels`, `stage.static_labels`, `stage.label_drop`, `stage.structured_metadata`, `stage.structured_metadata_drop`, `stage.sampling`, `stage.cri` (empty, no attributes), `stage.tenant` (`label`/`source`/`value`, each `Expressable`). **`discovery.kubernetes` sub-blocks**: `selectors` (incl. `field`/`label` as `Expressable`), `attach_metadata` (`node` + `namespace`). Other sub-blocks (e.g. `namespaces`) use `raw:`. **`*.relabel` sub-blocks**: `rule` (shared `$def` used by `loki.relabel`, `discovery.relabel`, and `prometheus.relabel` via cross-file `$ref` to `loki.schema.yaml#/$defs/ruleBlock`; the operator components' `rule` refs `#/$defs/rule` directly). **Literal-or-expression fields** use `Expressable` (`ast.rs`): a field typed `Expressable` (or `Option<…>`) accepts either a scalar literal or an inline expression object (`{env}`, `{function}`, `{operator}`, `{ref}`). In use on `stage.limit` rate/burst, `stage.drop` older_than, `stage.static_labels` values, `selectors` field/label, and the operator `scrape` block's `default_scrape_interval`/`default_scrape_timeout`. Schema side: `anyOf: [{type: }, {$ref: common/expression.schema.yaml}]` (safe — scalar vs object are disjoint). *Scalars only* — never `Expressable` (a literal map would collide with the expression object, the same overlap that forced `anyOf` in the raw `attributeValue`). This is an actively-expanding pattern; adopt per-field as needed. **`ExpressableList`** (`ast.rs`) is the list analog of `Expressable`, which stays sealed to scalars. The sealing exists because a literal *map* is shape-indistinguishable from the expression object; a literal *list* is not (array vs object are disjoint), so the untagged dispatch is unambiguous with `Literal` first. Used by cAdvisor's collector allowlists, which are wired to env vars via `encoding.from_json`. Fields widened to accept expressions during the raw-block conversion: `RelabelRule.replacement` (stamping `node` from `HOSTNAME`), `remoteWriteEndpoint.url` and `lokiWriteEndpoint.url` (env-coalesced destinations), `prometheus.scrape.scrape_interval`/`scrape_timeout`, `loki.source.api`'s `listen_port`, and the otelcol `client.endpoint` / `server.endpoint`. **Every sub-block `oneOf` ends with a `raw:` branch** — the escape hatch is non-negotiable in the design. Only the attributes we routinely use are documented per component — not exhaustive vs. alloy upstream. **Strict `additionalProperties: false`** is enforced; undocumented attributes are rejected at validation time with a hint pointing at the `raw:` escape vs. schema-extension choice. See [Authoring §The strict-attributes / raw-escape policy](../../../docs/content/reference/internal/pipelines/authoring.md#the-strict-attributes--raw-escape-policy). ## Rust sugar deserialization status The `ComponentBlock` enum in `pipeline.rs` dispatches to typed sugar structs (via `#[serde(rename = "loki.echo")]` etc.). Each typed component has a `pub struct …Block { fields }` + an `impl ToBlock` that normalizes to the generic `Block` AST for rendering. Sub-blocks follow the same pattern via per-component sub-block enums (`KubernetesSubBlock`, `RelabelSubBlock`). The `ToBlock` enum impls are generated by the `impl_to_block_dispatch!` macro (`ast.rs`, `pub(crate)`, `$crate::` paths) — list every variant in the invocation; a missed one is a compile error. **Done: every component in the typed-schema coverage list above round-trips through the typed path.** No pending sugar work. **Capsule types** (`components/capsule.rs`) make the bare-ref invariant structural: `LogsReceiver` (`forward_to`), `MetricsReceiver` (`forward_to` on prometheus.* — the metrics analog of `LogsReceiver`, list-wrapped via `metrics_receiver_list`), `RelabelRules` (`relabel_rules`), and `TargetEntry` (`targets`, untagged `Ref(Identifier) | Literal(IndexMap)`; `prometheus.scrape` reuses it). **`targets` refs are list-valued and must NOT be bare-array-wrapped.** A `discovery.*` export (`discovery.kubernetes.pods.targets`, `discovery.relabel.x.output`) is a `list(discovery.Target)`; a literal is a single `discovery.Target`. `targets = [ ]` is `list(list(Target))` and alloy **rejects it at load** (`conversion from '[]discovery.Target' is not supported`) — even though `alloy validate` *accepts* it. So `target_list` (capsule.rs) emits: a single ref directly (`targets = discovery.x.targets`), literals-only as an array (`[{…}]`), and multiple/mixed via `array.concat(ref, …, [{…}])`. (Earlier this list-wrapped and "verified against `alloy validate`" — a false green that broke a live agent; see the `alloy validate` gotcha below.) `forward_to`/`relabel_rules` are NOT affected — those refs are single capsules, so list-wrapping (`forward_to`) or direct assignment (`relabel_rules`) is correct. Schema side: `$defs/target` (discovery.schema.yaml, generic) vs `$defs/fileTargetEntry` (loki.schema.yaml, `required: [__path__]`) — strictness lives in the schema, the Rust type stays generic. A new capsule type is a newtype + `From<&T> for AttributeValue` via `Expression::name_to_ref` + a schema `$def` — `OtelcolConsumer` (the otelcol `output` block's `metrics`/`logs`/`traces`, list-wrapped via `otelcol_consumer_list`) is the worked example. ## Load-bearing invariants These are *non-obvious things that must stay true* — flagged here so reorders or refactors don't silently break them. Detail in [Authoring §Load-bearing invariants](../../../docs/content/reference/internal/pipelines/authoring.md#load-bearing-invariants). - **`AttributeValue` variant order**: `Null`/`Bool`/`Number`/`String`/`Array` *must* come before `Expression`. Serde's untagged struct deserializer accepts a sequence by positional-field assignment, so `["a", "b"]` would misroute to `Expression { raw: Some("a"), env: Some("b") }`. Regression-tested in `ast.rs`. - **`#[serde(deny_unknown_fields)]` on `Expression`**: keeps generic objects (`{mapping: ...}`) from silently matching `Expression` with all heads `None`. Also regression-tested. - **Ref-valued attributes render as bare refs, never quoted strings** — and this is now *enforced by type*: declare the field with a capsule type (`Vec`, `Option`, `Vec`) and convert via the capsule helpers / `Expression::name_to_ref`. Do NOT hand-build `Expression { ref_name: ... }` in new `to_block` impls; if a new capsule kind is needed, add it to `components/capsule.rs`. - **`raw:` escape is always the last `oneOf` branch** in every sub-block list. Adding a new typed branch goes *before* `raw`. - **A nested block list is `Vec` (or a typed sub-block enum), NEVER `Vec`.** The schemas express a nested list as `$ref: common/raw.schema.yaml` — the externally-tagged `{raw: {component, …}}` wrapper — while `Vec` deserializes the *unwrapped* `{component, …}`. Schema-valid YAML then passes validation and dies in serde with `missing field 'component'`, so the escape hatch reads as documented and is unusable. Four fields shipped that way in the raw-block conversion, because no pipeline exercises those escapes and nothing else notices. `RawOnlySubBlock` lives in `ast.rs`; `pipeline.rs::raw_escape_round_trips_in_every_raw_only_list` covers every such list and fails with that exact error if one regresses. - **A schema `anyOf`/`$ref` that admits an expression requires the matching Rust field to be `Expressable`.** The two drift independently and the mismatch is silent in both directions: an expression-capable schema over a plain `String` field accepts YAML that then fails serde, and the reverse quietly rejects valid config. Widen (or narrow) both together. - **`raw.schema.yaml`'s `attributeValue` uses `anyOf`, NOT `oneOf`**: an expression-shaped object (`{ref}`, `{env}`, `{operator}`, `{function}`) is *also* a valid generic `attributeObject`, and a `{value: ...}` is also a `commentedValue` — so exactly-one `oneOf` rejected every expression/commented raw value (this blocked the first real expression-valued raw block in `agent.yaml`). The Rust `AttributeValue` deserializer disambiguates by variant order + `deny_unknown_fields`; the schema just needs "is *some* legal raw value," which `anyOf` gives. Don't revert it to `oneOf`. Regression-tested in `pipeline.rs::raw_block_with_expression_values_round_trips`. - **`Error::Multiple` renders its children** (header + indented bullet per child, recursive). Earlier it displayed a bare "multiple errors" and swallowed the detail — `gen-pipelines` failures were undiagnosable. Tested in `error.rs`. - **A schema file's `$id` tail MUST match its path** (and the `ID_*` key in `validate.rs`). The Rust `Registry` registers each resource under an explicit key so it tolerates a mismatch, but `yaml-language-server` keys on the in-file `$id` and breaks (refs point at a URI no schema claims). Keep filename ↔ `$id` ↔ `ID_*` in lockstep. - **Known tooling bug — yaml-language-server `Maximum call stack size exceeded`**: fires on our (necessarily) recursive schemas during the LSP's meta-schema validation; it's upstream (fixed by redhat-developer/yaml-language-server#1269), NOT our content. The CLI validator is the source of truth — don't flatten the recursion to appease the editor. Full writeup + debugging recipe in [authoring.md](../../../docs/content/reference/internal/pipelines/authoring.md#known-tooling-issue-yaml-language-server-maximum-call-stack-size-exceeded). - **Schemas are the reference docs**: descriptions render in IDE hover and (eventually) in published reference; keep them user-facing. Each `$def` ends with a `See:` link to canonical alloy upstream. - **GOTCHA — schemas are embedded at compile time (`include_str!`).** After editing any `schemas/alloy/*.yaml`, you MUST `cargo build --bin mz-monitoring-build` before a manual `gen-pipelines`, or it validates against the **stale** embedded schema and reports phantom "doesn't match any typed schema" errors. `make pipelines` rebuilds the binary as a dependency (safe); `cargo test` recompiles the lib so tests see the fresh schema — which means **unit tests can pass while a manual render of the same construct fails**. If a render rejects something your tests accept, rebuild the binary first. (A new schema fragment file also needs registering in `validate.rs`: a `SCHEMA_*` `include_str!`, an `ID_*` const matching its `$id`, and an entry in the `Registry` list.) - **GOTCHA — `assert_renders` enforces alloy-fmt-canonical output.** It does an exact-bytes match AND (when `alloy` is on PATH) asserts the rendered output is what `alloy fmt` would produce. So a test fails if the renderer's output isn't canonical, even when your expected string matches the renderer. The renderer's alignment quirk (see Cleanup) makes some shapes non-canonical — write tests around shapes known to be canonical (single-line attribute groups, object literals) until the quirk is fixed. For a shape the renderer *can't* make canonical (single-line attr beside a multi-line array → alignment quirk; an inline object literal inside `array.concat(...)`), byte-check with a plain `assert_eq!(pipeline.render().unwrap(), …)` and a comment, rather than `assert_renders` — see the `targets`/`array.concat` tests in loki.rs/discovery.rs. - **GOTCHA — `alloy validate` does NOT catch value-type (capsule) mismatches; a real load does.** `alloy validate` checks syntax and resolves *bare* component references (dangling refs error), but it does **not** verify that an expression assigned to a capsule-typed argument actually yields that capsule. It green-lights `forward_to = [sys.env("X")]` and `targets = [discovery.x.targets]` (both wrong) — the errors (`should be capsule, got string`; `conversion from '[]discovery.Target' is not supported`) only surface under `alloy run` (initial graph eval). This produced two field bugs. When touching capsule-valued attributes (`forward_to`, `targets`, `relabel_rules`, future `otelcol.Consumer`), verify with an actual load, not just `validate` — e.g. `perl -e 'alarm 5; exec @ARGV' alloy run --disable-reporting --storage.path=/tmp/x --server.http.listen-addr=127.0.0.1:PORT cfg.alloy` and grep for `convert|capsule|failed to evaluate`. `assert_renders`' `alloy fmt` oracle does NOT load either, so passing tests are not proof of load-validity. ## Cleanup / refactor candidates - **`write_expression` uses a `rendered_expr` flag**: the idiomatic shape is a tuple-match over `(&env, &raw, &function, &ref_name, &operator)`, which encodes "exactly one head set" structurally. The flag pattern bit us once (forgotten assignments); a refactor would prevent recurrence. - **Renderer block-attribute alignment (task #15)**: current rule "any multi-line value disables alignment for the whole group" is too aggressive; alloy fmt aligns in more cases. Concretely, the only divergences in the rendered `agent.alloy` vs `alloy fmt` are (a) `rule` blocks where a single-line attr (`action`, `separator`, `replacement`) sits in a group that also contains the multi-line `source_labels` array, and (b) `loki.source.file`'s `targets` (single-line) next to the multi-line `forward_to` array. Everything else is canonical. Two ways out (an open decision): fix the alignment rule, or post-process rendered output through `alloy fmt` (we have alloy in CI now). Until then, `assert_renders` will reject tests that exercise those non-canonical shapes. - **Schema drift watch**: `description` blocks link to canonical alloy docs. When alloy upstream renames a field or adds one we use, the schema description and `properties` need a corresponding update. There's no automation here; it's a manual sweep when bumping alloy versions. - **`with_capacity` + push loops in `to_block` impls** (~5 sites): idiomatic form is `self.blocks.iter().map(ToBlock::to_block).collect::>>()?`. Cosmetic; sweep opportunistically. - **Capsule newtype fields are `pub`** (`LogsReceiver(pub Identifier)`): if ref-path validation is ever added, switch to private field + `fn new() -> Result`. ## Queued work (non-binding — directions, not commitments) Rough backlog from recent sessions; shapes may change. In loose priority: - **Fold agent logs into the OTLP exporter**: the agent now has `otelcol.exporter.otlp "gateway"` for metrics while logs still go out via `loki.write "gateway"`. Unifying them is tracked separately. - **`replace_map` sugar** on `discovery.relabel`/`loki.relabel`: a `{source_label: target_label}` map expanding to one `action: replace` rule each, in source order (relies on the `preserve_order` already enabled). Covers only the 1:1 replace case — multi-source / `separator` / `replacement` rules (e.g. `job`, `__path__`) stay explicit. Biggest readability win for the relabel-heavy blocks. - **Renderer alignment vs `alloy fmt`** (the open decision above): fix the rule, or post-process through `alloy fmt`. - **`$comment` / inline-comment rendering**: the schema already declares `$comment`/`commentedValue`, but `Block` has no comment field and the renderer drops them — pure plumbing. Doc-gen from comments is a stretch follow-on. - **Ref-resolution pass**: `forward_to`/`targets`/`relabel_rules` are free strings; nothing checks the referenced component exists (alloy validate catches it, but later + with worse messages). - **Reusable `Expressable` schema `$defs`** (`numberOrExpr`/`stringOrExpr`): fields currently inline the `anyOf`; a couple of named defs would DRY it. Cosmetic. - **Golden snapshot test** for the full rendered `agent.alloy`, once the above settle (so it doesn't churn). - **CI freshness**: the `pipelines` job in `.github/workflows/test.yaml` asserts committed `.alloy` matches a fresh render — keep rendered output committed. A few decisions are deliberately deferred: typing `gateway.yaml`'s ingress components (see Cleanup); `loki.write` remote/auth config (it's exceptional — shared by agents + gateways, may target remote sinks; the gateway sink currently renders `basic_auth`-free with a `GATEWAY_LOKI_DEST` env-coalesced `url`); and a real (build- or runtime-) parameterization mechanism for numeric knobs like `stage.limit` rates (current `encoding.from_json` env coercion is provisional).