Turbo Pipelines

Turbo Pipeline Configuration & Architecture

YAML configuration reference and architecture guide for Turbo pipelines. For interactive pipeline building, use /turbo-builder. For troubleshooting, use /turbo-doctor. For transform implementation, use /turbo-transforms.

CRITICAL: Always validate YAML with goldsky turbo validate <file.yaml> before showing complete pipeline YAML to the user or deploying.

Quick Start

name: my-first-pipeline
resource_size: s
sources:
  transfers:
    type: dataset
    dataset_name: base.erc20_transfers
    version: 1.2.0
    start_at: latest
transforms: {}
sinks:
  output:
    type: blackhole
    from: transfers

goldsky turbo validate pipeline.yaml   # Validate first
goldsky turbo apply pipeline.yaml -i   # Deploy + inspect

Prerequisites

• Goldsky CLI — curl https://goldsky.com | sh
• Turbo extension (separate binary) — curl https://install-turbo.goldsky.com | sh
• Logged in — goldsky login
• Secrets created for sinks if using PostgreSQL, ClickHouse, Kafka, etc. (see /secrets)

Architecture Decisions

Source Type Selection

Note: Kafka sources are used in production but are not documented in official Goldsky docs. Contact Goldsky support for topic names.

Data Flow Patterns

For detailed pattern diagrams, YAML examples, and multi-chain deployment guidance, read references/architecture-patterns.md.

Resource Sizing

Start small and scale up — defensive sizing avoids wasted resources.

Sink Selection

For full sink field specifications, see Sinks in the docs (one page per sink type).

Streaming vs Job Mode

Job mode rules: Runs to completion, auto-deletes ~1hr after finishing. Must delete before redeploying. Cannot pause/resume/restart.

Pipeline Splitting

Default to one pipeline with multiple sinks when the user wants the same source data sent to multiple destinations. A single Turbo pipeline supports multiple sinks natively — each sink has a from: field pointing at a source or transform by name, and sinks run independently (one failing does not block the others, and each can have its own batch_size / batch_flush_interval).

Do NOT split into separate pipelines just because there are multiple destinations. Generating one pipeline per sink duplicates the source ingestion, wastes resources, and decouples deployments that should be atomic.

Split into multiple pipelines only when: sources are fundamentally different (different chains with independent lifecycles), the destinations need different resource sizes, or the user explicitly asks for separate pipelines.

See references/architecture-patterns.md for fan-out (one source → multiple sinks) and fan-in (UNION ALL) YAML examples, and templates/multi-sink-pipeline.yaml / templates/fan-out-pipeline.yaml / templates/fan-in-pipeline.yaml for ready-to-adapt configs.

Configuration Reference

Pipeline Structure

name: my-pipeline          # Required: unique identifier (lowercase, hyphens)
resource_size: s            # Required: xs/s/m/l/xl/xxl
job: true                   # Optional: one-time batch (default: false = streaming)

sources:
  source_name:
    type: dataset           # or: kafka
    # ... source config

transforms:                 # Optional
  transform_name:
    type: sql               # or: script, handler, dynamic_table
    # ... transform config

sinks:
  sink_name:
    type: postgres           # or: clickhouse, kafka, webhook, s3_sink, etc.
    # ... sink config

Source Configuration

Dataset Source

sources:
  my_source:
    type: dataset
    dataset_name: <chain>.<dataset_type>
    version: <version>
    start_at: latest | earliest    # EVM chains
    # start_block: <slot_number>   # Solana only
    # end_block: <block_number>    # Optional: for bounded backfills
    # filter: >-                   # Optional: SQL WHERE for source-level pre-filtering
    #   address = '0x...' AND block_number >= 10000000

Use filter for contract addresses and block ranges (coarse pre-filtering). Use transform WHERE for fine-grained filtering.

For chain prefixes and dataset types, see /datasets.

Kafka Source

sources:
  my_source:
    type: kafka
    topic: base.raw.latest_balances_v2

No start_at or version fields. Optional: filter, include_metadata, starting_offsets.

Transform Configuration

SQL Transform

transforms:
  filtered:
    type: sql
    primary_key: id
    sql: |
      SELECT id, sender, recipient, amount
      FROM source_name
      WHERE amount > 1000

Transform Chaining

Chain transforms using from:

transforms:
  step1:
    type: sql
    primary_key: id
    sql: SELECT * FROM source WHERE amount > 100
  step2:
    type: sql
    primary_key: id
    from: step1
    sql: SELECT *, 'processed' as status FROM step1

For TypeScript, handler, and dynamic table transforms, see /turbo-transforms.

Throttle Transform

Caps the throughput of a stream by buffering records into batches and emitting each batch on a fixed minimum interval. Throttle does not modify data — every input record passes through unchanged. Use it to stay under rate limits of downstream sinks or external APIs, smooth bursty sources, or pace records into HTTP handlers.

transforms:
  throttled:
    type: throttle
    from: my_source
    max_batch_size: 100
    min_batch_interval: 10s

Effective max throughput ≈ max_batch_size / min_batch_interval records per second. Throttle limits the maximum rate, not the minimum — if upstream is slow, batches will be smaller and arrive less frequently.

Place throttle close to the bottleneck (just before the rate-limited sink or handler) so upstream transforms still process at full speed.

Sink Configuration

Quick examples for common sinks. For full field specs of all sink types, see Sinks in the docs (one page per sink type).

PostgreSQL

sinks:
  output:
    type: postgres
    from: my_transform
    schema: public
    table: my_table
    secret_name: MY_POSTGRES_SECRET
    primary_key: id

ClickHouse

sinks:
  output:
    type: clickhouse
    from: my_transform
    table: my_table
    secret_name: MY_CLICKHOUSE_SECRET
    primary_key: id

Pub/Sub (Turbo-only)

sinks:
  output:
    type: pubsub
    from: my_transform
    topic: my-topic
    secret_name: MY_PUBSUB_SECRET

The secret holds a GCP project id and service-account JSON. Topic must already exist in GCP.

Blackhole (Testing)

sinks:
  output:
    type: blackhole
    from: my_transform

Checkpoint Behavior

• Preserved by default when updating a pipeline
• Tied to source names — renaming a source resets its checkpoint
• Tied to pipeline names — renaming the pipeline resets all checkpoints

To reset checkpoints: rename the source or pipeline. Warning: this reprocesses all historical data.

Starter Templates

Template location: templates/ (relative to this skill's directory)

CLI Quick Reference

For lifecycle commands (pause/resume/restart/delete) and monitoring (inspect/logs), see /turbo-operations.

Troubleshooting

See references/troubleshooting.md for CLI hanging, validation errors, and runtime errors.

Related

• /turbo-builder — Interactive wizard to build pipelines step-by-step
• /turbo-doctor — Diagnose and fix pipeline issues
• /turbo-operations — Lifecycle commands and monitoring reference
• /turbo-transforms — SQL, TypeScript, and dynamic table transform reference
• /datasets — Dataset names and chain prefixes
• /secrets — Sink credential management