Substreams Dev

Substreams SVM Development Expert

Expert assistant for building Substreams projects on Solana/SVM — high-performance blockchain data indexing for Solana transactions, instructions, and program events.

Core Concepts

What is Substreams?

Substreams is a powerful blockchain indexing technology that enables:

• Parallel processing of blockchain data with high performance
• Composable modules written in Rust (map, store, index types)
• Protobuf schemas for typed data structures
• Streaming-first architecture with cursor-based reorg handling

SVM vs EVM Key Differences

Key Components

1. Manifest (substreams.yaml): Defines modules, networks, dependencies
2. Modules: Map (transform), Store (aggregate), Index (filter)
3. Protobuf: Type-safe schemas for inputs and outputs
4. WASM: Rust code compiled to WebAssembly for execution
5. IDL Decoders: Anchor/native IDL-based instruction decoders from substreams-solana-idls

Project Structure

my-solana-substreams/
├── substreams.yaml          # Manifest
├── proto/
│   └── v1/my-events.proto   # Schema definitions
├── src/
│   └── lib.rs               # Rust module code
├── Cargo.toml               # Rust dependencies
└── target/                   # Build output (gitignored)

Monorepo Structure (substreams-svm)

substreams-svm/
├── Cargo.toml                    # Workspace root
├── proto/                        # Shared protobuf definitions
│   └── v1/*.proto
├── common/                       # Shared Rust utilities
│   └── src/{lib.rs, db.rs, solana.rs}
├── spkg/                         # Pre-built SPKG dependencies
│
├── # Individual DEX modules
├── raydium/amm-v4/              # Raydium AMM V4
├── raydium/clmm/                # Raydium CLMM
├── raydium/cpmm/                # Raydium CPMM
├── raydium/launchpad/           # Raydium Launchpad
├── jupiter/v4/                  # Jupiter V4
├── jupiter/v6/                  # Jupiter V6
├── orca/whirlpool/              # Orca Whirlpool
├── meteora/{amm,daam,dllm}/     # Meteora pools
├── pumpfun/bonding_curve/       # PumpFun bonding curve
├── pumpfun/amm/                 # PumpFun AMM
├── pumpswap/                    # PumpSwap
├── moonshot/                    # Moonshot
├── pancakeswap/                 # PancakeSwap
├── lifinity/                    # Lifinity
├── phoenix/                     # Phoenix
├── openbook/                    # OpenBook
├── stabble/                     # Stabble
├── darklake/                    # Darklake
├── bonk-swap/                   # Bonk Swap
│
├── # Token modules
├── spl-token/                   # SPL Token events
├── native-token/                # SOL native transfers
├── metaplex/                    # Metaplex metadata
│
├── # Aggregate DB modules
├── db-svm-dex/               # DEX DatabaseChanges (all DEXs)
├── db-svm-transfers/         # Transfer DatabaseChanges
├── db-svm-balances/          # Balance DatabaseChanges
├── db-svm-accounts/          # Account DatabaseChanges
├── db-svm-metadata/          # Metadata DatabaseChanges
│
├── # Sink modules (ClickHouse / Postgres)
├── db-svm-dex-clickhouse/
├── db-svm-dex-postgres/
├── db-svm-transfers-clickhouse/
├── db-svm-transfers-postgres/
├── db-svm-balances-clickhouse/
├── db-svm-balances-postgres/
├── db-svm-accounts-clickhouse/
├── db-svm-accounts-postgres/
├── db-svm-metadata-clickhouse/
└── db-svm-metadata-postgres/

Prerequisites

Required CLI Tools

• substreams: Core CLI for building, running, and deploying
• buf: Required by substreams build for protobuf code generation

Authentication

substreams auth  # Interactive authentication
# Or set SUBSTREAMS_API_TOKEN environment variable

Rust Dependencies (Cargo.toml)

[dependencies]
substreams = "0.7.3"
substreams-solana = "0.14.2"
substreams-solana-idls = { git = "https://github.com/pinax-network/substreams-solana-idls", tag = "v1.0.0" }
substreams-database-change = "4.0.0"
prost = "0.13"
prost-types = "0.13"
bs58 = "0.5.1"

Common Workflows

Creating a New DEX Module

1. Define protobuf schema in proto/v1/<dex-name>.proto
2. Create crate directory with Cargo.toml and src/lib.rs
3. Add to workspace in root Cargo.toml
4. Implement map_events handler using IDL decoders
5. Create substreams.yaml manifest with program ID filter
6. Build: cargo build --target wasm32-unknown-unknown --release
7. Test: substreams run with small block range

Module Types

Map Module — Transforms Solana blocks into typed events:

- name: map_events
  kind: map
  inputs:
    - map: solana_common:blocks_without_votes
  blockFilter:
    module: solana_common:program_ids_without_votes
    query:
      string: "program:<PROGRAM_ID>"
  output:
    type: proto:my.dex.v1.Events

Store Module — Aggregates data across blocks:

- name: store_totals
  kind: store
  updatePolicy: add
  valueType: int64
  inputs:
    - map: map_events

Block Filtering

Solana Substreams use blockFilter with program IDs to efficiently skip irrelevant transactions:

blockFilter:
  module: solana_common:program_ids_without_votes
  query:
    string: "program:675kPX9MHTjS2zt1qfr1NYHuzeLXfQM9H24wFSUt1Mp8"

Binary Type

SVM Substreams use wasm-bindgen-shims binary type:

binaries:
  default:
    type: wasm/rust-v1+wasm-bindgen-shims
    file: ../../target/wasm32-unknown-unknown/release/my_module.wasm

Network

All SVM manifests use:

network: solana

IDL-Based Instruction Decoding

Unlike EVM which uses ABI event signatures, Solana Substreams use IDL decoders from the substreams-solana-idls crate:

use substreams_solana_idls::raydium;

// Unpack instruction data using IDL decoder
match raydium::amm::v4::instructions::unpack(instruction.data()) {
    Ok(raydium::amm::v4::instructions::RaydiumV4Instruction::SwapBaseIn(event)) => {
        // Process swap event
        Some(pb::Instruction {
            program_id: program_id.to_vec(),
            instruction: Some(pb::instruction::Instruction::SwapBaseIn(..)),
        })
    }
    _ => None,
}

Available IDL Decoders

The substreams-solana-idls crate provides decoders for:

• Raydium: amm::v4, clmm, cpmm, launchpad
• Jupiter: v4, v6
• Orca: whirlpool
• Meteora: amm, daam, dllm
• PumpFun: bonding_curve, amm
• PumpSwap
• Moonshot
• PancakeSwap
• Lifinity
• Phoenix
• OpenBook
• Stabble, Darklake, Bonk Swap

Solana Transaction Processing

Walking Instructions

use substreams_solana::{
    block_view::InstructionView,
    pb::sf::solana::r#type::v1::{Block, ConfirmedTransaction},
};

#[substreams::handlers::map]
fn map_events(block: Block) -> Result<pb::Events, Error> {
    Ok(pb::Events {
        transactions: block.transactions_owned()
            .filter_map(process_transaction)
            .collect(),
    })
}

fn process_transaction(tx: ConfirmedTransaction) -> Option<pb::Transaction> {
    let tx_meta = tx.meta.as_ref()?;
    let instructions: Vec<pb::Instruction> = tx.walk_instructions()
        .filter_map(|iview| process_instruction(&iview))
        .collect();

    if instructions.is_empty() {
        return None;
    }

    Some(pb::Transaction {
        signature: tx.hash().to_vec(),
        fee_payer: get_fee_payer(&tx).unwrap_or_default(),
        fee: tx_meta.fee,
        compute_units_consumed: tx_meta.compute_units_consumed(),
        instructions,
        ..Default::default()
    })
}

Common Helpers (from common crate)

use common::solana::{get_fee_payer, get_signers, is_failed, is_invoke, is_success, parse_program_id};

Address Encoding

Solana uses base58 encoding for addresses (not hex):

use substreams_solana::base58;

let address_string = base58::encode(&address_bytes);

Protobuf Schema Patterns

Standard Transaction Wrapper

syntax = "proto3";
package my.dex.v1;

message Events {
  repeated Transaction transactions = 1;
}

message Transaction {
  bytes signature                   = 1;
  bytes fee_payer                   = 2;
  repeated bytes signers            = 3;
  uint64 fee                        = 4;
  uint64 compute_units_consumed     = 5;
  repeated Instruction instructions = 6;
  repeated Log logs                 = 7;
}

message Instruction {
  bytes  program_id  = 1;
  uint32 stack_height = 2;
  oneof instruction {
    SwapInstruction swap = 3;
    // ... other instruction types
  }
}

Common Program IDs

Converted and distributed by TomeVault — claim your Tome and manage your conversions.