Pyroscope Architecture Reference

Pyroscope Architecture Reference

Complete reference for Grafana Pyroscope architecture, deployment modes, and scaling.

Deployment Modes

Monolithic Mode

Flag: -target=all (default)

All components run in a single process. Simplest configuration for getting started.

┌─────────────────────────────────────────────┐
│              Pyroscope (Single Process)     │
│                                             │
│  ┌──────────┐ ┌──────────┐ ┌─────────────┐ │
│  │Distributor│ │ Ingester │ │   Querier   │ │
│  └──────────┘ └──────────┘ └─────────────┘ │
│  ┌──────────┐ ┌──────────┐ ┌─────────────┐ │
│  │Compactor │ │Store-GW  │ │Query-Frontend│ │
│  └──────────┘ └──────────┘ └─────────────┘ │
└─────────────────────────────────────────────┘

Use Cases:

• Development and testing
• Small-scale deployments (~20GB profiles/day)
• Quick experimentation

Limitations:

• No horizontal scaling
• Multiple instances don't share data
• Not suitable for production at scale

Query URL: http://pyroscope:4040/

Microservices Mode

Flag: -target=[component_name]

Each component runs as a separate process, enabling horizontal scaling and fault isolation.

                    ┌───────────────┐
                    │  SDK / Alloy  │
                    └───────┬───────┘
                            │
                    ┌───────▼───────┐
                    │  Distributor  │  (Stateless)
                    └───────┬───────┘
                            │
                    ┌───────▼───────┐
                    │   Ingesters   │  (Stateful - 3+ replicas)
                    └───────┬───────┘
                            │
              ┌─────────────┼─────────────┐
              │             │             │
      ┌───────▼───────┐     │     ┌───────▼───────┐
      │   Compactor   │     │     │ Store-Gateway │
      └───────────────┘     │     └───────────────┘
                            │
                    ┌───────▼───────┐
                    │Object Storage │
                    │(S3/GCS/Azure) │
                    └───────────────┘

Query Path:
┌─────────────┐   ┌────────────────┐   ┌─────────────┐
│Query-Frontend│──▶│Query-Scheduler │──▶│   Querier   │
└─────────────┘   └────────────────┘   └─────────────┘

Use Cases:

• Production environments
• High-scale deployments
• Multi-tenant systems
• Independent component scaling

Query URL: http://pyroscope-querier:4040/

Core Components

Distributor (Stateless)

Purpose: Entry point for profile ingestion. Routes profiles to ingesters.

Responsibilities:

1. Data Validation - Validates incoming profiles
2. Series Sharding - Distributes series across ingesters using consistent hashing
3. Replication - Replicates each series to multiple ingesters

Replication Model:

• Dynamo-style quorum consistency
• Default replication factor: 3
• Write success requires n/2 + 1 confirmations (2 of 3)

Configuration:

distributor:
  push_timeout: 5s
  ingestion_tenant_shard_size: 4  # Shuffle sharding
  ring:
    kvstore: memberlist
    heartbeat_timeout: 5m

Scaling: Horizontally scalable, stateless

Ingester (Stateful)

Purpose: Receives and temporarily stores profiles before writing to long-term storage.

Responsibilities:

1. Profile Reception - Receives profiles from distributors
2. In-Memory Buffering - Stores recent data in memory
3. Batch Compression - Compresses and batches samples
4. Storage Upload - Writes blocks to object storage

Ring States:

Configuration:

ingester:
  num_tokens: 128
  heartbeat_interval: 5s
  heartbeat_timeout: 10s
  lifecycle:
    num_flush_instances: 3
    claim_on_rollout: true
    ring_timeout: 5m

Data Safety:

• Each profile replicates to 3 ingesters by default
• Single ingester failure causes no data loss
• Write-ahead log for crash recovery

Scaling: Stateful, minimum 3 replicas for production

Querier (Stateless)

Purpose: Handles query execution on the read path.

Responsibilities:

1. Query Evaluation - Executes profile queries
2. Data Fetching - Retrieves from ingesters (recent) and store-gateways (historical)
3. Result Aggregation - Merges results from multiple sources

Data Sources:

• Ingesters - Recent profile data (not yet flushed)
• Store-Gateways - Historical data from object storage

Configuration:

querier:
  query_store_after: 4h
  query_ingesters_within: 1h
  shuffle_sharding_ingesters_enabled: true
  max_query_lookback: 30d

Scaling: Horizontally scalable, stateless

Query Frontend (Stateless)

Purpose: API gateway for the read path. Optimizes query execution.

Responsibilities:

1. Query Reception - Receives incoming queries
2. Queue Management - Enqueues queries via query-scheduler
3. Result Aggregation - Collects and returns results

Benefits:

• Fair scheduling across tenants
• Query splitting and caching
• Horizontal scaling of query capacity

Configuration:

query_frontend:
  max_outstanding_per_tenant: 200
  querier_forget_delay: 15m

Scaling: Horizontally scalable, minimum 2 replicas for HA

Query Scheduler (Stateless)

Purpose: Manages query queue and distributes work to queriers.

Flow:

1. Query-frontend receives and processes queries
2. Query-frontend enqueues to query-scheduler
3. Query-scheduler maintains in-memory queue
4. Queriers pull and execute queries
5. Results route back through query-frontend

Configuration:

query_scheduler:
  max_outstanding_requests_per_tenant: 2048

Scaling: Minimum 2 replicas for HA

Compactor (Stateless)

Purpose: Optimizes storage by merging blocks and managing retention.

Responsibilities:

1. Block Compaction - Merges small blocks into larger ones
2. De-duplication - Removes duplicate data from replication
3. Retention Enforcement - Deletes expired data
4. Bucket Index - Maintains per-tenant block index

Compaction Process:

Stage 1 - Vertical Compaction:

• Merges blocks from same time range
• Eliminates duplicates from replication

Stage 2 - Horizontal Compaction:

• Combines adjacent time ranges
• Reduces index size significantly

Configuration:

compactor:
  data_dir: ./data
  block_ranges:
    - 1h
    - 2h
    - 8h
  compaction_interval: 30m
  compaction_concurrency: 4
  deletion_delay: 12h

Disk Requirements:

Minimum disk = compaction_concurrency × max_block_size × 2

Scaling: Stateless, shuffle sharding for horizontal scaling

Store Gateway (Distributed)

Purpose: Provides access to long-term storage for historical queries.

Responsibilities:

1. Block Management - Each instance manages subset of blocks
2. Query Serving - Retrieves profile data from object storage
3. Caching - Caches frequently accessed blocks

Configuration:

store_gateway:
  sharding_ring:
    enabled: true
    kvstore: memberlist
  sync_interval: 15m
  ignore_blocks_within: 3h

Scaling: Distributed block assignment enables horizontal scaling

Data Structures

Block Format

Each block contains profiling data for a single tenant:

<block-ulid>/
├── meta.json      # Block metadata (time range, etc.)
├── index.tsdb     # TSDB index (labels → profiles)
├── profiles.parquet  # Profile data (Parquet format)
└── symbols.symdb  # Symbol information for profiles

Characteristics:

• Unique ULID identifier
• Single-tenant isolation
• Google pprof protocol compatible
• Label-based organization

Bucket Index

Per-tenant file listing all blocks:

{
  "blocks": [...],
  "block_deletion_marks": [...],
  "updated_at": 1234567890
}

Benefits:

• Eliminates costly "list objects" API calls
• Store-gateways consult index directly
• Updated by compactor periodically

Hash Rings

Used for consistent data distribution:

Memberlist Configuration:

memberlist:
  bind_addr: 0.0.0.0
  bind_port: 7946
  join:
    - dnssrv+pyroscope-memberlist._tcp.pyroscope.svc.cluster.local
  gossip_interval: 1s
  gossip_nodes: 3
  retransmit_factor: 4

Ring Types:

• Ingester Ring - Distributes profiles across ingesters
• Store-Gateway Ring - Assigns blocks to store-gateways
• Compactor Ring - Coordinates compaction work

Shuffle Sharding

Limits which components handle specific tenants:

# Ingesters per tenant
distributor:
  ingestion_tenant_shard_size: 4

# Queriers per tenant
query_frontend:
  max_queriers_per_tenant: 2

# Store-gateways per tenant
store_gateway:
  tenant_shard_size: 3

# Compactors per tenant
compactor:
  compactor_tenant_shard_size: 1

Benefits:

• Failure isolation between tenants
• Resource fairness
• Reduced blast radius

Scaling Guidelines

Horizontal Scaling

Vertical Scaling

Production Replica Counts

Data Flow Details

Write Path

1. SDK/Alloy pushes profile to Distributor
2. Distributor validates and hashes profile labels
3. Distributor identifies target ingesters via hash ring
4. Profile replicated to N ingesters (default: 3)
5. Ingester buffers profile in memory
6. After block duration, ingester flushes to object storage
7. Compactor merges and optimizes blocks

Read Path

1. Query arrives at Query Frontend
2. Query Frontend enqueues to Query Scheduler
3. Querier pulls query from scheduler
4. Querier determines time range:
   - Recent: Query ingesters
   - Historical: Query store-gateways
5. Store-gateways fetch blocks from object storage
6. Results aggregated and returned via Query Frontend

Consistency Model

Write Consistency:

• Quorum-based (n/2 + 1 replicas)
• Default: 2 of 3 ingesters must confirm

Read Consistency:

• Eventual consistency for recent data
• Strong consistency for flushed data

Data Durability:

• 3-way replication prevents data loss
• WAL recovery for crash scenarios

Deployment Decision Matrix