Trino Admin Cluster Health

Trino Admin: Cluster Health

When to Use

• Cluster appears slow or queries are queuing unexpectedly
• Workers are reporting failures or dropping out of the cluster
• Memory pressure causing OOM query kills
• Setting up monitoring for a new Trino cluster
• Performing rolling upgrades or worker restarts

REST API Health Endpoints

Trino exposes a comprehensive REST API for monitoring without requiring a client connection.

COORDINATOR="http://trino-coordinator:8080"

# Cluster info: version, environment, uptime
curl -s $COORDINATOR/v1/info | jq .

# All registered nodes (coordinator + workers)
curl -s $COORDINATOR/v1/node | jq '.[].uri'

# Failed nodes (should be empty in healthy cluster)
curl -s $COORDINATOR/v1/node/failed | jq .

# Cluster-level stats: running queries, blocked queries, active workers
curl -s $COORDINATOR/v1/cluster | jq .

# All active queries with state
curl -s -H "X-Trino-User: admin" $COORDINATOR/v1/query | jq '.[] | {queryId, state, query}'

# Specific query details
curl -s -H "X-Trino-User: admin" $COORDINATOR/v1/query/<query_id> | jq .

# Kill a query
curl -X DELETE -H "X-Trino-User: admin" $COORDINATOR/v1/query/<query_id>

Key Health Metrics via REST

# Worker count and health
curl -s $COORDINATOR/v1/cluster | jq '{
  activeWorkers: .activeWorkers,
  runningQueries: .runningQueries,
  queuedQueries:  .queuedQueries,
  blockedQueries: .blockedQueries
}'

# Check if coordinator is ready to serve requests
curl -s $COORDINATOR/v1/info/state
# → "ACTIVE" = ready, "SHUTTING_DOWN" = graceful shutdown in progress

JMX MBeans for Prometheus

Configure Prometheus JMX exporter in jmx_exporter.yaml:

# prometheus-jmx-exporter.yaml
lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
  # Running and queued queries
  - pattern: 'trino.execution<name=QueryManager><>RunningQueries'
    name: trino_running_queries
    type: GAUGE

  - pattern: 'trino.execution<name=QueryManager><>(StartedQueries|FailedQueries)\.FiveMinute\.Count'
    name: trino_queries_$1_5m
    type: GAUGE

  - pattern: 'trino.execution<name=QueryManager><>ExecutionTime\.FiveMinutes\.(P50|P90|P99)'
    name: trino_query_execution_time_$1_ms
    type: GAUGE

  # Memory
  - pattern: 'trino.memory<type=ClusterMemoryPool, name=general><>FreeDistributedBytes'
    name: trino_free_memory_bytes
    type: GAUGE

  - pattern: 'trino.memory<name=ClusterMemoryManager><>QueriesKilledDueToOutOfMemory'
    name: trino_queries_killed_oom_total
    type: COUNTER

  # Workers
  - pattern: 'trino.failuredetector<name=HeartbeatFailureDetector><>ActiveCount'
    name: trino_active_workers
    type: GAUGE

  # Task throughput
  - pattern: 'trino.execution<name=SqlTaskManager><>InputDataSize\.FiveMinute\.Count'
    name: trino_input_data_rate_5m_bytes
    type: GAUGE

JVM args to enable JMX exporter:

# etc/jvm.config (add to existing config)
-javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent.jar=9090:/opt/jmx_exporter/prometheus-jmx-exporter.yaml

Prometheus Alert Rules

# trino-alerts.yaml
groups:
  - name: trino
    rules:
      - alert: TrinoWorkersLow
        expr: trino_active_workers < 3
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Trino has {{ $value }} active workers (threshold: 3)"

      - alert: TrinoQueryQueueHigh
        expr: trino_running_queries > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Trino query queue depth {{ $value }} > 80"

      - alert: TrinoOOMKills
        expr: increase(trino_queries_killed_oom_total[5m]) > 0
        labels:
          severity: warning
        annotations:
          summary: "Trino killed {{ $value }} queries due to OOM in last 5m"

      - alert: TrinoHighQueryFailureRate
        expr: |
          rate(trino_queries_FailedQueries_5m[5m]) /
          rate(trino_queries_StartedQueries_5m[5m]) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Trino failure rate {{ $value | humanizePercentage }}"

      - alert: TrinoFreeMemoryLow
        expr: trino_free_memory_bytes < 5 * 1024 * 1024 * 1024  # < 5GB free
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Trino cluster free memory {{ $value | humanize1024 }}B"

Query State Diagnosis

# Find all BLOCKED queries (may indicate memory/resource pressure)
curl -s -H "X-Trino-User: admin" $COORDINATOR/v1/query \
  | jq '.[] | select(.state == "BLOCKED") | {queryId, state, queryTextPreview}'

# Find long-running queries (running > 10 minutes)
curl -s -H "X-Trino-User: admin" $COORDINATOR/v1/query \
  | jq --argjson threshold 600000 \
    '.[] | select(.state == "RUNNING") | select(.elapsedTime.toMillis // 0 > $threshold)'

# Count queries by state
curl -s -H "X-Trino-User: admin" $COORDINATOR/v1/query \
  | jq 'group_by(.state) | map({state: .[0].state, count: length})'

Query states reference:

Log Monitoring

# Trino server log location
tail -f /var/trino/data/var/log/server.log

# Key patterns to alert on:
grep -E "WORKER_LOST|NodeManager|OutOfMemoryError|exceeded memory limit" server.log

# Worker registration events
grep "NodeManager" server.log | tail -50

# OOM query kills
grep "exceeded memory limit" server.log | tail -20

Key log patterns:

Graceful Worker Shutdown (Rolling Upgrade)

# 1. Put worker in graceful shutdown mode (new queries won't be scheduled)
curl -X PUT -H "Content-Type: application/json" \
  http://worker-host:8080/v1/info/state \
  -d '"SHUTTING_DOWN"'

# 2. Wait for all tasks to finish (poll until running tasks = 0)
watch -n 5 'curl -s http://worker-host:8080/v1/info/state'

# 3. Once state = INACTIVE, safely stop the process
bin/launcher stop

Configure shutdown grace period:

# etc/config.properties
shutdown.grace-period=2m

Worker Node Checklist

For each worker node, verify:

# JVM heap usage (should be < 80% of -Xmx)
curl -s http://worker:8080/v1/jmx/mbean/java.lang:type=Memory \
  | jq '.attributes[] | select(.name == "HeapMemoryUsage")'

# Number of active tasks
curl -s http://worker:8080/v1/task | jq 'length'

# Worker is registered with coordinator
curl -s http://coordinator:8080/v1/node | jq '.[].uri' | grep worker-host

Web UI Key Views

Navigate to http://trino-coordinator:8080/ui/:

Performance signals in Web UI:

• High "Blocked" percentage = memory or I/O bottleneck
• Many QUEUED queries = resource group concurrency limit hit
• wall_time >> cpu_time = waiting on network/I/O, not compute

Anti-Patterns

1. Running coordinator with node-scheduler.include-coordinator=true in production — coordinator becomes a bottleneck for both planning and execution; always set false for clusters with > 3 workers.
2. Not monitoring OOM kills — QueriesKilledDueToOutOfMemory counter silently increments; queries fail with cryptic errors; alert on this metric.
3. Ignoring BLOCKED query state — persistent BLOCKED queries hold memory slots and prevent other queries from running; set alert if any query is BLOCKED > 10 minutes.
4. Hard shutting down workers during query execution — kills in-flight tasks; always use graceful shutdown via /v1/info/state PUT.
5. Exposing JMX without authentication in production — JMX can expose sensitive cluster information; bind to localhost only or use TLS.

References

• Admin REST API: trino.io/docs/current/develop/client-protocol.html
• Web UI: trino.io/docs/current/admin/web-interface.html
• JMX connector: trino.io/docs/current/connector/jmx.html
• Related skills: [[trino-memory-and-spill-tuning]], [[trino-resource-group-governance]], [[trino-observability-platform]], [[trino-production-readiness-review]]