HolmesGPT Data Sources & Toolsets

HolmesGPT Data Sources & Toolsets

Complete reference for built-in and custom data source integrations.

Built-in Toolsets Overview

HolmesGPT includes 30+ pre-built integrations organized by category.

Kubernetes & Cloud Infrastructure

Monitoring & Observability

Prometheus Integration Details

HolmesGPT can query Prometheus metrics to diagnose performance issues.

Enable Prometheus Toolset:

toolsets:
  prometheus/metrics:
    enabled: true

env:
  - name: PROMETHEUS_URL
    value: "http://prometheus-server.monitoring.svc.cluster.local"
  # Optional: For authenticated Prometheus
  - name: PROMETHEUS_USERNAME
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: prometheus-username
  - name: PROMETHEUS_PASSWORD
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: prometheus-password

Example Prometheus Queries:

HolmesGPT translates natural language to PromQL automatically:

# Memory analysis
holmes ask "show memory usage trend for payment-service over last hour"
# Uses: rate(container_memory_usage_bytes{pod=~"payment.*"}[1h])

# CPU throttling detection
holmes ask "which pods have the highest CPU throttling?"
# Uses: rate(container_cpu_cfs_throttled_seconds_total[5m])

# Error rate analysis
holmes ask "what's the error rate for api-gateway?"
# Uses: rate(http_requests_total{status=~"5.."}[5m])

# Resource saturation
holmes ask "are any nodes running low on resources?"
# Uses: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

# Pod restarts
holmes ask "which pods have restarted most in the last 24 hours?"
# Uses: increase(kube_pod_container_status_restarts_total[24h])

Prometheus-Specific Runbook:

runbooks:
  - alert_name: "HighMemoryUsage"
    instructions: |
      ## Prometheus Queries to Run
      1. Current usage: `container_memory_usage_bytes{pod="<pod>"}`
      2. Usage trend: `rate(container_memory_usage_bytes{pod="<pod>"}[1h])`
      3. Limit comparison: `container_memory_usage_bytes / container_spec_memory_limit_bytes`

      ## Thresholds
      - Warning: > 80% of limit
      - Critical: > 90% of limit

  - alert_name: "HighLatency"
    instructions: |
      ## Prometheus Queries to Run
      1. P99 latency: `histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))`
      2. Request rate: `rate(http_requests_total[5m])`
      3. Error rate: `rate(http_requests_total{status=~"5.."}[5m])`

Grafana Loki Integration

toolsets:
  loki:
    enabled: true

env:
  - name: LOKI_URL
    value: "http://loki.monitoring.svc.cluster.local:3100"

Example Loki Queries:

# Search for errors
holmes ask "show errors from payment-service in the last 30 minutes"

# Analyze log patterns
holmes ask "what are the most common error messages in production?"

# Correlate with events
holmes ask "show logs around the time of the last pod restart"

Data & Messaging

Infrastructure & DevOps

ArgoCD Integration Details

HolmesGPT can investigate ArgoCD sync failures and application health issues.

Enable ArgoCD Toolset:

toolsets:
  argocd:
    enabled: true

env:
  - name: ARGOCD_SERVER
    value: "argocd-server.argocd.svc.cluster.local"
  - name: ARGOCD_AUTH_TOKEN
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: argocd-token

Create ArgoCD Token:

# Generate token for HolmesGPT service account
argocd account generate-token --account holmesgpt --id holmesgpt-token

# Store in secret
kubectl create secret generic holmesgpt-secrets \
  --namespace holmesgpt \
  --from-literal=argocd-token="<generated-token>"

Example ArgoCD Queries:

# Investigate sync failures
holmes ask "why is application payment-service out of sync?"

# Check application health
holmes ask "what ArgoCD applications are degraded?"

# Analyze sync history
holmes ask "show recent sync failures for production apps"

# Investigate specific error
holmes ask "analyze ArgoCD sync error: ComparisonError for deployment/api"

ArgoCD-Specific Runbook:

runbooks:
  - alert_name: "ArgoCDAppOutOfSync"
    instructions: |
      ## Investigation Steps
      1. Check app sync status: `argocd app get <app-name>`
      2. Review sync history: `argocd app history <app-name>`
      3. Check for resource conflicts
      4. Verify Git repository accessibility

      ## Common Causes
      - Manual changes to cluster resources
      - Git repository access issues
      - Resource conflicts with other controllers
      - Invalid manifests in Git

Documentation & Collaboration

Utilities

Enabling Toolsets

CLI Usage

# Use default toolsets
holmes ask "what pods are crashing?"

# Add custom toolset
holmes ask "query" -t /path/to/custom-toolset.yaml

# Multiple toolsets
holmes ask "query" -t toolset1.yaml -t toolset2.yaml

Helm Configuration

toolsets:
  # Kubernetes (recommended baseline)
  kubernetes/core:
    enabled: true
  kubernetes/logs:
    enabled: true

  # Monitoring
  prometheus/metrics:
    enabled: true
  grafana:
    enabled: false
  loki:
    enabled: false

  # DevOps tools
  argocd:
    enabled: true
  helm:
    enabled: true

  # Documentation
  confluence:
    enabled: false
  github:
    enabled: false

  # Utilities
  internet:
    enabled: false  # Disable for security
  bash:
    enabled: false  # Disable for security

Config File

# ~/.holmes/config.yaml
toolsets:
  - kubernetes/core
  - kubernetes/logs
  - prometheus/metrics
  - argocd

custom_toolsets:
  - ~/toolsets/my-custom.yaml

Custom Toolset Development

Basic Structure

# custom-toolset.yaml
toolsets:
  my-toolset:
    description: "Description for the LLM"
    prerequisites: "Required tools/access"
    tags:
      - kubernetes
      - monitoring
    installation: |
      Instructions for setup
    tools:
      - name: tool_name
        description: "What this tool does"
        command: |
          command to execute
        parameters:
          - name: param_name
            description: "Parameter description"
            required: true
        additionalInstructions: |
          How to interpret output

Variable Types

LLM-Inferred Variables (double braces)

command: |
  kubectl get pods -n {{ namespace }} -l app={{ app_name }}

The LLM will determine appropriate values based on context.

Environment Variables (single braces with $)

command: |
  curl -H "Authorization: Bearer ${API_TOKEN}" ${API_URL}/endpoint

Hidden from LLM, sourced from environment.

Example: Custom Grafana Toolset

toolsets:
  grafana-custom:
    description: "Query Grafana dashboards and panels"
    prerequisites: "Grafana API access"
    tags:
      - monitoring
      - grafana
    tools:
      - name: list_dashboards
        description: "List all Grafana dashboards"
        command: |
          curl -s -H "Authorization: Bearer ${GRAFANA_API_KEY}" \
            "${GRAFANA_URL}/api/search?type=dash-db" | jq '.[] | {uid, title}'

      - name: get_dashboard
        description: "Get a specific dashboard by UID"
        command: |
          curl -s -H "Authorization: Bearer ${GRAFANA_API_KEY}" \
            "${GRAFANA_URL}/api/dashboards/uid/{{ dashboard_uid }}" | jq '.dashboard'
        parameters:
          - name: dashboard_uid
            description: "Dashboard UID"
            required: true

      - name: query_panel
        description: "Query a specific panel's data"
        command: |
          curl -s -H "Authorization: Bearer ${GRAFANA_API_KEY}" \
            "${GRAFANA_URL}/api/ds/query" \
            -d '{"queries": [{"refId": "A", "datasourceId": \
            {{ datasource_id }}, "rawSql": "{{ query }}"}]}'
        parameters:
          - name: datasource_id
            description: "Datasource ID"
          - name: query
            description: "Query to execute"

Example: Custom Health Check Toolset

toolsets:
  service-health:
    description: "Check health of internal services"
    prerequisites: "Network access to services"
    tools:
      - name: check_endpoint_health
        description: "Check HTTP endpoint health"
        command: |
          curl -s -o /dev/null -w "%{http_code}" \
            http://{{ service }}.{{ namespace }}.svc.cluster.local:{{ port }}/health
        parameters:
          - name: service
            description: "Service name"
          - name: namespace
            description: "Kubernetes namespace"
          - name: port
            description: "Service port"
            default: "8080"
        additionalInstructions: |
          200 = healthy, 5xx = server error, 4xx = client error

      - name: check_database_connectivity
        description: "Test database connection"
        command: |
          kubectl run db-check --rm -i --restart=Never --image=postgres:15 \
            -- pg_isready -h {{ host }} -p {{ port }} -U {{ user }}
        parameters:
          - name: host
            description: "Database host"
          - name: port
            description: "Database port"
            default: "5432"
          - name: user
            description: "Database user"

Example: Kubernetes Diagnostics Toolset

toolsets:
  k8s-diagnostics:
    description: "Advanced Kubernetes diagnostic commands"
    tools:
      - name: check_node_pressure
        description: "Check for node resource pressure conditions"
        command: |
          kubectl get nodes -o json | jq '.items[] | {
            name: .metadata.name,
            conditions: [.status.conditions[] |
              select(.type | test("Pressure")) | {type, status}]
          }'

      - name: get_pod_resource_usage
        description: "Get actual resource usage for pods in namespace"
        command: |
          kubectl top pods -n {{ namespace }} --sort-by=memory
        parameters:
          - name: namespace
            description: "Target namespace"

      - name: check_pvc_status
        description: "Check PersistentVolumeClaim status"
        command: |
          kubectl get pvc -n {{ namespace }} -o json | jq '.items[] | {
            name: .metadata.name,
            status: .status.phase,
            capacity: .status.capacity.storage,
            storageClass: .spec.storageClassName
          }'
        parameters:
          - name: namespace
            description: "Target namespace"

      - name: get_events
        description: "Get recent events filtered by type"
        command: |
          kubectl get events -n {{ namespace }} \
            --sort-by='.lastTimestamp' \
            --field-selector type={{ event_type }} -o json | \
            jq '.items[-10:] | .[] |
              {time: .lastTimestamp, reason: .reason, message: .message}'
        parameters:
          - name: namespace
            description: "Target namespace"
          - name: event_type
            description: "Event type (Normal or Warning)"
            default: "Warning"

MCP Server Integration

HolmesGPT supports Model Context Protocol (MCP) servers for extended capabilities.

Configuration

# In Helm values.yaml
mcpServers:
  aws-mcp:
    command: "npx"
    args:
      - "-y"
      - "@aws-mcp/server"
    env:
      AWS_REGION: "us-east-1"

  github-mcp:
    command: "npx"
    args:
      - "-y"
      - "@github-mcp/server"
    env:
      GITHUB_TOKEN: "${GITHUB_TOKEN}"

Custom Docker Image for Additional Tools

When you need binaries not in the base image:

# Dockerfile
FROM us-central1-docker.pkg.dev/genuine-flight-317411/devel/holmes:latest

# Install custom tools
RUN apt-get update && apt-get install -y \
    postgresql-client \
    mysql-client \
    redis-tools

# Add custom scripts
COPY scripts/ /usr/local/bin/
RUN chmod +x /usr/local/bin/*.sh

# Add custom toolsets
COPY toolsets/ /etc/holmes/toolsets/

Build and use:

docker build -t my-holmes:latest .

# In Helm values
image:
  repository: my-registry/my-holmes
  tag: latest

Toolset Best Practices

1. Start Minimal: Enable only kubernetes/core and kubernetes/logs initially
2. Add Incrementally: Enable additional toolsets as needed
3. Security First: Disable internet and bash in production unless required
4. Use Environment Variables: Never hardcode secrets in toolset YAML
5. Test Custom Toolsets: Verify commands work before deploying
6. Document Prerequisites: Clearly state required permissions and access
7. Handle Large Outputs: Use llm_summarize transformer for verbose commands