HolmesGPT Integrations Reference

HolmesGPT Integrations Reference

Complete guide for integrating HolmesGPT with alerting platforms, messaging services, and incident management tools.

Slack Integration

Sending Analysis to Slack

HolmesGPT can send investigation results directly to Slack channels.

CLI Usage:

# Send to a specific channel
holmes ask "why is payment-service crashing?" \
  --destination slack \
  --slack-token xoxb-your-bot-token \
  --slack-channel "#incidents"

# Send alert investigation to Slack
holmes investigate alertmanager \
  --alertmanager-url http://alertmanager:9093 \
  --destination slack \
  --slack-token $SLACK_BOT_TOKEN \
  --slack-channel "#alerts"

Creating a Slack Bot

1. Go to api.slack.com/apps and create a new app
2. Under "OAuth & Permissions", add these Bot Token Scopes:
   • chat:write - Send messages
   • chat:write.public - Send to channels without joining
   • files:write - Upload files (for large outputs)
3. Install the app to your workspace
4. Copy the "Bot User OAuth Token" (starts with xoxb-)

Helm Configuration for Slack

env:
  - name: SLACK_BOT_TOKEN
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: slack-bot-token
  - name: SLACK_CHANNEL
    value: "#holmesgpt-alerts"

# In values.yaml destination configuration
destinations:
  slack:
    enabled: true
    default_channel: "#holmesgpt-alerts"

Slack Message Formatting

HolmesGPT formats messages with sections:

:mag: *HolmesGPT Investigation*

*Alert:* KubePodCrashLooping
*Namespace:* production

*Key Findings:*
• Pod restarted 5 times in the last hour
• OOMKilled events detected
• Memory usage exceeded limits

*Root Cause:*
Memory leak in application causing OOM kills

*Recommended Actions:*
1. Increase memory limits
2. Investigate memory leak
3. Enable memory profiling

PagerDuty Integration

Investigating PagerDuty Incidents

# Investigate open incidents
holmes investigate pagerduty \
  --pagerduty-api-key $PAGERDUTY_API_KEY

# Update incidents with analysis
holmes investigate pagerduty \
  --pagerduty-api-key $PAGERDUTY_API_KEY \
  --update

# Investigate specific incident
holmes investigate pagerduty \
  --pagerduty-api-key $PAGERDUTY_API_KEY \
  --incident-id P123ABC \
  --update

PagerDuty API Key Setup

1. Go to PagerDuty > Integrations > API Access Keys
2. Create a new API key with read/write access
3. Store securely in Kubernetes secret

kubectl create secret generic holmesgpt-secrets \
  --namespace holmesgpt \
  --from-literal=pagerduty-api-key="your-api-key"

Helm Configuration for PagerDuty

env:
  - name: PAGERDUTY_API_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: pagerduty-api-key

# Enable PagerDuty integration
integrations:
  pagerduty:
    enabled: true
    update_incidents: true
    add_notes: true

PagerDuty Integration Modes

OpsGenie Integration

Investigating OpsGenie Alerts

# Investigate open alerts
holmes investigate opsgenie \
  --opsgenie-api-key $OPSGENIE_API_KEY

# Update alerts with findings
holmes investigate opsgenie \
  --opsgenie-api-key $OPSGENIE_API_KEY \
  --update

# Filter by priority
holmes investigate opsgenie \
  --opsgenie-api-key $OPSGENIE_API_KEY \
  --priority P1,P2

OpsGenie API Key Setup

1. Go to OpsGenie > Settings > API key management
2. Create a new API key with appropriate permissions
3. Store in Kubernetes secret

env:
  - name: OPSGENIE_API_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: opsgenie-api-key

AlertManager Integration

Webhook Receiver Setup

Configure AlertManager to send alerts to HolmesGPT for automatic investigation.

AlertManager Configuration:

# alertmanager.yml
route:
  receiver: default
  routes:
    - match:
        severity: critical
      receiver: holmesgpt
      continue: true

receivers:
  - name: default
    # Your default receiver config

  - name: holmesgpt
    webhook_configs:
      - url: 'http://holmesgpt-holmes.holmesgpt.svc.cluster.local/api/investigate'
        send_resolved: false
        max_alerts: 10

Investigating AlertManager Alerts

# Basic investigation
holmes investigate alertmanager \
  --alertmanager-url http://alertmanager:9093

# With authentication
holmes investigate alertmanager \
  --alertmanager-url http://alertmanager:9093 \
  --alertmanager-username admin \
  --alertmanager-password $ALERTMANAGER_PASSWORD

# Filter by alert name
holmes investigate alertmanager \
  --alertmanager-url http://alertmanager:9093 \
  --alertname "KubePodCrashLooping"

# With custom runbook
holmes investigate alertmanager \
  --alertmanager-url http://alertmanager:9093 \
  -r ~/runbooks/kubernetes.yaml

Jira Integration

Investigating Jira Issues

# Investigate issues in a project
holmes investigate jira \
  --jira-url https://company.atlassian.net \
  --jira-username user@company.com \
  --jira-api-token $JIRA_TOKEN \
  --jira-project OPS

# Filter by JQL
holmes investigate jira \
  --jira-url https://company.atlassian.net \
  --jira-username user@company.com \
  --jira-api-token $JIRA_TOKEN \
  --jira-jql "project = OPS AND status = Open AND labels = incident"

Jira API Token Setup

1. Go to id.atlassian.com/manage-profile/security/api-tokens
2. Create a new API token
3. Store in secret

env:
  - name: JIRA_URL
    value: "https://company.atlassian.net"
  - name: JIRA_USERNAME
    value: "user@company.com"
  - name: JIRA_API_TOKEN
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: jira-api-token

GitHub Integration

Investigating GitHub Issues

# Investigate issues in a repository
holmes investigate github \
  --github-token $GITHUB_TOKEN \
  --github-repo owner/repo

# Filter by labels
holmes investigate github \
  --github-token $GITHUB_TOKEN \
  --github-repo owner/repo \
  --github-labels "bug,priority:high"

# Update issues with analysis
holmes investigate github \
  --github-token $GITHUB_TOKEN \
  --github-repo owner/repo \
  --update

GitHub Token Permissions

Required permissions for the GitHub token:

• repo - Full repository access (for private repos)
• public_repo - Public repository access only
• read:org - Read organization info (optional)

Webhook Configuration

Generic Webhook Output

Send HolmesGPT analysis to any webhook endpoint.

# Send to custom webhook
holmes ask "analyze cluster health" \
  --destination webhook \
  --webhook-url https://your-service.com/api/holmes \
  --webhook-headers "Authorization: Bearer $TOKEN"

Webhook Payload Structure

{
  "timestamp": "2024-01-15T10:30:00Z",
  "source": "holmesgpt",
  "type": "investigation",
  "data": {
    "query": "why is payment-service crashing?",
    "analysis": "Full analysis text...",
    "sections": {
      "alert_explanation": "...",
      "key_findings": "...",
      "root_causes": "...",
      "next_steps": "..."
    },
    "model": "sonnet",
    "token_count": 1500
  }
}

Webhook Helm Configuration

webhooks:
  - name: custom-webhook
    url: "https://your-service.com/api/alerts"
    headers:
      Authorization: "Bearer ${WEBHOOK_TOKEN}"
      Content-Type: "application/json"
    events:
      - investigation_complete
      - critical_alert

CI/CD Integration

GitHub Actions

name: Investigate Deployment Failure

on:
  workflow_run:
    workflows: ["Deploy"]
    types: [completed]

jobs:
  investigate:
    if: ${{ github.event.workflow_run.conclusion == 'failure' }}
    runs-on: ubuntu-latest
    steps:
      - name: Install HolmesGPT
        run: pip install holmesgpt

      - name: Configure kubectl
        uses: azure/k8s-set-context@v3
        with:
          kubeconfig: ${{ secrets.KUBECONFIG }}

      - name: Investigate Failure
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
        run: |
          holmes ask "analyze deployment failure for ${{ github.repository }}" \
            --destination slack \
            --slack-token $SLACK_BOT_TOKEN \
            --slack-channel "#deployments"

GitLab CI

investigate_failure:
  stage: post-deploy
  when: on_failure
  image: python:3.11
  script:
    - pip install holmesgpt
    - |
      holmes ask "why did deployment fail for $CI_PROJECT_NAME?" \
        --destination slack \
        --slack-token $SLACK_BOT_TOKEN \
        --slack-channel "#deployments"
  only:
    - main

Azure DevOps

- task: Bash@3
  displayName: 'Investigate Failure'
  condition: failed()
  inputs:
    targetType: 'inline'
    script: |
      pip install holmesgpt
      export ANTHROPIC_API_KEY=$(ANTHROPIC_API_KEY)
      holmes ask "analyze deployment failure for $(Build.Repository.Name)" \
        --destination slack \
        --slack-token $(SLACK_BOT_TOKEN) \
        --slack-channel "#azure-deployments"

Robusta Integration

HolmesGPT integrates seamlessly with Robusta for automated alert enrichment.

Robusta Playbook Configuration

# robusta_playbook.yaml
customPlaybooks:
  - triggers:
      - on_prometheus_alert:
          alert_name: KubePodCrashLooping
    actions:
      - holmes_investigate:
          model: sonnet
          add_to_timeline: true

Environment Variables for Robusta

env:
  - name: ROBUSTA_ACCOUNT_ID
    valueFrom:
      secretKeyRef:
        name: robusta-secrets
        key: account-id
  - name: ROBUSTA_SIGNING_KEY
    valueFrom:
      secretKeyRef:
        name: robusta-secrets
        key: signing-key

Best Practices

Secret Management

1. Never hardcode credentials - Use Kubernetes Secrets or external secret managers
2. Rotate tokens regularly - Set up token rotation policies
3. Use minimal permissions - Grant only required access levels
4. Audit access - Monitor and log integration usage

Rate Limiting Considerations

Error Handling

# Retry with exponential backoff
RETRY_COUNT=0
MAX_RETRIES=3

until holmes investigate pagerduty --pagerduty-api-key $KEY \
  || [ $RETRY_COUNT -eq $MAX_RETRIES ]; do
  RETRY_COUNT=$((RETRY_COUNT + 1))
  SLEEP_TIME=$((2 ** RETRY_COUNT))
  echo "Retry $RETRY_COUNT in $SLEEP_TIME seconds..."
  sleep $SLEEP_TIME
done

Multi-Integration Setup

# Complete multi-integration values.yaml
env:
  - name: ANTHROPIC_API_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: anthropic-api-key
  - name: SLACK_BOT_TOKEN
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: slack-bot-token
  - name: PAGERDUTY_API_KEY
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: pagerduty-api-key
  - name: GITHUB_TOKEN
    valueFrom:
      secretKeyRef:
        name: holmesgpt-secrets
        key: github-token
  - name: ALERTMANAGER_URL
    value: "http://alertmanager.monitoring.svc.cluster.local:9093"

integrations:
  slack:
    enabled: true
    default_channel: "#holmesgpt"
  pagerduty:
    enabled: true
    update_incidents: true
  alertmanager:
    enabled: true
    webhook: true
  github:
    enabled: true