Oodle Monitors

Oodle Monitors — CRUD and Alerting

This skill teaches the agent to build, validate, and update Oodle monitors so that alerts are actionable, scoped, and free from flapping.

Prerequisites

# Install + configure (see oodle-cli skill)
brew install oodle-ai/oodle/oodle
oodle configure
# or
export OODLE_API_KEY=<key>
export OODLE_INSTANCE=<instance>
export OODLE_DEPLOYMENT=<url>

Confirm authentication and that at least one monitor list call succeeds before creating new monitors:

oodle monitors list -o json | jq 'length'

Command Execution Order

Before running any oodle command:

1. Check whether the required resource ID or name is already in context.
2. If not, run the discovery command (e.g., oodle monitors list -o json).
3. If the result is ambiguous, ask the user to confirm before proceeding.
4. Run the target command with the resolved ID.
5. Do not run speculative commands (e.g., do not delete without first get-ing the resource).

Quick Reference

Common Operations

Listing monitors

# ✅ CORRECT — JSON output for scripting
oodle monitors list -o json

# ✅ CORRECT — narrow with --status to find only firing alerts
oodle monitors list --status alert -o json

# ✅ CORRECT — narrow with --labels for a specific team
oodle monitors list --labels env=prod,team=platform -o json

# ❌ WRONG — pulling everything then grepping
oodle monitors list | grep CPU

Reading a monitor before changing it

# ✅ CORRECT — fetch full JSON, edit, then update
oodle monitors get mon_123 -o yaml > monitor.yaml
$EDITOR monitor.yaml
oodle monitors update mon_123 -f monitor.yaml

# ❌ WRONG — building update payload from memory; overwrites unrelated fields
oodle monitors update mon_123 -f <(echo '{"options":{"thresholds":{"critical":90}}}')

Creating a monitor

A complete, valid monitor JSON:

{
  "name": "High CPU on web servers",
  "type": "metric alert",
  "query": "avg(last_5m):avg:system.cpu.user{env:prod,service:api} by {host} > 80",
  "message": "CPU above 80% on {{host.name}}. Runbook: https://runbooks.example.com/cpu\n@slack-ops",
  "labels": {"team": "platform", "env": "prod"},
  "options": {
    "thresholds": {
      "critical": 80,
      "critical_recovery": 70,
      "warning": 60,
      "warning_recovery": 50
    }
  }
}

# ✅ CORRECT
oodle monitors create -f monitor.json

# ❌ WRONG — no `type`, no `options.thresholds`, monitor will be rejected
oodle monitors create -f <(echo '{"name":"x","query":"y"}')

Updating a monitor

# ✅ CORRECT — get → edit → update
oodle monitors get mon_123 -o json > monitor.json
jq '.options.thresholds.critical = 85' monitor.json > monitor.new.json
oodle monitors update mon_123 -f monitor.new.json

# ❌ WRONG — sending only the changed field; missing fields become null
oodle monitors update mon_123 -f <(echo '{"options":{"thresholds":{"critical":85}}}')

Deleting a monitor

# ✅ CORRECT — verify first, then delete
oodle monitors get mon_123 -o json > /dev/null
oodle monitors delete mon_123 --force

# ❌ WRONG — speculative delete by name match
oodle monitors delete "$(oodle monitors list | grep CPU | head -1 | awk '{print $1}')" --force

Best Practices

Use a last_5m (or longer) evaluation window, not last_1m

Short windows cause alert flapping on normal traffic spikes. Use last_5m minimum for production alerts, last_15m for noisy metrics.

# ✅ CORRECT
"query": "avg(last_5m):avg:system.cpu.user{env:prod} by {host} > 80"

# ❌ WRONG — flaps on every brief spike
"query": "avg(last_1m):avg:system.cpu.user{env:prod} by {host} > 80"

Scope queries with explicit labels — never use {*}

{*} matches every series in the system and produces alerts for resources you don't own.

# ✅ CORRECT — scoped to a specific env + service
"query": "avg(last_5m):avg:system.cpu.user{env:prod,service:api} by {host} > 80"

# ❌ WRONG — alerts on every host in the org
"query": "avg(last_5m):avg:system.cpu.user{*} > 80"

Always set *_recovery thresholds below the trigger thresholds

Without recovery thresholds the monitor stays in alert state until the metric drops below the critical threshold exactly — small oscillations keep the alert active forever.

# ✅ CORRECT — clear recovery band (10pt below trigger)
"thresholds": {"critical": 80, "critical_recovery": 70, "warning": 60, "warning_recovery": 50}

# ❌ WRONG — no recovery values; monitor never cleanly recovers
"thresholds": {"critical": 80, "warning": 60}

Put a runbook URL and an @notifier handle in message

Alerts without an action are noise. Every monitor message must answer "what do I do?" and "who is paged?".

# ✅ CORRECT
"message": "CPU above 80% on {{host.name}} (env=prod, service=api).\nRunbook: https://runbooks.example.com/cpu\n@slack-ops @pagerduty-platform"

# ❌ WRONG — no actionable content, no routing
"message": "CPU is high"

Tag every monitor with at least team and env labels

Labels are how notification policies route alerts and how oodle monitors list --labels ... filters work.

# ✅ CORRECT
"labels": {"team": "platform", "env": "prod", "service": "api"}

# ❌ WRONG
"labels": {}

Failure Handling

References

• Oodle CLI repo
• Oodle docs