Monitoring Graphs

📈 Monitoring & Incident Graphing Skill 📊

This skill guides the agent through identifying high-signal metrics, extracting data efficiently, and creating professional annotated graphs.

🚨 DATA INTEGRITY: NEVER MAKE UP NUMBERS

• ONLY REAL DATA: Absolutely NEVER fake data, interpolate guesses, or hard-code values (like forcing a firewall block period to exactly 0) just to make a graph look "correct" or align with a narrative.
• The Data Is The Source Of Truth: If there is an outage, the raw metrics or the absence of metrics must prove it. Show the real raw availability.
• Handling Complete Blackouts: If a full blackout causes missing data points from the API, do not invent data. Instead, use Pandas reindex to fill the missing continuous time intervals with 0s.

🚀 Workflow

1. Metric Selection & Efficient Extraction 🕵️‍♀️

• Standard Time: Always use UTC for all timestamps by default. 🌐
• Performance Tip: Cloud Monitoring can be VERY SLOW for high-res data. 🐢
• Execution:
  1. Use Monitoring MCP to find relevant metrics for the incident time window.
  2. Download the data as CSV (e.g., out/incident/metric.csv) or directly parse JSON.
  3. Use a sub-agent (generalist) for large datasets to keep the main session history lean.
• Reference: See archetypes.md for "Apple-to-Apple" strategies and granularity tables.

2. Baseline Graph (Draft) 🖼️

• Rule: Always generate a raw "Draft" graph first to confirm the data shows a clear signal.
• Command:

  uv run ./scripts/plot_archetype.py --csv data.csv --out draft.png --title "Title"
  

• Verification: Ensure the graph is not "flat." The X-axis must explicitly show "Time (UTC)" so there is no ambiguity! 🕰️

3. Annotated Graph (Final) 🔴

• Requirement: Only proceed after the user confirms the draft is "good."
• Command:

  # Times are assumed to be UTC unless --timezone is specified
  uv run ./scripts/plot_archetype.py \
    --csv data.csv \
    --out final.png \
    --final \
    --start "YYYY-MM-DD HH:MM:SS" \
    --detect "YYYY-MM-DD HH:MM:SS" \
    --mitigate "YYYY-MM-DD HH:MM:SS" \
    --end "YYYY-MM-DD HH:MM:SS"
  

• Styling Guidelines (Annotations):
  • Red (#d93025): Breakages, outages, faults, or incident start/end. Use different line thicknesses (linewidth=1 to 3) or styles (:, --, -.) to denote severity.
  • Yellow (#f9ab00): Human detection or alert triggered.
  • Green (#1e8e3e): Fix, mitigation, or resolution applied.

📝 Safeguards & Tips

• Scaling Lines: Use --hmin and --hmax to show Cloud Run/GKE scaling limits (min/max instances).
• Auto-Coloring: Blue for ingress, Green for egress for Network I/O.
• Timeouts: Use timeout 60 for data extraction tool calls to avoid infinite loops. Also prefer MCP calls to Monitoring/Logging (faster) vs gcloud commands (slower).
• Clear Timing: Ensure that looking at the time variable allows a person to know EXACTLY what that time is.
  • For instance, if time is 03:00 ensure the TZ is specified in the X-axis label.
  • If the graph only contains HOURS (ie, all happens within a day) and no days, then ensure the day is also written in the X-axis label.
• Correlation in graphs: if you need to prove a time-correlation between heterogeneous graphs (eg error logs volume vs increase in traffic) remember:
  • Time is key - keep a single X-axis with consistent time across the 2+ graphs.
  • Example: assets/sample_correlation_data/improved_incident_graphs.png
• Revision and multiple files. Since images are very succulent, if you create a new image do NOT overwrite the old image, rather add _rev1, _rev2, .. to the file name.
  • If an image is proven to be wrong by further investigation, do NOT delete it, rather add it to Trash/ within the original folder, and possibly rename it to why it was wrong in the filename.

🐍 Juicy Python Patterns (From Real Investigations)

See scripts/reference_dual_plot.py for a complete example of parsing JSON and plotting.

1. Handling Missing Data (Reindexing)

# Force 0s for missing data in outages (Pandas)
full_range = pd.date_range(start=df.index.min(), end=df.index.max(), freq='1min')
df = df.reindex(full_range, fill_value=0)

2. Smoothing "Golden Signals"

# 5min rolling average for smoother 'Golden Signal' look
vol_smooth = grouped['Total'].rolling(5, min_periods=1).mean().fillna(0)
avail_smooth = grouped['Availability'].rolling(5, min_periods=1).mean().fillna(0)

3. Dual Axes (Availability & Traffic)

fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(15, 10), sharex=True)
ax1.plot(df.index, df.avail, color='#d93025', label='Success Rate') # Top: %
ax1.fill_between(df.index, df.avail, color='#d93025', alpha=0.1)
ax2.plot(df.index, df.vol, color='#1a73e8', label='Traffic') # Bottom: Volume
ax2.fill_between(df.index, df.vol, color='#1a73e8', alpha=0.1)