gabrielmoreira Slack Network Alerts
Slack OAuth Scopes Used
| Scope | Purpose |
|---|---|
chat:write |
Post alert messages to channels |
chat:write.customize |
Post with custom username/avatar per alert severity |
reactions:write |
Add emoji reactions to acknowledge/track alerts |
files:write |
Attach diagrams, reports, and log excerpts |
app_mentions:read |
Respond when users @mention NetClaw |
channels:join |
Join alert channels automatically |
Alert Severity Formatting
Use Slack Block Kit formatting to visually distinguish severity levels.
CRITICAL Alert Format
:rotating_light: *CRITICAL — Device Unreachable*
*Device:* R1 (10.1.1.1)
*Detected:* 2024-02-21 14:32 UTC
*Impact:* 47 downstream routes affected, 3 OSPF adjacencies lost
*Symptoms:*
• Ping 0% success (was 100%)
• OSPF neighbor state: DOWN
• BGP peer 10.1.1.2: IDLE
*Recommended Action:*
1. Check physical connectivity / power
2. Verify interface status on upstream device
3. Check for reload reason if device recovers
_Thread for investigation updates →_
HIGH Alert Format
:warning: *HIGH — Interface Flapping*
*Device:* SW1 | *Interface:* Gi1/0/24
*Flap Count:* 12 in last 30 minutes
*Connected To:* R2:Gi0/0/1 (discovered via CDP)
*Action Required:* Investigate physical layer — CRC errors detected (47 in 5 min)
WARNING Alert Format
:large_yellow_circle: *WARNING — Memory Pressure*
*Device:* R2 | *Memory:* 82% used (2.1G / 2.6G)
*Trend:* Increased from 71% over 24h
*Top Consumer:* BGP Router (843 MB)
_Monitor — no immediate action required_
INFORMATIONAL Alert Format
:information_source: *INFO — Configuration Change Detected*
*Device:* R1 | *User:* admin (10.0.0.50)
*Time:* 2024-02-21 09:15 UTC
*Change:* 3 lines modified in running-config
_Captured in GAIT session abc123_
Alert Workflows
Health Check Alert Flow
After running a health check (pyats-health-check skill), post results to the designated channel:
- Run the full health check procedure
- Determine overall severity (CRITICAL > HIGH > WARNING > HEALTHY)
- Format the alert using the appropriate severity template above
- Post to the network operations channel
- If CRITICAL or HIGH, add :rotating_light: or :warning: reaction to the message
- If CRITICAL, start a thread with detailed diagnostics
Threshold-Triggered Alerts
When any health check metric exceeds thresholds:
| Metric | WARNING | HIGH | CRITICAL |
|---|---|---|---|
| CPU 5min avg | > 50% | > 75% | > 90% |
| Memory used | > 70% | > 85% | > 95% |
| Interface errors | > 0 CRC | > 100/min | Resets incrementing |
| Packet loss | > 0% | > 5% | > 20% |
| NTP offset | > 100ms | > 500ms | > 1s or unsync |
| BGP peer | Flapping | 1 peer down | Multiple peers down |
| OSPF adjacency | Flapping | 1 adj lost | Area partition |
Multi-Device Fleet Alert Summary
After a fleet-wide health check, post a single summary:
:bar_chart: *Fleet Health Summary — 2024-02-21 14:00 UTC*
*Devices Checked:* 8 | :red_circle: 1 CRITICAL | :warning: 2 HIGH | :large_yellow_circle: 3 WARNING | :white_check_mark: 2 HEALTHY
┌──────────┬──────────┬──────┬────────┬──────────┬─────────────┐
│ Device │ CPU │ Mem │ Intf │ NTP │ Overall │
├──────────┼──────────┼──────┼────────┼──────────┼─────────────┤
│ R1 │ :white_check_mark: 12% │ :warning: 78%│ :white_check_mark: │ :white_check_mark: │ :warning: WARNING │
│ R2 │ :white_check_mark: 8% │ :white_check_mark: 45%│ :red_circle: Gi2 │ :white_check_mark: │ :red_circle: CRITICAL │
│ SW1 │ :warning: 67% │ :white_check_mark: 52%│ :white_check_mark: │ :red_circle: unsync │ :red_circle: HIGH │
└──────────┴──────────┴──────┴────────┴──────────┴─────────────┘
_Details in thread →_
Security Alert Flow
After a security audit (pyats-security skill), post findings:
:shield: *Security Audit — R1*
*Findings:* 2 CRITICAL | 2 HIGH | 3 MEDIUM | 1 LOW
:red_circle: *CRITICAL:*
• [C-001] SSHv1 enabled — MITM vulnerability
• [C-002] No VTY access-class — management plane exposed
:warning: *HIGH:*
• [H-001] No OSPF authentication on Gi1
• [H-002] SNMP community 'public' with no ACL
_Full report attached as file →_
Reaction-Based Acknowledgment
Use emoji reactions to track alert status:
| Reaction | Meaning |
|---|---|
| :eyes: | Alert acknowledged — someone is looking |
| :wrench: | Fix in progress |
| :white_check_mark: | Resolved |
| :no_entry: | False positive / suppressed |
| :hourglass: | Waiting on change window |
| :busts_in_silhouette: | Escalated to team |
When a user reacts with :eyes: on an alert, NetClaw can respond in the thread:
Acknowledged by @engineer1 at 14:35 UTC. Tracking in thread.
File Attachments
Attach supporting data as Slack files:
- Topology diagrams — Draw.io PNG/SVG after topology discovery
- Health reports — Full text report as .txt attachment
- Config diffs — Pre/post change diffs
- Log excerpts — Relevant log sections for investigation
- Markmap mind maps — Protocol hierarchy visualizations
Channel Strategy
| Channel | Purpose | Alert Types |
|---|---|---|
| #netclaw-alerts | Critical/High alerts only | Device down, security critical |
| #netclaw-reports | Scheduled reports | Health checks, audits, reconciliation |
| #netclaw-changes | Change notifications | Config changes, CR updates |
| #netclaw-general | General interaction | Ad-hoc queries, help |
Message Threading
- Always thread follow-up investigation steps under the original alert
- Post the initial alert as a top-level message
- Post diagnostics, show command outputs, and resolution steps as thread replies
- Post final resolution summary as both a thread reply and a top-level message
Integration with GAIT
Every alert should reference the GAIT session:
:information_source: _Tracked in GAIT session `abc123` — commit `def456`_