Open Horizons SRE investigation
Turn one observed reliability symptom into tested hypotheses and an evidence-backed owner without changing workloads.
When to invoke
- Investigate an Open Horizons outage, error spike, latency regression, or unhealthy workload.
- Analyze Prometheus, Grafana, Azure Monitor, Application Insights, Kubernetes events, or logs.
- Establish impact and likely root cause after a deployment.
- Define mitigation requirements, observability gaps, and post-fix verification.
Prerequisites and context
Require the symptom, affected service/environment, time window, user impact, recent changes, and
available telemetry. Use read-only access and redact credentials, tokens, PII, and sensitive payloads.
Procedure
- Establish incident status, severity, impact, start time, affected services, and current mitigation.
- Build a timeline from deployment, alert, metric, event, trace, and log timestamps.
- Check service health and high-signal indicators before collecting broad logs.
- Form two or three causal hypotheses and define one discriminating check for each.
- Run targeted read-only queries, preserve query and time range, and record confirming or
falsifying evidence.
- Identify the most likely owning component and distinguish immediate mitigation from permanent fix.
- Define post-fix verification, alert gaps, SLO impact, and runbook updates; return changes to owners.
Criteria
| Conclusion |
Requirement |
| Confirmed root cause |
Evidence demonstrates mechanism and a discriminating check rules out alternatives |
| Likely cause |
Strong evidence exists but one material dependency remains unverified |
| Hypothesis |
Plausible explanation without sufficient evidence |
| Blocked |
Required telemetry or authorization is unavailable |
Output template
## SRE investigation result
**Status:** INVESTIGATING | MITIGATED | ROOT-CAUSE-CONFIRMED | BLOCKED
**Impact:** <users/services/severity>
**Window:** <start/end and timezone>
### Hypotheses
| Hypothesis | Discriminating check | Evidence | Result |
| --- | --- | --- | --- |
### Outcome
- Most likely owner: <component/agent>
- Mitigation requirement: <action or none>
- Permanent-fix requirement: <action or unknown>
- Verification: <query/check>
- Observability gaps: <list or none>
Limits
- Do not edit code, dashboards, alerts, SLOs, runbooks, manifests, or infrastructure.
- Do not restart, scale, roll back, delete, or reroute workloads.
- Do not treat temporal correlation as causation or expose sensitive telemetry.
- Do not make security conclusions; route indicators to the security reviewer.
Related primitives
| Name |
Type |
Use it when |
open-horizons-sre-investigator |
agent |
A read-only incident owner should execute this method. |
observability-stack |
skill |
Product-specific metric, dashboard, or alert references are needed. |
pipeline-diagnostics |
skill |
Evidence points to CI/CD failure rather than runtime behavior. |
open-horizons-deployment-operator |
agent |
An approved mitigation or rollback must execute. |
Quality gate
1---2name: open-horizons-sre-investigation-23description: Investigates one Open Horizons reliability symptom or incident through read-only telemetry, timeline construction, causal hypothesis testing, ownership assignment, and verification planning. Use when investigating outages, latency, errors, unhealthy workloads, alerts, SLO breaches, or deployment regressions.4---56# Open Horizons SRE investigation78Turn one observed reliability symptom into tested hypotheses and an evidence-backed owner without changing workloads.910## When to invoke1112- Investigate an Open Horizons outage, error spike, latency regression, or unhealthy workload.13- Analyze Prometheus, Grafana, Azure Monitor, Application Insights, Kubernetes events, or logs.14- Establish impact and likely root cause after a deployment.15- Define mitigation requirements, observability gaps, and post-fix verification.1617## Prerequisites and context1819Require the symptom, affected service/environment, time window, user impact, recent changes, and20available telemetry. Use read-only access and redact credentials, tokens, PII, and sensitive payloads.2122## Procedure23241. Establish incident status, severity, impact, start time, affected services, and current mitigation.252. Build a timeline from deployment, alert, metric, event, trace, and log timestamps.263. Check service health and high-signal indicators before collecting broad logs.274. Form two or three causal hypotheses and define one discriminating check for each.285. Run targeted read-only queries, preserve query and time range, and record confirming or29 falsifying evidence.306. Identify the most likely owning component and distinguish immediate mitigation from permanent fix.317. Define post-fix verification, alert gaps, SLO impact, and runbook updates; return changes to owners.3233## Criteria3435| Conclusion | Requirement |36| --- | --- |37| Confirmed root cause | Evidence demonstrates mechanism and a discriminating check rules out alternatives |38| Likely cause | Strong evidence exists but one material dependency remains unverified |39| Hypothesis | Plausible explanation without sufficient evidence |40| Blocked | Required telemetry or authorization is unavailable |4142## Output template4344```markdown45## SRE investigation result4647**Status:** INVESTIGATING | MITIGATED | ROOT-CAUSE-CONFIRMED | BLOCKED48**Impact:** <users/services/severity>49**Window:** <start/end and timezone>5051### Hypotheses52| Hypothesis | Discriminating check | Evidence | Result |53| --- | --- | --- | --- |5455### Outcome56- Most likely owner: <component/agent>57- Mitigation requirement: <action or none>58- Permanent-fix requirement: <action or unknown>59- Verification: <query/check>60- Observability gaps: <list or none>61```6263## Limits6465- Do not edit code, dashboards, alerts, SLOs, runbooks, manifests, or infrastructure.66- Do not restart, scale, roll back, delete, or reroute workloads.67- Do not treat temporal correlation as causation or expose sensitive telemetry.68- Do not make security conclusions; route indicators to the security reviewer.6970## Related primitives7172| Name | Type | Use it when |73| --- | --- | --- |74| `open-horizons-sre-investigator` | `agent` | A read-only incident owner should execute this method. |75| `observability-stack` | `skill` | Product-specific metric, dashboard, or alert references are needed. |76| `pipeline-diagnostics` | `skill` | Evidence points to CI/CD failure rather than runtime behavior. |77| `open-horizons-deployment-operator` | `agent` | An approved mitigation or rollback must execute. |7879## Quality gate8081- [ ] Impact, time window, and recent changes are explicit.82- [ ] Multiple hypotheses and discriminating checks were considered.83- [ ] Queries and evidence are timestamped and sanitized.84- [ ] No repository or workload state changed.85- [ ] Owner, mitigation, permanent-fix requirement, and verification are explicit.
paulasilvatech