Open Horizons SRE investigation
Turn one observed reliability symptom into tested hypotheses and an evidence-backed owner without changing workloads.
When to invoke
- Investigate an Open Horizons outage, error spike, latency regression, or unhealthy workload.
- Analyze Prometheus, Grafana, Azure Monitor, Application Insights, Kubernetes events, or logs.
- Establish impact and likely root cause after a deployment.
- Define mitigation requirements, observability gaps, and post-fix verification.
Prerequisites and context
Require the symptom, affected service/environment, time window, user impact, recent changes, and
available telemetry. Use read-only access and redact credentials, tokens, PII, and sensitive payloads.
Procedure
- Establish incident status, severity, impact, start time, affected services, and current mitigation.
- Build a timeline from deployment, alert, metric, event, trace, and log timestamps.
- Check service health and high-signal indicators before collecting broad logs.
- Form two or three causal hypotheses and define one discriminating check for each.
- Run targeted read-only queries, preserve query and time range, and record confirming or
falsifying evidence.
- Identify the most likely owning component and distinguish immediate mitigation from permanent fix.
- Define post-fix verification, alert gaps, SLO impact, and runbook updates; return changes to owners.
Criteria
| Conclusion |
Requirement |
| Confirmed root cause |
Evidence demonstrates mechanism and a discriminating check rules out alternatives |
| Likely cause |
Strong evidence exists but one material dependency remains unverified |
| Hypothesis |
Plausible explanation without sufficient evidence |
| Blocked |
Required telemetry or authorization is unavailable |
Output template
## SRE investigation result
**Status:** INVESTIGATING | MITIGATED | ROOT-CAUSE-CONFIRMED | BLOCKED
**Impact:** <users/services/severity>
**Window:** <start/end and timezone>
### Hypotheses
| Hypothesis | Discriminating check | Evidence | Result |
| --- | --- | --- | --- |
### Outcome
- Most likely owner: <component/agent>
- Mitigation requirement: <action or none>
- Permanent-fix requirement: <action or unknown>
- Verification: <query/check>
- Observability gaps: <list or none>
Limits
- Do not edit code, dashboards, alerts, SLOs, runbooks, manifests, or infrastructure.
- Do not restart, scale, roll back, delete, or reroute workloads.
- Do not treat temporal correlation as causation or expose sensitive telemetry.
- Do not make security conclusions; route indicators to the security reviewer.
Related primitives
| Name |
Type |
Use it when |
open-horizons-sre-investigator |
agent |
A read-only incident owner should execute this method. |
observability-stack |
skill |
Product-specific metric, dashboard, or alert references are needed. |
pipeline-diagnostics |
skill |
Evidence points to CI/CD failure rather than runtime behavior. |
open-horizons-deployment-operator |
agent |
An approved mitigation or rollback must execute. |
Quality gate
1---2name: open-horizons-sre-investigation3description: Investigates one Open Horizons reliability symptom or incident through read-only telemetry, timeline construction, causal hypothesis testing, ownership assignment, and verification planning. Use when investigating outages, latency, errors, unhealthy workloads, alerts, SLO breaches, or deployment regressions.4---56<!-- Generated from harness/github-copilot/plugins/open-horizons-platform/skills/open-horizons-sre-investigation/SKILL.md by harness/claude-code/scripts/convert_from_copilot.py. Edit the source, not this file. -->78# Open Horizons SRE investigation910Turn one observed reliability symptom into tested hypotheses and an evidence-backed owner without changing workloads.1112## When to invoke1314- Investigate an Open Horizons outage, error spike, latency regression, or unhealthy workload.15- Analyze Prometheus, Grafana, Azure Monitor, Application Insights, Kubernetes events, or logs.16- Establish impact and likely root cause after a deployment.17- Define mitigation requirements, observability gaps, and post-fix verification.1819## Prerequisites and context2021Require the symptom, affected service/environment, time window, user impact, recent changes, and22available telemetry. Use read-only access and redact credentials, tokens, PII, and sensitive payloads.2324## Procedure25261. Establish incident status, severity, impact, start time, affected services, and current mitigation.272. Build a timeline from deployment, alert, metric, event, trace, and log timestamps.283. Check service health and high-signal indicators before collecting broad logs.294. Form two or three causal hypotheses and define one discriminating check for each.305. Run targeted read-only queries, preserve query and time range, and record confirming or31 falsifying evidence.326. Identify the most likely owning component and distinguish immediate mitigation from permanent fix.337. Define post-fix verification, alert gaps, SLO impact, and runbook updates; return changes to owners.3435## Criteria3637| Conclusion | Requirement |38| --- | --- |39| Confirmed root cause | Evidence demonstrates mechanism and a discriminating check rules out alternatives |40| Likely cause | Strong evidence exists but one material dependency remains unverified |41| Hypothesis | Plausible explanation without sufficient evidence |42| Blocked | Required telemetry or authorization is unavailable |4344## Output template4546```markdown47## SRE investigation result4849**Status:** INVESTIGATING | MITIGATED | ROOT-CAUSE-CONFIRMED | BLOCKED50**Impact:** <users/services/severity>51**Window:** <start/end and timezone>5253### Hypotheses54| Hypothesis | Discriminating check | Evidence | Result |55| --- | --- | --- | --- |5657### Outcome58- Most likely owner: <component/agent>59- Mitigation requirement: <action or none>60- Permanent-fix requirement: <action or unknown>61- Verification: <query/check>62- Observability gaps: <list or none>63```6465## Limits6667- Do not edit code, dashboards, alerts, SLOs, runbooks, manifests, or infrastructure.68- Do not restart, scale, roll back, delete, or reroute workloads.69- Do not treat temporal correlation as causation or expose sensitive telemetry.70- Do not make security conclusions; route indicators to the security reviewer.7172## Related primitives7374| Name | Type | Use it when |75| --- | --- | --- |76| `open-horizons-sre-investigator` | `agent` | A read-only incident owner should execute this method. |77| `observability-stack` | `skill` | Product-specific metric, dashboard, or alert references are needed. |78| `pipeline-diagnostics` | `skill` | Evidence points to CI/CD failure rather than runtime behavior. |79| `open-horizons-deployment-operator` | `agent` | An approved mitigation or rollback must execute. |8081## Quality gate8283- [ ] Impact, time window, and recent changes are explicit.84- [ ] Multiple hypotheses and discriminating checks were considered.85- [ ] Queries and evidence are timestamped and sanitized.86- [ ] No repository or workload state changed.87- [ ] Owner, mitigation, permanent-fix requirement, and verification are explicit.
paulasilvatech