Expose job failures in one place
Failures that live in the wrong inbox are failures nobody acts on. Put every scheduled and background job's last known status on one page, so "what is broken right now" is one glance, not a search.
Procedure
- Choose the single surface: a dashboard panel, a status page, or a table in the operations DB. One, not three.
- Have each job write a row on every run to a shared table:
CREATE TABLE job_status(
job text PRIMARY KEY, last_started timestamptz,
last_finished timestamptz, status text, rows_affected bigint,
detail text, runbook text);
- Make every job report through the same helper so the row shape is identical — a per-job bespoke format defeats the single view.
- Derive the status from evidence:
status='ok' means the run finished and produced the expected shape, not merely that it exited 0.
- Show three columns at minimum: last finished (age), status, and rows affected. Compare age to the schedule to catch silent stops.
- Sort by "most broken first": stale and failing at the top, healthy at the bottom, so the page answers "what needs me" immediately.
- Include the owning team and the runbook link per row.
- Do not require a human to open N tools: pull external jobs (cron on other hosts, cloud schedulers, CI crons) into the same table via their APIs.
- Alarm from the aggregate ("more than 0 jobs failing for 10m") so a job with no heartbeat monitor is still caught.
- Keep history: prune only rows older than your retention window, so a weekly failing job is visible on Monday from Saturday's rows.
- Test the surface by forcing a job to fail and confirming the row and the alarm both update.
Pitfalls
- Failure alerts split across three channels, so each team sees only its own and nobody sees the whole.
- Storing only success/failure with no timestamp, so a job that has not run for a week looks fine.
- A dashboard that requires a query per job and is too slow to trust.
- Per-job log formats, so the aggregate view needs a different parser per job.
- Reporting exit code 0 for a run that did nothing (zero rows) — the dangerous silent success.
- A status page that is read-only and unauthenticated, leaking internal service names.
Verification
psql -c "select job, now()-last_finished as age, status, rows_affected
from job_status order by (status<>'ok') desc, age desc limit 20"
# pass: every known job appears, the failing and stale ones sort first
Report the surface URL, the jobs it covers, and the query behind it.
1---2name: expose-job-failures-in-one-place3description: Use when failed jobs are scattered across email, logs, and dashboards so nobody has a single view. Builds one surface listing every automation's last run and status.4metadata:5 source: https://skillmd.com/skills/jiayaoqijia/expose-job-failures-in-one-place6---78# Expose job failures in one place910Failures that live in the wrong inbox are failures nobody acts on. Put every scheduled and background job's last known status on one page, so "what is broken right now" is one glance, not a search.1112## Procedure13141. Choose the single surface: a dashboard panel, a status page, or a table in the operations DB. One, not three.152. Have each job write a row on every run to a shared table:16 CREATE TABLE job_status(17 job text PRIMARY KEY, last_started timestamptz,18 last_finished timestamptz, status text, rows_affected bigint,19 detail text, runbook text);203. Make every job report through the same helper so the row shape is identical — a per-job bespoke format defeats the single view.214. Derive the status from evidence: `status='ok'` means the run finished and produced the expected shape, not merely that it exited 0.225. Show three columns at minimum: last finished (age), status, and rows affected. Compare age to the schedule to catch silent stops.236. Sort by "most broken first": stale and failing at the top, healthy at the bottom, so the page answers "what needs me" immediately.247. Include the owning team and the runbook link per row.258. Do not require a human to open N tools: pull external jobs (cron on other hosts, cloud schedulers, CI crons) into the same table via their APIs.269. Alarm from the aggregate ("more than 0 jobs failing for 10m") so a job with no heartbeat monitor is still caught.2710. Keep history: prune only rows older than your retention window, so a weekly failing job is visible on Monday from Saturday's rows.2811. Test the surface by forcing a job to fail and confirming the row and the alarm both update.2930## Pitfalls3132- Failure alerts split across three channels, so each team sees only its own and nobody sees the whole.33- Storing only success/failure with no timestamp, so a job that has not run for a week looks fine.34- A dashboard that requires a query per job and is too slow to trust.35- Per-job log formats, so the aggregate view needs a different parser per job.36- Reporting exit code 0 for a run that did nothing (zero rows) — the dangerous silent success.37- A status page that is read-only and unauthenticated, leaking internal service names.3839## Verification4041 psql -c "select job, now()-last_finished as age, status, rows_affected42 from job_status order by (status<>'ok') desc, age desc limit 20"43 # pass: every known job appears, the failing and stale ones sort first4445Report the surface URL, the jobs it covers, and the query behind it.