Operational Value Designer

Operational Value Grader

Design one deterministic operational-value grader that reports absolute operational attainment for each workflow run. Keep exactly one authoritative primary value, and declare any normalized diagnostic metrics separately.

Operational value is the degree to which the workflow's intended repository outcome is attained for the opportunity assigned to a run, demonstrated by accepted repository evidence under a frozen contract. It is not execution quality, output volume, safe-output creation, or an agent's assessment.

Output

Create one executable evaluator at:

.github/graders/WORKFLOW-NAME-operational-value.sh

Configure the workflow:

graders:
  operational-value:
    run: .github/graders/WORKFLOW-NAME-operational-value.sh

The grader's primary operational value (value) is absolute attainment in [0,1]. A comparable frozen baseline may be reported separately as baselineValue; gh-aw derives deltaFromBaseline. Never define the primary operational value as a difference from baseline.

Design Procedure

1. Validate OWNER/REPO and resolve .github/workflows/WORKFLOW-NAME.md. Do not infer inputs from the workspace or remotes.
2. Recover adoption-time intent from the workflow's first commit and first parent. Use only adoption-time workflow content and pre-adoption evidence to choose opportunities, accepted evidence, formulas, targets, or a baseline.
3. Define how every workflow run binds to one operational case:
   • produce a stable opportunityKey;
   • prevent overlapping ownership where possible;
   • preserve repeated keys when duplicate runs target the same opportunity so downstream analysis can cluster or deduplicate them;
   • treat reruns with the same GitHub run ID as the same subject.
4. Freeze accepted evidence, evidence repositories, matching rules, zero-versus-missing behavior, and maturesAt computation.

• Declare only the workflow permission scopes required to collect that evidence. The evaluator receives GH_TOKEN with the agent job's declared permissions; gh-aw does not add evidence permissions automatically.

5. Choose exactly one direct primary metric in [0,1]. Higher must always mean greater attainment. Optional diagnostic metrics may provide normalized outcome context, but must remain separate rather than being combined into the primary value. Keep trace graders and activity counts separate.

6. If comparable pre-adoption evidence exists, score it with the same metric and freeze it under baseline. Otherwise use attainment-only with a null baseline value.

7. Implement the evaluator interface below and run:

.github/skills/operational-value-designer/scripts/verify-operational-value-evaluator.sh .github/graders/WORKFLOW-NAME-operational-value.sh gh aw compile .github/workflows/WORKFLOW-NAME.md

## Evaluator Interface

The evaluator uses Bash 3.2-compatible Bash plus `jq` and supports:

- `--definition`: print the frozen schema-version 4 contract.
- `--metric`: read one evidence object on stdin and print a deterministic number in `[0,1]` or `null`.
- `--grade-run`: read a run request on stdin and print one operational-value observation.

`--grade-run` receives:

```json
{
  "schemaVersion": 1,
  "run": {
 "id": "12345",
 "attempt": 1,
 "repository": "OWNER/REPO",
 "workflow": "Workflow name",
 "ref": "refs/heads/main",
 "sha": "...",
 "eventName": "schedule",
 "createdAt": "2026-08-23T11:58:00Z"
  },
  "evidenceAt": "2026-08-23T12:00:00.000Z",
  "case": null,
  "event": null,
  "config": {}
}

It returns:

{
  "value": 0.75,
  "opportunityKey": "issue:42",
  "case": {"issue": 42},
  "evidenceCutoff": "2026-08-23T12:00:00.000Z",
  "maturesAt": "2026-08-30T12:00:00.000Z",
  "provenance": [
    {"repository": "OWNER/REPO", "kind": "issue", "ref": "42"}
  ],
  "diagnostics": {"repository-health": 0.8}
}

The function must cap evidenceCutoff at the earlier of evidenceAt and maturesAt. A run is never intrinsically pending: the operational value is an as-of observation and may be recomputed until maturity. After maturity, the cap makes the result stable.

Regrade a Historical Run

Recompute a run at an explicit evidence time with the same evaluator used by the original run:

gh aw graders operational-value RUN-ID \
  --evidence-at 2026-08-30T12:00:00.000Z \
  --json

Add --repo [HOST/]OWNER/REPO to select the GitHub host for the current repository checkout. The command downloads the original grader artifact, reuses its operational case and complete run subject, and refuses to execute unless the archived evaluator matches both digest records and the evaluator at the recorded commit in the trusted checkout. It prints a new observation and never modifies the original artifact.

Build a Historical Report

Build a report across all completed runs from the contract's adoption time:

gh aw graders operational-value report WORKFLOW-NAME

This writes JSON, SVG, and Markdown artifacts under reports/operational-value. It applies the current evaluator digest to every run for comparability and backfills pre-grader runs by passing their run subject with case: null and event: null. The evaluator must reconstruct assignment from accepted evidence when the case is null.

Mature numeric results are cached by repository, workflow, evaluator digest, and Monday-based UTC week. Unavailable, immature, and failed evaluations are retried. Independent weeks are evaluated concurrently; use --concurrency to control evaluator executions (the default is 8). Use --refresh to bypass cached observations, --until to choose the evidence endpoint, and --output or --cache-dir to relocate generated files.

Reports preserve all run-level points. Weekly primary means retain the latest observation per repeated opportunityKey within each week. Declared diagnostics are plotted independently using their latest or mean aggregation. Missing evidence remains null, and changes over time or from a baseline do not establish causation.

Definition Contract

--definition must contain:

• schemaVersion: 4 and grader: "operational-value";
• repository, workflow name, source path, and adoption commit/time;
• operational-value statement;
• evidence opportunity, assignment, accepted evidence, repositories, collection, maturation, zero rule, and missing rule;
• one primary metric with formula and validation examples;
• optional diagnosticMetrics, each with a unique ID, name, formula, higher_is_better direction, and latest or mean aggregation;
• baseline mode, value, cutoff, and provenance.

For baseline-comparable, baseline value must be in [0,1] and have immutable provenance. For attainment-only, baseline value and cutoff must be null.

Interpretation Rules

• value answers “what operational value did this run attain for its assigned opportunity?”
• deltaFromBaseline answers “how far is this observation above or below the frozen pre-adoption reference?”
• Diagnostics provide separate normalized context and never contribute to value.
• Neither establishes that the workflow caused the outcome.
• Compare runs only under the same evaluator digest and evidence horizon.
• Identify a replayed observation by (runId, evaluatorDigest, evidenceAt).
• Do not treat repeated observations of one run, duplicate opportunity keys, or overlapping state windows as independent samples.