Pm AI Monitor To Oversight

PM AI Monitor to Oversight

Overview

Use this skill when an AI or agent monitor is expected to notice concerning behavior, policy violations, unsafe tool use, data exposure, or workflow drift. It separates a monitor prediction from observed behavior, human judgement, control action, and downstream outcome. The output is an oversight contract; it is not a working monitor, a safety guarantee, or an automatic block.

When to use

Use this skill when:

• a long-running, tool-using, delegated, or high-impact AI workflow may need asynchronous monitoring or a synchronous gate;
• a team wants to turn a monitor alert into human review, containment, escalation, or a learning writeback;
• a PM needs to decide what the monitor can observe, what it cannot see, how quickly it must signal, and what evidence supports a control claim;
• a monitor score, alert count, quiet period, or apparent recall is being treated as proof of safe behavior;
• the product needs to distinguish the monitored agent's behavior from the monitor's false positive, false negative, blind spot, or manipulation risk.

Use pm-ai-run-to-observability when the primary gap is trace/event identity. Use pm-ai-signal-to-intervention after a live signal is already validated and needs a bounded response. Use pm-ai-incident-to-runbook when multiple signals already indicate a journey-level incident. Use pm-ai-risk-to-control for a broader pre-launch hazard review. Use pm-ai-prompt-injection-to-defense when the primary gap is the attack path or authority defense rather than monitor scope and oversight.

Do not use

Do not use this skill to:

• build or tune a monitor, call a provider, inspect hidden reasoning, store chain-of-thought, create an alert, block a run, or execute a containment;
• label a monitor prediction as ground truth, an alert as an incident, or a quiet period as safety without independent evidence and eligible exposure;
• publish monitor precision, recall, safety, harmlessness, causal, adoption, or production claims from a synthetic fixture, one alert, a benchmark, or a provider statement;
• route an irreversible or high-impact action into automation before the authorized reviewer, approval boundary, and manual fallback are explicit;
• expose private traces, customer data, credentials, tokens, private URLs, hidden reasoning, or sensitive tool arguments in a public packet.

Use Observed, Predicted, Reviewed, Not reviewed, Not run, Not provided, Conflict, Unknown, Need evidence, and Not covered when the evidence is missing.

Core boundary

A monitor alert is a prediction about an observed scope. It is not proof of the target behavior, harm, root cause, or safe completion.

Keep these layers separate:

Workflow

1. Define the oversight decision

Write one sentence:

For [AI workflow and user job], decide whether [monitor signal] should enter [review, contain, gate, escalate, or learn] under [authority and evidence boundary].

Capture:

If scope, authority, or reviewer ownership is unknown, stop at Need evidence.

2. Classify the concern without overclaiming

Use a primary concern class and keep UNKNOWN when multiple paths remain:

These are hypotheses or review categories. The monitor does not get to supply its own ground truth.

3. Bound monitor visibility and coverage

Record whether the monitor sees:

• input, user/developer instructions, retrieved content, files, and tool outputs;
• model output, action proposal, tool arguments, call/result linkage, handoffs, approvals, retries, and final response;
• downstream execution, permission outcome, external receipt, and user state;
• only a sample, a delayed event, a redacted field, an aggregate, or a subset of tenants, tools, locales, environments, or risk classes.

Never infer that an event was safe because it was outside the observation scope. Write Coverage: Unknown when the join cannot be established.

4. Choose the timing and state machine

Separate timing from authority:

Use explicit states:

Not started → Observed → Predicted → Review pending → Reviewed: concern / benign / unknown → Containment candidate → Contained or released → Outcome pending → Outcome reconciled.

Predicted, Contained, and Outcome reconciled are not interchangeable.

5. Evaluate the monitor as a control

Before a safety or release claim, ask for:

1. a positive slice with the concern present;
2. a negative slice with similar benign behavior;
3. adversarial or red-team cases for blind spots and monitor manipulation;
4. independent labels or control outcomes, not only monitor self-labels;
5. coverage and eligible exposure for each slice;
6. alert latency, reviewer capacity, acknowledgement delay, and containment success or failure;
7. privacy, retention, access, redaction, and audit receipts.

Do not turn one matched employee escalation, one benchmark, or one perfect synthetic task into a general recall claim. Treat false-negative coverage as an open question when control evaluation is absent.

6. Route to oversight

This skill may propose a route. It does not send alerts, block actions, revoke credentials, publish an incident, or write to an external system.

7. Write the oversight contract

monitor_id:
target_workflow:
user_job:
concern_class:
monitor_prediction:
observation_scope:
eligible_exposure:
coverage_gaps:
timing: Asynchronous alert | Synchronous gate | Human review
latency_budget:
reviewer_and_authority:
state:
independent_oracle:
positive_slice:
negative_slice:
control_or_red_team_receipt:
privacy_retention_redaction:
route: Observe | Review | Containment candidate | Synchronous gate candidate | Escalate | Learn | Need evidence
stop_rule:
rollback_or_containment:
outcome_status:
not_claimed:

Output contract

Return these sections in order and preserve missingness:

Oversight decision

State the user job, target, concern, signal status, reviewer, authority, and one route in five lines or fewer.

Monitor and observation boundary

List what the monitor can see, what it cannot see, sampling, delay, redaction, tenant/tool scope, and the eligible exposure.

Signal versus outcome

Separate monitor prediction, observed behavior, human review, control action, downstream state, and outcome oracle. Include every missing join.

Review and containment states

Show alert timing, reviewer acknowledgement, escalation, containment candidate, manual fallback, release-to-continue, and outcome-pending states.

Control evidence

List positive, negative, adversarial, and expected-variance cases; independent labels; coverage; denominator; latency; reviewer capacity; privacy; and the false-positive/false-negative evidence status.

Safety-case decision

Choose Observe, Review, Containment candidate, Synchronous gate candidate, Escalate, Learn, or Need evidence. State what the decision does not prove.

Not covered and review ask

List unsupported safety, recall, precision, harmlessness, causal, adoption, and production claims. End with one sanitized evidence request.

Edge cases

• The monitor alerts after the action: preserve the delay and downstream state; do not claim the monitor prevented harm.
• The monitor is silent: report eligible exposure and coverage before interpreting silence; unknown scope is not a clean run.
• The monitor and reviewer disagree: preserve both labels, adjudication, and the criterion version; do not discard disagreement to improve a score.
• The agent follows untrusted content: separate prompt-injection evidence, tool authority, monitor visibility, and resulting action; do not let content instruct the monitor's rubric.
• A monitor wants hidden reasoning: reject the requirement and design around observable actions, outputs, tool calls, approvals, and receipts.
• The action is irreversible: stop at a synchronous gate or authorized human approval candidate with a manual fallback.
• The reviewer is unavailable: use an explicit expiry and safe fallback; do not silently fail open for a high-impact action.
• A positive benchmark is perfect: check task realism, negative cases, adversarial adaptation, coverage, and open-world false negatives.
• Sensitive content is in the trace: retain a private authorized pointer; publish only redacted IDs, classes, versions, and evidence status.

Rationalizations to reject

Final check

Before handing off, confirm:

• target workflow, user job, concern, monitor version, and observation scope are explicit;
• eligible exposure, coverage gaps, sampling, delay, and privacy boundary are explicit;
• monitor prediction, observed behavior, reviewer label, control action, and downstream outcome are separate;
• timing, reviewer authority, acknowledgement, escalation, expiry, fallback, and containment are explicit;
• positive, negative, adversarial, and expected-variance evidence is named or marked Not run;
• false-positive and false-negative evidence is not invented;
• the route has one owner, one stop rule, one next receipt, and one rollback or containment boundary;
• no hidden reasoning, customer data, credential, secret, private URL, or automatic external write entered the package;
• ## Not covered lists unsupported safety, recall, precision, causal, adoption, and production claims.

For the field definitions, source ledger, state matrix, and handoff template, read references/monitor-oversight.md.

Not covered

• No live monitor, alerting system, model call, dashboard, chain-of-thought, customer trace, provider result, control evaluation, or red-team exercise is implemented or claimed.
• A monitor prediction, quiet period, reviewer label, or containment receipt does not prove safety, recall, precision, harmlessness, causality, adoption, or production readiness.
• No action is blocked, paused, revoked, escalated, or written to an external system by this skill.
• No real user, tool argument, credential, private URL, source document, hidden reasoning, or production log belongs in a public fixture.