Pm AI Context To Contract

PM AI Context to Contract

Use this skill when an AI or agent may be seeing too much, too little, the wrong thing, or an untrusted thing for a user job. It turns a context change into a reviewable contract: what enters the model's working context, why it is there, how it is selected, which source wins when facts conflict, what must be removed or redacted, and how the team will know the change helped.

The output is a product decision packet, not a prompt rewrite, context-builder implementation, model benchmark, provider recommendation, or production guarantee. Context is treated as a changing system boundary rather than a single prompt string.

When to use

Use it when:

• an agent change adds or removes system instructions, retrieved knowledge, tools, MCP servers, conversation history, memory, task state, or examples;
• a result changes after context grows, a session crosses a compaction point, a tool returns verbose data, or a source becomes stale;
• a team needs to decide between preloaded context and just-in-time retrieval, raw history and a structured summary, or a full tool set and a task subset;
• a user-facing AI feature needs source visibility, data-use boundaries, uncertainty, human control, or a safe fallback;
• a context optimization reduces tokens or latency but its quality, privacy, trust, or task-completion effect is not yet known;
• a long-running agent needs a handoff capsule or durable state instead of depending on a hidden conversation window.

Use pm-ai-evaluation-plan when the primary work is a quality rubric and test dataset. Use pm-ai-cost-to-guardrail when the primary decision is economics or latency per successful outcome. Use pm-ai-trace-to-regression when one concrete run already failed. Use pm-ai-task-boundary when the open question is what a person versus an AI should own. Use pm-decision-to-spec after the context decision needs a broader product and UX handoff.

Do not use this skill to expose hidden chain-of-thought, read private customer content, install a retrieval or memory system, call a model or provider, change a tool permission, choose a vendor, or claim that more context is better without outcome evidence.

Guardrails

1. Frame one user job, success oracle, decision owner, observation window, and context change. Do not review a context bundle without naming the outcome it is meant to support.
2. Inventory six components separately: instructions, knowledge, tools, memory, state, and query. Do not hide state in memory or treat tool descriptions and tool results as harmless metadata; both enter context.
3. For every source record authority, provenance, freshness, version, privacy class, tenant or access boundary, selection rule, and evidence status. If a field is missing, write Not provided rather than guessing.
4. Declare inclusion and exclusion rules. A source that is relevant in theory may still be omitted because it is stale, redundant, unauthorized, untrusted, too large, or outside the job scope.
5. Set a proposed input budget by component and reserve room for the query, output, tool results, retries, and safe fallback. Never hard-code a provider context limit as a universal product truth.
6. Make conflict resolution explicit. A newer document does not automatically outrank a policy, and a retrieved paragraph does not become an instruction merely because it is inside the context.
7. Treat retrieved pages, files, tool results, issue comments, and memory as potentially untrusted data. Prompt injection, hidden instructions, cross-tenant content, secrets, and unsafe links are release blockers when they can change behavior or leak data.
8. Keep user-visible context visibility proportional to risk: show source identity, time or freshness, tool activity, limitations, and controls when useful; never reveal private raw content or hidden chain-of-thought as a trust substitute.
9. Test context changes with the same success oracle, relevant slices, a negative set, and component ablations. Fewer tokens or a prettier answer is not proof of better context.
10. Keep raw evidence separate from distilled context. A summary can guide the next call, but high-risk decisions need a path back to the original source or trace.
11. Record sanitized context-manifest fields for diagnosis: component IDs, source IDs, versions, counts, token ranges, selection reasons, truncation, retrieval, compaction, fallback, and outcome. Do not store raw prompts, secrets, customer text, or private URLs in a general analytics packet.
12. Keep a reversible route visible: restore the last known-good context version, narrow the source set, disable a memory write, expose fewer tools, ask for clarification, or hand the job to a person.

Core definitions

Use these calculations only after scope is declared:

available_context_budget
  = declared_input_budget - reserved_output_and_tool_overhead

component_budget_share
  = selected_component_budget / available_context_budget

context_retention_rate
  = critical_facts_preserved_after_compaction / critical_facts_before_compaction

If the denominator, critical-fact set, input budget, or version boundary is missing, write Not measurable. Do not infer context quality from token count.

Workflow

1. Frame the decision and user outcome

Write one sentence:

We need to decide whether ... context change can support the user job ... within ... trust, privacy, quality, latency, and cost boundaries.

Name the current workaround, decision owner, affected journey, success oracle, observation window, baseline context version, proposed context version, and the evidence that could change the decision.

2. Map the six context components

Create one row for each source or source family. At minimum record:

Keep memory separate from state: a preference or durable note is not the same thing as the current approval, task checkpoint, or world state.

3. Register source authority, freshness, and trust

Build a source ledger with IDs such as S-001, T-004, M-002, and Q-001. For every row record source, owner, retrieval or write time, version, access scope, effective window, and evidence status. Label user text, retrieved text, tool output, and memory as data unless the product has an explicit rule that promotes a field to an instruction.

Write a conflict rule before reading the result. For example: current approved policy outranks an old article; a user claim can describe intent but cannot override a system safety rule; a tool result can supply data but cannot change its own permission. These are examples, not universal defaults.

4. Set the budget and selection policy

Declare an input budget with a safety margin and reserve room for the query, expected output, tool responses, retries, and fallback. Allocate budget by component, then define what happens when a component exceeds it:

• retrieve fewer, more diverse, higher-signal sources;
• preserve identifiers and citations while dropping redundant prose;
• expose a task-specific subset of tools;
• compact history into a versioned handoff capsule;
• move durable state to an approved artifact and load it just in time;
• stop and ask for clarification instead of silently truncating critical rules.

Do not turn a budget into a provider recommendation. The PM decision is which information the job needs and what can safely be omitted.

5. Design dynamic context and long-horizon continuity

Choose among preloaded, just-in-time, hybrid, compacted, cached, or artifact-backed context. Record the tradeoff between freshness, latency, access control, retrieval failure, and observability. A compaction rule must state what it preserves: goals, decisions, constraints, source IDs, unresolved risks, approvals, and next action.

If memory can be written, define who or what may write it, the validation gate, retention or deletion path, provenance, and how a user can correct or reset it. If a session crosses a context version, start a new trace segment instead of blending old and new state.

6. Map failure, privacy, and recovery states

At minimum cover:

empty context -> source unavailable -> stale source -> conflicting source
-> oversized context -> compaction -> untrusted tool/retrieval result
-> privacy or permission mismatch -> safe fallback or human handoff

For each state define what the user sees, what the system records, what is blocked, how to recover, and whether the result can count as a completed job. Do not hide an omission, silent truncation, cross-tenant mismatch, or memory write behind a successful-looking final answer.

7. Define visibility and human control

For a user-facing flow, specify the minimum useful evidence: source identity, freshness, selected tool or data category, missing-context warning, uncertainty, edit or reset control, approval boundary, and manual fallback. Show an activity or evidence summary rather than private reasoning traces. High-impact or irreversible actions remain subject to the approval contract in pm-ai-approval-to-flow.

8. Evaluate the context change

Reuse the same completion oracle across baseline and candidate. Include:

• representative user-job slices and hard negative or no-context cases;
• stale, conflicting, unauthorized, injected, and malformed sources;
• context-component ablations such as no memory, no retrieval, fewer tools, compacted history, or just-in-time retrieval;
• source visibility, faithfulness, unsupported-claim, privacy, tool-routing, fallback, latency, and cost checks when applicable;
• a small human review set when the failure cannot be safely judged by a deterministic check.

Pre-commit Ship, Iterate, Hold, Rollback, or Need evidence rules. Keep component scores and context-manifest evidence separate from the final answer score.

9. Instrument and write back

Specify safe fields for each run: trace ID, context version, component IDs, source versions, selected counts, token ranges, retrieval or tool events, memory read/write, compaction, truncation, access decision, fallback, outcome oracle, privacy class, and latency. Link a failure to the appropriate regression, incident, evaluation, cost, or release-learning surface. Keep the raw artifact under its approved access boundary.

10. End with one review ask

Choose exactly one decision and name the owner, unresolved context risk, and next evidence that could change it.

Output contract

Return these sections in this order. Keep unsupported fields explicitly Not provided, Not measurable, Not measured, Proposed, Not run, Unknown, or Not covered.

Decision on the desk

State the decision, owner, user job, outcome, baseline and candidate context versions, proposed budgets, and the evidence boundary.

User job and context outcome

Define the success oracle, eligible job, current workaround, affected journey, and what a completed outcome means. Do not use token count as the outcome.

Context inventory

List instructions, knowledge, tools, memory, state, and query with source IDs, purpose, selection rule, authority, privacy, freshness, budget, omission risk, and fallback.

Source, freshness, and trust contract

Show provenance, version, effective window, conflict rule, access boundary, untrusted-data handling, and what is Observed, Estimated, Proposed, or Not measured.

Budget and selection rules

Show total input budget, reserved output/tool/fallback room, component shares, ranking or filtering rules, exclusion rules, truncation behavior, and the rollback path when the budget is exceeded.

Dynamic state, memory, and compaction

Describe preloaded versus just-in-time sources, state versus memory, write and reset policy, handoff capsule, compaction preservation set, and version boundary.

Failure, privacy, and recovery

Cover empty, stale, conflicting, injected, oversized, unauthorized, malformed, and unavailable contexts; state user-visible behavior, safe records, recovery, and human fallback.

Evaluation and release gate

List baseline/candidate, slices, negative cases, component ablations, oracle, quality/trust/privacy/tool/fallback gates, p50/p95 latency or cost links when relevant, and the predeclared decision rule.

Instrumentation and writeback

List the privacy-safe manifest and trace fields, owner, QA method, retention, and where failures or learnings become regressions, incidents, eval cases, cost guardrails, or release observations.

Not covered

Name every unexecuted provider call, raw-data review, user session, retrieval quality test, context-window measurement, model comparison, deployment, adoption, traffic, ROI, safety, or star claim.

Review ask

End with one of Ship, Iterate, Hold, Rollback, or Need evidence, one owner, and one next evidence request.

Edge cases

• Only a token count exists: record a budget signal, not a quality result; add a completion oracle and context-manifest evidence.
• The context window is large: still set a product budget; larger capacity does not prove that redundant or stale context improves the job.
• Average latency is fine but retrieval is slow: inspect p95 and the fallback boundary; separate source freshness from response speed.
• A source is relevant but stale: show its age, block or downgrade it by policy, and preserve a safe fallback; never silently blend it with current policy.
• Instructions conflict: use the declared authority rule and make the conflict a test case; do not let source order become an accidental policy.
• Tool or retrieval output contains instructions: treat it as untrusted data until a separate policy explicitly promotes a field; check for prompt injection and unsafe tool chaining.
• Memory contains a wrong fact: quarantine or delete the memory entry, record provenance, and test a clean session; do not ask the model to simply ignore it without a reset path.
• Compaction drops a critical decision: hold or rollback the change; add a preservation test for goals, approvals, constraints, source IDs, and risks.
• Too many tools are exposed: narrow the task-specific subset and measure tool selection, fallback, and negative routing; do not hide the change in a prompt rewrite.
• Silent truncation occurs: make the boundary observable and fail safe; never count a truncated run as equivalent to the baseline.
• Cross-tenant or permission mismatch: block the run, preserve only safe metadata, and route to security or incident handling.
• Just-in-time retrieval is fresher but slower: compare p95, abandonment, source correctness, and fallback together; do not optimize one signal alone.
• Context reduction lowers cost but quality is unknown: use pm-ai-cost-to-guardrail for the economics and keep this skill at Hold or Need evidence until the same outcome oracle is tested.
• Small or synthetic sample: label it as directional or fictional; do not infer production reliability, adoption, or star impact.
• Raw evidence is too sensitive: store approved source IDs, categories, hashes, or redacted excerpts only, with an access-controlled path to the original when required.

Final check

Before returning the contract, confirm:

• the user job, outcome oracle, owner, baseline, candidate, and observation window are explicit;
• all six context components are considered, including absent components;
• each source has provenance, authority, freshness, version, privacy, access, selection, budget, omission risk, and fallback status;
• memory and state are not blended, and tool schemas/results are visible in the inventory;
• inclusion, exclusion, conflict, injection, and truncation rules are written before interpreting results;
• the budget reserves room for output, tool responses, retries, and fallback;
• compaction, just-in-time retrieval, caching, or artifacts have a stated freshness, latency, access, and recovery tradeoff;
• user-facing visibility, uncertainty, correction/reset, approval, and manual fallback are covered when the job is high impact;
• baseline/candidate tests use the same oracle, include negative cases and component ablations, and separate quality from token or latency signals;
• manifest and trace fields are privacy-safe and can locate a failing layer;
• fictional, synthetic, internal, observed, proposed, and production evidence are labelled;
• Not covered contains every unexecuted claim and unsupported adoption, traffic, ROI, safety, or star conclusion;
• the final line contains one decision, one owner, and one next evidence ask.