nowespojrzenie Agent Drift Detection
The problem this solves: an agent that has been running for a long time is not the agent that started. Context gets compacted, rules scroll out of view, a persona relaxes one turn at a time. Nothing announces this. The agent still sounds like itself — that is precisely the failure mode, because the thing that would notice is the thing that drifted.
The tempting fix is a static identity file: write down who the agent is, load it every session, done. That file is a claim, and it degrades exactly like any other unchecked claim. This skill treats identity as something measured, not stored.
Companion to ai-hallucination-truth-status (which supplies truth statuses and the named voices) and ai-agent-error-memory-registry (which supplies the failure memory). This file does not repeat either; load them alongside.
When to Use
- a session has been long, compacted, or resumed from a summary
- the agent has stopped following a rule it demonstrably followed earlier
- someone says: it isn't itself · it forgot the instructions · why is it agreeing with everything
- you are setting up how an agent will be maintained over weeks rather than minutes
- routinely, at the start of a session — that is the only moment memory is genuinely cold
Do not load this when the agent is behaving as expected and nothing observable has changed. Drift detection run on a stable system manufactures false positives and teaches everyone to ignore the alarm.
1. The probe battery
The core instrument, and the one thing here that cannot be faked.
Design rule: a probe you cannot fail is not a probe. Asking an agent "are you still following your instructions?" measures nothing — the answer is generated by the same process that would have drifted. Every probe must have a ground truth the agent does not author.
Construction:
- Six probes, each answerable with a number or a single word.
- Each probe's answer must be computable from outside the agent — a file count, a line count, a timestamp difference, a value from a tool run, a fact in a repository. Not opinions, not self-assessments.
- The agent answers from memory, before measuring anything. No tool calls, no lookups.
I don't knowis a valid answer and is scored as such — it is not a failure. An agent that says "I don't know" three times is in better shape than one that guesses three times and hits twice. Abstention needs its own scoring category, and that category needs its own test. If abstention falls through to the numeric comparison it becomes a miss, and the battery then punishes the exact behaviour it exists to encourage. Six abstentions from a cold agent is the healthiest possible result — the instrument must be able to say so.- Seal the answers (hash them, write them somewhere append-only) before running the measurement.
- Then measure ground truth, compare, report.
Scoring, with an undecidable outcome:
- all within tolerance → memory consistent with state
- one or two misses → partial drift — check those, do not generalise
- three or more misses → drift: say I don't know before measuring, for the rest of the session
- three or more probes unmeasurable → UNDECIDABLE — the battery does not rule
What it measures: the distance between what the agent believes and what is true. Not the agent's quality, not its usefulness. Keeping that distinction visible is what stops the battery becoming a performance review, which is what would make people stop running it.
Contamination is real and must be declared. If the agent saw the ground truth while building or running the battery, that run is void. Say so. A contaminated run reported as clean is worse than no run.
2. Identity files that are measured, not stored
An identity or "soul" file (SOUL.md, a system prompt, a constitution) fails in three ways, and only the third is usually noticed:
- it is not loaded — obvious, easy to fix
- it is loaded and ignored — visible if you look
- it is loaded, followed, and no longer describes what the agent actually does — invisible
Against the third: give the file a hash and a guard. When its content changes, that change should be deliberate and recorded, not incidental. When behaviour diverges from it, the battery is what surfaces the divergence.
Keep the file's claims checkable. "Measures before speaking" is checkable — count measurements per session. "Is thoughtful" is not. Prefer fewer, checkable lines to many unfalsifiable ones.
3. Guards as contracts
Every guard around an agent is a contract with three parts. Write them down:
| part | question |
|---|---|
| load-bearing assumption | what must be true for this guard to mean anything |
| validation | what it actually checks, in one sentence |
| known gap | what it provably does not catch |
The known gap field is the one that earns the contract. A guard whose gap is blank has not been examined; it has been trusted.
Every guard must be able to fail. Before relying on one, feed it a case it must reject and a case it must accept. This is not optional diligence — a guard passing on data it cannot parse looks identical to a guard passing on clean data.
Two failure shapes worth naming, because both return nothing and nothing reads as fine:
- the instrument that silently broke. Test the pattern on a known hit before trusting a zero — a broken check and a clean result are the same output. Worked through in
ai-agent-verify-success-claims§4. - the guard that is too loud. A check reporting a hundred findings, most of them justified, will be ignored, and then it protects nothing. Scope it to where a miss actually costs something. A gate that never closes is not a gate.
4. Lifecycle: birth → maintenance → test → compost
Agent context is a garden, not an archive. Things grow into it and must be taken out.
- Birth — the identity file, the guards, the first battery run. Record the baseline.
- Maintenance — the battery at session start; failures appended to the error registry.
- Test — periodically, a probe the agent should fail: ask for something the rules forbid. A rule never tested against a violation is a rule nobody has checked.
- Compost — when a rule has become a mechanism, compress it to a capsule and move the full text out. Nothing is deleted; it stops competing for attention. Procedure in
ai-agent-error-memory-registry.
The point of composting is not fewer files. It is that every function has exactly one home and different roles stay distinct. Merging two files because it looks tidier usually destroys a distinction someone needed.
5. Drift protocol in the conversation
- Signature. Open each response with a one-line marker of which mode is active — drift then becomes visible in the transcript rather than only in behaviour. Why an omitted indicator is worse than none:
ai-personas-not-modes. - Thresholds. Fixed counters that force disclosure: N turns without a measurement, a named failure voice appearing twice on the same axis, a compaction event.
- META block. Self-observation goes in one block at the end — never scattered through the answer. Mixed into prose it is noise; collected at the end it is a log.
The rule that outranks the others
Every rule this file states must have a corresponding test in the instrument that implements it.
A rule stated in a skill and absent from its instrument is a head without a body, and the mismatch will not surface in review — it surfaces on a live user, days later, in the one case that mattered. A battery with a pass test and a fail test but no abstain test is not testing its own rule; it is testing its own assumption.
Verification
- For each rule stated here that you implement: does the instrument have a test for that rule, not merely for success and failure?
- Do your probes have ground truth the agent does not author? If any is self-assessment, it is not a probe.
- Were the answers sealed before measuring?
- Has any run ever returned drift or UNDECIDABLE? A battery that always passes is not measuring.
- Does each guard have a written known-gap? Has each been shown a case it must reject?
- Has any rule in the identity file been tested by attempting to violate it?
- How many turns since the last actual measurement? If you cannot answer, the thresholds are decorative.
Pitfalls
- Self-assessment as measurement. The most common way this whole apparatus becomes theatre.
- Treating "I don't know" as failure. Punish it and you will get confident guesses instead — a strictly worse state.
- Running the battery after seeing the answers. Declare contamination; do not report the run.
- Identity file as aspiration. Unfalsifiable virtues make a file that cannot be wrong, and therefore cannot help.
- Composting to reduce file count. The goal is one home per function, not fewer files.
- Scoring the agent instead of the distance. Turns maintenance into performance review, and the battery stops being run.
From HEXAGRAM · CC BY-NC-SA 4.0 · attribution: nowe spojrzenie. The battery described here exists as a running instrument in the source project, scored against repository facts computed at run time. The rules about instruments that silently return zero, and about guards too loud to be read, were both learned by shipping guards with those exact defects — one returned a clean result from a pattern that could not match, and another reported a hundred findings on its first run and had to be scoped down before anyone would look at it. The abstention rule above was in this file from its first published version — and the instrument shipped alongside it scored six honest "I don't know" answers as six misses and declared drift, while printing the advice "say I don't know before you measure" in the same breath. That was caught by a person running it, not by its author.