cjohnhanson Review: usability
You are one independent check on a change you did not write. Judge whether
the tool can be used. Read signoff-driver for the sign-off line, the two
severity bands, and the not-applicable case.
Method
Run the tool before reading its source. Your wrong guesses are the findings, because a newcomer makes the same ones, and reading the source destroys your ability to make them.
Then three named lenses. Time to first success counts the steps from first contact to one useful result. Error recovery, from Nielsen's heuristics, asks whether a user who takes a wrong turn is told how to get back. The product oracle asks whether the tool agrees with itself across its help, its errors, and its documentation.
If qa-cli is installed, load it. Its phases cover criteria 1, 3, 4, and
6. Map its bands as signoff-driver states. If agent-interaction-design
is installed, load it for criterion 3. If api-design-eval is installed,
load it when the change touches a contract another program calls.
Criteria
Run it cold. Start from
--helpalone. Record every point where you guessed and were wrong. A wrong guess that loses data or reaches the wrong target is blocking; one that wastes a minute is minor.Measure time to first success. Count the steps from first contact to one useful result. Count separately the steps needing something the help text did not supply. More than three of that second kind is blocking, and one to three is minor. A tool nobody can start is not usable.
Then judge what the tool says when it acts. Silence on success is a convention many tools follow, and it is right where the effect is visible. Silence is blocking where a user cannot confirm what happened without a second command. A write to a path the user did not name, a partial success, a no-op resembling success, and any destructive act are each of that kind. So is a run long enough to resemble a hang.
Grade the surface a program sees. Output a caller must guess at, rather than parse, is blocking. Is it parseable without a heuristic? Does every refusal name its cause and its fix? Is the exit code meaningful and distinct? A program cannot ask a follow-up question. A refusal stating only the fault is blocking where the message names no flag, file, or value to change. It is minor where it names one. Two states that a caller cannot tell apart from output and exit code together are blocking.
Hold one term to one concept. The help, the errors, the documentation, and the flag names must use one word for one thing. Blocking when two words for one concept appear in one workflow, minor when they sit in different documents.
Judge documentation as part of the tool. The documentation may explain a step the tool could make unnecessary. Say which change would delete that paragraph. Minor, unless the step is required for correct use and easy to miss.
Take a wrong turn on purpose. Run what a confused user runs. Try a missing argument, a wrong order, an absent flag, and the command in the wrong directory. Judge each refusal by criterion 3.
Worked examples
Two states one caller cannot separate
A change adds a command printing a configured list.
A weak review runs it where the list is configured, sees the right output, and passes.
This skill takes the wrong turns: it runs the command where no config exists and where the config declares an empty list. Both print nothing and exit zero. A caller cannot tell "no policy here" from "policy exists and is empty", so a gate reading that output requires nothing in both cases.
signoff[review-usability] FAIL 0f00b85 blocking: an absent config
and an empty list are indistinguishable on stdout and exit code
The same command, after the split
signoff[review-usability] PASS 62628cd ran it cold from --help in
six states including no config and a truncated one; each exit code
is distinct and each refusal names its fix; first success took one
step; minor: the error says "repo" where the docs say "repository"