Using Tooltester

Test Runner Guide for Agents

This document describes the Pyromaniac test system so that agents can understand how to run tests, interpret results, and diagnose failures.

Read references/pyromaniac-tooltester.md for the repository-specific workflow before adding or debugging testcode/tests-*.txt suites.

Overview

The test system uses a Perl script testcode/test.pl to execute test suites. Each suite is described by a file testcode/tests-<name>.txt. Tests run a command-line tool, capture its output, and compare it against expected output in files under testcode/expect/.

Running Tests

Direct make invocation

Usually commands are managed through a Makefile, using a command like:

make tests TEST=<suite-name>

The TEST= argument maps to testcode/tests-<name>.txt. Omit TEST= to run all suites.

Direct invocation

The tool can be directly invoked on test script like this:

riscos-tooltester mytool testdir

Commonly the tests are held in a directory testcode or testdata.

Test Specification File Format

Test suite files are named tests-<suite-name>.txt. They are line-oriented; # begins a comment. Statements have the form <Keyword>: <argument>.

Structure

Tests are grouped; a Group: block sets defaults that each Test: within it inherits. A Test: overrides those defaults. A - prefix on a statement (with no argument) clears a group-level setting.

Group: Descriptive Group Name

Command: $TOOL --some-flags --command "$ARG1"
Expect: expect/mymodule/default-expect

Test: Basic case
Args: hello

Test: Error case
Args: badarg
RC: 1
Expect: expect/mymodule/error-case

All Statements

Substitution Variables

These expand inside statement arguments:

Test Output and Interpreting Results

Live output format

Each test prints a single line like:

Group Name / Test Name ... OK
Group Name / Test Name ... FAIL: Expected output did not match
Group Name / Test Name ... CRASH: Expected RC 0, got 139

• OK — test passed
• FAIL — test ran but output or RC was wrong
• CRASH — the command was killed by a signal

At the end of each suite, totals are printed:

Pass: 58  Fail: 0  Crash: 0  Skip: 0

Diff output on failure

When output does not match, a diff is shown immediately below the FAIL line. The diff uses a line-number prefix and a marker character:

Example:

 1 : Group header line
 2 + data: [{'number': 0, 'enabled': False, ...}]
   - data: [{'triggered': False, 'enabled': False, ...}]

This means line 2 in actual output has number first but the expected file has triggered first.

The -actual file

Whenever output does not match the expected file, test.pl writes the (post-replacement) actual output to <expect-file>-actual. For example:

testcode/expect/pymodules/timermanager/sysrq-list-claimed-actual

This is the most useful file for diagnosing failures — it shows exactly what the tool produced after replacements were applied. Compare it to the expected file using a normal diff:

diff testcode/expect/mymodule/mytest testcode/expect/mymodule/mytest-actual

When the test passes, the -actual file is deleted. Stale -actual files from previous runs may linger; only trust them if you just ran the relevant test.

Expect Files

Expect files live under expect/<suite-name>/<test-name>. They contain the verbatim expected output. No special syntax — plain text.

Common conventions:

• Normalised placeholder tokens replace variable data (e.g. SHA, BRANCH, XXXXXXXX, DATE, TESTDATA)
• RISC OS addresses appear normalised by the replacement script rather than raw hex

Replacement Scripts

A Replace: <path> statement applies a script to the actual output before comparing it to the expected file. This is used to normalise variable data such as:

• Git commit SHAs
• Memory addresses
• Branch names (master vs main)
• Dates and timestamps
• Absolute paths

Replacement scripts are similar in intent to sed. They live alongside the expect files, by convention named <test-name>_replacements.

Replacement script syntax

• Lines starting with # or containing only whitespace are ignored.
• Lines beginning with % are directives:
  • %include <file> — include another replacement file inline
• All other lines are rules. Each rule has an optional condition followed by an action, separated by whitespace.

Conditions

Append ! to negate a condition: /regex/! means "if line does NOT match".

Conditions can be combined: 3-10 /pattern/ means "lines 3–10 that also match the pattern".

Actions

Examples

# Normalise all commit SHAs
s/\[(main|master)( \(root-commit\) | )[a-f0-9]+\]/[BRANCH$2SHA]/

# Replace absolute test data paths
s!/[^ ]+/testcode/testdata!TESTDATA!g

# Skip git hint lines
/hint: .*/ d

# Normalise memory addresses (wildcard addresses)
s/([^0-9a-f\-])[0-9a-f]{8}/$1XXXXXXXX/gi

# Include a shared replacement file
%include status_replacements

Shared replacement files

The testcode/expect/ directory might contain reusable replacement files:

Per-suite shared files are typically placed alongside their expect files and included with %include.

Common Patterns and Tips

Diagnosing a failure

1. Run the suite: make tests TEST=<name> 2>&1 | tee output.log
2. Look at the FAIL: line message in the output or log.
3. Read expect/<suite>/<test>-actual — this is exactly what the tool produced after replacements.
4. Diff it against the expected file to see what changed.
5. If the difference is in variable data (addresses, timestamps, SHAs) and no replacement is normalising it, add a substitution rule to the replacement script.
6. If the difference is in structural output (different ordering, new or removed lines, different text), update the expected file to match the new correct output.

When to update expected files vs fix code

• Update the expected file when output is correct but differs due to environmental variation (e.g. git version differences, dictionary order changes, argparse version differences).
• Fix the code when the output is wrong.
• Add/update a replacement script when the output varies between runs for legitimate reasons (SHAs, timestamps, addresses, absolute paths).

Return code failures

If a test fails with Expected RC 0, got N, the tool crashed or errored. Check the -actual file for any error message.

Missing -actual file after failure

If a test fails with an RC mismatch but no expected output mismatch (no Expect: statement, or RC failure prevents comparison), there will be no -actual file. Re-run with extra verbosity or add temporary debug output to diagnose.

Test Data

Test data files used by tests live in testcode/testdata/. Absolute paths to this directory are normalised by replacement scripts (typically replaced with the token TESTDATA).

Suite Naming Conventions

Suite names follow a hierarchical pattern:

• pymodules-<module> — tests for Python extension modules
• graphics-<area> — graphics subsystem tests
• filesystem-<area> — filesystem tests
• core, core-write — core OS tests

Test suites are commonly split out and included from an all encompassing tests.txt which includes all the files. This allows individual suites to be run independantly.

The suite name is the part after tests- in the filename, e.g. testcode/tests-pymodules-iic.txt → suite name pymodules-iic.