Testing
Test the claim at the cheapest layer that can falsify it.
Match evidence to the claim
- Pure logic belongs in a fast unit test without ROS, a robot, or a simulator.
- A node or launch claim needs the expected processes and interfaces to appear,
not merely a successful build.
- Behavior that depends on physics, sensors, timing, or closed-loop control
needs a deterministic scenario in simulation or on the target robot.
- A learned-policy pipeline smoke proves that data, training, evaluation, and
metrics connect. A regression claim needs a known checkpoint and a stated
performance bar.
- Add costlier layers only when the risk or claim requires them. A demo does
not need a fleet-scale qualification suite, but it still needs a real smoke.
Read ROS2-AND-SIM.md for ROS launch, simulator, map, and
headless-CI concerns. Read POLICY-EVAL.md for learned-policy
smokes and regression gates. Tool-specific syntax remains in official docs and
the owning ros2, simulator, or lerobot skill.
Make the result trustworthy
- Seed controllable randomness, use simulation time consistently, bound the
run, and record the fixture and environment that produced the result.
- Assert observable behavior and interfaces, not duplicated configuration
literals or log wording.
- Guard measurement tools against an empty run. Exiting successfully after
measuring zero devices or episodes is a test-harness failure.
- Keep benchmarks and long evaluations outside the default suite unless their
cost is justified on every change.
- Use
test-assets when fixtures, worlds, datasets, recordings, or goldens need
provenance and maintenance rules.
If a suite is flaky, slow, green without useful evidence, or fails only on a
particular runner, read FAILURES.md.
Keep remote cost proportional
- Run the same pipeline locally at tiny scale before any paid remote test.
- Keep GPU- or hardware-dependent tests explicit and schedulable; do not make a
default CI job depend on unavailable hardware.
- A remote success should preserve the exact image, model, data, seed,
hardware, result, and cost window needed to understand the claim.
Done
- The changed behavior has evidence at the lowest meaningful layer.
- A robotics app's smoke exercises the behavior it exists to demonstrate, not
only process health.
- CI runs the stable, affordable evidence by default and clearly separates
slow, paid, hardware, and manual checks.
1---2name: testing3description: Choose proportional evidence for robotics software before claiming it works.4---56# Testing78Test the claim at the cheapest layer that can falsify it.910## Match evidence to the claim1112- Pure logic belongs in a fast unit test without ROS, a robot, or a simulator.13- A node or launch claim needs the expected processes and interfaces to appear,14 not merely a successful build.15- Behavior that depends on physics, sensors, timing, or closed-loop control16 needs a deterministic scenario in simulation or on the target robot.17- A learned-policy pipeline smoke proves that data, training, evaluation, and18 metrics connect. A regression claim needs a known checkpoint and a stated19 performance bar.20- Add costlier layers only when the risk or claim requires them. A demo does21 not need a fleet-scale qualification suite, but it still needs a real smoke.2223Read [ROS2-AND-SIM.md](ROS2-AND-SIM.md) for ROS launch, simulator, map, and24headless-CI concerns. Read [POLICY-EVAL.md](POLICY-EVAL.md) for learned-policy25smokes and regression gates. Tool-specific syntax remains in official docs and26the owning `ros2`, simulator, or `lerobot` skill.2728## Make the result trustworthy2930- Seed controllable randomness, use simulation time consistently, bound the31 run, and record the fixture and environment that produced the result.32- Assert observable behavior and interfaces, not duplicated configuration33 literals or log wording.34- Guard measurement tools against an empty run. Exiting successfully after35 measuring zero devices or episodes is a test-harness failure.36- Keep benchmarks and long evaluations outside the default suite unless their37 cost is justified on every change.38- Use `test-assets` when fixtures, worlds, datasets, recordings, or goldens need39 provenance and maintenance rules.4041If a suite is flaky, slow, green without useful evidence, or fails only on a42particular runner, read [FAILURES.md](FAILURES.md).4344## Keep remote cost proportional4546- Run the same pipeline locally at tiny scale before any paid remote test.47- Keep GPU- or hardware-dependent tests explicit and schedulable; do not make a48 default CI job depend on unavailable hardware.49- A remote success should preserve the exact image, model, data, seed,50 hardware, result, and cost window needed to understand the claim.5152## Done5354- The changed behavior has evidence at the lowest meaningful layer.55- A robotics app's smoke exercises the behavior it exists to demonstrate, not56 only process health.57- CI runs the stable, affordable evidence by default and clearly separates58 slow, paid, hardware, and manual checks.
robium-ai