# Rseng Numerical Accuracy > Covers floating-point correctness in research code: why 0.1 + 0.2 != 0.3, choosing absolute vs relative tolerances in tests, accumulation error and safe summation, precision choices (float32 vs float64), catastrophic cancellation, NaN and infinity handling, and cross-platform or cross-library result drift. Use PROACTIVELY when floating-point comparisons fail mysteriously, when writing numerical tests or choosing tolerances, when results differ across machines, compilers, BLAS builds or library versions, or when precision or numerical stability questions arise in analysis or simulation code. - Skill: `fdiblen/rseng-numerical-accuracy` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add fdiblen/rseng-numerical-accuracy` - Raw SKILL.md: https://api.skillmd.com/api/skills/fdiblen/rseng-numerical-accuracy/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Research & Search - License: CC-BY-4.0 - Author: fdiblen (https://skillmd.com/u/fdiblen) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/fdiblen/rseng-numerical-accuracy --- # Numerical accuracy in research code Floating-point numbers are approximations with well-defined rules, and research conclusions can hinge on respecting them. The two recurring failures: treating floats as exact (== comparisons, accumulating error blindly) and treating all differences as noise (tolerances loosened until tests pass). Both are avoidable with a small set of habits - and both matter more in research than elsewhere, because the numbers ARE the result. ## The ground rules - Never compare floats with == (except against an exactly representable sentinel like 0.0 you assigned yourself). Use tolerance-based comparison: `math.isclose`, `numpy.isclose`, or the testing helpers below. - Decimal literals are usually not representable: 0.1 is stored as the nearest binary fraction, which is why 0.1 + 0.2 != 0.3. Format-rounding for display hides this; arithmetic does not. - NaN propagates and never equals anything, including itself; test with isnan, and decide explicitly whether NaN in data means missing, invalid or bug (rseng-data-management's missing-data discipline). Silent NaN propagation into published numbers is the classic silent failure. - Precision-degrading operations (subtracting nearly equal numbers - catastrophic cancellation; summing numbers of very different magnitude) lose precision structurally; restructure the formula (e.g. use expm1/log1p, two-pass variance algorithms) rather than adding digits. ## Tolerances: choose, do not tune - Relative tolerance compares magnitudes ("within 1e-9 of each other, proportionally") - right for values far from zero. Absolute tolerance is required near zero, where relative comparison degenerates. Robust comparisons combine both (numpy.testing.assert_allclose(rtol=, atol=)). - Derive tolerances from the problem: input data precision, algorithm order, condition number of the operation - and record the justification in a comment or the test name. A tolerance someone loosened until CI passed documents nothing and hides regressions (rseng-testing). - Scientific sign-off: when a characterization or migration test needs a tolerance decision (rseng-legacy-code), make it with the domain expert - it is a statement about the science, not the code. ## Precision and reproducibility across platforms - float64 is the scientific default; float32 halves memory and can double throughput (and is common on GPUs - rseng-gpu-computing) but carries ~7 decimal digits: justify it per-array, not globally. Mixed precision is an optimization to apply deliberately (rseng-performance-profiling). - Bit-identical results across machines are NOT promised by IEEE-conformant code: compiler flags, SIMD width, BLAS implementation, thread count and reduction order all legitimately change last bits. Cross-platform tests therefore assert within tolerances, never bitwise (a policy mature projects like the astronomy stack encode in their contribution rules). - What IS controllable: pin library versions (rseng-reproducible-environments), fix seeds for stochastic parts, avoid fast-math-style flags for result-bearing code, and document the platform in published results (rseng-publishing-releasing). - Parallel reductions reorder sums; if run-to-run variation appears under threading, that is why - use deterministic-reduction options where offered, or widen tolerances knowingly. ## Accumulation and safe patterns - Long naive sums accumulate error linearly; library sums (numpy) use pairwise summation - prefer them over hand loops. Kahan compensated summation is available when a manual loop is unavoidable and precision matters. - Prefer numerically stable library routines (lstsq over normal equations, logsumexp over exp-sum-log) - the stable formulation is usually one function call away. - Validate against known solutions: analytic cases, conservation laws and invariants make the best numerical tests because their expected error is reasoned, not guessed. ## Working with this skill This skill is source-independent: it encodes IEEE-754 floating-point practice as applied in scientific computing. Learn more (verified): - https://floating-point-gui.de - the floating-point guide - https://numpy.org/doc/stable/reference/routines.testing.html - NumPy testing helpers (assert_allclose and friends) - https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html - Goldberg, What Every Computer Scientist Should Know About Floating-Point Arithmetic ## Related skills Check whether any of these applies before moving on: - rseng-debugging - diagnosing cross-platform result differences - rseng-gpu-computing - float32 and mixed precision trade-offs - rseng-legacy-code - characterization-test tolerance sign-off - rseng-performance-profiling - precision changes as deliberate optimization - rseng-reproducible-environments - pinned libraries limit result drift - rseng-testing - tolerance-based numerical test design