Math Foundation

Math Foundation — Deep Engineering Guide

Every frame pushes millions of floats through vectors, matrices, quaternions, and collision tests. Math is the substrate: correct under 16.6 ms, deterministic for netcode, and SIMD-friendly or it dies. This skill covers the core math library every engine needs.

1. The Canonical Type Set

Rules:

• Value types, struct (not class), no vtables — SIMD can then reinterpret.
• NaN/Inf guards: default init to 0; assert on non-finite inputs in hot paths.
• Deterministic math when needed (see §6).

2. SIMD: The Practical Diet

2.1 The SIMD Types

float4 maps to __m128 (SSE), float4x4 to rows of __m128. Use intrinsic headers you control (or DirectXMath/Eigen/SIMD eigen) rather than accidental scalar loops.

struct alignas(16) float4 { float x, y, z, w; };
// SSE: _mm_add_ps(a, b) etc.
inline float4 FMA(float4 a, float4 b, float4 c) {
    return _mm_fmadd_ps(a, b, c);   // a*b + c — 1 instr, no rounding split
}

2.2 Alignment & Layout

• alignas(16) on every SIMD type (avoid memcpy warnings, use aligned_alloc(16, …)).
• Hot arrays (vertex pos, transforms) live in SoA SIMD lanes (§ in memory skill).
• Avoid -ffast-math unless you know: it breaks NaN propagation (netcode determinism).

2.3 When SIMD Matters Most

Matrix×Vector, quaternion×vector, AABB test packs, and any tight loop with uniform ops. At 100k transforms: SIMD mat×vec = ~4–8x over scalar.

3. Quaternions: The Correct Rotation

3.1 Why Not Euler

Gimbal lock (order-dependent), non-unique representations, bad interpolation. Quaternions: 4 floats, no gimbal lock, slerp smooth, compose cheaply.

3.2 The Core Ops

quat mul(q1, q2);        // compose: apply q2 first, then q1. ~8 mults via SIMD
vec3  rotate(q, v);      // q v q^-1  →  t=2 cross(q.xyz,v);  v + q.w*t + cross(q.xyz,t)
quat  slerp(a, b, t);    // shortest arc, accounting for sign (a·b<0 → negate b)
quat  nlerp(a, b, t);    // cheap approx (normalize lerp); fine for small arcs
quat  fromAngleAxis(angle, axis);     // normalized
quat  fromMat4(m);/mat4 toMat4(q);    // round-trip must preserve axis/angle

3.3 The Sign Problem

q and -q represent the same rotation. Correct slerp/nlerp: if dot(a,b) < 0, flip b sign first (else the path follows the long way). Meets every engine bug report: "model spins 360° on t=0..1".

3.4 Normalization Drift

Composing 1k quats drifts norm → normalize periodically (per-frame per-transform), cheap with rsqrt.

4. Transforms & Coordinate Conventions

4.1 Transform Composition

World transform = T(parent) * R * S (translation × rotation × scale, parent-first). Store as Transform { pos, rot, scale }; combine for children.

Convention decision (critical, document it):

• Row vectors × matrices (D3D-style) or column vectors (GL style)? Pick ONE for the whole engine + renderer; document in the codebase header; mismatch = mirrored sprites.

4.2 Handedness

• Engine convention: e.g., right-handed, Z-up (DOOM/Unreal) vs Z-forward left-handed (Unity-ish) — pick and convert at import (see import-and-cook).
• All camera/projection/shadow conversions flow through one place to avoid 50 copies of the flip.

4.3 Fast Paths

• World→local: combine inverse T * R^-1 * S^-1 without building a full inverse matrix.
• AABBs re-computed from transform (or a 3-vector math) — cheaper than matrix multiply then recompute box.
• LOD/stream culling: cheap sphere test first, then exact.

5. Numerical Stability

5.1 The Sneaky FP Facts

• a+b+c != (a+b)+c (float non-associativity).
• 0.1f is not 0.1.
• Large + small = the small is lost (catastrophic cancellation near equal values).

5.2 Mitigations

• Use double for accumulation templates (ray-tracing accumulators, physics) — speed cost acceptable there.
• Long lerps / slerps: interpolate t relative to start, not absolute.
• epsilon comparisons with std::numeric_limits<T>::epsilon() * errorScale, never a == b.
• Retain a "stable normal" path: recompute normals from the transform each frame.

5.3 Determinism (Netcode Critical!)

• No -ffast-math, no FMA being used in some builds but not others, same float ops per platform.
• Fixed-point integer math for lockstep position where feasible (see determinism.md).
• Deterministic sqrt/sin implementations cross-compiler (see multiplayer netcode determinism references).

6. Collision Primitives

6.1 The Primitive Zoo

6.2 Prioritize Cheap → Expensive

Collision/streaming queries:

1. Sphere vs frustum (cheap) → skip.
2. AABB/AABB exact.
3. OBB/OBB via SAT only when needed.
4. Triangle/ray only on candidate hits.

6.3 Ray-AABB Slab (fastest shape test)

bool RayAABB(Ray r, AABB b, float& tmin, float& tmax) {
    float t0 = DBL_MIN, t1 = DBL_MAX;
    for (int a = 0; a < 3; ++a) {
        float inv = 1.0f / r.d[a];
        float tNear = (b.min[a] - r.o[a]) * inv;
        float tFar  = (b.max[a] - r.o[a]) * inv;
        std::swap if inverted...
        t0 = max(t0, tNear); t1 = min(t1, tFar);
        if (t0 > t1) return false;
    }
    return true;
}

The 3-iteration slab test beats general AABB intersection on hot paths.

7. Curves & Interpolation

7.1 The Easing Family

Linear → use EaseInOut for UI, camera; avoid linear for non-linear feel. Include easeOutBack(s), easeInOutCubic, etc. Table-driven t function + cheap.

8. The Math Library Checklist (Lead-Level)

1. Value types, SIMD-aligned, deterministic (no fast-math).
2. Quaternions everywhere for rotation; sign-correct slerp; periodic normalization.
3. One matrix convention documented end to end.
4. Collision tests ranked cheap→expensive; slab/static fast paths.
5. Numerical stability patterns (double accumulators, relative epsilon, no ==).
6. Easing/curves table.
7. Unit tests: properties (round-trips, orthonormality, no drift), fuzz a few.

9. Unit-Testing the Library

• Algebraic identities: quat compose distributes; inverse round-trips; slerp endpoints exact.
• Determinism: run same sim under MSVC+GCC+Clang, compare hashes.
• SIMD vs scalar reference within epsilon.
• Fuzz: random transforms → inverse → reconstruct, assert within tolerance.
• NaN/Inf injection: assert allfinite guards fire.

Ship a math_self_test executable run in CI.

10. References

• references/simd.md — SIMD intrinsics, alignment, SoA, FMA/fast-math, specialized mat×vec
• references/quaternion.md — algebra, composition, slerp/nlerp, axis-angle, drift fix, matrix round-trip
• references/transforms.md — conventions, handedness, T·R·S, world/local inverse, AABB from transform
• references/numerical-stability.md — FP facts, cancellation, determinism, epsilon discipline
• references/collision-primitives.md — sphere/aabb/obb/ray/plane/frustum tests, slab ray-AABB, SAT
• references/curves-and-interp.md — lerp/slerp, easing family, Catmull-Rom, Beziers, frame-rate independent