Information Entropy

Information Entropy (cross-cutting/numerics/information-entropy)

Use when the task is the information-theoretic content of a symbol distribution for aerospace data-channel work: the Shannon entropy in bits per symbol of a measured count distribution or a stated probability mass function, the binary entropy function for a two-symbol source, the uniform-distribution entropy bound as log2 of the symbol count, and the minimum source-coding bit rate of a symbol stream as entropy times symbol rate. This leaf is the pure information-measure utility of the numerics pack: deterministic, stdlib only, no RNG. It pairs with flight-test-operations/planning/pcm-telemetry-decommutation, where the recovered channel symbol stream is the stream whose entropy and minimum bit rate this leaf sizes. It does NOT fit probability models, estimate frequency spectra, sample or estimate densities, or build coding trees: the numerics siblings below own those operations.

Domain quick reference

• Shannon entropy: H = -sum(p_i * log2(p_i)) bits per symbol over the normalized probabilities; a zero-probability symbol contributes 0 * log2(0) = 0. Raw counts are normalized by their total first, so counts and probabilities give identical results.
• Binary entropy function: b(p) = -p log2(p) - (1-p) log2(1-p) for a two-symbol source with symbol probability p. Endpoints give b(0) = b(1) = 0 (fully deterministic source) and the peak is b(0.5) = 1 bit. The function is symmetric, b(p) = b(1-p).
• Uniform bound: H_uniform = log2(N) for N symbols, the maximum entropy any distribution over N symbols can reach, so H <= log2(N) always. Redundancy against the bound is 1 - H / log2(N): 0 for a uniform source, 1 for a deterministic one.
• Minimum source-coding bit rate: R_min = H * r bits per second for a stream of r symbols per second. Shannon source coding says the average code length per symbol cannot go below H, so R_min is the floor bit rate the data channel must carry and quantization must respect.
• Units: bits per symbol for entropy, symbols per second for the symbol rate, bits per second (bps) for the minimum bit rate.
• NACA TR-824 anchors the numerics-pack reference convention; the relations above are standard information-theory methodology, summary-only.

Workflow

1. Assemble the symbol distribution of the data channel: raw symbol counts from a measurement window, or the probability mass function when it is known directly.
2. Run shannon_entropy on the distribution to get entropy_bits and the normalized probabilities; the function accepts counts or probabilities and normalizes internally.
3. For a two-symbol source (single bit per symbol, on-off channel), run binary_entropy with the probability of one symbol instead.
4. Run uniform_entropy with the symbol count to get the log2 upper bound for the comparison.
5. Size the channel floor: min_bit_rate with the per-symbol entropy and the symbol rate in symbols per second.
6. For the consolidated verdict, run entropy_summary with the distribution and symbol rate: entropy, symbol count, uniform bound, redundancy, and minimum bit rate in one dict.
7. Confirm the deterministic checks with the contract test scripts/test_information_entropy.py.

Worked example

A telemetry data channel carries four symbol classes with p = [0.5, 0.25, 0.125, 0.125] at 1000 symbols per second (module outputs shown).

• shannon_entropy(p) returns entropy_bits 1.75 bits per symbol and normalized [0.5, 0.25, 0.125, 0.125]; the same call on counts [4, 2, 1, 1] returns the same 1.75 bits.
• binary_entropy(0.9) returns 0.4689955935892811, 0.4690 bits within the worked-example bound; binary_entropy(0.5) returns exactly 1.0.
• uniform_entropy(4) returns 2.0 bits and uniform_entropy(8) returns 3.0 bits, the log2 upper bounds.
• min_bit_rate(1.75, 1000.0) returns 1750.0 bps.
• entropy_summary(p, 1000.0) returns entropy_bits 1.75, n_symbols 4, uniform_bound_bits 2.0, redundancy 0.125 (1 - 1.75/2), and min_bit_rate_bps 1750.0.
• The same channel carrying a uniform 8-symbol source at 1000 symbols per second needs min_bit_rate(3.0, 1000.0) = 3000 bps, so the four-symbol message reduces the channel floor by 42 percent ((3000 - 1750) / 3000 = 0.4167).
• A skew distribution [0.9, 0.05, 0.03, 0.02] gives 0.6175431233120147 bits per symbol, well below the 2.0 bit bound for 4 symbols.

Pitfalls

• Feeding any negative probability or count, or a zero-sum input: the module raises ValueError, as it does for empty input and binary p outside [0, 1].
• Expecting entropy above log2(N): H always lies between 0 and log2(N), so a uniform 4-symbol source gives exactly 2.0 bits and skew sources sit below (0.6175 for [0.9, 0.05, 0.03, 0.02]) - a result above the bound means invalid input, not a dense source.
• Misreading binary_entropy at its endpoints: b(0.0) = b(1.0) = 0.0, b(0.5) = 1.0 exactly, and the function is symmetric, b(p) = b(1-p).
• Treating counts and probabilities as interchangeable without the normalization rule: counts normalize identically to probabilities ([5, 5] and [0.5, 0.5] both give 1.0 bit), but every entry must be non-negative and the sum nonzero.
• Quoting the entropy as the channel rate: the floor is min_bit_rate = entropy times symbol rate (1750 bps at 1000 symbols/s), and the four-symbol saving over the uniform 8-symbol source (3000 bps) only holds at the same symbol rate.
• Calling entropy_summary on fewer than 2 symbols or with a negative entropy or symbol rate: it raises ValueError, and redundancy is only meaningful in [0, 1] for a valid distribution (0 uniform, 1 deterministic).

Verification

• Confirm shannon_entropy([0.5, 0.25, 0.125, 0.125]) returns entropy 1.75 bits and that H is always between 0 and log2(N): deterministic inputs give exactly 0, the uniform input equals uniform_entropy(N).
• Confirm counts normalize identically to probabilities: [5, 5] gives 1.0 bit, the same as [0.5, 0.5].
• Confirm binary_entropy(0.5) = 1.0, endpoints binary_entropy(0.0) and binary_entropy(1.0) = 0.0, and the symmetry b(p) = b(1-p).
• Confirm entropy_summary redundancy is 0 for a uniform source and 1 for a fully deterministic source, and sits in [0, 1] for every valid distribution.
• Confirm ValueError rejection: empty input, any negative probability or count, zero-sum input, binary p outside [0, 1], uniform n_symbols below 1, negative entropy or symbol rate in min_bit_rate, and entropy_summary on fewer than 2 symbols.
• Confirm determinism: identical inputs give identical outputs; the module never uses random numbers.
• Run the contract test offline: python3 scripts/test_information_entropy.py (32 tests, deterministic).

Related leaves

• cross-cutting/numerics/probability-distributions: distribution parameter estimation and pdf/cdf work, the model step that can feed this leaf its probability mass function (no entropy there).
• cross-cutting/numerics/descriptive-statistics: location and spread summaries of measured symbol counts before the entropy step.
• cross-cutting/numerics/monte-carlo-sampling: seeded random draws and empirical density estimation, not information measures.
• cross-cutting/numerics/fast-fourier-transform and cross-cutting/numerics/power-spectral-density: frequency-domain spectra, a different content view from per-symbol information.
• cross-cutting/numerics/hypothesis-testing: significance tests on symbol-count tables, complementary to the entropy summary.
• flight-test-operations/planning/pcm-telemetry-decommutation: recovers the telemetry channel symbol stream whose entropy and minimum bit rate this leaf computes.

Behavior contract (gate 3)

Run the deterministic contract test (stdlib unittest, offline):

python3 scripts/test_information_entropy.py

The test covers the spec worked-example anchors (Shannon entropy 1.75 bits for p = [0.5, 0.25, 0.125, 0.125], binary entropy 0.4690 at p = 0.9, minimum bit rate 1750 bps at 1000 symbols per second, the 3000 bps uniform-8 comparison and its 42 percent reduction), the identities (uniform distribution entropy equals log2(N), deterministic sources give 0, b(0.5) = 1, endpoint and symmetry behavior of the binary entropy function, counts matching probabilities), the max-entropy bound H <= log2(N), dict key contracts, determinism, and ValueError rejection of empty, negative, zero-sum, out-of-range and sub-scale inputs. Runs in well under a second.

Compliance

• Standards referenced, not reproduced: NACA TR-824 anchors the numerics-pack public-domain reference convention; Shannon entropy and the source-coding bound are standard information-theory methodology (paraphrase-only) per standards-map.yaml.
• compliance: STANDARDS-REF, gated: false.