Entropy
When to Use
Use this skill when working on entropy problems in information theory.
Decision Tree
Shannon Entropy
- H(X) = -sum p(x) log2 p(x)
- Maximum for uniform distribution: H_max = log2(n)
- Minimum = 0 for deterministic (one outcome certain)
scipy.stats.entropy(p, base=2) for discrete
Entropy Properties
- Non-negative: H(X) >= 0
- Concave in p
- Chain rule: H(X,Y) = H(X) + H(Y|X)
z3_solve.py prove "entropy_nonnegative"
Joint and Conditional Entropy
- H(X,Y) = -sum sum p(x,y) log2 p(x,y)
- H(Y|X) = H(X,Y) - H(X)
- H(Y|X) <= H(Y) with equality iff independent
Differential Entropy (Continuous)
- h(X) = -integral f(x) log f(x) dx
- Can be negative!
- Gaussian: h(X) = 0.5 * log2(2pie*sigma^2)
sympy_compute.py integrate "-f(x)*log(f(x))" --var x
Maximum Entropy Principle
- Given constraints, max entropy distribution is least biased
- Uniform for no constraints
- Exponential for E[X] = mu constraint
- Gaussian for E[X], Var[X] constraints
Tool Commands
Scipy_Entropy
uv run python -c "from scipy.stats import entropy; p = [0.25, 0.25, 0.25, 0.25]; H = entropy(p, base=2); print('Entropy:', H, 'bits')"
Scipy_Kl_Div
uv run python -c "from scipy.stats import entropy; p = [0.5, 0.5]; q = [0.9, 0.1]; kl = entropy(p, q); print('KL divergence:', kl)"
Sympy_Entropy
uv run python -m runtime.harness scripts/sympy_compute.py simplify "-p*log(p, 2) - (1-p)*log(1-p, 2)"
Key Techniques
From indexed textbooks:
- [Elements of Information Theory] Elements of Information Theory -- Thomas M_ Cover & Joy A_ Thomas -- 2_, Auflage, New York, NY, 2012 -- Wiley-Interscience -- 9780470303153 -- 2fcfe3e8a16b3aeefeaf9429fcf9a513 -- Anna’s Archive. What is the channel capacity of this channel? This is the multiple-access channel solved by Liao and Ahlswede.
Cognitive Tools Reference
See .claude/skills/math-mode/SKILL.md for full tool documentation.
Source: parcadei/Continuous-Claude-v3 → .claude/skills/math/information-theory/entropy/SKILL.md
1---2name: entropy3description: "Problem-solving strategies for entropy in information theory"4allowed-tools: "[Bash, Read]"5category: general-purpose6source_repo: parcadei/Continuous-Claude-v37source_path: ".claude/skills/math/information-theory/entropy/SKILL.md"8source_url: https://github.com/parcadei/Continuous-Claude-v3/blob/HEAD/.claude/skills/math/information-theory/entropy/SKILL.md9metadata:10 source: https://skillmd.com/skills/parcadei/entropy11---121314# Entropy1516## When to Use1718Use this skill when working on entropy problems in information theory.1920## Decision Tree2122231. **Shannon Entropy**24 - H(X) = -sum p(x) log2 p(x)25 - Maximum for uniform distribution: H_max = log2(n)26 - Minimum = 0 for deterministic (one outcome certain)27 - `scipy.stats.entropy(p, base=2)` for discrete28292. **Entropy Properties**30 - Non-negative: H(X) >= 031 - Concave in p32 - Chain rule: H(X,Y) = H(X) + H(Y|X)33 - `z3_solve.py prove "entropy_nonnegative"`34353. **Joint and Conditional Entropy**36 - H(X,Y) = -sum sum p(x,y) log2 p(x,y)37 - H(Y|X) = H(X,Y) - H(X)38 - H(Y|X) <= H(Y) with equality iff independent39404. **Differential Entropy (Continuous)**41 - h(X) = -integral f(x) log f(x) dx42 - Can be negative!43 - Gaussian: h(X) = 0.5 * log2(2*pi*e*sigma^2)44 - `sympy_compute.py integrate "-f(x)*log(f(x))" --var x`45465. **Maximum Entropy Principle**47 - Given constraints, max entropy distribution is least biased48 - Uniform for no constraints49 - Exponential for E[X] = mu constraint50 - Gaussian for E[X], Var[X] constraints515253## Tool Commands5455### Scipy_Entropy56```bash57uv run python -c "from scipy.stats import entropy; p = [0.25, 0.25, 0.25, 0.25]; H = entropy(p, base=2); print('Entropy:', H, 'bits')"58```5960### Scipy_Kl_Div61```bash62uv run python -c "from scipy.stats import entropy; p = [0.5, 0.5]; q = [0.9, 0.1]; kl = entropy(p, q); print('KL divergence:', kl)"63```6465### Sympy_Entropy66```bash67uv run python -m runtime.harness scripts/sympy_compute.py simplify "-p*log(p, 2) - (1-p)*log(1-p, 2)"68```6970## Key Techniques7172*From indexed textbooks:*7374- [Elements of Information Theory] Elements of Information Theory -- Thomas M_ Cover & Joy A_ Thomas -- 2_, Auflage, New York, NY, 2012 -- Wiley-Interscience -- 9780470303153 -- 2fcfe3e8a16b3aeefeaf9429fcf9a513 -- Anna’s Archive. What is the channel capacity of this channel? This is the multiple\-access channel solved by Liao and Ahlswede.7576## Cognitive Tools Reference7778See `.claude/skills/math-mode/SKILL.md` for full tool documentation.7980---8182**Source:** [`parcadei/Continuous-Claude-v3`](https://github.com/parcadei/Continuous-Claude-v3) → `.claude/skills/math/information-theory/entropy/SKILL.md`