Rényi entropy basics
What does it mean that H_α is non-decreasing on page 13 of the teaching guide? Does α = 0.5 give more uncertainty than ∞?
Asked 18 Oct 2026
The guide says non-increasing: \(H_{0.5} \ge H_1 \ge H_2 \ge H_\infty\). As \(\alpha\) grows the entropy can only stay the same or go down, so \(\alpha=0.5\) gives at least as much entropy as \(\alpha=\infty\).
All \(H_\alpha\) describe the same distribution; what changes is how it is measured. Small \(\alpha\) counts every outcome including rare ones; large \(\alpha\) mostly counts the most likely outcome; at \(\alpha=\infty\) only the top probability matters.
Example: for \((0.5, 0.25, 0.25)\), \(H_1 = 1.5\), \(H_2 = 1.415\), \(H_\infty = 1\) bit. For a uniform \(X\) all are equal to \(\log k\).
Explain probability of guessing and complete the check-your-understanding question on page 14.
Asked 18 Oct 2026
With one guess, the best strategy names the most likely value: \(P_{\text{guess}}(X) = \max_x P(X=x)\), and \(H_\infty(X) = -\log_2 P_{\text{guess}}\). Each bit of min-entropy halves the chance of guessing right.
After seeing \(Y\): \(P_{\text{guess}}(X\mid Y) = \sum_y \max_x P(x,y)\), so \(I_\infty(X;Y)\) is the log-ratio of guessing probabilities after and before.
Understanding the paper’s claims
Explain: “for binary features the feature-side min-entropy relevance vanishes whenever the feature is absent from at least half of the samples of every class…”
Asked 15 Oct 2026
Full answer: Two directions of relevance, which covers the zero criterion, the scaling law, the bounds and the index walk.
In what order should I read the files? I want to run the experiments first.
Asked 16 Oct 2026