Rényi mRMR: which direction of Arimoto information?

Style guide

Every component and rule from the design system, rendered.

Corrected draft ready · Updated 18 Oct 2026

3 min readMy notes
About this page

Open Aa in the top bar and change the theme, font, size or width to see each one react. Numbers in the tables below are placeholders that show the format, not results. Each block shows how to write it in the page’s Markdown file.

Text

Body text is set in the reader’s chosen serif at the chosen size, with lines of about 66 characters. Italic introduces a term or adds emphasis; bold marks the one phrase a skimmer must not miss. Links look like this. Quotation marks are curly: “direction”, not “direction”. Ranges use an en dash: pages 3–7, 2019–2024.

A subsection heading

Second-level headings start sections; third-level headings split them. Headings are in sentence case with no full stop.

A run-in group label

  • Lists are for three or more parallel items.
  • Each item is short and starts the same way.
  • Two items stay a sentence.

A quotation from a paper or from the chat, with its source after it.

Code and file names: Renyi_mRMR_reproduce_corrected.ipynb, npm run dev.

Numbers

Number formats. How each kind of number is written on the site.
Kind Written Not
Negative value (true minus) −0.42 -0.42
Multiplication, powers of ten 3 × 4, 1.2 × 10⁵ 3x4, 1.2e5
Number and unit (no-break space) 30 min, 2.4 GB 30min
Percent 10% 10 %
Range 30–90 min 30-90 min
Large number (thin space from 5 digits) 12 500 12,500
Four digits 4096 4 096
Mean ± spread (say SD or SE) 0.912 ± 0.004 0.91234±0.0041
p-value p = .012, p < .001 p = 0.000
Confidence interval 95% CI [0.88, 0.93] (0.88-0.93)
Date 18 Oct 2026 10/18/26

In a sentence: the experiment ran on three datasets over 20 splits and took 30–90 min. A value of a mathematical object goes in maths: \(\alpha = 2\), \(q = 0.05\), \(n = 1000\).

Key numbers

  • 57.9% → 89.7%BASEHOCK AUC at α = ∞, feature-side to class-side
  • −22 to −33 ptsFeature-side α = ∞ vs Shannon, text sets
  • 20Random splits per experiment

Charts

A chart is for a trend or a comparison a table can’t show at a glance. At most three series; hover for values; “Show the data” opens the numbers.

Feature-side ÷ class-side relevance on BASEHOCK

  • Predicted, q^(α−1)
  • Observed median
Line and dots on a log scale: a smooth prediction against a few measurements.
Show the data
Rényi order αPredicted, q^(α−1)Observed median
0.513.212.1
20.00570.0068
51.1 × 10⁻⁹3.1 × 10⁻⁹

Test AUC by selector, BASEHOCK and RELATHE

  • BASEHOCK
  • RELATHE
Bars for a comparison across categories. Placeholder values.
Show the data
selectorBASEHOCKRELATHE
mRMR0.8840.812
JMI0.8710.826
Feature side0.5790.601
Class side0.8970.833

Maths

Inline maths sits in the sentence at text size: the Arimoto information \(I_\alpha(X;Y) = H_\alpha(X) - H^{\mathrm{A}}_\alpha(X\mid Y)\) is non-negative, and \(\mathbb{E}[\mathbbm{1}\{f=1\}] = q\) for every \(q \in (0,1)\).

A numbered display, referred to later as \(\eqref{eq:arimoto}\):

\begin{equation}\label{eq:arimoto} H^{\mathrm{A}}_\alpha(X\mid Y) = \frac{\alpha}{1-\alpha}\,\log \sum_{y} P(y) \Bigl(\sum_{x} P(x\mid y)^{\alpha}\Bigr)^{1/\alpha} . \end{equation}

Aligned steps, aligned at the equals sign:

\begin{align*} I_\infty(X;Y) &= \log \sum_y \max_x P(x,y) - \log \max_x P(x) \\ &= \log \frac{\sum_y \max_x P(x,y)}{\max_x P(x)} \\ &\ge 0 . \end{align*}

A piecewise definition:

\[\delta^\star(q) \coloneqq \begin{cases} 0 & \text{if } q \le \tfrac12, \\ 2q - 1 & \text{otherwise.} \end{cases}\]

A long display scrolls sideways inside its own box on a phone instead of shrinking:

\[\sum_{k=1}^{K} \Bigl[ I_\alpha(C; f_k) - \frac{1}{|S_{k-1}|} \sum_{j \in S_{k-1}} I_\alpha(f_k; f_j) \Bigr] \;\le\; \sum_{k=1}^{K} I_\alpha(C; f_k) \;\le\; K \log |\mathcal{C}| \quad \text{for all } \alpha \in (0, \infty] .\]

Rare symbols

Typeset: \(\mathbb{R}^n\), \(\mathbb{E}\), \(\mathbb{P}\), \(\mathds{1}\), \(\mathbbm{1}\), \(\mathcal{F}\), \(\mathscr{L}\), \(\mathfrak{g}\), \(X \indep Y \mid Z\), \(\llbracket 1, n \rrbracket\), \(\ket{\psi}\), \(\braket{\phi|\psi}\), \(\dv{f}{x}\), \(\pdv{f}{x}{y}\), \(\norm{x}_2\), \(\abs{x}\), \(\cancel{x}\), \(\bm{x}^\top \mathbf{A} \bm{x}\), \(\iiint\), \(\oint\), \(\aleph_0\), \(\hbar\), \(\ell\), \(\lesssim\), \(\succeq\), \(\rightleftharpoons\), \(\overset{\mathrm{d}}{\to}\), \(\xrightarrow[n\to\infty]{\mathrm{a.s.}}\), \(\operatorname*{arg\,max}_{f}\).

Typed straight into prose: ∀ ∃ ∈ ∉ ⊂ ⊆ ℝ ℕ ℤ 𝔼 ℙ 𝟙 ⟨x, y⟩ ⫫ ≔ ≜ ≈ ≲ ≪ ⊗ ⊕ ∂ ∇ ∮ ∞ ℵ₀ ħ ℓ α β γ Γ Δ Θ λ μ σ Σ φ ψ Ω.

Theorems and proofs

Definition (min-entropy). For a discrete random variable \(X\), the min-entropy is \(H_\infty(X) = -\log \max_x P(x)\).

Assumption. Every class has at least one training example, so \(\pi_c > 0\) for all \(c\).

Theorem (vanishing relevance). If \(q_c \le \tfrac12\) for every class \(c\), then \(I_\infty(f;C) = 0\), whatever the gap between the class rates.

Proof. The best guess of \(f\) is “absent” in every class and before seeing the class, so the numerator and denominator of the ratio above are both \(1-q\), and the logarithm is \(0\).

Lemma. \(H_\alpha(X)\) is non-increasing in \(\alpha\).

Corollary. \(H_\infty(X) \le H_2(X) \le H_1(X)\).

Remark. The class side does not vanish this way, but has its own blind zone when classes are unequal.

Theorem 4.2 (Alvim et al.). A fixed number from another paper: n="4.2" keeps it out of this page’s count.

Tables

Table 1. Test AUC by selector and dataset, mean ± SD over 20 splits. Best per column in bold, second underlined. Placeholder values.
Selector BASEHOCK RELATHE SEMEION Time (min)
mRMR (Shannon) 0.884 ± 0.006 0.812 ± 0.009 0.941 ± 0.003 4.1
JMI (Shannon) 0.871 ± 0.007 0.826 ± 0.008 0.936 ± 0.004 6.3
Feature side, \(\alpha=\infty\) 0.579 ± 0.011 0.601 ± 0.014 0.925 ± 0.005 3.8
Class side, \(\alpha=\infty\) 0.897 ± 0.005 0.833 ± 0.007 0.939 ± 0.003 3.9

AUC: area under the ROC curve. SD: standard deviation across splits.

Callouts

Key idea

The direction of Arimoto information, not the order \(\alpha\), decides how rare features are treated.

Note

Background or context that helps but is not the main line.

Check before submitting

A claim the chat marked as unverified, or a known limitation.

If a reviewer asks

“Why not just tune \(\alpha\)?” Because at \(\alpha=\infty\) feature-side relevance is zero for most words, whatever their class gap.

Question and answer

Is $H_\alpha$ non-increasing in $\alpha$?

Asked 18 Oct 2026

Yes. Larger \(\alpha\) weights the most likely outcome more, so the entropy can only stay the same or fall.

Steps

  1. Open the reproduction notebook and choose Run All.
  2. Read the corrected paper section by section next to the outputs.
  3. Use the teaching guide to prepare for questions.