Rényi mRMR: which direction of Arimoto information?

Rényi mRMR: which direction of Arimoto information?

An audit and rewrite of a draft on Rényi-entropy mRMR feature selection. The direction of Arimoto mutual information, not the order α, decides how rare features are treated.

Corrected draft ready · Updated 18 Oct 2026

feature selectionRényi entropymRMRinformation theory

1 min readMy notes
  • ~40issues flagged in the first draft
  • 57.9% → 89.7%BASEHOCK AUC at α = ∞, feature-side → class-side
  • 2920 / 2920index-walk picks predicted exactly
  • 5 × 20datasets × stratified splits

The question

For \(\alpha \ne 1\), Arimoto’s mutual information \(I_\alpha(X;Y) = H_\alpha(X) - H_\alpha(X\mid Y)\) is not symmetric, so mRMR has two possible relevance terms: feature-side \(I_\alpha(f;C)\) and class-side \(I_\alpha(C;f)\). The draft used the feature side and reported that some Rényi orders beat Shannon. The revised paper asks which direction to use, and whether any order really helps.

Key idea

The direction decides how rare features are treated. For a rare word of frequency \(q\), feature-side relevance scales like \(q^\alpha\), class-side like \(q\) (as Shannon does). No order \(\alpha\) beats Shannon mRMR robustly.

Main findings

  • Zero criterion: feature-side min-entropy relevance is zero whenever the feature is absent from at least half the samples of every class.
  • Scaling law: feature-side \(\sim q^\alpha\), class-side \(\sim q\), with non-asymptotic bounds for \(\alpha > 1\).
  • Index walk: on sparse text, the feature-side \(\alpha=\infty\) selector returns features in file order.
  • Class-side blind zone: with unequal class sizes, class-side \(I_\infty\) is zero for words favouring the majority class.
  • The Palamidessi–Romanelli joint rule has the best average rank of eleven selectors; pairwise min-entropy mRMR does worse than Shannon on all three text sets.

Where to start

  1. Run the reproduction notebook and compare with the executed copy.
  2. Play with the explorer at the top of the HTML paper, then read Two directions of relevance.
  3. Read the corrected paper with your outputs side by side.
Check before submitting
  • Full text of Endo–Kudo, and of Liu & Hu 2009/2010 and Chowdhury 2017 (claims rest on abstracts).
  • Page numbers for Liu & Hu 2009 are still missing.
  • The PCMAC and RELATHE newsgroup pairs were inferred from the dataset names.
  • Affiliation line is a placeholder; edit the “Use of AI tools” paragraph to match what you did.
  • Run the journal’s similarity check (iThenticate/Turnitin) before submitting.