Skip to content

Method Comparisons

This page is a hub for practical comparisons between procedures.

Comparison Checklist

  1. Fix random seeds for reproducibility.
  2. Evaluate both error control and power.
  3. Run enough replications to reduce Monte Carlo noise.
  4. Compare methods under both null-heavy and signal-heavy regimes.

Suggested Baseline Set

  • Addis
  • Saffron
  • LordThree
  • BatchBH

Minimal Experiment Skeleton

import random

from online_fdr.p_values import Addis
from online_fdr.p_values import Saffron

rng = random.Random(42)
p_values = [rng.random() for _ in range(500)]

methods = {
    "addis": Addis(alpha=0.05, wealth=0.025, lambda_=0.25, tau=0.5),
    "saffron": Saffron(alpha=0.05, wealth=0.025, lambda_=0.5),
}

discoveries = {name: sum(method.test_one(p) for p in p_values) for name, method in methods.items()}
print(discoveries)