FDR control and FDP certification¶
This example starts from one fixed test family and compares four decision rules:
- unadjusted pointwise thresholding;
- Benjamini-Hochberg (BH), used by standard
select(...); - Storey's adaptive BH, read from the same snapshot's adjusted p-values; and
- Benjamini-Yekutieli (BY), applied manually through SciPy.
It then constructs a simultaneous post-hoc upper bound for realized FDP over a queried threshold grid.
For certification directly from a fitted detector, use
certificate = detector.fdp_bounds(x_family, confidence=0.95).
The example below reuses the snapshot already computed for the BH comparison.
Complete example¶
import numpy as np
from scipy.stats import false_discovery_control
from sklearn.ensemble import IsolationForest
from nonconform import ConformalDetector, Split
from nonconform.metrics import false_discovery_rate, statistical_power
rng = np.random.default_rng(42)
x_reference = rng.normal(size=(3_000, 4))
x_family = np.vstack(
[rng.normal(size=(160, 4)), rng.normal(loc=5.0, size=(40, 4))]
)
y_family = np.r_[np.zeros(160, dtype=int), np.ones(40, dtype=int)]
alpha = 0.1
detector = ConformalDetector(
detector=IsolationForest(n_estimators=100, random_state=42),
strategy=Split(n_calib=1_000),
seed=42,
).fit(x_reference)
bh_selected = np.asarray(detector.select(x_family, alpha=alpha))
result = detector.last_result
assert result is not None
assert result.p_values is not None
p_values = result.p_values
pointwise_selected = p_values <= alpha
storey_selected = result.adjusted_p_values(method="storey") <= alpha
by_adjusted = false_discovery_control(p_values, method="by")
by_selected = by_adjusted <= alpha
for name, selected in {
"pointwise": pointwise_selected,
"BH": bh_selected,
"Storey-BH": storey_selected,
"BY": by_selected,
}.items():
print(
name,
{
"discoveries": int(selected.sum()),
"realized_fdp": float(false_discovery_rate(y_family, selected)),
"power": float(statistical_power(y_family, selected)),
},
)
certificate = result.fdp_bounds(confidence=0.95)
print(certificate.to_frame([0.005, 0.01, 0.025, 0.05, 0.1]).to_string(index=False))
cutoff = 0.01
mask = certificate.select(cutoff)
print("FDP upper bound at cutoff:", certificate.bound_at(cutoff))
print("Discoveries at cutoff:", int(mask.sum()))
The pointwise rule does not account for the 200 simultaneous tests. BH is less conservative than BY when its independence or positive-dependence conditions apply. BY supports a broader dependence class, but neither procedure repairs invalid p-values or adaptive family construction.
Storey-BH estimates the share of true nulls and runs BH at a correspondingly
higher level. storey_selected equals
detector.select(x_family, alpha=alpha, method="storey") for this snapshot
without scoring again. Choose one procedure before inspecting the family; see
Adaptive BH with Storey's estimator
for its scope.
The labeled realized_fdp is the FDP of this one family. FDR is the expected
FDP over repetitions. Do not interpret one low realized FDP as validation of
the FDR theorem.
Why the FDP certificate is different¶
The certificate is simultaneous over p-value thresholds within its documented scope. This supports post-hoc threshold exploration while attaching a high-confidence realized-FDP upper bound. It does not select by BH and does not replace the expected-FDR target.
The native snapshot API requires unmodified snapshots from unweighted Split or detached calibration with
Empirical p-values. It rejects weighted, KDE, conditional-calibration, and
resampling-strategy result bundles.
The ordinary call accepts only confidence and prepares one default MC-THC
envelope with 1,000 fresh Monte Carlo draws and boosting enabled. Its
simultaneous coverage probability is joint over the data and independent
sampling, not conditional on every realized envelope or fixed seed. Repeated
queries reuse the envelope; repeated constructions can vary and must not be
cherry-picked for more favorable bounds. The synthetic family contains 40
planted anomalies and uses 1,000 held-out calibration observations so that the
example shows a useful nonempty certificate. Labels evaluate the example;
one favorable batch does not establish coverage.
For advanced envelope settings, use FDPCertificate.from_p_values(...) and
choose the method and all settings before inspecting the final curve. A fixed
Monte Carlo seed gives reproducibility, not a separate conditional coverage
guarantee. The expert deterministic method="ks" uses the dependent-law
transductive DKW construction, rather than the ordinary i.i.d. one-sample KS
distribution.
For weighted covariate shift, use the separate weighted WCS example. For online hypotheses and single-stream change evidence, see the distinctions in the FDR guide.