Skip to content
Edit this page View source of this page

FDR control and FDP certification

This example starts from one fixed test family and compares four decision rules:

  • unadjusted pointwise thresholding;
  • Benjamini-Hochberg (BH), used by standard select(...);
  • Storey's adaptive BH, read from the same snapshot's adjusted p-values; and
  • Benjamini-Yekutieli (BY), applied manually through SciPy.

It then constructs a simultaneous post-hoc upper bound for realized FDP over a queried threshold grid.

For certification directly from a fitted detector, use certificate = detector.fdp_bounds(x_family, confidence=0.95). The example below reuses the snapshot already computed for the BH comparison.

Complete example

import numpy as np
from scipy.stats import false_discovery_control
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split
from nonconform.metrics import false_discovery_rate, statistical_power

rng = np.random.default_rng(42)
x_reference = rng.normal(size=(3_000, 4))
x_family = np.vstack(
    [rng.normal(size=(160, 4)), rng.normal(loc=5.0, size=(40, 4))]
)
y_family = np.r_[np.zeros(160, dtype=int), np.ones(40, dtype=int)]
alpha = 0.1

detector = ConformalDetector(
    detector=IsolationForest(n_estimators=100, random_state=42),
    strategy=Split(n_calib=1_000),
    seed=42,
).fit(x_reference)

bh_selected = np.asarray(detector.select(x_family, alpha=alpha))
result = detector.last_result
assert result is not None
assert result.p_values is not None
p_values = result.p_values

pointwise_selected = p_values <= alpha
storey_selected = result.adjusted_p_values(method="storey") <= alpha
by_adjusted = false_discovery_control(p_values, method="by")
by_selected = by_adjusted <= alpha

for name, selected in {
    "pointwise": pointwise_selected,
    "BH": bh_selected,
    "Storey-BH": storey_selected,
    "BY": by_selected,
}.items():
    print(
        name,
        {
            "discoveries": int(selected.sum()),
            "realized_fdp": float(false_discovery_rate(y_family, selected)),
            "power": float(statistical_power(y_family, selected)),
        },
    )

certificate = result.fdp_bounds(confidence=0.95)
print(certificate.to_frame([0.005, 0.01, 0.025, 0.05, 0.1]).to_string(index=False))
cutoff = 0.01
mask = certificate.select(cutoff)
print("FDP upper bound at cutoff:", certificate.bound_at(cutoff))
print("Discoveries at cutoff:", int(mask.sum()))

The pointwise rule does not account for the 200 simultaneous tests. BH is less conservative than BY when its independence or positive-dependence conditions apply. BY supports a broader dependence class, but neither procedure repairs invalid p-values or adaptive family construction.

Storey-BH estimates the share of true nulls and runs BH at a correspondingly higher level. storey_selected equals detector.select(x_family, alpha=alpha, method="storey") for this snapshot without scoring again. Choose one procedure before inspecting the family; see Adaptive BH with Storey's estimator for its scope.

The labeled realized_fdp is the FDP of this one family. FDR is the expected FDP over repetitions. Do not interpret one low realized FDP as validation of the FDR theorem.

Why the FDP certificate is different

The certificate is simultaneous over p-value thresholds within its documented scope. This supports post-hoc threshold exploration while attaching a high-confidence realized-FDP upper bound. It does not select by BH and does not replace the expected-FDR target.

The native snapshot API requires unmodified snapshots from unweighted Split or detached calibration with Empirical p-values. It rejects weighted, KDE, conditional-calibration, and resampling-strategy result bundles.

The ordinary call accepts only confidence and prepares one default MC-THC envelope with 1,000 fresh Monte Carlo draws and boosting enabled. Its simultaneous coverage probability is joint over the data and independent sampling, not conditional on every realized envelope or fixed seed. Repeated queries reuse the envelope; repeated constructions can vary and must not be cherry-picked for more favorable bounds. The synthetic family contains 40 planted anomalies and uses 1,000 held-out calibration observations so that the example shows a useful nonempty certificate. Labels evaluate the example; one favorable batch does not establish coverage.

For advanced envelope settings, use FDPCertificate.from_p_values(...) and choose the method and all settings before inspecting the final curve. A fixed Monte Carlo seed gives reproducibility, not a separate conditional coverage guarantee. The expert deterministic method="ks" uses the dependent-law transductive DKW construction, rather than the ordinary i.i.d. one-sample KS distribution.

For weighted covariate shift, use the separate weighted WCS example. For online hypotheses and single-stream change evidence, see the distinctions in the FDR guide.