Statistical Concepts¶
A practitioner reference for the statistical terms used throughout nonconform. Each entry says what the term means, how it shows up in the library, and what can go wrong in practice.
For more detail, see Understanding Conformal Inference.
P-values¶
What it is: A number in \([0,1]\) whose null distribution is super-uniform when the test's assumptions hold. Smaller values represent stronger evidence against that null; a p-value is not an effect size or a posterior probability.
In nonconform: A conformal p-value compares a test score with calibration scores. Smaller values mean stronger evidence that the point does not look like the calibration data.
Guarantee: For \(a\in[0,1]\), a valid null p-value satisfies \(\Pr(p \le a) \le a\). For split conformal anomaly testing, this is ordinarily a marginal statement over the random calibration set and a null test point, conditional on an independently fitted, fixed scoring rule.
Common mistake: A small p-value is not a probability that the point is an anomaly. It is evidence against the point behaving like the calibration reference population.
Classical vs. Randomized:
Empirical()defaults totie_break="classical", which gives discrete p-values in steps of \(1/(n+1)\) and includes calibration scores tied with the test score, giving deterministic conservative behavior.- Valid
tie_breakvalues are"classical"and"randomized"(orTieBreakMode.CLASSICAL/TieBreakMode.RANDOMIZED);Noneis invalid. - For smoother p-values, use
Empirical(tie_break="randomized"). Probabilistic()returns a continuous KDE estimate of score-tail probability. It is model-based and does not inherit the empirical rank's exact finite-sample validity. The library does not attach a general asymptotic-validity claim to it.
E-values¶
What it is: A non-negative evidence value where larger values indicate stronger evidence against a null hypothesis. Unlike p-values, e-values are designed to be averaged across certain dependent analyses when their validity conditions hold.
In nonconform: DerandomizedSplits uniformly averages evidence from repeated
conformal splits and selects discoveries using e-BH. See the
derandomized e-value guide
for usage, tie handling, diagnostics, and expert interfaces.
Guarantee: The construction targets an aggregate null-evidence condition
that is sufficient for e-BH FDR control under the method assumptions; it does
not claim that each constructed value is individually an ordinary e-value. The
scope requires exchangeable inliers/null test points, a strict or independently
randomized score ordering, one fixed test family in consistent row order, valid
integrated unweighted Split score maps, and one final e-BH filtering step.
Common mistake: Do not treat e-values as p-values or threshold them at ordinary p-value cutoffs. E-BH selects using the ordering and scale of the evidence together with a target FDR level.
False Discovery Rate (FDR)¶
What it is: Let \(V\) be the number of false discoveries and \(R\) the total number of discoveries in one testing family. Its realized false discovery proportion is \(\mathrm{FDP}=V/\max(R,1)\). The false discovery rate is the repeated-sampling expectation \(\mathrm{FDR}=\mathbb{E}[\mathrm{FDP}]\).
Why it matters: When you test many observations, some normal points will look anomalous by chance. FDR control targets the expected false-positive proportion among discoveries, for example at most 5% in expectation when the assumptions hold.
In nonconform: detector.select(X_test, alpha=...) applies BH for p-value
strategies, e-BH for DerandomizedSplits, and WCS in weighted mode. alpha is a
nominal FDR target, not a bound on the FDP of every realized family. The
corresponding guarantee requires the evidence conditions and assumptions of
the selected procedure. The historical
false_discovery_rate(...) metric returns realized FDP for supplied labels.
detector.fdp_bounds(x, confidence=0.95) instead prepares an immutable
simultaneous upper curve for realized FDP across p-value thresholds in one
fixed family. Its default envelope is sampled once using 1,000 fresh Monte Carlo
draws; queries reuse it. Coverage is joint over the data and independent
sampling, not conditional on every fixed seed or realized envelope. A seed in
the expert FDPCertificate.from_p_values(...) factory gives reproducibility;
it does not establish a separate conditional guarantee. Keep the scoring map,
family, and construction settings fixed, and do not cherry-pick repeated
constructions. certificate.select(0.05) applies a p-value cutoff, not an FDP
target. See the FDP guide.
Raw-score False-Positive Rate (FPR)¶
What it is: For an anomalous-higher score \(S\), the FPR at threshold \(t\) is \(\Pr(S\ge t)\) when the future observation is a genuine inlier. It is a per-inlier false-alarm probability, not the fraction of false discoveries among all selected observations.
In nonconform: detector.fpr_bounds() returns an immutable
FPRCertificate. Its simultaneous upper curve can be queried with
bound_at(t) and inverted with threshold_for(target_fpr). Apply the resulting
rule with certificate.select(scores, target_fpr=...).
Guarantee: Calibration and future inlier scores must be independent and
identically distributed (i.i.d.) conditional on a scoring map fixed independently
of the calibration data. Calibration observations must be clean inliers from
the deployment inlier distribution; exchangeability alone is insufficient.
Under these assumptions, with probability at least confidence over the
calibration draw, conditional on the independently fitted score map, the
certificate bounds the population inlier FPR simultaneously at every threshold.
On that event, inversion gives a threshold whose future-inlier false-alarm
probability is at most target_fpr. This is a guarantee conditional on the
realized calibration sample, rather than marginal validity averaged over
calibration samples.
Native certificates support unweighted Split calibration, including detached
calibration, independently of the configured p-value estimator. They use the
exact finite-sample one-sided KS distribution, evaluated deterministically;
ties make the bound conservative. Its finite-threshold floor is about 3.85%
with 1,000 calibration scores at 95% confidence. A target below the floor returns
an infinite threshold and an empty selection.
Common mistake: A 5% FPR target does not imply 5% FDR in a batch containing anomalies, and it does not bound the probability of at least one false alarm in an arbitrarily large batch or a realized false-alarm proportion in every finite batch. It also provides no recall, FNR, or total-cost guarantee.
Anytime False-Alarm Control (Ville Bound)¶
What it is: A sequential false-alarm guarantee for nonnegative evidence processes. If \(M_t\) is a valid nonnegative supermartingale under the null and \(M_0\le1\), then for any \(\lambda>0\):
In nonconform: ExchangeabilityMonitor supplies randomized sequential-rank
p-values to the configured martingale. AlarmConfig(ville_threshold=lambda)
uses this style of anytime alarm thresholding for the product exchangeability
martingale.
AlarmConfig(restarted_ville_threshold=lambda) applies the same Ville threshold
to a restarted mixture e-process (evidence process) designed for sensitivity to
changes that begin later in the monitored stream. This is not a uniform power
improvement because later restart times receive less prior mass. The restart
prior is the
weighting over possible restart times; see
Exchangeability Martingales
for the documented default.
This guarantee applies to false alarms over time on a single stream. For multiple testing settings across many hypotheses or streams, use dedicated FDR procedures; see Exchangeability Martingales and FDR Control.
Repeatedly calling ConformalDetector.compute_p_value() uses one fixed
calibration ECDF and does not by itself supply the conditional sequential
validity needed here. Enabling both ordinary and restarted Ville alarms also
requires error allocation if action is taken when either alarm fires.
Exchangeability¶
What it is: Data points are exchangeable if shuffling their order does not change their joint distribution. For many practical workflows, "same population, same measurement process, no systematic time/order effect" is the operational check.
Why it matters: With a scoring rule fitted independently and then frozen, exchangeability of calibration scores and the relevant true-null test score supports the standard split-conformal rank argument.
Operational evidence: Calibration and target null observations are sampled under the same mechanism, preprocessing and inclusion rules do not depend on their eventual scores, and there is no unmodeled ordering or group effect. These checks support exchangeability but cannot prove it from one dataset.
Common violations: Distribution shift, temporal dependence, duplicated or clustered observations treated as independent rows, outcome-dependent sampling, and preprocessing fitted using calibration or target information.
Calibration Set¶
What it is: Reference observations scored by a fixed detector to form the
empirical null comparison set. In Split, these are held out from detector
fitting; detached calibration accepts a separately supplied set.
Why it matters: The calibration set provides the "baseline" for computing p-values. Test scores are compared against calibration scores.
How big should it be: Empirical conformal p-values move in steps of
\(1/(n+1)\) with n calibration samples. Small calibration sets give coarse
p-values and can make FDR selection conservative or powerless. There is no
universal minimum: choose the count from the required testing resolution,
family size, detector fitting needs, and measured stability.
Statistical Power¶
What it is: The probability of rejection under a specified alternative. Across a realized labeled family, the analogous descriptive quantity is recall or true positive rate, \(\mathrm{TP}/(\mathrm{TP}+\mathrm{FN})\).
In nonconform: statistical_power(y_true, predictions) retains its v1 name
but computes the realized true positive rate for those arrays. Repeated
simulation or sampling under a declared alternative is needed to estimate
power.
Trade-off: For a fixed scoring system and data design, relaxing the rejection threshold typically increases both power and false positives. Choose the FDR level from the operational cost of false alarms versus missed anomalies.
Covariate Shift¶
What it is: A shift from a calibration covariate distribution \(P_X\) to a target distribution \(Q_X\) while the relevant conditional data-generating mechanism remains invariant. Weighted conformal methods use the density ratio \(w(x)=dQ_X/dP_X\) and require target support to be covered by calibration support.
Solution: Use weighted conformal prediction only when the shift is plausibly covariate shift with sufficient support overlap and reliable weights. If the anomaly mechanism changes, weighting alone does not restore the guarantees. See Weighted Conformal.
Key Relationships¶
| Concept | Statistical role | Depends on |
|---|---|---|
| p-value | Supports a level-\(a\) test through null super-uniformity | Null and calibration assumptions; calibration size sets rank resolution |
| FDR | Expected FDP of a declared testing family | Null p-value validity, dependence, family definition, and selection procedure |
| FPR certificate | High-probability simultaneous upper bound on population inlier FPR conditional on calibration | Score map fixed independently of calibration, clean i.i.d. calibration and future inlier scores, confidence, and supported Split scope |
| Ville threshold | Bounds ever-crossing probability for one valid stream | e-process validity and threshold choice |
| Restarted Ville threshold | Applies the Ville bound to a restart-mixture e-process | Component e-process validity and restart prior |
| Power | Rejection probability under an alternative | Alternative distribution, scorer, calibration, and decision rule |
| Exchangeability | Supplies rank symmetry used for conformal validity | Joint sampling, dependence, ordering, and preprocessing |
References¶
For mathematical foundations and implementation context:
- Understanding Conformal Inference - conformal p-values and exchangeability assumptions.
- FDR Control - multiple testing, BH selection, and dependence assumptions.
- Weighted Conformal - covariate-shift workflows.
- Shafer & Vovk (2008) - conformal prediction tutorial.
- Bates et al. (2023) - conformal p-values for outlier testing and a uniform raw-score FPR bound.
- Benjamini & Hochberg (1995) - the original FDR procedure.
- Jin & Candès (2023) - weighted conformal p-values and WCS.
- Ramdas et al. (2023) - anytime-valid inference with e-values and martingales.