Skip to content
Edit this page View source of this page

Statistical Concepts

A practitioner reference for the statistical terms used throughout nonconform. Each entry says what the term means, how it shows up in the library, and what can go wrong in practice.

For more detail, see Understanding Conformal Inference.


P-values

What it is: A number in \([0,1]\) whose null distribution is super-uniform when the test's assumptions hold. Smaller values represent stronger evidence against that null; a p-value is not an effect size or a posterior probability.

In nonconform: A conformal p-value compares a test score with calibration scores. Smaller values mean stronger evidence that the point does not look like the calibration data.

Guarantee: For \(a\in[0,1]\), a valid null p-value satisfies \(\Pr(p \le a) \le a\). For split conformal anomaly testing, this is ordinarily a marginal statement over the random calibration set and a null test point, conditional on an independently fitted, fixed scoring rule.

Common mistake: A small p-value is not a probability that the point is an anomaly. It is evidence against the point behaving like the calibration reference population.

Classical vs. Randomized:

  • Empirical() defaults to tie_break="classical", which gives discrete p-values in steps of \(1/(n+1)\) and includes calibration scores tied with the test score, giving deterministic conservative behavior.
  • Valid tie_break values are "classical" and "randomized" (or TieBreakMode.CLASSICAL / TieBreakMode.RANDOMIZED); None is invalid.
  • For smoother p-values, use Empirical(tie_break="randomized").
  • Probabilistic() returns a continuous KDE estimate of score-tail probability. It is model-based and does not inherit the empirical rank's exact finite-sample validity. The library does not attach a general asymptotic-validity claim to it.

E-values

What it is: A non-negative evidence value where larger values indicate stronger evidence against a null hypothesis. Unlike p-values, e-values are designed to be averaged across certain dependent analyses when their validity conditions hold.

In nonconform: DerandomizedSplits uniformly averages evidence from repeated conformal splits and selects discoveries using e-BH. See the derandomized e-value guide for usage, tie handling, diagnostics, and expert interfaces.

Guarantee: The construction targets an aggregate null-evidence condition that is sufficient for e-BH FDR control under the method assumptions; it does not claim that each constructed value is individually an ordinary e-value. The scope requires exchangeable inliers/null test points, a strict or independently randomized score ordering, one fixed test family in consistent row order, valid integrated unweighted Split score maps, and one final e-BH filtering step.

Common mistake: Do not treat e-values as p-values or threshold them at ordinary p-value cutoffs. E-BH selects using the ordering and scale of the evidence together with a target FDR level.


False Discovery Rate (FDR)

What it is: Let \(V\) be the number of false discoveries and \(R\) the total number of discoveries in one testing family. Its realized false discovery proportion is \(\mathrm{FDP}=V/\max(R,1)\). The false discovery rate is the repeated-sampling expectation \(\mathrm{FDR}=\mathbb{E}[\mathrm{FDP}]\).

Why it matters: When you test many observations, some normal points will look anomalous by chance. FDR control targets the expected false-positive proportion among discoveries, for example at most 5% in expectation when the assumptions hold.

In nonconform: detector.select(X_test, alpha=...) applies BH for p-value strategies, e-BH for DerandomizedSplits, and WCS in weighted mode. alpha is a nominal FDR target, not a bound on the FDP of every realized family. The corresponding guarantee requires the evidence conditions and assumptions of the selected procedure. The historical false_discovery_rate(...) metric returns realized FDP for supplied labels.

detector.fdp_bounds(x, confidence=0.95) instead prepares an immutable simultaneous upper curve for realized FDP across p-value thresholds in one fixed family. Its default envelope is sampled once using 1,000 fresh Monte Carlo draws; queries reuse it. Coverage is joint over the data and independent sampling, not conditional on every fixed seed or realized envelope. A seed in the expert FDPCertificate.from_p_values(...) factory gives reproducibility; it does not establish a separate conditional guarantee. Keep the scoring map, family, and construction settings fixed, and do not cherry-pick repeated constructions. certificate.select(0.05) applies a p-value cutoff, not an FDP target. See the FDP guide.


Raw-score False-Positive Rate (FPR)

What it is: For an anomalous-higher score \(S\), the FPR at threshold \(t\) is \(\Pr(S\ge t)\) when the future observation is a genuine inlier. It is a per-inlier false-alarm probability, not the fraction of false discoveries among all selected observations.

In nonconform: detector.fpr_bounds() returns an immutable FPRCertificate. Its simultaneous upper curve can be queried with bound_at(t) and inverted with threshold_for(target_fpr). Apply the resulting rule with certificate.select(scores, target_fpr=...).

Guarantee: Calibration and future inlier scores must be independent and identically distributed (i.i.d.) conditional on a scoring map fixed independently of the calibration data. Calibration observations must be clean inliers from the deployment inlier distribution; exchangeability alone is insufficient. Under these assumptions, with probability at least confidence over the calibration draw, conditional on the independently fitted score map, the certificate bounds the population inlier FPR simultaneously at every threshold. On that event, inversion gives a threshold whose future-inlier false-alarm probability is at most target_fpr. This is a guarantee conditional on the realized calibration sample, rather than marginal validity averaged over calibration samples.

Native certificates support unweighted Split calibration, including detached calibration, independently of the configured p-value estimator. They use the exact finite-sample one-sided KS distribution, evaluated deterministically; ties make the bound conservative. Its finite-threshold floor is about 3.85% with 1,000 calibration scores at 95% confidence. A target below the floor returns an infinite threshold and an empty selection.

Common mistake: A 5% FPR target does not imply 5% FDR in a batch containing anomalies, and it does not bound the probability of at least one false alarm in an arbitrarily large batch or a realized false-alarm proportion in every finite batch. It also provides no recall, FNR, or total-cost guarantee.


Anytime False-Alarm Control (Ville Bound)

What it is: A sequential false-alarm guarantee for nonnegative evidence processes. If \(M_t\) is a valid nonnegative supermartingale under the null and \(M_0\le1\), then for any \(\lambda>0\):

\[ \Pr\left(\sup_t M_t \ge \lambda\right) \le \frac{1}{\lambda}. \]

In nonconform: ExchangeabilityMonitor supplies randomized sequential-rank p-values to the configured martingale. AlarmConfig(ville_threshold=lambda) uses this style of anytime alarm thresholding for the product exchangeability martingale. AlarmConfig(restarted_ville_threshold=lambda) applies the same Ville threshold to a restarted mixture e-process (evidence process) designed for sensitivity to changes that begin later in the monitored stream. This is not a uniform power improvement because later restart times receive less prior mass. The restart prior is the weighting over possible restart times; see Exchangeability Martingales for the documented default.

This guarantee applies to false alarms over time on a single stream. For multiple testing settings across many hypotheses or streams, use dedicated FDR procedures; see Exchangeability Martingales and FDR Control.

Repeatedly calling ConformalDetector.compute_p_value() uses one fixed calibration ECDF and does not by itself supply the conditional sequential validity needed here. Enabling both ordinary and restarted Ville alarms also requires error allocation if action is taken when either alarm fires.


Exchangeability

What it is: Data points are exchangeable if shuffling their order does not change their joint distribution. For many practical workflows, "same population, same measurement process, no systematic time/order effect" is the operational check.

Why it matters: With a scoring rule fitted independently and then frozen, exchangeability of calibration scores and the relevant true-null test score supports the standard split-conformal rank argument.

Operational evidence: Calibration and target null observations are sampled under the same mechanism, preprocessing and inclusion rules do not depend on their eventual scores, and there is no unmodeled ordering or group effect. These checks support exchangeability but cannot prove it from one dataset.

Common violations: Distribution shift, temporal dependence, duplicated or clustered observations treated as independent rows, outcome-dependent sampling, and preprocessing fitted using calibration or target information.


Calibration Set

What it is: Reference observations scored by a fixed detector to form the empirical null comparison set. In Split, these are held out from detector fitting; detached calibration accepts a separately supplied set.

Why it matters: The calibration set provides the "baseline" for computing p-values. Test scores are compared against calibration scores.

How big should it be: Empirical conformal p-values move in steps of \(1/(n+1)\) with n calibration samples. Small calibration sets give coarse p-values and can make FDR selection conservative or powerless. There is no universal minimum: choose the count from the required testing resolution, family size, detector fitting needs, and measured stability.


Statistical Power

What it is: The probability of rejection under a specified alternative. Across a realized labeled family, the analogous descriptive quantity is recall or true positive rate, \(\mathrm{TP}/(\mathrm{TP}+\mathrm{FN})\).

In nonconform: statistical_power(y_true, predictions) retains its v1 name but computes the realized true positive rate for those arrays. Repeated simulation or sampling under a declared alternative is needed to estimate power.

Trade-off: For a fixed scoring system and data design, relaxing the rejection threshold typically increases both power and false positives. Choose the FDR level from the operational cost of false alarms versus missed anomalies.


Covariate Shift

What it is: A shift from a calibration covariate distribution \(P_X\) to a target distribution \(Q_X\) while the relevant conditional data-generating mechanism remains invariant. Weighted conformal methods use the density ratio \(w(x)=dQ_X/dP_X\) and require target support to be covered by calibration support.

Solution: Use weighted conformal prediction only when the shift is plausibly covariate shift with sufficient support overlap and reliable weights. If the anomaly mechanism changes, weighting alone does not restore the guarantees. See Weighted Conformal.


Key Relationships

Concept Statistical role Depends on
p-value Supports a level-\(a\) test through null super-uniformity Null and calibration assumptions; calibration size sets rank resolution
FDR Expected FDP of a declared testing family Null p-value validity, dependence, family definition, and selection procedure
FPR certificate High-probability simultaneous upper bound on population inlier FPR conditional on calibration Score map fixed independently of calibration, clean i.i.d. calibration and future inlier scores, confidence, and supported Split scope
Ville threshold Bounds ever-crossing probability for one valid stream e-process validity and threshold choice
Restarted Ville threshold Applies the Ville bound to a restart-mixture e-process Component e-process validity and restart prior
Power Rejection probability under an alternative Alternative distribution, scorer, calibration, and decision rule
Exchangeability Supplies rank symmetry used for conformal validity Joint sampling, dependence, ordering, and preprocessing

References

For mathematical foundations and implementation context: