User guide¶
nonconform supports two primary workflows built on anomaly scores:
- Batch discovery control: construct p-values or e-values for a fixed family and select anomalies with a justified FDR procedure.
- Sequential change monitoring: generate randomized sequential conformal p-values, accumulate evidence with a martingale, and trigger configured alarms.
Choose the workflow first. Their evidence, error targets, and evaluation metrics are different.
Start from your task¶
| Task | Read first | Then |
|---|---|---|
| Select anomalies in one batch | Conformal inference | FDR control |
| Stabilize repeated split selections | Derandomized e-values | Batch evaluation |
| Monitor an ordered stream for change | Exchangeability martingales | Streaming evaluation |
| Handle modeled covariate shift | Weighted conformal | FDR control |
| Choose split, CV, jackknife, or bootstrap | Conformalization strategies | Choosing strategies |
| Integrate a detector | Detector compatibility | Common workflows |
| Diagnose a failure | Troubleshooting | Input validation |
Foundations¶
| Page | Purpose |
|---|---|
| Statistical concepts | Short definitions of p-values, exchangeability, FDR, power, covariate shift, and Ville control |
| Conformal inference | Rank construction, data roles, marginal and conditional validity, ties, and batch versus sequential p-values |
| Conformalization strategies | Exact mechanics and statistical scope of Split, DerandomizedSplits, CrossValidation, jackknife, and bootstrap |
| Choosing strategies | Decision process based on validity needs, resolution, model-fit budget, and empirical evaluation |
Applied workflows¶
| Page | Purpose |
|---|---|
| Detector compatibility | scikit-learn, PyOD, custom protocols, blocked batch-adaptive models, and score polarity |
| Weighted conformal | Covariate-shift assumptions, density-ratio estimators, weight diagnostics, and WCS |
| FDR control | BH, BY, WCS, derandomized e-values, post-hoc FDP certificates, repeated batches, and online FDR distinctions |
| Derandomized e-values | DerandomizedSplits with automatic repetitions, e-values, and e-BH |
| Exchangeability martingales | Sequential randomized ranks, betting martingales, alarms, and Ville scope |
Evaluation and operations¶
| Page | Purpose |
|---|---|
| Batch evaluation | Reproducible labeled families with oddball and correct FDP/power bookkeeping |
| Streaming evaluation | False-alarm, detection-delay, online-testing, and window-family designs |
| Input validation | Enforced constraints, calibration resolution, fitted state, and weighted batch identity |
| Best practices | Leakage prevention, family and episode design, reproducibility, and production review |
| Logging | Actual logger namespaces and progress controls |
| Troubleshooting | Symptom-led diagnosis without weakening guarantees post hoc |
Guarantees are conditional statements
Standard split-conformal p-values require exchangeability of calibration and true-null test scores conditional on a fixed training-only scorer. FDR control additionally requires the dependence assumptions of the selection procedure. Weighted workflows require a correct shift model, overlap, and suitable weights. Sequential Ville guarantees require conditionally valid sequential p-values and a valid e-process. Passing API validation does not establish any of these assumptions.
Review checklist¶
Before relying on a result, confirm:
- the null population, testing family, or monitoring episode was defined in advance;
- fitting, calibration, tuning, and final evaluation roles are separated;
- learned preprocessing is part of the fitted scoring construction;
- detector score polarity and fixed-state behavior are verified;
- empirical p-value resolution is adequate for the actual family;
- the selected FDR, FDP-bound, weighted, or sequential theorem matches the implementation and data design;
- labeled evaluation data did not choose the result later reported on it; and
- assumptions and failure modes accompany every statistical claim.
For agents and advanced users, the API reference exposes signatures and public docstrings, while Common workflows provides independently runnable examples.