User Guide¶

Welcome to the comprehensive online-fdr user guide. This section provides in-depth explanations of concepts, methods, and best practices for online false discovery rate control.

Overview¶

Online FDR control is essential when hypotheses arrive sequentially and decisions must be made immediately. The package has two evidence surfaces: p-value procedures for small-is-strong evidence and e-value procedures for large-is-strong evidence.

Guide Structure¶

This user guide is organized into the following sections:

Concepts¶

Fundamental concepts and terminology

• False Discovery Rate (FDR) vs Family-Wise Error Rate (FWER)
• Online vs Batch testing paradigms
• Alpha spending and alpha investing principles
• Dependency structures and their implications

Sequential Testing¶

P-value methods that test one hypothesis at a time

• Alpha Investing Family (GAI, SAFFRON, ADDIS)
• LORD Family (LORD3, LORD++, D-LORD, etc.)
• LOND methods for different dependency structures
• Alpha spending approaches

E-Value Methods¶

Batch and online methods for large-is-strong evidence

• e-BH for fixed batches of valid e-values
• e-LOND for online streams of valid e-values
• Calibration, merging, e-process, and generation tooling
• Construction-level assumptions and roadmap items

Batch Testing¶

P-value methods that test multiple hypotheses simultaneously

• Benjamini-Hochberg and adaptive variants
• Methods for dependent test statistics
• When to choose batch vs online approaches

Data Generation¶

Simulation utilities for testing and validation

• Built-in data generating processes
• Creating custom simulation scenarios
• Power analysis and method evaluation

Performance Evaluation¶

Tools for assessing method performance

• FDR and power calculations
• Metrics for online settings
• Benchmarking and comparison frameworks

onlineFDR Parity¶

Method-level parity and intentional differences

• Mapping to Bioconductor onlineFDR release semantics
• Explicit Parity / IntentionalDivergence / Extension labels
• Notes on true-online API differences in this package

Quick Navigation¶

Common Workflows¶

1. Choosing the Right Method¶

graph TD
    A[Start] --> B{Online or Batch?}
    B -->|Online| C{Independence assumption?}
    B -->|Batch| D{Known null proportion?}
    C -->|Independent| E[ADDIS/SAFFRON]
    C -->|Dependent| F[LORD/LOND variants]
    C -->|Conservative| G[Alpha Spending]
    D -->|Known| H[Benjamini-Hochberg]
    D -->|Unknown| I[Storey-BH]
    D -->|Dependent| J[Benjamini-Yekutieli]

2. Parameter Selection¶

Most methods require careful parameter tuning:

• Alpha level (alpha): Your desired FDR level (typically 0.05 or 0.1)
• Initial wealth (W): Controls early power (start with alpha/2)
• Lambda (lambda): Candidate threshold for ADDIS/SAFFRON (try 0.25-0.5)
• Tau (tau): Discarding threshold for ADDIS (try 0.5)

3. Performance Monitoring¶

Track key metrics during online testing:

from online_fdr.core.utils import MemoryDecayFDR

# Initialize tracking
fdr_tracker = MemoryDecayFDR(delta=0.99, offset=0)

# During testing
for p_value, true_label in data_stream:
    decision = method.test_one(p_value)
    current_fdr = fdr_tracker.score_one(decision, true_label)

    # Optional: Stop if FDR exceeds threshold
    if current_fdr > target_alpha * 1.2:  # 20% buffer
        print("Warning: FDR approaching target level")

Best Practices¶

Do's¶

• Choose methods appropriate for your dependency structure
• Validate with simulations before real applications
• Monitor FDR in real-time for early stopping
• Use consistent random seeds for reproducibility
• Document method parameters and their rationale

Don'ts¶

• Don't switch methods mid-stream without theoretical justification
• Don't ignore dependency when present
• Don't use overly aggressive parameters without validation
• Don't forget to account for multiple testing in downstream analyses

Integration Examples¶

With Scikit-learn Pipelines¶

from sklearn.pipeline import Pipeline
from sklearn.feature_selection import SelectKBest
from online_fdr.p_values import Addis

class OnlineFDRSelector:
    def __init__(self, alpha=0.05):
        self.method = Addis(alpha=alpha, wealth=alpha/2, lambda_=0.25, tau=0.5)

    def fit_transform(self, X, y):
        # Apply statistical tests and online FDR control
        selected_features = []
        for i in range(X.shape[1]):
            p_value = statistical_test(X[:, i], y)  # Your test here
            if self.method.test_one(p_value):
                selected_features.append(i)
        return X[:, selected_features]

With A/B Testing Frameworks¶

class OnlineABTester:
    def __init__(self, alpha=0.05):
        self.method = Addis(alpha=alpha, wealth=alpha/2, lambda_=0.25, tau=0.5)

    def analyze_variant(self, control_data, variant_data):
        p_value = statistical_test(control_data, variant_data)
        decision = self.method.test_one(p_value)

        return {
            'significant': decision,
            'p_value': p_value,
            'current_alpha': getattr(self.method, 'last_rejection_threshold', None)
        }

Advanced Topics¶

For specialized use cases, see:

• Theory Section for mathematical foundations
• API Reference for complete method documentation
• Examples for domain-specific applications
• Contributing Guide for adding new methods

Getting Help¶

If you're stuck or need clarification:

1. Check the FAQ in each section
2. Browse existing GitHub Issues
3. Start a GitHub Discussion
4. Consult the referenced papers for theoretical details

Ready to dive deeper? Choose a section above or continue with Concepts for the theoretical foundations.