Online Fallback Procedure¶

>>> # Create online fallback instance
>>> fallback = OnlineFallback(alpha=0.05)
>>> # Test p-values sequentially
>>> decision1 = fallback.test_one(0.01)  # First test
>>> decision2 = fallback.test_one(0.03)  # Higher power if previous rejected
>>> print(f"Decisions: {decision1}, {decision2}")

>>> # Sequential testing with FWER guarantee
>>> p_values = [0.001, 0.8, 0.02, 0.9, 0.005]
>>> decisions = [fallback.test_one(p) for p in p_values]
>>> print(f"FWER-controlled decisions: {decisions}")

  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101

class OnlineFallback(AbstractSequentialTest):
    """Online Fallback procedure for controlling the familywise error rate (FWER) online.

    The online fallback procedure, developed by Tian and Ramdas (2021), provides strong
    FWER control for testing an a priori unbounded sequence of hypotheses one by one
    over time without knowing the future. It ensures that with high probability there
    are no false discoveries in the entire sequence.

    This procedure is uniformly more powerful than traditional Alpha-spending methods
    and strongly controls the FWER even under arbitrary dependence among p-values.
    It uses a gamma sequence to allocate alpha budget over time and includes a "fallback"
    mechanism that increases future testing power after making discoveries.

    The method unifies algorithm design concepts from offline FWER control and online
    false discovery rate control, providing a powerful adaptive approach for sequential
    hypothesis testing scenarios.

    Args:
        alpha: Target familywise error rate (e.g., 0.05 for 5% FWER). Must be in (0, 1).

    Attributes:
        alpha0: Original target FWER level.
        last_rejected: Whether the previous hypothesis was rejected.
        seq: Gamma sequence for alpha allocation over time.
        num_test: Number of hypotheses tested so far.
        alpha: Current alpha level for the next test.

    Examples:
        >>> # Create online fallback instance
        >>> fallback = OnlineFallback(alpha=0.05)
        >>> # Test p-values sequentially
        >>> decision1 = fallback.test_one(0.01)  # First test
        >>> decision2 = fallback.test_one(0.03)  # Higher power if previous rejected
        >>> print(f"Decisions: {decision1}, {decision2}")

        >>> # Sequential testing with FWER guarantee
        >>> p_values = [0.001, 0.8, 0.02, 0.9, 0.005]
        >>> decisions = [fallback.test_one(p) for p in p_values]
        >>> print(f"FWER-controlled decisions: {decisions}")

    References:
        Tian, J., and A. Ramdas (2021). "Online control of the familywise error rate."
        Statistical Methods in Medical Research, 30(4):976-993.

        ArXiv preprint: https://arxiv.org/abs/1910.04900
    """

    def __init__(
        self,
        alpha: float,
    ):
        super().__init__(alpha)
        self.alpha0: float = alpha
        self.last_rejected: bool = False
        self.seq = DefaultLordGammaSequence(c=0.07720838)

    def test_one(self, p_val: float) -> bool:
        """Test a single p-value using the online fallback procedure.

        The online fallback procedure adjusts the alpha level based on whether the
        previous test was rejected (fallback mechanism) and allocates additional
        alpha using a gamma sequence. This creates higher power for future tests
        when discoveries are made.

        Args:
            p_val: P-value to test. Must be in [0, 1].

        Returns:
            True if the null hypothesis is rejected (discovery), False otherwise.

        Raises:
            ValueError: If p_val is not in [0, 1].

        Examples:
            >>> fallback = OnlineFallback(alpha=0.05)
            >>> fallback.test_one(0.01)  # First test, uses base alpha
            True
            >>> fallback.test_one(0.03)  # Second test, higher power after discovery
            True
            >>> fallback.test_one(0.04)  # Third test, even higher power
            False

        Note:
            The alpha level increases when the previous test was rejected, implementing
            the "fallback" mechanism that provides additional testing power.
        """
        validity.check_p_val(p_val)
        self.num_test += 1

        self.alpha = self.last_rejected * self.alpha
        self.alpha += self.alpha0 * self.seq.calc_gamma(self.num_test)

        is_rejected = p_val < self.alpha
        self.last_rejected = bool(is_rejected)  # Fix SIM210: Use bool() instead of True if else False

        return is_rejected

>>> fallback = OnlineFallback(alpha=0.05)
>>> fallback.test_one(0.01)  # First test, uses base alpha
True
>>> fallback.test_one(0.03)  # Second test, higher power after discovery
True
>>> fallback.test_one(0.04)  # Third test, even higher power
False

 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101

def test_one(self, p_val: float) -> bool:
    """Test a single p-value using the online fallback procedure.

    The online fallback procedure adjusts the alpha level based on whether the
    previous test was rejected (fallback mechanism) and allocates additional
    alpha using a gamma sequence. This creates higher power for future tests
    when discoveries are made.

    Args:
        p_val: P-value to test. Must be in [0, 1].

    Returns:
        True if the null hypothesis is rejected (discovery), False otherwise.

    Raises:
        ValueError: If p_val is not in [0, 1].

    Examples:
        >>> fallback = OnlineFallback(alpha=0.05)
        >>> fallback.test_one(0.01)  # First test, uses base alpha
        True
        >>> fallback.test_one(0.03)  # Second test, higher power after discovery
        True
        >>> fallback.test_one(0.04)  # Third test, even higher power
        False

    Note:
        The alpha level increases when the previous test was rejected, implementing
        the "fallback" mechanism that provides additional testing power.
    """
    validity.check_p_val(p_val)
    self.num_test += 1

    self.alpha = self.last_rejected * self.alpha
    self.alpha += self.alpha0 * self.seq.calc_gamma(self.num_test)

    is_rejected = p_val < self.alpha
    self.last_rejected = bool(is_rejected)  # Fix SIM210: Use bool() instead of True if else False

    return is_rejected

Overview¶

The Online Fallback procedure provides strong familywise error rate (FWER) control for testing an a priori unbounded sequence of hypotheses one by one over time. Unlike FDR procedures, it ensures that with high probability there are no false discoveries in the entire sequence.

Key Features¶

• FWER control: Strong control under arbitrary dependence
• Fallback mechanism: Increased power after making discoveries
• Unbounded sequences: No need to specify total number of tests
• Adaptive power: Testing power increases with discoveries
• General dependence: Valid under arbitrary p-value dependence

Mathematical Framework¶

The online fallback procedure maintains an alpha wealth process \(\{W_t\}\) where:

1. Initial wealth: \(W_0 = \alpha\)
2. Alpha allocation: \(\alpha_t = \gamma_t \cdot W_{t-1}\) (if no recent discovery)
3. Fallback mechanism: \(\alpha_t = W_{t-1}\) (if previous test was rejected)
4. Wealth update: \(W_t = W_{t-1} - \alpha_t\) (spend alpha)
5. Discovery reward: \(W_t = W_t + \alpha_t\) (earn back on rejection)

Algorithm Details¶

For test \(t = 1, 2, \ldots\):

1. Calculate alpha level:
2. If previous test rejected: \(\alpha_t = W_{t-1}\) (full wealth)

3. Otherwise: \(\alpha_t = \gamma_t \cdot \alpha_0\) (gamma sequence)

4. Make decision: Reject \(H_t\) if \(P_t \leq \alpha_t\)

5. Update wealth:

6. Spend: \(W_t = W_{t-1} - \alpha_t\)
7. If rejected: \(W_t = W_t + \alpha_t\) (earn back)

Fallback Mechanism¶

The key innovation is the "fallback" property: - After discovery: Next test uses all remaining wealth - High power: Discoveries enable aggressive follow-up testing
- FWER preservation: Wealth accounting ensures error control - Momentum: Recent discoveries create testing momentum

Implementation¶

The OnlineFallback class uses: - LORD gamma sequence for baseline alpha allocation - Boolean flag to track recent discoveries - Wealth-based alpha calculation with fallback

Examples¶

Basic FWER Control¶

from online_fdr.spending import OnlineFallback

# Initialize online fallback procedure
fallback = OnlineFallback(alpha=0.05)

# Test p-values sequentially
p_values = [0.3, 0.01, 0.02, 0.4, 0.003]
decisions = []

for i, p_val in enumerate(p_values, 1):
    decision = fallback.test_one(p_val)
    decisions.append(decision)
    print(f"Test {i}: p={p_val:.3f}, alpha={fallback.alpha:.4f}, reject={decision}")

print(f"Total discoveries: {sum(decisions)}")
print(f"FWER controlled at level {fallback.alpha0}")

Demonstrating Fallback Effect¶

# Show how alpha increases after discovery
fallback = OnlineFallback(alpha=0.05)

print("Without discovery:")
fallback.test_one(0.8)  # No rejection
print(f"Next alpha: {fallback.alpha:.4f}")

print("\\nWith discovery:")
fallback_2 = OnlineFallback(alpha=0.05)
fallback_2.test_one(0.01)  # Rejection
print(f"Next alpha after discovery: {fallback_2.alpha:.4f}")

Sequential Testing Scenario¶

# Simulate drug development pipeline
fallback = OnlineFallback(alpha=0.05)

# Stage 1: Screening compounds
screening_p = [0.6, 0.4, 0.02, 0.7, 0.1]  # One promising compound
decisions = [fallback.test_one(p) for p in screening_p]
print(f"Screening phase: {sum(decisions)} compounds selected")

# Stage 2: Detailed testing (higher power due to fallback)
detailed_p = [0.03, 0.008, 0.15]  # Follow-up on promising leads
for p in detailed_p:
    decision = fallback.test_one(p)
    print(f"Detailed test: p={p}, alpha={fallback.alpha:.4f}, reject={decision}")

Comparison with Other Methods¶

Applications¶

The online fallback procedure is particularly suitable for:

• Drug discovery: Sequential compound testing
• Clinical monitoring: Safety signal detection
• Quality control: Manufacturing process monitoring
• A/B testing: Sequential feature testing with strict error control
• Genomic screening: Gene discovery with strong error control

Theoretical Properties¶

Theorem (FWER Control): Under arbitrary dependence among p-values, the online fallback procedure controls FWER at level \(\alpha\).

Key insight: The wealth process \(\{W_t\}\) forms a supermartingale under the global null hypothesis, ensuring \(\mathbb{E}[V_\infty] \leq \alpha\) where \(V_\infty\) is the total number of false discoveries.

Advantages and Limitations¶

Advantages¶

• Strong FWER control under arbitrary dependence
• Increased power after discoveries (fallback mechanism)
• No need to pre-specify number of tests
• Intuitive wealth-based interpretation

Limitations¶

• More conservative than FDR methods when many discoveries expected
• Requires careful wealth management
• May exhaust alpha budget quickly without discoveries
• Less suitable when false discoveries are acceptable

Best Practices¶

1. Use when strict error control is required (no false discoveries acceptable)
2. Monitor wealth levels to avoid early exhaustion
3. Consider prior information about expected discovery rates
4. Plan sequential testing strategy to maximize fallback benefits
5. Document decision process for regulatory compliance

References¶

Tian, J., and A. Ramdas (2021). "Online control of the familywise error rate." Statistical Methods in Medical Research, 30(4):976-993.

Ramdas, A., T. Zrnic, M. Wainwright, and M. Jordan (2018). "SAFFRON: an adaptive algorithm for online control of the false discovery rate." Proceedings of the 35^th International Conference on Machine Learning, 80:4286-4294.

Foster, D. P., and R. A. Stine (2008). "α-investing: a procedure for sequential control of expected false discoveries." Journal of the Royal Statistical Society: Series B, 70(2):429-444.