False coverage rate

Definitions

Not keeping the FCR means \text{FCR}q when q= \frac{V}{R} = \frac{\alpha m_0}{R} , where m_0 is the number of true null hypotheses, R is the number of rejected hypothesis, V is the number of false positives, and \alpha is the significance level. Intervals with simultaneous coverage probability 1-q can control the FCR to be bounded by q.

Classification of multiple hypothesis tests

Main article: Classification of multiple hypothesis tests

The problems addressed by FCR

Selection

Selection causes reduced average coverage. Selection can be presented as conditioning on an event defined by the data and may affect the coverage probability of a CI for a single parameter. Equivalently, the problem of selection changes the basic sense of P-values. FCR procedures consider that the goal of conditional coverage following any selection rule for any set of (unknown) values for the parameters is impossible to achieve. A weaker property when it comes to selective CIs is possible and will avoid false coverage statements. FCR is a measure of interval coverage following selection. Therefore, even though a 1 − α CI does not offer selective (conditional) coverage, the probability of constructing a no covering CI is at most α, where

: \Pr[\theta \not\in \mathrm{CI},\ \text{CI constructed}] \leq \Pr[\theta \not\in \mathrm{CI}] \leq \alpha

Selection and multiplicity

When facing both multiplicity (inference about multiple parameters) and selection, not only is the expected proportion of coverage over selected parameters at 1−α not equivalent to the expected proportion of no coverage at α, but also the latter can no longer be ensured by constructing marginal CIs for each selected parameter. FCR procedures solve this by taking the expected proportion of parameters not covered by their CIs among the selected parameters, where the proportion is 0 if no parameter is selected. This false coverage-statement rate (FCR) is a property of any procedure that is defined by the way in which parameters are selected and the way in which the multiple intervals are constructed.

Controlling procedures

Bonferroni procedure (Bonferroni-selected–Bonferroni-adjusted) for simultaneous CI

Simultaneous CIs with Bonferroni procedure when we have m parameters, each marginal CI constructed at the 1 − α/m level. Without selection, these CIs offer simultaneous coverage, in the sense that the probability that all CIs cover their respective parameters is at least 1 − α. unfortunately, even such a strong property does not ensure the conditional confidence property following selection.

FCR for Bonferroni-selected–Bonferroni-adjusted simultaneous CI

The Bonferroni–Bonferroni procedure cannot offer conditional coverage, however it does control the FCR at

FCR-adjusted BH-selected CIs

In BH procedure for FDR after sorting the p values P(1) ≤ • • • ≤ P(m) and calculating R = max{ j : P( j) ≤ j • q/m}, the R null hypotheses for which P(i) ≤ R • q/m are rejected. If testing is done using the Bonferroni procedure, then the lower bound of the FCR may drop well below the desired level q, implying that the intervals are too long. In contrast, applying the following procedure, which combines the general procedure with the FDR controlling testing in the BH procedure, also yields a lower bound for the FCR, q/2 ≤ FCR. This procedure is sharp in the sense that for some configurations, the FCR approaches q.

1. Sort the p values used for testing the m hypotheses regarding the parameters, P(1) ≤ • • • ≤P(m).

2. Calculate R = max{i : P(i) ≤ i • q/m}.

3. Select the R parameters for which P(i) ≤ R • q/m, corresponding to the rejected hypotheses.

4. Construct a 1 − R • q/m CI for each parameter selected.

References

Footnotes

Other Sources

References

1. (March 2005). "False Discovery Rate–Adjusted Multiple Confidence Intervals for Selected Parameters". Journal of the American Statistical Association.
2. [http://primage.tau.ac.il/libraries/theses/exeng/free/2979859.pdf Theoretical Results Needed for Applying the False Discovery Rate in Statistical Problems]. April, 2001 (Section 3.2, Page 51)