FALSE DISCOVERY RATE
From a probabilistic perspective, the interpretation of the Bonferroni correction is that we choose a new threshold for the P-value where we don’t expect anything to pass by chance. In practice, however, with more than
25,000 GO terms used in gene set enrichment analysis and 25,000 genes measured in each experiment, there are situations where we just won’t be able to get a small enough P-value to pass the Bonferroni corrected threshold.
In these cases, we might choose another threshold where we know that some of the tests that pass are actually false discoveries (expected under the null hypothesis). We might be willing to accept some of these mistakes, as long as the fraction of tests that are false discoveries is known (and small). Changing the P-values according to this strategy is known as controlling the false discovery rate (FDR) or performing an FDR correction.
The most famous and widely used FDR correction is the so-called Benjamini-Hochberg correction (Benjamini and Hochberg 1995). This correction is applied as follows:
- 1. Calculate the P-values for each of the tests that are performed.
- 2. Rank the tests by their test statistics so that the most significant is 1st, the second most significant is 2nd, etc.
- 3. Move down the ranked list, and reject the null hypothesis for the r most significant tests, where the P-value is less than ra/m, where m is the number of tests and a is the FDR threshold.
In this case, a is no longer a P-value, but rather an estimate of the fraction of the tests that are falsely rejected, or P(H0 was true|H0 was rejected). Notice the similarity between the formula to choose the FDR P-value threshold and the Bonferroni. For r = 1, there is only one test with a small P-value; the FDR is equivalent to the Bonferroni correction. However, if there are 100 tests with small P-values, now the threshold will be 100 times less stringent than the Bonferroni correction. However, with FDR, we accept that 100 x a of those tests will actually be false.
Applying the Benjamini-Hochberg (BH) procedure to the search for genes that are differentially expressed between T cells and other cells, at a = 0.05, we have more than 13,000 genes passing the threshold. Since 5% of these are false discoveries, our estimate for the number of genes is actually closer to 12,000. Notice that this is very close to what we guessed by looking at the P-value distribution mentioned earlier, and more than 2.5x the estimate we got using the Bonferroni correction.
There are other interesting ways of controlling the FDR (Storey 2002) but the BH procedure is widely used because it is simple to calculate.