-
Begin with N of 10, increase by 10 until p < 0.05 or max N reached.
-
This design has inflated type-I error.
-
Lower p-value threshold needed to ensure specified type-I error rate.
-
The number of interim analyses and max N affect the type-I error rate.
-
Threshold can be identified using simulation.
A recent Facebook post to a statistician group highlighted a basic science article (in a Nature journal: https://www.nature.com/articles/s41467-017-02765-w#Sec10) that used a ‘foolproof’ study design: “we continuously increased the number of animals until statistical significance was reached to support our conclusions.” Many of the statisticians in the group were exasperated! The group was rightly critical of the authors’ brazen approach, but also due to the resulting inflated type-I error rate. However, as the following code demonstrates, this approach needs only a simple modification to become a valid (i.e., by preserving the specified type-I error rate) adaptive trial design.
For simplicity (and completeness), consider a block-randomized two sample comparison using a t-test. Then, begin with a sample size of 10 in each group and increase by 10 until p < 0.05 or until a maximum sample size is reached.
The code chunk below simulates this study design. This process was repeated many times under the null hypothesis (no difference in mean outcome) and the test results used to estimate the actual type-I error rate, as well as the median (IQR) total sample size. When the maximum sample size is 100 per group, the type-I error rate is about 19% and the median (IQR) sample size is 100 (100, 100) per group. If the sample size increment is 5 instead of 10, the type-I error rate is about 24%. When the maximum sample size is 1000 per group, type-I error rate is about 37% and the median (IQR) sample size is 1000 (200, 1000) per group. Thus, smaller sample size increment (more frequent interim analysis) and larger maximum sample size both result in a larger type-I error rate.
19%, 24%, and 37% are obviously unacceptably large type-I error rates. However, the specified type-I error rate (usually 5%) can be achieved by modifying the p-value threshold. The final code chunk below demonstrates that, by using a p-value threshold of 0.01 (found by guess-and-check), maximum sample size of 100, and sample size increment at 10, the type-I error rate is about 5%. This also has the effect that the maximum sample size is reached in about 95% of simulated studies, under the null.
``
1 | |
can the type-I error be fixed by adjusting threshold?
repeat procedure
out <- replicate(5000, alg(alp=0.01, nmx=100), simplify=FALSE)
estimate type-I error
mean(sapply(out, [[, ‘rej’))
distribution of total sample size (pairs)
table(sapply(out, [[, ‘n’))
Related