Power Analysis and Meta-Analysis are Falsely Reassuring Solutions for Concerns about Statistical Power
Colloquium | April 19 | 12:10-1:15 p.m. | 5101 Tolman Hall
Leif Nelson, Professor, Haas School of Business
Transparency, disclosure, and preregistration have revealed that some statistically underpowered studies may be propped up by selective reporting and p-hacking. Conscientious researchers therefore have renewed investment in statistical power. One approach is to conduct traditional power analysis to correctly set the sample size. I argue that this tool is not helpful and often biased toward samples which are too small. A different approach is to conduct underpowered studies, but meta-analyze across a collection. I argue that this tool is strikingly prone to false-positive inferences. Statistical power is a hugely important problem, but the most widely suggested tools do not offer a helpful solution.