The Threat Posed By Small Inaccuracies in Evaluations that Seek to Detect Small Effects

Evaluators of education interventions are increasingly designing studies to detect impacts that researchers once considered too small to be meaningful. While the need to detect smaller impacts is based on compelling arguments, a new report from the Institute of Education Sciences examines how it creates a new challenge for researchers: the need to guard against small inaccuracies that systematically lead researchers to overestimate or underestimate the effects. The report assessed the threat of small inaccuracies in evaluations using two of the most rigorous designs commonly used in education research—randomized controlled trials (RCTs) and regression discontinuity designs (RDDs). In the case of RCTs, the report examined inaccuracies due to missing outcome data.  These analyses were based on models of missing data used by the What Works Clearinghouse. In the case of RDDs, the report examined bias due to the use of incorrect regression models. These analyses used simulations that were based on data collected in several prior education evaluations. The report finds that small inaccuracies can pose a real threat in evaluations that need to detect small impacts. Specifically, it finds that: • For RCTs, evaluators must either achieve much lower rates of missing data than before or offer a strong justification for why missing data are unlikely to be related to study outcomes; and • For RDDs, state-of-the-art statistical methods can protect against inaccuracies from incorrect regression models, but this protection comes at a cost – much larger sample sizes are needed in order to detect small effects when using these methods. The report recommends other strategies researchers can use to avoid or reduce inaccuracies in studies designed to detect small impacts. It concludes that the threat of small inaccuracies pose in these contexts is real but surmountable—so long as it is not ignored.