Conventional value-added models may be misleading, but are likely to generate substantial achievement gains

 

Conventional value-added models (VAMs) compare average test scores across schools after adjusting for students’ demographic characteristics and previous scores. The resulting VAM estimates are biased if the available control variables fail to capture all cross-school differences in student ability. 

This paper introduces a new test for VAM bias that asks whether VAM estimates accurately predict the achievement consequences of random assignment to specific schools. 

Test results from admissions lotteries in Boston suggest conventional VAM estimates may be misleading. This finding motivates the development of a hierarchical model describing the joint distribution of school value-added, VAM bias, and lottery compliance. The researchers used this model to assess the substantive importance of bias in conventional VAM estimates and to construct hybrid value-added estimates that optimally combine ordinary least squares and instrumental variables estimates of VAM parameters. 

Simulations calibrated to the Boston data show that, bias notwithstanding, policy decisions based on conventional VAMs are likely to generate substantial achievement gains. Estimates incorporating lotteries are less biased, however, and yield further gains.