Assessing the accuracy of evaluating principals using student test scores

States and districts need ways of measuring principal performance that correctly identify effective principals. A new report from the Institute of Education Sciences examines the accuracy of test-based measures of principal performance that could be implemented broadly.

The study assessed the predictive validity of these measures—the extent to which ratings from these measures accurately reflect principals’ contributions to student achievement in future years. Performance measures should have high predictive validity to be useful for informing personnel decisions about principals. Because such decisions determine which individuals will lead schools in subsequent years, measures should accurately identify which principals are likely to perform well in the future.

The report finds little evidence that test-based performance measures accurately predict principals’ future contributions to student achievement. Using statewide data from Pennsylvania for the 2007-08 to 2013-14 school years, the study found:

•    Performance measures that reflected students’ end-of-year achievement, without accounting for their past achievement, provided no information for predicting principals’ contributions to student achievement in the following year;

•    Performance measures that accounted for students’ past achievement by measuring their growth provided, at most, a small amount of information for predicting principals’ contributions in the following year; and

•    Averaging performance measures across multiple recent years did not improve the accuracy of these measures.

These findings suggest that states and districts should exercise caution when using these measures to make major decisions about principals. If a principal evaluation system is designed to use student test scores, measures based on student growth are preferable because they are stronger predictors than end-of-year achievement.


Details:

The study conducted two sets of analyses using Pennsylvania's statewide data on students and principals from 2007/08 to 2013/14. First, using data on 2,424 principals, the study assessed the extent to which ratings from each measure are stable by examining the association between principals' ratings from earlier and later years. Second, using data on 123 principals, the study examined the relationship between the stable part of each principal's rating and his or her contributions to student achievement in future years. Based on results from both analyses, the study simulated each measure's accuracy for predicting principals' contributions to student achievement in the following year.

The study found that the two performance measures that did not account for students' past achievement—average achievement and adjusted average achievement—provided no information for predicting principals' contributions to student achievement in the following year.

The two performance measures that accounted for students' past achievement—school value-added and adjusted school value-added—provided, at most, a small amount of information for predicting principals' contributions in the following year, with less than one-third of each difference in value-added ratings across principals reflecting differences in their future contributions.