Head start accountability systems may be missing how classroom quality varies within preschool centers

American Educational Research Association

Main Findings:

• The high-stakes accountability policies used to monitor the quality of Head Start preschool centers may miss important variation in classroom quality within centers, which could lead to incorrect representations of center quality and inaccurate decisions about which programs need to re-compete for their funding.
• Federal and state accountability systems randomly select a portion of classrooms within a given center to determine center-wide quality, under the assumption that quality is consistent across a center's classrooms. However, across Head Start programs, the authors found that one third to one half of the variation in quality is due to differences between classrooms within a center, as opposed to quality differences from center to center.
• The authors' analysis suggests that 37 percent of centers in their sample would have received different funding decisions by the major accountability system for Head Start, the Head Start Designation Renewal System (DRS), depending on which half of classrooms in a center were randomly included in the assessment of quality.
• Furthermore, average center-level quality, as determined by current accountability policies, was not found to be related to measures of children's development, calling into question the common approach of averaging classroom quality within centers to represent children's experiences. Instead, differences in classroom instructional quality within a center was significantly related to differences in children's academic and social skills.
• "Head Start has a long history of applying research to program improvement," said study coauthor Terri J. Sabol, an assistant professor of human development and social policy at Northwestern University. "Indeed, the major accountability system for Head Start, the Head Start Designation Renewal System, drew from best practices in research on the importance of classroom quality, including structural and process quality, for children. However, our findings suggest that there are still key issues to address in how we measure quality and use measurements to hold programs accountable and to encourage quality improvements."
• "Head Start has done an impressive job investing in and improving quality over the past decades and, by and large, is meeting the needs of young children," said study coauthor Emily C. Ross, who is currently a policy fellow for the American Association for the Advancement of Science and the Society for Research in Child Development. "This study speaks specifically to concerns about how accountability systems monitor quality. It suggests the importance of ensuring that quality is measured and represented accurately so that, ultimately, all children can have a high-quality experience."

Details:

• Head Start is the nation's largest publicly funded preschool program for low-income children, serving more than 1 million children each year with a federal annual budget of more than $9 billion. It is administered by the U.S. Department of Health and Human Services, which awards grants to individual agencies that operate local center-based programs. Nationwide, there are currently about 1,700 Head Start agencies, which provide program services to about 15,000 Head Start centers with more than 41,000 classrooms.
• Many have argued that a way to boost and sustain the effectiveness of Head Start is by regulating quality to improve children's direct experiences with the program. Head Start programs are increasingly subject to high-stakes accountability systems.
• Most policies designed to regulate and rate Head Start quality--including the DRS and state policy quality initiatives such as the Quality Rating and Improvement Systems (QRIS)--do so at the level of the agency or center.
• Both Head Start DRS and QRIS randomly select a subset of classrooms within a given center (or agency) to represent quality, on the assumption that a subset of classrooms selected within a center generalizes to all classrooms within the center. Yet variation among classrooms within the same center and agency is possible under the current guidelines, which are set by Congress through the Head Start Program Performance Standards.
• For their study, the authors examined variation in common indicators of classroom quality in current accountability systems--including class size, child-adult ratio, teacher education, the global environment, and instructional support--using a large, nationally representative sample of Head Start centers.
• The authors assessed data on 196 centers, 596 classrooms, and 4,130 students from the Department of Health and Human Services' 2006 and 2009 cohorts of the Head Start Family and Child Experiences Survey (FACES), to determine the extent to which classroom quality varies within centers, whether current accountability practices provide an accurate or fair representation of center quality, and how classroom- and center-level quality relates to children's outcomes.
• The analysis was not an exact replication of the entire Head Start DRS, but rather a simulation focused on a potential issue about quality ratings that has implications for how accountability systems are structured. Also, the issue about how classrooms are selected in centers or agencies applies only to the larger centers/agencies that do not assess quality for all classrooms but, rather, randomly select a subset of classrooms to represent quality.
• "Despite decades of research on Head Start, there is surprisingly little research on the extent to which classroom quality varies within Head Start centers," said Sabol. "Our results indicate that a number of current choices in how Head Start centers are evaluated may interfere with fairness and accuracy in accountability systems."
• "We recognize the cost and time burden of collecting data for all classrooms within a center," Sabol said. "However, with the high stakes of many accountability systems, it is important to get the structure of the systems correct. The next step for the research field is to figure out how many classrooms within a center should be sampled to get an accurate representation of quality."
• "Teachers and classrooms matter," Sabol said. "Although it is important to select a high-quality school, our results suggest that it's also important--if not more important--to find high-quality teachers within schools. These findings apply to school administrators, as well, in terms of offering support services to ensure that all teaching within a school is of high quality."