English Learners, One Study, and the Dangers of Translating Research into Practice

It hit the news with a bang: Immigrants learned English in half the time when they were held back in third grade, proclaimed one headline about the National Bureau of Economic Research working paper by David Figlio of Northwestern University and Umut Özek of the American Institutes for Research. Repeating 3rd Grade Could Help Struggling English-Learners, blared another. The findings, while clear, are also inconsistent with the cumulative body of research, which suggests that grade retention is not an effective means of improving academic achievement. Yet the study from Figlio and Özek, who are economists, finds that Florida English learners retained in Grade 3 enjoyed multiple academic advantages including better English skills, quicker attainment of proficiency, and higher rates of advanced course-taking in middle and high school. Florida’s policy calls for students to be retained in the third grade if they score too low on the state’s English language arts exam. Like the news coverage, the critiques of the study came hard and fast. Within weeks, a statement entitled The danger of using a single study to inform policy for English learners was released by a group of 32 education scholars with expertise in teaching those whose first language is not English, including National Education Policy Center Fellows Francesca López of the University of Arizona, and Deborah Palmer of the University of Colorado Boulder.  Shortly after the statement was released, Figlio and Özek contributed to an Education Week commentary that echoed many of the points made by the other education researchers. For instance, they noted: Florida’s policy not only retains lower-achieving students in third grade, it also provides additional services including summer school, 90 minutes of daily reading instruction, and placement with an “effective teacher.” This multi-intervention approach makes it difficult to tell what is making a difference. Is it grade retention, or is it the summer school? The teacher? The intensive reading instruction? An NEPC review by NYU professor Joseph Robinson-Cimpian raised similar concerns regarding another working paper about Florida’s Grade 3 retention policy. The study uses a statistical method called regression discontinuity analysis that compares English learners who scored just above the cutoff for retention with those who scored just below it. The benefit of this method is that these two groups of students are similar in many ways. But a drawback is that the study results may not apply to English learners with much higher or lower scores. The commentary concluded:

While grade retention coupled with other interventions might improve some outcomes for some English-learners, we urge policymakers and practitioners to exercise great caution before introducing "brute force" instruments for educating a population with such a history of educational marginalization and mistreatment.

NEPC’s mission is “guided by the belief that the democratic governance of public education is strengthened when policies are based on sound evidence.” But this study of grade retention serves as a cautionary tale about the importance not only of relying on sound evidence but of (a) applying the overall body of sound evidence, and (b) applying evidence in a sound manner. It is rarely wise to devise policies based on a single study, no matter how rigorous, convincing, or large. Further, it is crucial to ensure that key aspects of the study design (like the fact that the sample consists entirely of students on the cusp of the retention and resources intervention) do not get lost in translation as research is transformed into policy and practice with the potential to impact millions of real students in real schools.