Math Assessment Accommodations for English Language Learners

WestEd's Regional Educational Laboratory West (REL West) examined the effect of linguistic modification on middle school students' ability to show what they know and can do on mathematics assessments.

Linguistic modification involves reducing the complexity of the English used in a test in ways that support clarity without simplifying or significantly altering the targeted construct assessed. This can help ensure that math tests more accurately measure students' math understanding.

This study resulted in a report prepared by REL West staff.

Research Questions

The primary research question asked if the effect of linguistic modification on students' performance on two sets of math items (original and linguistically modified) varied across three subgroups of students:

• Students in grades 7 and 8 who were English language learner students (EL)
• Non-English language arts proficient non-EL (NEP) students
• English language arts proficient non-EL (EP) students.

If the linguistic modification increased the accessibility of EL and NEP students to the assessed math content, one would expect a significant score gain between the original and linguistically modified item set for EL and NEP students while no or minimal effect for EP students. More specifically, the primary research question was as follows:

• Does the effect of linguistic modification on students' math performance (as measured by raw scores or IRT theta estimates) vary across the three subgroups of students (EL, NEP, and EP students)? If so, did the linguistic modification improve the math understanding scores for the EL and NEP students relative to the EP students?

Secondary research questions provided information to support findings from the primary research question, and to examine the degree to which linguistic modification retains the integrity of the targeted math constructs and whether the relationship between the underlying construct(s) and the associated items differed across the three student subgroups within and between item sets. Secondary research questions were as follows:

• For each item set, do any items exhibit differential item functioning (DIF) between EL students and EP students and between NEP students and EP students? How do the DIF findings differ between the original and linguistically modified item sets? In other words, when comparing both EL and NEP students with EP students with similar math achievement levels, do the probabilities of the students answering individual items correctly differ on the test with linguistically modified items as compared to the test with original items? Does linguistic modification reduce the number of items showing DIF?
  
• Does the number of factors that underlie student responses to an item set (original or linguistically modified) differ for EL, NEP, and EP students? Do item-factor relationships differ across the three student subgroups? If more than one factor underlies performance on each item set, do the correlations among factors differ across the three student subgroups? Does linguistic modification reduce the number of factors or affect the item-factor relationship? For the EP students, do raw scores on the original and the linguistically modified item sets correlate similarly with scores from the state's standardized tests of math achievement?

Methodology

Intervention: The intervention was linguistic modification.

Design and Sample: This study followed a 2 x 3 x 2 fully crossed factorial design. The factors were item set (original or linguistically modified), student subgroup (EL, NEP, and EP students), and student grade level (grades 7 and 8).

The study consisted of three phases: item set development, item set refinement, and item set administration. The item set development involved expert judgment — experts in mathematics, mathematics education, English language development, applied linguistics, standards, assessment, and curriculum and instruction, and with experience with the EL student population. These experts selected the original item set and created a matched set of linguistically modified items. Item set refinement involved cognitive interviews of 5 EL and 4 non-EL students and a small scale pilot (64 EL, 48 non-EL students).

Results from the cognitive interviews and small scale pilot informed the finalization of the item sets (original and matched linguistically modified) for administration in this study.

Item sets were randomly assigned to students in grades 7 or 8, regardless of apparent language status (EL or non-EL student) or other subgroup membership (ELA proficient or non-ELA proficient). A total of 4,617 students in either 7th or 8th grade participated in the study. The sample included 1,214 EL students (who have moderate to high levels of English language proficiency), 1,625 NEP students, and 1,778 EP students. Within each subgroup, approximately half of the students were administered the original (non-linguistically modified) item set while the other half were administered the linguistically modified item set.

Analysis: A series of analysis of variance (ANOVA) based on four common scoring approaches was conducted to address the primary research question about the impact of linguistic modification on student performance. Each of the four ANOVAs was applied to the full 2 x 3 x 2 crossed factorial design, where item set (original or linguistically modified), student subgroup (EL, NEP, or EP students), and grade level (7 or 8) were the three factors. The dependent variable in each case was student performance, as measured by raw score total or theta estimates of math understanding from the 1-, 2-, and 3-PL IRT models. The interaction between student subgroup and item set was of particular interest in these ANOVAs because it addressed the study's primary research hypothesis.

Five types of secondary analyses based on the raw score approach were performed. The purpose of conducting these secondary analyses was to provide additional information about linguistic modification and its effect using different sources of information. These analyses were:

• Classical item-level descriptive analyses
• Reliability analyses
• Analysis of differential item functioning
• Analysis of the factor structure of the item sets
• Analyses of test correlations

Outcomes

The results of the study indicated the following:

• As implemented in the study, linguistic modification did not alter the targeted math construct assessed.
• Differences across EL, NEP, and EP students in the effects of linguistic modification on students' math performance depended on the scoring approach used (i.e., how scores for each student were calculated or estimated).
• For each student subgroup, the mean difference in performance on the two item sets was greatest for EL students, followed by NEP students.
• For all three student subgroups, one dominant factor (math understanding) was found to underlie performance on both item sets; however, the measurement structure between the underlying factor and the items differed across student subgroups.

The results provide mixed support for the notion that linguistic modification of middle school mathematics items increase EL students' ability to show what they know and can do in mathematics. Nonetheless, this study contributes to the body of knowledge informing assessment practices and accommodations appropriate for EL students, facilitating future development of more valid and reliable measures of what EL students know and can do.