Development of Assessment Systems

Public comment by Gerald M. Eads II, Ph.D.

Race to the Top Assessment Meeting

Sponsored by the United States Department of Education

Atlanta, Georgia

November 17, 2009

Gerald Eads currently conducts research for the Professional Standards Commission, Georgia’s teacher certification agency. He has served as head of testing for the Virginia Department of Education, research faculty for the State Data & Research Center at the Georgia Institute of Technology, and assistant for policy research to the director at the Georgia Office of Educational Accountability. His doctoral training was in experimental and educational psychology and measurement. The opinions offered herein are entirely those of the author and do not represent the position of any agency.

1. Validity

The RT3 assessment executive summary notes that “the framework would focus on the design and quality of assessment systems and not accountability policies” yet it is the accountability policies which drive validity – and validity is at the very core of the issue of quality. The testing system you are considering must be validated for each of your purposes: instructional improvement, measuring school, principal, and teacher effectiveness, and predicting college “readiness.” It would behoove us, for example, to require state consortia to demonstrate that the testing system differentiates among teachers on meaningful dimensions. Just because “scores go up” does not mean in and of itself that anything of value changes – higher graduation rates, increased college success, lower unemployment rates, employment persistence, etc. Unless we can demonstrate externally referenced value, a testing system is nothing more than the proverbial boat _ _ that is to say, a hole in the water into which we pour money. Developing an assessment system without considering policy intent and implication is little different from building the atom bomb and refusing to address the consequences. Your Framework begins with policy validity issues, not the least of which is “Individual student achievement as measured against standards that build toward college and career readiness by the time of high school completion” – we cannot know that the tests are valid for that purpose unless we undertake the requisite longitudinal work to determine the relationship of the test scores to the desired outcomes. I urge you to require such study.

2. Test Design

“Standards” are nothing more than minimum competencies. It matters not whether the “standards” are set at the 10th, 30th, or 90th percentile levels – they are still just Newspeak for minimum competencies. The executive summary talks of both standards and growth – the latter requiring full_range testing – in the same breath. If “standards” become as they are now nothing more than different levels of test performance, then it will be hard to avoid setting performance levels for different goals – college, tech school, direct job placement – on the same tests, similar to what some European countries do. It’s called tracking. We rarely consider the consequences of socalled “high standards” – dropout rates, narrowed curriculum, and so on, and thus we rarely try to address those problems at the policy level, but leave the classroom teacher to struggle with the aftermath of our short_sightedness. We have yet to establish that in fact high_stakes tests of any nature actually improve education. I dearly hope you require states to study the long_term impact of testing policy. If scores go up, and teachers get hired and teachers get fired, and nothing else changes – then the millions spent on testing will be for naught.

3. Testing Time

We must, as noted in the USED summary, study the impact of testing time on instruction, particularly so if we are to consider multiple components during the school year. I’ve looked at the testing calendars for several districts in this state – we begin serious disruption of instruction in March, but it starts earlier. It is not only the time that students spend in actual testing but the time when some students are sent to testing and teachers must adapt their instruction for the remaining class so that tested students are not punished for their absence – or, at least, everyone is shortchanged equally. Every hour a child is being tested is an hour that child is not being taught. I am not aware of any research that attempts to determine the impact of lost instructional time due to testing. I would hope that you require attention to test administration design and efficiency such that the disruption to instruction is kept to a minimum.

4. Societal Considerations

We focus our testing on “standards” in basic areas. The USED summary suggests we begin development in language arts and mathematics. It’s been adequately demonstrate that testing in only a few high stakes areas narrows instruction. Today’s accountability movement seems to focus only on the development of skills to produce “good workers” with virtually no consideration for other purposes of public education, for example, to produce good citizens. What is the impact on the society of reductions in the arts, language and social sciences? We pay lip service to a desire to test “critical thinking” yet we seem determined to remove from schooling those things worth thinking about. Perhaps it would be useful to undertake study of the relationship between test scores and indicators of positive and negative societal outcomes such as incidence of voting and arrest. Teachers in public schools serving upper socioeconomic strata spend little time preparing students for 10th and 20th percentile tests, but if teachers in poor schools must spend all their time getting their students to pass minimum competencies in English and math, how does that impact instructional time spent in other areas, and do those changes, if any, have impact on such things as citizenship behaviors? I do indeed understand that we must “start somewhere,” and that there are reasonable arguments for beginning this effort with reading and math, but let us make sure that Mr. Obama’s initiative does not do to our public schools what someone else did to Afghanistan, getting things off to a reasonable start and then walking away.