Review Worth Sharing Identifies Flaws in NC Voucher Evaluation
An evaluation of an education program typically gives some information about whether or not a program is working. But a recent evaluation
of North Carolina’s school voucher program is so flawed
methodologically that it fails to explain whether the state’s
Opportunity Scholarships help or harm a student’s education, according
to a review by Kris Nordstrom, an education policy consultant on the Education and Law Project at the North Carolina Justice Center, a social justice-focused research and advocacy organization.
Nordstrom’s review is part of a new NEPC feature called Reviews Worth Sharing,
which are not commissioned or edited by NEPC but that we believe
contribute to our goal of helping policymakers, reporters, and others
assess the social science merit of reports and judge their value in
guiding policy. The views and conclusions addressed belong entirely to
The review finds that
methodological flaws in the evaluation make it impossible to accurately
compare North Carolina private school students who receive the vouchers
with their public school counterparts who do not. It is also possible
that the private school students who participated in the analysis were
not representative of the average voucher student. That’s because the
working paper only examined a small, non-random handful of voucher
students (89 individuals, or 1.6 percent of all voucher recipients) who
volunteered to be tested for the evaluation. In addition, just over half
of the private schools attended by these 89 recipients were Catholic.
Yet only 10 percent of all North Carolina voucher schools are Catholic.
The evaluation did use a
statistical method called propensity-score matching to create a public
school comparison group that was designed to be similar to the pool of
private school volunteers. However, Nordstrom identifies five main flaws
with this comparison:
The private school students who volunteered to participate in
the evaluation were recruited by a pro-voucher advocacy organization, Parents for Educational Freedom in North Carolina. The evaluation does not clarify to what extent, if any, the organization cherry-picked the volunteers or their schools.
The public school students likely came from lower-income families
than the voucher recipients. Evaluation authors said that they accounted
for this difference by incorporating prior year’s test results into the
analysis. But that assumes that income differences did not impact
performance in the ensuing school year.
The public school students likely attended schools with higher
poverty rates than the private school students would have been
attending, absent the vouchers. Again, evaluation authors said that they
accounted for this difference by incorporating prior year’s test
results into the analysis, but that (again) assumes that the differences
did not impact performance in the ensuing school year.
It is possible that the public and private school students had
different levels of motivation when taking the test. While voucher
recipients might have perceived that their performance could impact
their ability to remain in their private schools, the public school
students likely viewed the exam as a meaningless exercise.
The test used in the evaluation was not aligned to North Carolina’s
Standard Course of Study. If it was aligned more closely with the
private schools’ curricula, that could give the voucher recipients an
North Carolina’s voucher program is scheduled to grow by $10 million per year, to $144.8 million in 2027-28.
Yet as Nordstrom concludes:
Carolina General Assembly lawmakers are about to conclude yet another
legislative session without implementing meaningful evaluation and
accountability measures on state voucher programs. Despite the N.C.
State report, unfettered expansion of vouchers continues, and
policymakers, educators, and parents still don’t know whether the
program is working or not.