In a recent television interview with Bill O’Reily, President Obama discussed the D.C. Opportunity Scholarship program and stated the private school vouchers “didn’t actually make that much of a difference” and have not “significantly improved the performance of kids in these poorest communities.”
President Obama seems to be relying on the final report of the D.C. Opportunity Scholarship Program which stated, “there is no conclusive evidence that the OSP affected student achievement.” But it seems Obama’s (and some of the media’s) familiarity with the report ends here.
It’s true the final report did not find statistically significant reading gains, but earlier reports over the first three years did. The report also found large gains in graduation rates. And importantly, even the lead author of the final report, Patrick Wolf, supports expanding the D.C. voucher program. A deeper understanding of the report explains why.
First, the study examines the impact of being offered a voucher (after applying, qualifying and winning in the lottery process) – not the impact of using a voucher. This was done to set a really high bar for determining whether the vouchers made a difference. To achieve statistically significant achievement results, all the kids who won a voucher and used it to attend private schools had to score high enough to lift the scores of all the kids who won a voucher but stayed in public schools.
Next, random assignment studies (as great as they are) suffer from a major methodological flaw called “the real world.” Students were randomly assigned to a control group (no voucher offered) and a treatment group (a voucher was offered). Students in the control didn’t have vouchers, but that didn’t stop them from enrolling in private schools or charter schools. Students who were offered vouchers weren’t required to use them and if they did, they didn’t have to stay in the private schools.
By the final year report, 47 percent of students in the control group (who were not offered a voucher) ended up in private schools or charter schools at some point during the study. Regarding the treatment group, 78 percent of the students offered a voucher used a voucher, but only 27 percent used it to attend a private school during every year of the study. That means 51 percent of students offered a voucher used it inconsistently — returning to public, charter and private schools as they pleased.
In other words, one could summarize the study as examining the impact of some students using school choice vs. slightly fewer students using school choice. The DC study is not, as President Obama believes, proof that vouchers do not work.
It is absolutely amazing that simply being offered a voucher correlated with statistically significant achievement gains in reading during the first few years of the study. And frankly, with so many students exercising choice in both comparison groups, we shouldn’t be surprised that the results eventually became statistically insignificant.
But even the statistical significance needs a caveat. Scientific research requires researchers to be at least 95 percent confident the observed results did not occur at random. Researchers use something called the p-value to determine this. If the p-value is at, or smaller than, 0.05, they can claim to be 95 percent confident the results did not occur at random. If the p-value is higher than 0.05, they must claim the results were not statistically significant.
As it turns out, the D.C. Opportunity Scholarship researchers could only claim to be 94 percent confident that the observed gains for reading were not random in the final report. The effect size was roughly the same as in previous years, but results missed the threshold for statistical significance by a hair.
Finally (and often overlooked by voucher opponents), the research team discovered that simply being offered a voucher resulted in a 12-point improvement in graduation rates. More impressively, students who actually used their voucher to attend a private school saw a massive, 21-point gain in graduation rates.
Obama may say vouchers “didn’t do very much,” but I’d wager the graduates of the D.C. Opportunity Scholarship program feel differently.
