Suspect statistics and misinterpreted data in ‘Reinventing Maine Government’ report?

Suspect statistics and misinterpreted data in ‘Reinventing Maine Government’ report?

In a Press Herald op-ed on February 13, Tony Payne claims that Maine spends 25% more per-pupil on K-12 education than the national average.  To substantiate this claim he quotes from the Reinventing Maine Government report released last year by Alan Caron’s group Envision Maine which appears to have evolved from the Caron’s GrowSmart organization, largely remembered for commissioning the 2006 Brookings report which served as the impetus for Governor Baldacci’s school consolidation initiative.  

As the figures cited by Payne seemed at odds with those reported elsewhere, I reread the Envision Maine report closely.  The report’s conclusions have been widely acclaimed, particularly among leaders in the business community such as Payne and David Flanagan, both of whom served on the report’s advisory board.  So I was disturbed to find in the report what look like both substantial errors in the statistics and deliberate misinterpretations in their presentation.

The education section of the report opens with this:

In 2008, Maine spent $2 billion, in combined state and local dollars, on K-12 education. That worked out to $13,513 per student – 25 percent more than the national average of $10,259 per student, and more per student than all but nine other states.

I see no source for these figures.  But I do recognize the $10,259 national number as consistent with that reported by the US Census June 2010 report on Public Education Finances .  But if you look up the table 8 in that report which contains that $10,259 US figure, you will find that Maine's per-pupil expenditure is shown as $11,572 -- not $13,513. The correct figure places Maine only three states higher than the national average in per-pupil expenditures.

So there are many different ways to calculate school expenditures.  Some will exclude federal funds, some will include debt and capital expenses.  But for comparisons to be meaningful, the basis simply has to be consistent.

Now rereading the convoluted wording of the paragraph above from the report, one has the distinct realization that its author knew that it was a mis-comparison.  Worse, one may conclude that it was deliberate.  That's certainly  my conclusion until someone can cite a common source with both figures.

Not settling with that, the same introductory paragraph continues.

During the last few years, eighth grade math scores plummeted from 1st place nationally a decade ago to 24th place in 2007.

But, if you check the 8th grade Math NAEP scores from which this characterization is drawn, you might be forgiven for your surprise in finding that, despite this "plummeting," these same scores actually are higher now than they've ever been. Maine's 8th grade Math scores have increased from where they were when we led the nation in 2000 yet, because test scores in other states also rose at a higher rate, the report's author chose to characterize Maine's achievement as "plummeting."  

A true assessment is shown at the same linked page. Maine's current 8th-grade math scores statistically lag only 8 states, statistically exceed the scores of 23 states, and are statistically indistinguishable from the remaining 20. Yet from this, the Envision Maine report chooses to characterize Maine's math scores as being in free fall.

So, instead of these NAEP scores - which consistently show Maine ranking about 13th nationally in absolute scale scores -  the report instead shifts focus to Maine's average score on the SAT which are mandatory nowhere else. Average state scores on the SAT are a direct function of the participation rate.  If you're unconvinced of that, check out a chart of the competing college ACT scores which will show Maine ranking FIFTH nationally, strictly because the ACT is taken only by about 5% of Maine's students, a self-selected group which almost certainly is over-represented by ambitious overachievers.

Having now misrepresented both the expense and the performance of a school system that would otherwise be understood as exemplary given Maine's relative income and poverty, the report then goes on to compare these inflated per-pupil expenditures to those of other rural states, with no further regard to relative school achievement.  

The report rattles off 12 states with both lower population density and lower per-pupil expenses without acknowledging that, on average, those states also represent substantially lower test scores than Maine's.

The report then goes on to use incorrect figures for Maine's student-teacher ratio which now stands at 12.2 and also makes the same comparison between Maine's number and the national figure of 15.8, rather than comparing it with the ratio of 12.9 in states that do as well or better than Maine on the tests.  Is it responsible to claim that our current position in the top third nationally in achievement can be improved by moving towards the national median of 15.8 when the states that outperform us maintain a ratio not much different from ours?

So, in short: the report twists bogus numbers to portray the state as doing worse than it is while paying more than we do. Then, from this erroneous position,  the report theorizes that we could do better with fewer schools and fewer teachers as long as we complement them with a parallel system of new charter schools fed from the same limited resources.

It's enough to make a reader cynical about Envision Maine's motivation.

Brian Hubbell,

February 15, 2011