MDIschools.net

Mon, Jan 31, 2011 at 5:19 PM
To: David Silvernail
Cc: Senator Brian Langley, Representative David Richardson, Representative Peter Johnson, Committee Analyst Philip McCarthy,

David,

I'm gravely disappointed that MEPRI's report The Condition of K - 12 Public Education in Maine - 2011 on pp 63-64 misleadingly compares Maine's average SAT scores unfavorably with national averages with no mention that such a comparison is entirely invalid.

As you surely know, the relative difference in average scores is entirely the consequence of the difference in SAT participation rates. Nationally, the SAT is taken by fewer than half of high school students, a self-selected group of college applicants, while Maine requires it of all students. The test pools are hugely different, therefore the comparison is without statistical meaning.

The college board itself prefaces its published results with a disclaimer against interstate comparisons of SAT scores and explains exactly why here:

Media and others often rank states, districts, and schools on the basis of SAT scores despite repeated warnings that such rankings are invalid. The SAT is a strong indicator of trends in the college-bound population, but it should never be used alone for such comparisons because demographics and other nonschool factors can have a strong effect on scores. If ranked, schools and states that encourage students to apply to college may be penalized because scores tend to decline with a rise in percentage of test-takers.

David W. Grissmer, in The Continuing Use and Misuse of SAT Scores, Psychology, Public Policy, and Law, 2000, explains more.

As you certainly understand, it is critically important that policy makers not be mislead into drawing damaging conclusions about the condition of Maine's schools from such invalid comparisons.

Brian Hubbell,
Bar Harbor

--------------------
Tue, Feb 1, 2011
From: David Silvernail
To: Brian Hubbell

Brian, we recognize that SAT performance can be related to participation rates. That is why we reported these participation rates in the Conditions book.

But one has to be careful to attribute all difference in SAT scores to participation rates. For instance, here are 2010 SAT aggregate scores for states with participation rates similar to Maine:

State Participation Rate SAT Aggregate Scores

1. Maine 92% 1389
2. Massachusetts 86% 1547
3. New York 85% 1461
4. Connecticut 84% 1536

For these states with comparable participation rates the aggregate SAT scores are between 1-2 standard deviations above Maine's scores.

David Silvernail
Center for Education Policy, Applied Research, & Evaluation
University of Southern Maine

---------------------

Tue, Feb 1, 2011
From: Brian Hubbell
To: David Silvernail
Cc: Representative Pete Johnson, Representative David Richardson, Committee Analyst, Phillip McCarthy, Senator Brian Langley

David,

If those are the SAT comparisons that MEPRI takes to be significant, then those are the numbers the report should discuss explicitly.

With only the comparison to the national average, the report invites the same specious conclusion that the Governor - solely on that basis - repeatedly made during his campaign that Maine schools "are in the bottom third nationally in achievement."

I think the attached chart accurately makes the point you suggest - relating Maine's SAT performance to other New England states - while honestly representing my own concern about the greater correlation with participation rate.

To get the true picture, however, I think you need to accompany this with another graph plotting state average SAT score vs. median household income.

-Brian

---------------------------

Tue, Feb 1, 2011 at 2:09 PM
From: Brian Hubbell
To: David Silvernail
Cc: Representative Pete Johnson, Representative David Richardson, Committee Analyst Phillip McCarthy, Senator Brian Langley

...And, extending on my last comment, here's what is most important to bear in mind when comparing Maine's SAT performance with that of Connecticut, Massachusetts, and New York:

1. By far the strongest correlation to relative SAT score is household income.
   • SAT Scores and Family Income, New York Times
   • College Board Total Profile Report (see table 11 on page 4)
2. Here are the 3-year average median incomes for the northeastern states - those whose schools Maine most closely emulates.
   • Maine $48,032
   • New York $50,372
   • Vermont $50,619
   • Rhode Island $53,584
   • Massachusetts $59,981
   • Connecticut $65,213
   • New Hampshire $66,654

-Brian