Sneaky changes in NECAP documentation

The NECAP-as-graduation-test has occupied a lot of my attention recently.  As I have written before, the NECAP test is a fundamentally different kind of test than one you would use as a graduation test.  The questions you’d put on a graduation test are exactly the ones that the test designers consider a waste of time and leave off.  This is a matter of relatively simple statistics, and even if it were not, there are plenty of psychometricians (testing experts) who agree with me.

In discussions of this matter, it’s tempting to quote a page from the “Guide to Using the 2012 NECAP Reports” on the subject, and several people have drawn my attention to this passage:

“NECAP is only one indicator of student performance and results of a single NECAP test administration should not be used for referring students to special education or for making promotion and/or graduation decisions.” (page 6)

At a hearing on the matter a couple of weeks ago, a Senator read that passage to Deborah Gist, who replied by pouncing on him to emphasize that the word “single” was the key word in that sentence. She pointed out that giving kids who flunk the opportunity to take the test again complies fully with this caution.

At the time, I wondered how any sentient speaker of English could read that sentence and think the critical word in it was “single.” To me, it seems like a caution against using the test as a graduation test or a special ed placement test. In truth, the sentence is a tad gratuitous, since the statistics of the test say the same thing, and say it in much stronger language. It seems odd to read the sentence any other way. However, if it was my career and reputation that depended on reading it in just the right way, I suppose I too could find a way to claim that never has the word “single” played such an important role in any sentence of the English language.

So imagine my surprise when I learned that the word “single” was added to that sentence in 2011. Measured Progress, the company that designed the NECAP test, publishes a “Guide to Using the NECAP Reports” each year. For the most part the report is just boilerplate, updated each year by changing it slightly to accommodate some of the changes to the test. That year, for example, was the first year for the writing test in the 3-8 grades, so there was some text about that. But before February 2011, when the guide was reporting on the 2009 test, the sentence above — same page, identical rest of the paragraph — read like this:

“NECAP is only one indicator of student performance and should not be used for referring students to special education or for making promotion and/or graduation decisions.”

Let’s have a big hurrah here for the internet archive’s Wayback Machine, from which I learned that the old version was still on the RIDE web site as late as January 18 of this year, and that the change was made for the report on the 2010 results, in early 2011.

What’s interesting to me is that the earlier sentence seems pretty clear — and to be clearly different than it became after 2011. There is no wiggle room in “should not be used.”

More important, this is how the text read back when the NECAP was adopted as a graduation requirement. At that time, it seems that the Department of Education was fairly clearly contradicting the advice of the NECAP designers — who subsequently changed that advice!  Are we to assume that the technical documentation for this test is only advisory?  Or maybe not proofed very well?  Which other simple declarative statements in the documentation are ok for the department to ignore?  Can schools ignore some of it?  How about students?

Or is it only the people who pay Measured Progress who can get them to change their advice?

The guides for the NECAP science tests were never changed — after all, they’re not used for graduation tests — so they continue to read just as the reading and math guide did before 2011. (The 2011 science report is here.  A friend downloaded the 2012 report a few weeks ago, but there appears to be no link to it any more on this page, so maybe they’re changing that one now, too.)

What we’re talking about here is dishonesty. This isn’t the same as simple dishonesty, or lying. This is intellectual dishonesty, and here’s the problem with that. The world is what it is. The facts of the world do not care about your opinion, or your triumph in some argument. Intellectual honesty is important in science because it’s the only way to get our understanding of the world to approach the world.  Fudge your results, and you’ll find that your cure for cancer doesn’t work, that your miracle glue is really an explosive, or that your economic policy just makes things worse. This is why science is supposed to progress by scientists checking and criticizing each others results: that’s how you maintain intellectual honesty. Sometimes the disputes get personal or political and distract from the real aim, but the real aim is to get at the truth via intellectual honesty, enforced by the scientific community.

The truth is that the NECAP wasn’t designed to be a graduation test, and this was obvious from the very beginning. It has been coerced into the role not because it was good for kids, but because it was cheaper than designing a dedicated graduation test. The features that make it a bad graduation test are objectively true facts about the test and its design. Neither editing technical documentation, committee-hearing filibusters, or cutting off public comment at Board of Education meetings will change those facts.

I have no doubt at all that the commissioner can fend off challenges from the public over these matters, indefinitely. But reality will — as it usually does — have the last word. And children will pay the price. The question for Board of Education members, legislators, school administrators, teachers, and parents is which side they want to be on.