June 24, 2013

PSU, ACLU petition RIDE: ‘Don’t test me, bro’

Deprecated: Function get_magic_quotes_gpc() is deprecated in /hermes/bosnacweb08/bosnacweb08bf/b1577/ipg.rifuturecom/RIFutureNew/wp-includes/formatting.php on line 4387

The Providence Student Union, the ACLU, the RI Disability Law Center and 14 other organizations with a vested interest in equitable public education in the Ocean State are formally asking RIDE to stop using the NECAP test as a graduation requirement.

“The new Board of Education has never had the opportunity to fully discuss, much less take a position on, the actions of its predecessor – the Board of Regents – in approving high stakes testing,” said RI ACLU Executive Director Steve Brown, a frequent contributor to RI Future. “Through this petition, we are hopeful that the Board will take a stand and agree with the many organizations signing this petition that high stakes testing is bad policy.”

Board of Education Chairwoman Eva Mancuso told RI Future in May that the new board would reconsider the policy.

“I think that’s an important issue to come before the board,” she told me in May. “I certainly want to look at that issue.”

She also said: “I don’t think it’s the best test.” And added, “40 percent of kids are not going to not graduate from high school if I have anything to do about it.”

Using the NECAP test as a graduation requirement has emerged as one of the most controversial initiatives of Deborah Gist’s embattled tenure as education commission of Rhode Island.

Not only has using high stakes tests as a graduation requirement become more controversial across the country, the issue is further strained in Rhode Island because there are unanswered questions about the validity of the NECAP test in particular to measure individual student performance.

Tom Sgouros, Rick Richards and other RI Future contributors have painstakingly detailed how it is designed to measure school, not individual aptitude. The Providence Student Union made national news when it challenged adults to take a version of the NECAP test.

Here’s the full press release sent from the Providence Student Union today:

A coalition of 17 organizations representing youth, parents, the disability community, civil rights activists, college access organizations and other constituencies have filed a formal petition with the state Board of Education to initiate a public rule-making process over a proposal to rescind Rhode Island’s controversial new high-stakes testing graduation requirement. Under the Administrative Procedures Act, the Board will have 30 days to respond to the petition.

“The clock is ticking, and the futures of literally thousands of Rhode Island teens are hanging in the balance,” said Steven Brown, ACLU of RI Executive Director. “The new Board of Education has never had the opportunity to fully discuss, much less take a position on, the actions of its predecessor – the Board of Regents – in approving high stakes testing. Through this petition, we are hopeful that the Board will take a stand and agree with the many organizations signing this petition that high stakes testing is bad policy.”

Questions about the validity of high stakes testing as a graduation requirement have been a source of great concern and debate in recent months. In a cover letter accompanying the petition, the organizations echoed the views of many students and teachers that, rather than educating students, the policy has led to too much time being spent teaching to the test. In fact, earlier this month RIDE supported legislation that explicitly authorizes school districts to pull students out of core classroom instruction to prep for the test, if doing so is deemed to be in the student’s “best interest.” The groups also point to RIDE’s own failure to meet 32 of 33 goals it set for itself in improving achievement for traditionally vulnerable students as “ample proof of the validity of our concerns.”

RIDE has repeatedly assured worried parents that many students at risk of not graduating need not fear the testing requirement. But the signatories, like many citizens across the state, remain concerned – especially for the significant cohort of ELL and special education students.

“Use of high-stakes testing has a disproportionate impact on students with disabilities and is counter to what we know works best for these students,” said Anne Mulready, supervising attorney at the RI Disability Law Center. “Our state and school districts have made significant investments in building the capacity to provide individualized instruction for students with disabilities that focuses on individual student strengths and learning styles, as required by federal and state law. But these investments are being needlessly squandered by the use of a high-stakes test to determine who gets a high school diploma.”

The Board of Education has been in existence for six months, but has never formally discussed or voted on this controversial requirement, despite the extensive public comment the subject has received at Board meetings. Under the Administrative Procedures Act, the Board has thirty days to initially respond to the petition, either by denying or it by initiating a public rule-making process, where testimony will be accepted and the Board can, as the groups note, “consider in a timely but deliberate manner whether to accept, modify, or reject this proposal.”

As Hector Perea, a member of the Providence Student Union noted, “The petition does not make the Board take a stand on high-stakes testing. It just pushes the Board to start a public process where they have to, at the very least, think about whether to debate the issue. We think the thousands of concerned students and parents of Rhode Island deserve at least that.”

Among the groups signing the petition are The Autism Project, Children’s Policy Coalition, College Visions, NAACP Providence Chapter, Providence Student Union, ACLU of Rhode Island, Rhode Island Disability Law Center, Rhode Island Teachers Of English Language Learners, Urban League of Rhode Island, and Youth in Action.

June 24, 2013June 25, 2013

Letter from Measured Progress: All is Well!

Deprecated: Function get_magic_quotes_gpc() is deprecated in /hermes/bosnacweb08/bosnacweb08bf/b1577/ipg.rifuturecom/RIFutureNew/wp-includes/formatting.php on line 4387

On June 3, 201, Commissioner Gist received a letter from the Principal Founder of Measured Progress concerning the NECAP. It said, in part:

“While graduation decisions were not a consideration when the NECAP program was designed, the NECAP instruments are general achievement measures that are reliable at the student level”

First of all, it is interesting to speculate why such a letter would be sent at this particular time, well after setting the policy requiring the use of NECAP for graduation decisions. I speculate that the letter was requested to reassure a restive Board of Regents, but that is just my guess.

Still, if this is intended as reassurance from Measured Progress, it can only be read as tepid. First, the letter acknowledges that the NECAP was never designed to measure the learning of individual students. It was, instead, designed as a general achievement measure. Unspoken is the reality that, if the NECAP had been designed to measure the learning of individual students, it would have been designed much differently. But, that question, which drags in issues of test validity, was not asked and was not addressed.

There is not a word about test validity in the letter. That is, there is no claim that the test provides information that predicts “college and career” readiness any better than a large number of other contending measures: grades, recommendations, work or leadership experience, portfolios, senior projects, or socio-economic background.

Actually, test scores track socio-economic background so closely that it would be difficult to do a good job of distinguishing the two in a validity study.

So, there is no claim in the letter that the test is more useful than information that is already available. But there is the important claim that the test is reliable at the student level. And, after all, it is the reliability of the NECAP score that contributes so much to its attraction– that attraction being the simplicity of reducing a complex history of learning into two numbers–one for reading and one for math. After all, what could be more objective that a single number? Like the current balance of a bank account, this number tells us how much reading and math the student knows.

But the test score number is not like the current balance of a bank account, which is an exact number. Instead, it is an estimate of how much a student knows. Part of the test score is what the student really knows—the true score–and part of the test score is the mistakes the student makes—getting something wrong he/she really knows, or getting something right that he/she really does not know. These mistakes create error in the test score–the more error in the test score, the less reliable it is.

When testing companies like Measured Progress talk about reliability, they talk about the reliability of the test. They mean that, using different analytical techniques, they can tell how much measurement error the test contributes to the score of a student.

Using a camera as an analogy, this is like telling someone how much the lens distorts a picture. In photography, where the subject doesn’t contribute distortion to the picture, this is all you need to know. If, to pick a number, the test is reliable at the .85 level for students, that means that .15, or 15% of the test score is error.

The usual way to deal with the error is to turn it into an error band around the reliable portion of the score. Thus, when RIDE creates a cut-score for graduation, it puts an error band around it and takes the score at the bottom of the error band as the cut-score. Voila, fair and true cut scores!

But in testing, the person tested has long been acknowledged as a source of distortion, or variation, or measurement error (see Thorndike, 1951). Beyond the test itself, the person tested contributes random variation based on “health, motivation, mental efficiency, concentration, forgetfulness, carelessness, subjectivity or impulsiveness in response and luck in random guessing”.

If you ask teachers, parents, or anyone else who actually knows students, one of the first things they bring up is how differently students behave from day to day. They worry about whether a student will have a good day or a bad day when they take the NECAP. They assert as commonplace knowledge that the same student can get very different scores on the same test on different days. This kind of variation is called test-retest error.

Yet there is no reporting on this source of measurement error in the NECAP Technical Report. Partly, this is because getting test-retest reliability entails serious logistical problems—large numbers of students need to take parallel forms of a test in a relatively short period of time. It’s difficult and prohibitively expensive.

But recent improvements in techniques for analyzing tests (Boyd, Lankford & Loeb, 2012) have changed this and, all of a sudden, we can begin to understand the reliability of students when they take “general achievement measures”, i. e., standardized achievement tests.

To return to our camera analogy, in addition to understanding how much distortion the lens produces, we can now begin to understand how much distortion the object of observation causes. Now, instead of one layer of error, we have two layers of error and they impact each other as multipliers. If, for example, the lens is .85, or 85%, reliable, and the subject is also .85, or 85%, reliable, the total reliability is .85 X .85, or .72.

Reliability of .72 means that more than a quarter of the score (28%) is error. In other words, taking the student into account, the test is a lot less reliable than we thought it was when we only took the test into account. As the authors cited above report:

“we estimate the overall extent of test measurement error is at least twice as large as that reported by the test vendor…”

The test referred to by the authors– developed by CTB-McGraw Hill–is very similar to the NECAP.

All of this casts stronger doubt on the wisdom making the NECAP a graduation requirement. Not only is the NECAP flawed in the several ways discussed in this column before—it discourages students, victimizes the weaker students in the system, constricts curriculum, and degrades teaching and learning–but one of its chief virtues, its reliability, is seriously oversold.

Underestimating test reliability is bad for a student graduation requirement, but we should also consider the impact on the whole accountability structure: teacher assessments are based not on just one student test, but several, so increases in unreliability puts the evaluation system in doubt. Likewise, accountability associated with schools—the measures defining Priority Schools and, school progress and gap closing, to name a few. The whole house of cards is now exposed to a stiff breeze.

June 24, 2013June 25, 2013

RIte Care cuts could prove to be a killer

Deprecated: Function get_magic_quotes_gpc() is deprecated in /hermes/bosnacweb08/bosnacweb08bf/b1577/ipg.rifuturecom/RIFutureNew/wp-includes/formatting.php on line 4387

Do you support the compromises made to put together this year’s state budget? Would you support them if you knew they will cause people to die? Statistically speaking, one cut is likely to cause as many as 30 deaths over the next few years.

The problem with discussions of the state budget is that they’re usually conducted in the abstract. We talk about budget numbers and cutting a little here and moving this number into that column and it’s all rather academic and somewhat bloodless. To make it a little less bloodless, I’d like to look at just one number and see what it really means. And since we’re talking about blood, let’s look at the Medicaid cut.

As was reported here, the Finance Committee’s budget drops Medicaid coverage for about 6,500 parents to save a bit more than $4 million. The deal is that these are people whose incomes are between 133% and 175% of the Federal poverty line, or $25,975 to $34,177. Their children will continue to receive health care through Medicaid and RIte Care, but these parents will have to buy insurance on the new Health Benefits Exchange, part of Obamacare. There is a federal subsidy available, that will keep the monthly costs down to $100-150/month, but each person will be liable for out-of-pocket medical costs expected to average around $2,000 per year. There’s a good summary here.

So, now to the math: someone who earns $26,000 each year will see an increase in monthly payments equal to a little more than half a month’s gross pay per year, plus out-of-pocket costs. If he or she is healthy, maybe that’s all. But if there’s an illness or injury, we’re looking at an additional month’s pay in medical costs. So how would you feel to know that your salary next year is going to be short 1.5 months gross over this year? In fairness, the House budget includes $500,000 to provide some subsidy. More math: $500,000 divided by 6,500 is about eight cents short of $77. That will surely help.

Still more math: According to a 2008 study by the Urban Institute, using numbers from the Census Bureau and the Institute of Medicine, about 137,000 people died between 2000 and 2006 from a lack of insurance. These are estimated excess deaths due to late or skipped treatments for disease and injury, or missed diagnoses. This is a mortality rate of a bit less than half a percent for the uninsured.

Unfortunately, 6,500 is a fairly big number. A bit less than half a percent of that still comes up a bit more than 30 people. If none of those 6,500 get insurance, or if they forego treatment because of its expense, we can expect about 30 people to die in the next few years as a result of this change in policy. Presumably lots of them will get coverage, but I doubt that all will. If as many as 29 people out of 30 — 6,283 out of 6,500 — manage to scrape up enough to pay for health insurance and also pay for all the doctor visits they might have used under Medicaid, there will still be one death.

In other words, we can reasonably expect that people will die because of the House Finance vote last Tuesday. Thank you, Helio Melo and Gordon Fox.

Ok, so this is not a large number of deaths in the grand scheme of state business (unless it happens to be you, of course), but the point of doing a study like the Urban Institute’s is that mortality is just the readily countable tip of the iceberg, and that for each preventable death, we can expect a great deal of pointless suffering from untreated chronic conditions, illness, and injuries.

This is what’s so infuriating about what passes for debate on the state budget around here. Because the legislative leadership has so thoroughly assimilated the idea that the only way to conduct state business is simply to do what business wants — tax policy, education policy, health care, whatever — it seems impossible to construct an argument in favor of something as simple as preventing death and suffering. Sometimes it seems the only effective way forward is to argue in terms of cost-benefit analyses and advancing the state’s economy. But in fact, sometimes things must be done simply because they are the right thing to do, and other things must be condemned because they are not. What kind of state do you want to live in?

So next time you hear someone saying how important it is that we pay back all our bonds, even the ones borrowed to fund stupid insider deals, don’t ask “is this important?” Ask, “Is this more important than the death and suffering of some poor people?” Is cutting the sales tax on liquor to improve sales in liquor stores near Massachusetts more important than the death and suffering of poor people? How about letting businesses accelerate depreciation? Is preserving all the tax cuts rich people got between 1997 and 2010 more important than that death and suffering? See how much clearer it gets?

Needless to say, people will object to having their budget choices portrayed in this manner, but do you sympathize with them, or with the poor people who will not get decent health care here, in what is still among the richest countries on earth?