The conviction of teachers and administrators in Atlanta of racketeering should caution us all about the dangers of value-added metrics and high-stakes testing. There is no reason to think that the convicts were uniquely depraved. There is, however, abundant evidence to show that they worked within a misguided and poisonous institutional culture that was both depraved and stupid. Many succumbed to the poison.
The depravity is exemplified by an administrator who wore gloves while changing students’ answers at her school, though I don’t mean to say she would have been ‘less’ guilty if she had not worn gloves.
But the stupidity! An abundance of testimony confirms that the attention of the district was morbidly fixed on test results, which common sense and research could have told them was counterproductive. These results give a skewed picture of what students know, and their use in making consequential decisions was an invitation to corrupting pressures, which, sure enough, came to be exerted and felt.
The New York Times has almost figured this out—but not quite. It took an economics reporter to sniff out the problem, for the Times’s education reporters, since the removal of the much-missed Michael Winerip from the education beat, have as usual failed to get the story. The discovery of the problem in its economic and sociological aspects goes back to the academic ‘lawgivers’ of the mid-1970s who gave us Goodhart’s Law (economics) and Campbell’s Law (sociology). The problem is that when a measurement is used not just to measure but also to make consequential decisions, the users exert a corrupting pressure on the process of measurement. The Times reporter amusingly but incorrectly says that Goodhart’s Law means that “a performance metric is useful as a performance metric so long as it isn’t used as a performance metric.” Actually, it means that performance metrics are useful when they are not tied to consequential decisions. To put the seeming paradox correctly we should say, “a performance metric is useful only so long as no one uses it to reward or punish people.”
A performance metric should be reliable, which VAMs are not; but the main issue of this posting is not the inherent unreliability of standardized testing to determine what students know; it is that misconceived and misused evaluations of students’ performance can lead to corruption. The Atlanta verdict shows that this corruption can be criminal.
And the alternative to corrupt and worthless ‘metrics’? 1) Suck it up and turn down the RAT[1] grants. 2) Return from evaluations of teachers based on ‘metrics’ to evaluations based on discernment. 3) Dust off your Deming:
• Drive out fear
• Eliminate slogans, exhortations, targets.
• Eliminate management by numbers, numerical goals. Substitute leadership.