Compasses and Roadmaps, Educational and Moral

Today’s text is a line by Arne Duncan:  “I am a deep believer in the power of data to drive our decisions. Data gives us the roadmap to reform. It tells us where we are, where we need to go, and who is most at risk.” The line has a certain resonance, though not quite the pithy eloquence of Lady Bracknell’s that “statistics…are laid down for our guidance.” Lady Bracknell has the advantage of Duncan in that her ventriloquist was Oscar Wilde, but both dummies have made a fundamental mistake. Statistics do not guide us, and they do not tell us where we need to go. They are results or epiphenomena, like the tracks of subatomic particles in a bubble chamber. Saying that they tell us where we need to go is like saying that the bubble chamber tells the particles where they need to go. Statistics may provide useful information, but deciding where to go is a judgment, not a statistic. Confusing these things can lead to mistakes, and it can provide cover for an abdication of responsibility: it’s not me, it’s the numbers.

Thus, I am troubled by Duncan’s analogy of reform by statistics to finding a destination on a roadmap. It implies that reform is a kind of AAA TripTik^®: just follow the nice map. Actually, as I have argued, a better analogy is Chesterton’s, that it is a comparative of which we have not settled the superlative. Indeed, I would argue that it is a comparative of which we have not settled the comparative.

What else can explain the utter failure of No Child Left Behind and the incipient failure of its offspring RAce to the Top? If data were the roadmap to reform, by this summer 100% of US high school graduates would be proficient in all their subjects, as “proven” by their ability to get good scores on multiple choice tests in two of those subjects. Some superlative: an unachievable goal. Some comparative: a method of “measuring” achievement that is absurd, as shown by, among other cases, the Tennessean music teachers rated by the ability of their students to answer multiple choice questions about English prose composition.

But even the data themselves are suspect. Mr. Duncan says they can tell who is “at risk,” but the “value”-“added” “metrics” used to determine peer-groups of schools whose students are most “at risk” assign like weightings to factors whose coefficients of correlation are very, very unlike; and they make consequential determinations on the basis of correlations as weak as -0.114.

What is more, a naïve believer in Duncan’s assertion would have no clue of what the statistics of “proficiency” leave out, for they are all based on multiple-choice tests. The problem is that these tests do not capture a lot of what takes place, or should take place, in a good classroom[1].

As Donald Campbell foresaw, the prospect of high-stakes testing tends to exert a corrupting pressure on courses away from breadth of learning and towards test preparation in the narrowest sense. In the article from which Duncan’s comment was taken, its author, Professor Meredith Broussard, focuses on the narrowness of the preparation, which often turns out to be the study of textbooks written by the publishers of the tests. To no one’s surprise, the mysteries of multiple choice Professor Broussard cites as “solved” by study of the expensive textbooks amount to what Orwell called “preparation for a confidence trick.” Like Amy Chuan, Broussard wants success for her child and looks for it without stepping outside an intellectual terrain in which these tests and their deleterious effects are taken for granted. She further notes that success in these tests usually comes from studying expensive textbooks and taking expensive courses of test preparation, options that are generally not open to the students “most at risk.”

I do not mean to imply that Broussard and Chuan are confidence tricksters; rather, Broussard at least seems to have been taken in by the confidence game established by the real confidence men, people like Duncan. (It is possible that instead of being a confidence man, Duncan simply can’t think properly.) In my view, if he bears any resemblance to a compass, it is because we can tell directions from him by noting that he is the southern end of a northbound horse.