Slipping in the Egregious Vocables

If someone told you that a counterfeiter should be pardoned because he was clever enough to make 20s that fooled a change machine the speaker had designed, you would be right to start laughing—or to check your pocket to see that your wallet was still in it before he left the room.  I’m torn, then, between laughter and looking for my wallet when I read a report that the makers of the e-Rater essay grading machine think students capable of gaming it are demonstrating higher-level thinking anyway, and so should be allowed a pass.

For it turns out that the machine can be gamed, though a thoughtful person could have anticipated the game. A critic might have guessed that a problem would crop up in the manufacturer’s substitution of inane machine-gradable proxy values for the human power of recognizing, let us say, stylish complexity or depth, which a good essay might be expected to have. The proxy values reward no such thing, for they can’t detect it. Instead, they reward long sentences, big words, and five-sentence paragraphs, which good writing has, except when it hasn’t.

To see a different but related example of the problem of proxy values in rating, you can substitute for an e-Rater your computer’s Flesch-Kincaid Readability Tool and experience how it can misgrade the readability of a piece of writing. If you download Shakespeare’s Sonnet LX (“Like as the waves make towards the pebbl’d shore”) and scan it with F-K, you will “find” that it gives the sonnet a 2nd-grade “readability.” To test the value of that “judgment,” go ahead and share the poem with your second graders and see how long the discussion lasts.

The marking machine also has a tin ear and is a sucker for lies, as we might have expected. If presented the five samples of awful writing that open George Orwell’s “Politics and the English language,” it would probably pass them. Hence, sentences like this one by Lancelot Hogben

Above all, we cannot play ducks and drakes with a native battery of idioms which prescribes such egregious collocations of vocables as the basic put up with for tolerate or put at a loss for bewilder.

would get a pass even though Orwell’s dismissal of it has become justly famous: “He is playing ducks and drakes with a battery that can write prescriptions.”

Les Perelman, a director of writing at MIT, reports that he has had some other revealing fun with the e-Rater and can now offer advice on how to fool it. Since e-Rater prefers big words, he says, he advises gamers to use “egregious” rather than “bad” as Hogben does (though Hogben’s sample is a rich lode of other fool’s gold too). He also says you can write nonsense so long as it “looks” as if it has been written well. He argued in one top-scored essay that universities are going broke because they overpay their teaching assistants, who go off on private jets to South Seas holidays. Who knows? It might even pass an essay claiming that Joel Klein respects teachers or that No Child Left Behind leaves no child behind.



Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.