Testing 2

We can tell a good exam by what it does and by what it does not do:


  • distinguish between remembering and understanding
  • require students not just to know stuff but to use it
  • distinguish between sound evaluation and unjustified holding-forth
  • require at least some qualitative distinction among the grades awarded, to be judged by the teacher
  • require good writing whenever it requires writing at all
  • allow the marriage of cow and bull (see below)
  • provide opportunity for learning as well as for demonstration

Does not

  • reward guesswork
  • reward bull or baloney
  • comprise disjecta membra
  • invite the “montillation effect” (see below)

Since there is so much positive to say about a good exam, we can afford to turn first and briefly to what a good exam does not do, thereby getting it out of the way. In my last blog but one, I discussed the rewarding of guesswork and hope that I made a good case against the kind of test that does so.

What I didn’t speak about is another kind of guesswork, called “bull.” In using bull the student essayist slings concepts, generalizations, and abstractions untamed and undisciplined by connections to the world of persons, places, and things, usually in the hope of slipping by with a passable grade. Professor William G. Perry discussed this problem in his essay “Examsmanship and the Liberal Arts.” The essay is very much of its time and place, but it discusses what seems to be an ever-present problem: how to balance the need for concept-work and ideation with the need for an anchor in reality that a command of relevant detail provides. He calls familiarity with such detail “cow” to suggest a balance with the “bull” of empty ideas, and with tongue in cheek recommends a marriage in every good essay. Where an essay is inflated with senseless abstraction and other kinds of hot air, it needs deflation with a sharp red pencil. And, of course, a chaotic collection of snippets that never rises above recall and naming must be downgraded too.

In a prior posting I discussed the “montillation effect,”  in which a student can get  a good or even perfect score in a “test” of “knowledge” that that student doesn’t even understand. We have to be sure that in testing comprehension we are not just inviting students to mimic it, repeating what they have heard about the “montillation of traxoline.”

Another couple of problems come to mind when I think about vocabulary lists. Suppose that you have given your students some words with definitions to be learned: base – a number that is raised to a power; horse – large solid-hoofed herbivorous mammal; network – a thing reticulated or decussated at equal intervals with interstices between the intersections; reed – fibrous core of rattan used in basket weaving; syzygy – a pair of Gnostic aeons male and female. Now, I’ve exaggerated a problem in such lists: that it might leave a student lost—farblondzhet! How does one gain purchase on some of this stuff? The temptation will be simply to memorize the definitions without knowing what they mean. That temptation will be especially strong in a student who knows that he will be examined on his vocabulary by, say, a set of matching questions linking definition and word, a task that has nothing to do with understanding.

The second problem lies in coming to terms with context: even though syzygy is a great word for playing hangman, most of its uses will depend on a grounding that the student is unlikely to have or ever to get: astrology and Gnosticism. Many more words than syzygy will seem bewilderingly uncontextualized to a 9th-grader, a fact that teachers of “test-taking skills” may be reluctant to admit. Learned this way, such words will seem to students like beached monsters of the deep. The study of words learned in context is far likelier to lead to their being useful and used aptly in a carefully prepared essay question, which itself makes their use more likely. A properly set essay question will invite the use of vocabulary learned during the study of a text containing new words.

Setting good essay questions is harder than one thinks, but it is essential to elicit the best possible responses from students—responses that show they have been doing real live thinking while working up their answer. Studying for my teaching credential, I was given distinctions between essay and “objective” tests that were mainly of utility for the teacher: setting an essay test was “easy” but we would be paid back in having to grade responses. Harder was the construction of a good multiple-choice “objective” exam, but it led to more reliable and less troublesome assignment of grades. For the present I don’t want to get into the false distinction between an “objective” test and an essay test, but I do want to assert that forming a feasible yet demanding essay question takes careful thought. That sort of thinking does become easier with familiarity, unlike putting together a hundred multiple-choice questions, which will always require a certain amount of grunt-work. But the question, however long in formation, should demand thinking of all students while offering opportunities for imaginative, thoughtful, well-informed thinking and writing (or their lack) that will distinguish each grade of student from the others.

A word, or more than a word, might be said about letting tests give students a chance to learn as well as to perform. There are a lot of ways to ensure that this happens, but one way that can be really effective is to give out essay questions ahead of time and require that students work up notes to answers in the classes before the exam. If you are expecting to have time for two essays during the exam, set five or ten questions beforehand and give them out. Some questions can test understanding and application of material covered explicitly; some might require them to extend their thinking to deal with problems or issues not explicitly covered. Class then becomes a chance for work, solitary or in groups, silent or talkative at need, with the teacher intervening in individual or group cases, maybe even giving a special mini-lesson to the class as a whole. It is amazing how productive one or two weeks of such preparation can be. Let them bring notes taken in their own hand (but not printed or copied) to the exam, and watch the notes multiply, though students must be warned that the notes may not constitute, in effect, a pre-written essay. Choosing which questions to have them answer is fairly simple: roll a ten-sided Dungeons and Dragons die or use some other random process. That gets the juices flowing even more than—one hopes—they already were.

Implied in the use of questions requiring students to handle stuff that may in some respects be new to them is a rejection of the use of “study guides” before tests. High school students’ study guides should be their own notes. If they don’t learn how to anticipate intellectual demands and respond to them with a readiness they have achieved themselves, they are missing one of the main learning-opportunities an exam can provide.

So, yes, students planning on university or demanding work must learn to handle encounters with surprise. Having done so, they will have the salutary experience, as Professor Barzun says, of disorientation, bewilderment, or momentary dismay followed by a “rallying of forces.” If students’ response to an exam must oscillate between boredom and cave-in, something is wrong in the preparation that such an exam or the study for it  has given them.

Leave a Reply