The Phantom VAMs

One cinematic curiosity of the Great Depression is The Phantom President (1932). It had songs by Richard Rogers & Lorenz Hart. It starred Jimmy Durante, Claudette Colbert and, amazingly, George M. Cohan for his only appearance in a “talking picture.” It opened to warm reviews by among others The New York Times. In spite of all these seeming advantages, it remains remarkably unmemorable: whatever makes a movie wonderful seemed to be missing here, even though all the “ingredients” look right[1]. The double-lead role by Cohan is as a politician of more than usually vacuous dullness and his look-alike, a typical charismatic mountebank. They work together…H-m-m…maybe teachers will recognize many types in this movie.

And the Teachers’ Typology extends from the people (and horses) supplied so generously in education to the ideas typical of the field and their typical expression. Dr. Johnson once said of a contemporary that he was “dull, naturally dull, but he must have taken great pains to become as we now see him.” One constraint in educationist writing is the Iron Law of Dullness that it must obey in order to be taken seriously. Young people seeking graduate degrees in education or teaching credentials must endure,  must (pretend to) welcome such writing.  Maybe its promoters think that dullness will guarantee against mountebanks and balonists[2]. In fact, it tends to guarantee that those who succeed in the field are those who do not have to try too hard to make their productions dull.

This is one reason why we must be grateful, when we find them, to scholars who have mastered the necessary dullness while preserving their critical intelligence and producing remarkable work. Two such scholars are Morgan S. Polikoff and Andrew C. Porter, authors of a study called “Instructional Alignment as a Measure of Teaching Quality.” Their paper concludes that “the correlations of value-added with observational measures of pedagogical quality, student survey measures, and instructional alignment were small.[3]” Of all the hypotheses they suppose might explain this smallness of correlation, the one they find most plausible is that “the tests used for calculating VAM are not particularly able to detect differences in the content or quality of classroom instruction.[4]” They follow the discussion of this hypothesis with the relatively frisky observation that “[a]lthough we would not expect perfect correlation of instructional measures with VAM, correlations at or near 0 should raise concern.”

Indeed, they should raise more than concern, as the authors imply at the study’s end: “[T]his study contributes to a growing literature suggesting state tests may not be up to the task of differentiating effective from ineffective (or aligned from misaligned) teaching[5]”.  The authors note by the way in their conceptual discussion some prior research finding that the usefulness of tests varies with their “distance” from the learning being “measured” and that “state or national standardized tests… are likely to show the weakest” connection to what is sometimes called “Opportunity to Learn” (OTL)[6].

Let me summarize, using the authors’ own words for the final point:

1.     There is little or no significant correlation between comparatively reliable ways of judging effective teaching and “Value”-“Added” “Metrics”.

2.     There is little or no significant correlation between “aligned” instruction and VAMs.

3.     “VAMs are not associated with either the content or quality of instruction[7]”.

The authors ask a final question: “If VAMs are not associated with either the content or quality of instruction, what are they measuring?”

My answer: They are measuring nothing. They are a phantom entity.



[1] Though one moment, memorable to a certain college student and some of his classmates, showed a fade shot in which “the southern end of a northbound horse,” as it would have been called in 1932, fades to a political speaker spouting pious platitudes. This little fade comes back to me again and again when I read or hear the recitation of educationist pieties. I am waiting for someone to study why the field of education is home to so many northbound horses that they begin to assume the qualities of a type.)

[2] balonist (bə-lōn΄-ist) n.: one who offers or requires baloney. Not to be confused with a balloonist, whose hot air is confined to his balloon. Cf. “Baloney Bingo”. Richard van de Lagemaat offers a workshop in “Baloney Detection across the Curriculum,” but not at schools of education (q.v.). (from The Didact’s Dictionary.)

[3] p. 13

[4] p. 16

[5] p. 16

[6] Ruiz-Primo, M. A., Shavelson, R. J., Hamilton, L., & Klein, S. P. (2002). On the evaluation of systemic science education reform: Searching for instructional sensitivity. Journal of Research in Science Teaching, 39, 369–393.

[7] p.16

Leave a Reply