T
he title of Jerry Z. Muller’s latest book, The Tyranny of Metrics, seems designed to gladden the hearts of lovers of Shakespeare, Milton, and the Bible: It promises a cri de coeur on behalf of humanism against a bloodless techscape of data analytics. Who calls out tyranny these days (apart, that is, from everyone who believes Donald Trump guilty of it)? Only those who dare to oppose it. Thus the stage is set for an epic confrontation between word and number.
Alas, a better title for the book would be Why Metrics Don’t Measure Up. It’s not metrics as such that vex Muller but the misuse of metrics, as well as the unintended consequences that flow when managers rely on them too crudely in assessing and rewarding performance. Far from overthrowing the tyrant, Muller ends up trying to counsel him on how to do his job better.
The Tyranny of Metrics is a short book, and it is typically a virtue when authors take the time to be concise, a practice that comes naturally to few. To work from an authoritative body of knowledge, to discern a theme worthy of exploration, and to eliminate all that is extraneous from its judicious development—these are the achievements of a master. They constitute the greatest gift an author can bestow on a reader. Unfortunately, there is another kind of short book, and that is a book that should have been a magazine article. Here, the problem is filling out enough pages to achieve plausibility between cloth covers. This is the problem The Tyranny of Metrics faces and doesn’t quite manage to overcome. The text of The Tyranny of Metrics is perhaps 40,000 words long. But in truth, about an eighth of that would have been sufficient for Muller’s argument. In a nutshell—indeed, the first chapter is called “The Argument in a Nutshell”—Muller holds that society has fallen prey to “metric fixation,” whose “key components” are the beliefs that numerical indicators of performance can replace experienced judgment; that transparency, or publicizing such indicators, brings accountability for performance; and that “the best way to motivate people…is by attaching rewards and punishments to their measured performance.” Now, to be sure, Muller does not intend “to claim that measurement is useless or intrinsically pernicious.” But he argues that a fixation on metrics may lead the fixated into numerous flaws. These include measuring what’s easiest to measure rather than assessing whether it’s the best gauge of desired outcomes; measuring “inputs” (say, how much you spend on education) rather than outputs (how much students learn); and excessive standardization (measuring college graduation rates in calculating “human capital” without acknowledging “the fact that all B.A.s are not the same”).
Worse, when metrics lead to rewards for good performance and punishment for poor performance, human beings will “game the metric.” They will engage in “creaming” by selecting an easier path to a higher score (surgeons who avoid operating on patients with complications, for example). They will seek lower standards for success (administrators who boost high-school graduation rates by lowering requirements). They will omit or distort data (police departments that reduce the rate of serious crime by booking potential felony offenses as misdemeanors). And they will outright cheat (teachers who alter student scores upward on standardized tests to show improved year-end performance).
Muller, a professor at Catholic University and the author of Capitalism and the Jews and many other works of economic history, offers a series of examples of these problems and how they play out in areas from medicine to the military. He calls them “case studies,” but that really isn’t what they are; a case study would entail detailed examination of both the pluses and minuses of measuring and counting. Muller offers a nod to the positive aspects, but it’s drawing up the indictments that interests him.
Muller has done no original research on these examples but rather is reporting on the work others have done. What is perhaps most striking, then, is how familiar the problems associated with metrics are. Muller tells the story of New York City’s undertaking an experiment in performance-based teacher pay, only to have a program evaluation conclude that there was “no evidence that teacher incentives increase student performance, attendance,” etc. But what we really have here (and there are examples throughout The Tyranny of Metrics) is a metric rebuttal to “metric fixation.” The New York City case actually seems to be an example of numbers doing what they are supposed to do.
As for the possibility of a broader critique, Muller does offer some quotations from big-picture-type thinkers thundering against—well, it’s not entirely clear what they’re against except straw men. Here’s Isaiah Berlin: “To demand or preach mechanical precision, even in principle, in a field incapable of it is to be blind and to mislead others.” Fine. But does this mean our practitioners of politics should not authorize experts to evaluate through clinical trials the safety and efficacy of new medications before they go on the market? No, that’s not what Berlin means, or Muller by quoting him.
I am firmly a member of the camp of Shakespeare, Milton, and the Bible. But I am also a member of the editorial committee of the Democracy Fund’s Voter Study Group, where a debate is raging over whether a “k-means cluster analysis” of issue importance among respondents offers a better way to understand voter behavior than demographic analysis. It’s not poetry, but it’s a debate worth having.
