Pontius Jr.’s Metrics That Make a Difference: How to Analyze Change and Error presents a framework for evaluating both prediction error and differences more generally. This framework advocates for paying attention to the specific research context when choosing metrics of evaluation and motivates an extra layer of complexity to avoid what might be lurking in aggregate measures.

Pointius Jr.’s framework is based on three elements of difference: quantity (which measures the absolute number of differences), exchange (which measures how many differences are swaps, i.e. one observation goes from A to B while another one goes from B to A), and shift (which measures how many differences are not simple swaps, i.e. one observation goes from A to B while another one goes from B to C). These measures can also be used to describe model performance or change over time or space. By partitioning differences into categories, he shows readers that it does not just matter what percent of the time things are different (or in the model evaluation context, we are wrong), but how things are different (or how we are wrong). The spatial and temporal examples from his background in geography are especially motivating for when simple metrics can hide what is really going on.

The structure of the book is based on the type of variables in the comparison. This layout is helpful for those dipping into the book for some advice on a particular analysis of interest, but he also explains how some scenarios can be reduced to others, emphasizing the connections between sections. Many chapters come with a companion example chapter, complete with pointers to relevant software, to give readers a feel for how all the formulas play out in practice.

This is a compact text meant for close reading, slowly working through each graphic and sets of equations. However, the discussion questions at the end of each chapter stay at the high-level, emphasizing the conceptual to help readers tunnel out of the weeds. This book would be a helpful supplementary text for exploring alternative metrics for model performance beyond those traditionally taught. For example, Pontius Jr.’s quantity, exchange, shift ideas expand the concepts underlying the traditional contingency table, and he compares and contrasts the more traditional Relative Operating Characteristic (ROC) curve with the Total Operating Characteristic (TOC) curve he advocates for. Since the book’s case studies drive the point home that metric choice should be tailored to a specific research goal and its final “commandments” chapter provides practical advice that motivates the complexities of this more nuanced framework, this would also be a helpful reference text for those working on an final project for a statistical modeling class or for those working on a research project, especially if working with geographic data of some kind.

Deciding how to evaluate a model or assess change is nontrivial, and this book makes a convincing case for pushing us to go beyond commonly used metrics to dig deeper and uncover more nuanced insights.