Measurements: The Qualitative Work of Quantitative Work

Katie Pine

Max Liboiron

Editor’s Note: Katie Pine and Max Liboiron continue this week’s theme of makers, hackers, and engineers with a post about the politics and performativity of measurements, central to the practice of many engineers and scientists.

Katie Pine (@khpine) is a postdoc in Intel Labs Cultural Transformation Lab, and is currently in residence at UC Irvine.  Katie’s work bridges Computer Supported Cooperative Work, Organization Studies, and Science & Technology Studies.  At present her NSF-funded research examines micro-foundations of IT-enabled accountability policy and practice in the healthcare domain.

Max Liboiron (@maxliboiron) is a postdoc at Northeastern University’s Social Science Environmental Health Research Institute (SSEHRI) and a co-founding member of the Superstorm Research Lab, a mutual aid research collective. Liboiron studies “techniques of definition,” the tools and practices used by scientists and activists to make emerging, contested, amorphous forms of environmental harm manifest.

---

From common core to quantified self, measurement is increasingly part and parcel of our daily lives.  We use number-driven measurements to make visible, manage, and regulate increasingly nuanced aspects of daily life, work, public institutions, and our environment.

However, measurements are never mere faithful representations of nature, but have social and political origins and ramifications.  We are exploring two aspects of measurement that often go unnoticed: first, the situated, complex work that goes into making measurements work in the first place (and the fact that this work is inherently social, cultural, and political), and second, the idea that measurements themselves can be seen as performative, creating and re-creating the very things they are intended to make visible.

Representational theory defines measurement as “the correlation of numbers with entities that are not numbers,” a process of transformation, translation, and even interpretation at the level of sampling and gathering data. What is selected for measurement and what is not, how measurements are standardized, what counts as an important unit of measure, and how measurements are used all have stakes for the systems of which they are part.

Moser & Law (2006) argue that current metaphors for information as “flow” are inaccurate, as these metaphors presume that information is immutable, something that is created and exists in the world and thus can be taken up, passed around, and used for calculation.  Moser and Law instead argue that we can see information as something that is inherently mutable and relational, that changes its shape as it is circulated and used.  To put it more simply, information never fully has meaning on its own – it becomes meaningful and usable when a particular person or group make decisions about what the information is and how they can use it.

A good example comes from a recent study on counting rates of infection in hospitals (Dixon Woods et Al., 2012).  The authors found that an act as seemingly simple as counting infections was actually highly social and cultural – the answer to the question “what counts?” varied widely from one hospital to another, calling into question the current focus in healthcare (and investment of healthcare dollars) on quality measures as a tool for achieving reforms such as infection reduction in practice. Making meaning of numbers requires acts of both calculation and judgment, what Moser & Law call “qualculation.”

In this perspective, we can see data as fluid, emergent, and relational through processes of judging and calculating, even at the most basic level:

Judgment and calculation also have much in common. This is because each makes relations between elements that are materially heterogeneous and different in kind. Each needs to simplify those heterogeneities and order – perhaps homogenise – them. Each, therefore, works by setting limits to what will count as “information”. Each does this by setting boundaries to what is taken to be important and what is not. In short, we are arguing that judgment and calculation both work by arraying and manipulating entities within a single spatio-temporal frame. In this way they achieve what we will call qualculability. (Moser and Law 59)

The observation that measurement “works” through effortful situated judgments and decision making practices leads naturally to another question. What effect is the measurement itself exerting?  How is the practice of measuring something in a particular way actually changing and shaping the world?  The practice of measurement itself changes what people see and understand, what people value, how attention and effort are directed and, at the core, what the thing is that is being measured – it is performative.

Performativity is a key topic in language studies, gender studies, science studies, and economic sociology.  In thinking about measurements as performative, we are drawn to the work of Michel Callon, who thinks about performativity in economic markets. Callon describes how the very models that we use to describe markets actually shape the markets themselves.  It is important to recognize that models never fully “frame” the markets they describe, and the wiggle-room that is left between model and market is where change and adaptation occur.  But a key takeaway in thinking about measurement is that the very act of creating and engaging in measurement of something serves to create and re-create that thing in a particular way, with particular values and stakes for those involved.

We’ll show how measurements are performative in the next two case studies.

Katie’s case study: The Joint Commission Healthcare Quality Measures

The Joint Commission (TJC) is the primary hospital accreditation organization in the U.S.  TJC is very influential because the U.S. government uses TJC accreditation as a criteria for allowing hospitals to accept medicare and medicaid.  The primary mission of TJC is to accredit healthcare organizations based on an evaluation of certain criteria outlined by TJC.  For many decades TJC’s process focused on whether an organization was compliant with certain minimum acceptable standards.  In the mid 1960s, TJC’s focus shifted dramatically: rather than certifying minimum standards for hospitals, they sought to assess the quality of care being provided in hospitals.

In the new world of quality improvement in hospitals, adherence to specifications of process is a product that must be made measurable.  Rather than a simple tool for evaluating performance and comparing performances delivered by multiple hospitals, the new quality measures are expected to both evaluate performance and also serve as an active tool for changing practice on the ground.  Yet, the definition of “quality” itself is being tied to a specific process- that of quality measurement- and specific care practices that are seen as emblematic of “quality” produced by a healthcare organization as a whole.

At present, many organizations use quality measures in an attempt to quantify process to assess health care quality.  TJC is still perhaps the most influential; they rolled out the Oryx initiative in 1997, which developed “core measure sets” for multiple areas.  Hospitals are required to collect and report data on these measures, and data about hospital performance on measures is made available to the public.  Hospitals that perform poorly face potentially huge consequences. In recent years, quality measures are increasingly being woven into the very fabric of policies that govern the provision of U.S. healthcare.  In Texas, new legislation has been proposed that would tie medicare reimbursement rates to a hospital’s performance on quality measures. Consumers and the general public use publicly available measures to make decisions about health care.  Poor performance comes at great cost, as we saw recently when UCLA medical center did badly on a leapfrog survey.  But, quantifying practices is tricky business; in my own primary domain of research, obstetrics, a fundamental problem (just one of many) has been encountered with definitions.  There is wide variation in how clinicians understand clinical events such as “onset of labor” and “gestational age;” yet, these are the data that form the basis for high-stakes measures.

Max’s case study: Previous Sewage Contamination (PSC)

In the mid-nineteenth century, increased industrial activity and urbanization lead to the contamination of waterways used for public water supplies. Scientists were asked to determine whether a waterway was fit for consumption, but, in the professional opinion of Edward Frankland, a British water chemist for the Royal Institution, science was not always up to the task. Water analysis could not define the safety of water, chiefly because the presence and habits of germs, a new concept in the field, were largely unknown. Frankland believed germs could withstand filtration, chemical reagents, dilution, condensation, and other popular purification methods. Thus, even if a bacteriological test found no living germs in a sample, Frankland reasoned that a few germs may have survived purification and were just not present in the sample taken. These resilient germs could start an epidemic.

Thus, in 1867, Frankland introduced the concept of “previous sewage contamination,” or PSC, meant to represent the amount of sewage a river had received upstream. It was a number obtained by measuring the total amount of nitrogen compounds in a water sample, which in turn indicated the amount of organic material that had been in the water. This organic material could come from sewage or peat or other sources (science could not differentiate between them, and Frankland maintained the differentiation was “hygienically irrelevant”). PSC was meant to indicate whether there had ever been sewage in the water, and thus a potential health danger, regardless of whether the water had been purified.

In effect, PSC was a metric used to advocate for a definition of safety that differed substantially from the status quo, which pushed for post-purification as the preferred technique of definition. Thus, PSC was an activist measurement. As a member of several Royal commissions on water quality, Frankland had the ability to instate PSC in water analysis reports received by Londoners. The idea was that citizens and other stakeholders would become disgusted, fearful, or enraged about the inevitable presence of “previous sewage contamination” in their water, and demand better water. Since PSC would be present in any purified source, as all local waterways were used as extensions of sewers and had other organic materials in them besides, “better water” would entail either changing the source of London’s water supply, or the legislated cessation of all sewage disposal into waterways. The latter was Frankland’s goal.

(A version of this is also on The Discard Studies Blog)

Qualculation and its objects

It is no coincidence that both the examples above deal with delineating harm and health. Morbidity (ie, death) has always been a fairly stable metric of failed health (though, as Bowker and Star show in Sorting Things Out, cause of death has not been), but to monitor the more subtle indicators of health (whatever they may be), there is ample room for disagreement.

In both examples, qualcaluation, the “making definite” of judgments via measurement creates a new type of object. In one case, it creates “good” and “bad” hospitals and practices, and in the second case, it creates pollution and potential harm where before there was none. Activism is all about intervening in material conditions, and Franklin knew his judgment, expressed as a measurement, would be extrapolated off the page to make things happen in the world of things.  Advocacy via measurement is not unique to activism– in healthcare, quality improvement stakeholders lobby for the adoption of particular measures because measurement is not just a way of seeing and knowing what is happening but an active attempt to regulate practice.  For example, maternity care advocates considered TJC’s adoption of a core measure set for obstetrical care two years ago a coup, because maternity care had previously not been on the national agenda for quality improvement.

The Politics of Qualculation

Politics, to borrow Arjun Apparturi’s definition, is the set of relations, assumptions and contests pertaining to power. In the example of hospitals, politics result from the creation of qualculated healthcare and decides which hospitals stay open and how many resources they get, while in the water quality case, politics drive the judgment behind the qualculation to being with. Moreover, Frankland believed in the pollution and he made it apparent, while in hospitals, “poor quality” health care was not apparent until after it was quantified. This isn’t to say that “continuous quality improvement” does not entail politics in its creation. Far from it. If quantification entails judgment as a matter of course–that is, if qualculation is a characteristic of all data, which we are arguing is the case–it’s turtles all the way down. Seeing and knowing things about quality (of health care, of water) always already have politics woven in, and rest on previous layers of politics and judgment.

One of the unique abilities of scholars in the humanities and social sciences is to denaturalize the layered, “given” ontologies born of quantification. Our job is to back up the truck and question the ground it stands on. In this case, we want to back-construct measurements to see where they come from and how the thing being measured came to exist in the first place. This job is critical (in both senses of the term) because, as is evident from the examples, one of which is explicitly activist and the other which has high stakes, qualculation has politics. Measurements are never neutral, but inflected with power as techniques for defining what is in the world.

Historians and sociologists of mathematics have addressed the symbolic or political meanings of systems of measuring units, their standardization, and their enactment. However, the knowledge involved in the production of measured quantities and the mathematical operations with these quantities has hardly been treated. This is the back construction mentioned above.  In thinking together through our cases, of environmental pollution and healthcare performance measurement, we are unpacking historical and contemporary uses of measuring standards to cast light on the distinct social and political uses of measurement standards, identify strategies devised by actors to deal with the values they obtain, and speculate about how measurements might be used as tools of activism and advocacy.

References

Bowker, G. C., & Star, S. L. (1999). Sorting things out: Classification and its consequences. Boston: MIT press.

Dixon-Woods, M., Bion, L.M., & Tarrant, C. (2012). What counts? An ethnographic study of infection data reported to a patient safety program. Milbank Quarterly, 90(3), 548-591.

Hamlin, Christopher. (1990) Edward Frankland: The Analyst as Activist. Berkeley: University of California Press.

Moser, I., & Law, J. (2006). Fluids or flows? Information and qualculation in medical practice. Information Technology & People, 19(1), 55-73.

Callon, M. (2007). What does it mean to say economics is performative?  In  MacKenzie, D. A., Muniesa, F., & Siu, L. (Eds.). Do economists make markets?: on the performativity of economics. Princeton: Princeton University Press.

Pine, K. & Morton, C. unpublished manuscript. Understanding the landscape of maternal quality measures through a case study of early elective deliveries, a U.S. perinatal core measure.