Measurements: The Qualitative Work of Quantitative Work

Katie Pine

Max Liboiron

Editor’s Note: Katie Pine and Max Liboiron continue this week’s theme of makers, hackers, and engineers with a post about the politics and performativity of measurements, central to the practice of many engineers and scientists.

Katie Pine (@khpine) is a postdoc in Intel Labs Cultural Transformation Lab, and is currently in residence at UC Irvine.  Katie’s work bridges Computer Supported Cooperative Work, Organization Studies, and Science & Technology Studies.  At present her NSF-funded research examines micro-foundations of IT-enabled accountability policy and practice in the healthcare domain.

Max Liboiron (@maxliboiron) is a postdoc at Northeastern University’s Social Science Environmental Health Research Institute (SSEHRI) and a co-founding member of the Superstorm Research Lab, a mutual aid research collective. Liboiron studies “techniques of definition,” the tools and practices used by scientists and activists to make emerging, contested, amorphous forms of environmental harm manifest.

---

From common core to quantified self, measurement is increasingly part and parcel of our daily lives.  We use number-driven measurements to make visible, manage, and regulate increasingly nuanced aspects of daily life, work, public institutions, and our environment.

However, measurements are never mere faithful representations of nature, but have social and political origins and ramifications.  We are exploring two aspects of measurement that often go unnoticed: first, the situated, complex work that goes into making measurements work in the first place (and the fact that this work is inherently social, cultural, and political), and second, the idea that measurements themselves can be seen as performative, creating and re-creating the very things they are intended to make visible.

Representational theory defines measurement as “the correlation of numbers with entities that are not numbers,” a process of transformation, translation, and even interpretation at the level of sampling and gathering data. What is selected for measurement and what is not, how measurements are standardized, what counts as an important unit of measure, and how measurements are used all have stakes for the systems of which they are part.

Moser & Law (2006) argue that current metaphors for information as “flow” are inaccurate, as these metaphors presume that information is immutable, something that is created and exists in the world and thus can be taken up, passed around, and used for calculation.  Moser and Law instead argue that we can see information as something that is inherently mutable and relational, that changes its shape as it is circulated and used.  To put it more simply, information never fully has meaning on its own – it becomes meaningful and usable when a particular person or group make decisions about what the information is and how they can use it.

A good example comes from a recent study on counting rates of infection in hospitals (Dixon Woods et Al., 2012).  The authors found that an act as seemingly simple as counting infections was actually highly social and cultural – the answer to the question “what counts?” varied widely from one hospital to another, calling into question the current focus in healthcare (and investment of healthcare dollars) on quality measures as a tool for achieving reforms such as infection reduction in practice. Making meaning of numbers requires acts of both calculation and judgment, what Moser & Law call “qualculation.”

March-April 2014: Studying Hackers, Makers, and Engineers

This month’s theme – ethnographies of hackers, makers, and engineers – is edited by Morgan G. Ames, who made the transition from being a hacker to studying them herself.

Stories abound of the translation work that ethnographers in industry do to make the experiences of those they observe – the ‘users’ – legible to engineers, designers, marketers, and others in the businesses that employ them. Lucy Suchman, one of the first and most inspiring critical ethnographers in industry, referred to the expectations she encountered for this translation-work in her own job at Xerox PARC as ‘throwing results over the wall.’ These results may take the form of the dreaded ‘implications for design’ that ethnographers are sometimes asked to generate, whether for employers or ACM conferences.

What happens, though, when the ethnographic gaze is turned back on those who are usually the beneficiaries of this translation-work? Lucy Suchman explored this in her groundbreaking book, Plans and Situated Actions, drawing on her experiences at Xerox PARC in the 1980s. Alongside Lucy, a growing number of ethnographers from both academia and industry have been exploring the cultures of scientists, analysts, and others like them in a bid to help the outside world better understand them – and for them to better understand themselves.

The March-April edition of Ethnography Matters will continue this ethnographic inversion by featuring guest authors who are exploring the cultures of hackers, makers, and engineers. The authors hail from the Intel Science and Technology Center for Social Computing, a five-university alliance advised by Intel’s Genevieve Bell (another inspiring industry ethnographer). This group is working on new ways of bridging the academy-industry divide, and many of its members are exploring various aspects of hacker, maker, and engineer cultures.

Caution: Hackers Thinking!

The first post, by Nick Seaver (@npseaver), makes the case for studying technologists as a form of ‘studying up’ – of putting those who wield power under the ethnographic lens. He relates this to his own research on music recommendation systems like Pandora and Spotify.

Second,  Austin Toombs (@altoombs) tells us how he “fell in” to doing ethnographic research on a local hackerspace, how he has navigated the line between ethnography, participation, and activism, and what ethnography has taught him about hackers and himself.

Katie Pine (@khpine) and Max Liboiron (@maxliboiron) then discuss their work in health informatics and civil engineering cultures to show how measurement itself is performative, and how ethnography is particularly well-suited to accounting for this performativity.

Lilly U. Nguyen (@deuxlits) tells us how in her own work on the ethnography of software in Vietnam, she both studies and embodies “diaspora” – and she shares the insights that diaspora has given her.

Marisa Leavitt Cohn then considers how NASA engineers approach concerns of legacy, inheritance, and survival of computational practices as they contemplate the end of life of the mission.

Silvia Lindtner (@yunnia) and Amelia Guimarin (@femhacktweets) round out the theme with a post that shares three stories of hackers and makers in China. Their observations complicate the celebratory story of hacking/making, giving us a richly detailed look at some of the real challenges and triumphs in this very active space.

 

On the Importance of Ethnography in Education: an interview with Mizuko ‘Mimi’ Ito

Mizuko ‘Mimi’ Ito

Editor’s Note: We finish off this month’s theme on ethnography in education with an interview with Mizuko ‘Mimi’ Ito (@mizuko). Mimi has some impressive experience with the topics covered this month: she is the Research Director at the Digital Media and Learning Hub, the John D. and Catherine T. MacArthur Foundation Chair in Digital Media and Learning, and a Professor in Anthropology and Informatics at UC Irvine (after getting two PhDs from Stanford). And she is as kind and generous as she is brilliant.

In this interview, Mimi provides insights on bridging disciplines – from ethnography to economics – and institutions – from academia to industry. She also discusses the challenges and opportunities in forging new research agendas and shaping a field, something with which she has a lot of personal experience. We are thrilled to share Mimi’s insights with you to round out this month’s theme on ethnography in education. To learn more about Mimi, check out her many books and reports, summarized at the end of the interview.

---

Morgan: You’ve worked on a lot of compelling projects using ethnographic methods. What do you see as the strengths of ethnography?

Mimi: I think, for me, I was always in an unusual bucket as an ethnographer because I’ve always done research ‘at home’ and I haven’t taken on the frame of culture in quite the same was as ethnographers do, but I’ve adopted and adapted the perspectives and political commitments and methods of ethnography, and for that has worked very well in studying youth media. My approach has been to study youth culture and media as a space of cultural difference within a particular society. These technologies are new and children and youth occupy a somewhat segregated culture. Feminist ethnographies look at social stratification, and my approach shares affinities with those.

When I started out, there wasn’t a lot of work in anthropology looking at children and youth cultures, and I found that the perspectives of ethnography was really useful for looking at these subaltern and disempowered groups. A lot of my perspectives came from my training in anthropology about how to give voice to the unique ingenuity and perspectives of those who are disempowered. The role of youth in most societies as a relatively oppressed and marginalized population has been relatively under-studied in anthropology. The field has done a great job of studying regional inequities, and gender, race, and class, but has been remarkably silent about the everyday oppression that most societies have based on age.

Why Digital Inequality Scholarship Needs Ethnography

Christo Sims

Editor’s Note: We are excited to kick off this month’s theme on what ethnography can bring to education research with a post by Professor Christo Sims (@christosims). Christo has insights from a public school in New York City that was meant to foster digital inclusion across gender, racial, and socioeconomic barriers, but ended up entrenching these barriers instead. His story shows how ethnographic research can answer difficult questions and broaden the usual dialogues about digital inequality in education in fundamental – and important – ways.

---

Why Digital Inequality Scholarship Needs Ethnography

By Christo Sims

Digital inequality scholarship is well-intentioned. It debunks myths about digital media’s inherent egalitarianism and draws attention to the digital dimensions of social inequalities. Digital inequality scholars have shown, for example, that people with access to networked media use those technologies in different ways, some of which are thought to be more beneficial than others. They have highlighted how differences in skills and quality of access shape use. And they have rightly attacked the stereotype of the digital generation. These are important contributions for which we should be grateful.

Yet digital inequality scholarship is also limited in some fundamental, and I believe hazardous, ways. To defend these claims, I will draw on an in-depth ethnographic study of an ambitious attempt to combat digital inequality: a new, well-resourced, and highly touted public middle school in Manhattan that fashions itself as “a school for digital kids.” It is hard to imagine a more concerted attempt to combat digital inequality, and yet the school paradoxically helped perpetuate many of the very social divisions it hoped to mend. In-depth ethnographic studies can help us understand these outcomes, and they can provide us with tools for forming more accurate conceptions of relations between digital media and social inequalities.

Recruitment flier for the Downtown School

July 2013: Ethnography in Education

Guest Editor
Morgan G. Ames

Welcome to this month’s theme on ethnography in education research! From the promise of radio learning nearly a century ago, to the recent hype around One Laptop Per Child, to the current excitement around massive open online courses (MOOCs), education has been a site of constant reform efforts – or, as education researcher Larry Cuban puts it, “tinkering.” While using “big data” to evaluate these reforms has its allure (and can be useful in ethnographic research, as Jenna and Ayman have shown us in previous posts), ethnography is unique in being able to dig below the surface and uncover the complicated processes and contingent effects of education and education reform.

This month’s authors highlight how ethnography can uncover unexpected results or answer difficult questions about some of the thorniest problems in education reform, especially the persistence of various kinds of inequality. Our first article, by Christo Sims (@christosims), tackled this question head-on in an ethnography of a technology-focused public school in New York that inexplicably had many of its less advantaged students transfer out. With his research, Christo was able to say why this was happening and what it means for other efforts for digital inclusion.

Coming up next, we will hear from Ricarose Roque (@ricarose), who is working to break down some of the stubborn gender, racial, and socioeconomic divides in computer science and bring the programming environment Scratch to a more diverse community. She will talk about some of the unexpected benefits parents experienced in the qualitative focus groups she has been conducting as part of her research.

Later in the month, Sheila Frye (@sheila_frye) will tell us about her research on interactive eBooks, which promote active reading habits – a crucial part of literacy – to children who may not learn this skill otherwise. Sheila uses ethnography to take a close look at both the benefits and the potential drawbacks of interactive eBooks. Her enthusiasm for ethnographic methods is infectious; she is one of the few graduate students we know who LOVES her dissertation work!

Performing Success: When mythologies about a technology dominate first impressions

Morgan Ames

 Editor’s Note: We are lucky to have Morgan G. Ames @morgangames back from her fieldwork in South America to contribute a post to March edition of Stories to Action. Morgan gives us an insider’s view of a One Laptop Per Child’s (OLPC) project in Paraguay. Her insights reveal how ethnographic work creates a critical eye to reveal the truth behind what she calls “performing success.”  Her story helps us see how the real benefits that users experience with a technology are often covered up with mythologies that we tell about the device. The result of her work provides invaluable insights for OLPC.

Morgan shared this story below at Microsoft’s annual Social Computing Symposium organized by Lily Cheng at NYU’s ITP. Watch the video of her talk. After her presentation, Morgan also hosted the geek version of My Little Pony or Porn Star (take the test if you haven’t yet!)  in having us guess the technology referred to in overly optimistic quotes about new technologies. You can play along by watching the video of Morgan hosting the game with the conference attendees. Morgan created a tumblr, Techutopianism, dedicated to tracking technology utopian quotes!

_______________________________________________________________

This vignette problematizes the value of first impressions by illustrating an example of participants’ desire to perform success to visitors, especially high-profile ones. In the process, it shows the value of ethnographies, as more sustained research initiatives which ideally last long after the novelty effect of the visitor and of the (techno-)social interactions they are studying have worn off.

The day started like many schooldays in Paraguay. It was a Tuesday in late October, 2010, well into spring, and several months into my fieldwork studying the medium-size One Laptop Per Child (OLPC) project there. The sun was beating down and temperatures had already climbed into the high 20s C when we rolled up to the school at 8am with our visitor in tow, just in time for the start of classes.

The visitor, one of OLPC’s founding members and chief software architects, was in the country for a whirlwind five-day visit. The local non-governmental organization (NGO) in charge of the project, Paraguay Educa, had carefully filled his itinerary with meetings with high-ranking officials they hoped to convince to support the project as well a visit to Itaipu Dam, one of its most high-profile donors – and this school visit.

I was excited and intrigued that this visitor was going to actually visit a school and spend time in a classroom. After several months of fieldwork, I had noted a number of positive aspects about the project, especially due to the sustained efforts the NGO had been putting into teacher training, community outreach, and laptop maintenance, but I had also noted a number of troubling issues, some of them caused by OLPC’s design or support choices. Would he see these issues, and if so, would he act on making them better?