Interrogating Ethnography again

Steve Lubet has an amusing and depressing little essay about the preposterous defenses some social scientists offer when confronted with evidence that dubious — to put it mildly — research methods appear to be common among ethnographers:

 It had not previously occurred to me, however, that someone with a doctorate in social science might lack a fundamental understanding of basic mathematics, such as the role of the denominator. Alas, that seems to have been the case with UCLA’s professor Stefan Timmermans, whose review, titled “Hypocriticism,” appeared in the journal Contemporary Sociology and was cited approvingly in this year’s Annual Review of Sociology.

In what he evidently intended as a devastating riposte, Timmermans wrote:

I tallied up all the fact-checking in this book and grouped them into three categories:

* Lubet doubts ethnographer/finds ethnographer is wrong: 23

* Lubet finds ambiguity/unable to factcheck: 10

* Lubet finds ethnographer is correct/shows ethnographer fact-checking own claims: 22

Considering that this is selectively drawn from 50 ethnographies, each making ‘‘many thousands’’ (p. 136) of claims, Lubet’s indictment of ‘‘ethnographic malpractice’’ (p. 1) falls flat.

Now, it is true that a problem rate of 23 out of thousands would be inconsequential, but it is always possible to trivialize an observation by assuming an exaggerated denominator. What Timmermans did not choose to recognize – although clearly stated multiple times in the book – was that my sample consisted of only those claims that could possibly be fact-checked. Of that number, at least half turned out to be dubious – 23/45 – which is a troubling error rate, to say the least.

My emphasis.

One of the dozens of ethnographies Lubet interrogates is of course Alice Goffman’s On the Run: a book absolutely loaded with “data” that turned out to be very obviously made up when people investigated the matter. At the time, I suggested that the methods used by ethnographers were, given the nature of of academic publishing, rather problematic, given that the overwhelming majority of such work is simply not subject to any kind of independent verification.

Lubet’s work more than suggests that this is in fact the case.