Tuesday, August 28, 2007

On Response Bias

Portfolio's Jeff Bercovici accuses Eric Alterman of "playing stupid" when writing about the results of the Pew Research Center's analysis of the public's news interests over the last two decades. One component of the findings is that people are not particularly interested in what the study categorizes as "tabloid" fare. Quoting Alterman on this point about Americans' professed tastes, Bercovici responds:

Wow. They profess that, do they? Well, then, it must be so.

Of course it's not. Though he doesn't say so here, I'm guessing Alterman is familiar with the concept of response bias, which pollsters have to grapple with every time they design a questionnaire. Survey respondents routinely answer questions as they think they're supposed to rather than as they actually would if they were being candid. So an Us Weekly subscriber, asked if she would like to see more Paris Hilton stories on the nightly news, says no rather than have the interviewer think she's an idiot. But put her alone in front of a TV when a Paris story comes on and she'll respond the honest way: with her eyeballs.

Fair enough; this is the easy rebuttal, and it's the one that popped into my head upon first reading of the study's results. But it's so easy, in fact, that if you read the second part of the report -- like Alterman did, one hopes -- you'll find a whole section dedicated to explaining why Michael Robinson, the study's author, doesn't think it holds up (see pages 7-10).

Part of Robinson's argument is that the public doesn't seem to be particularly embarrassed about other news-watching habits -- their fascination with weather, for instance, or their lack of interest in foreign news -- so why lie about tabloid news? Another is that ratings at times suggest the public has a pretty solid appetite for hard news. The strongest argument, however, is that, given the realities of the marketplace, all news outlets are doing is competing for very small niche audiences, and capturing them can yield big ratings gains. So even though you see a bump for, say, Paris Hilton stories, it's not necessarily the case that the majority of Americans are into it; indeed, on the whole, relatively few Americans watch news -- cable news in particular -- to begin with.

Here's Robinson:
It is true that when CNN rushed to cover [Anna Nicole] Smith’s death, CNN tripled its audience -- and its rating -- from the day before. For CNN -- and its cable-news competition -- that is the story about audience. And in the feverishly competitive world of cable news, that "dramatic" increase in CNN’s rating helps explain why in the three-week saga of Smith’s death and internment, cable news sources devoted 22% of their entire newshole to her.

This is what a shift in ratings will do to news content. But how big a shift is this in terms of the national audience? On CNN, the audience initially increased by a factor of three, an increase of approximately a million people -- less than a single Nielsen ratings point. And from the sociologist’s perspective, that shift represents less than one half of one percent of the nation.
Now, I'm not sure if these are knockdown arguments (I'm definitely skeptical of the first) but they're certainly not frivolous (particularly the last). With limited column space, one could see why Alterman might not address the issue of response bias, but Robinson thought about it, and he came up with a set of claims to respond to people like Bercovici.

Perhaps Bercovici can actually read the report and acquaint himself with these arguments -- as opposed to, say, jumping to the accusation that Alterman is being deliberately obtuse and the implication that Robinson and Pew don't understand basic polling issues? Just a thought.

UPDATE: Alterman responds to Bercovici.

UPDATE 2: Bercovici updates his analysis. Incidentally, I think he's pretty much right in what he says. As I mentioned in my original post, I'm definitely skeptical of the argument that proceeds from the public's professed interest in weather and the lack of interest in foreign news; it's a different matter for someone to fess up to being into trash. Also, even assuming the study is sound and not irretrievably tainted by response bias, Alterman's confusion as to why this stuff exists is strange, for the reasons Bercovici identifies. News organizations function off and pursue niche audiences; they don't program for America At Large.