Monday, November 27, 2006

Nano and the need for good social science

Public opinion is in the eye of the beholder. Or at least that’s what it seems like after recent reports of a supposed study of public opinion on nanotechnology in U.S. and European media outlets. In particular, a number of blogs and media organizations ran stories with headlines like this during the past week:

Nanotechnology risks need more study, German survey finds.”

I am not quite sure who is to blame, but I am still in awe over the fact that a single small-group meeting gets misconstrued into a “study of public opinion” and triggers international coverage. The one group that is not to blame are the organizers, who never claimed to provide an accurate assessment of public opinion. In fact, the project, launched by the Bundesinstitut für Risikobewertung (BfR) [German Federal Institute for Risk Assessment], was designed as a consensus conference and explicitly labeled as such:

The staging of a consumer conference puts BfR’s statutory remit on risk communication into practice by directly involving groups of consumers in the discussions about the risks and benefits prior to the introduction of a broadly based consumer application of this technology. This is the first time that a public agency in Germany has used this tool.”

None of the blogs or media outlets who covered the story, of course, bothered to read the original description of the meeting on BfR’s web site. Otherwise they would have noticed that what most media reports call a “survey” was a single meeting of 16 consumers. More importantly, these were not 16 randomly selected consumers, which would hardly qualify as a survey either. Rather, BfR contacted a random 6,000 citizens of the larger Berlin/Brandenburg area in northeast Germany. Of those 6,000 first contacts, 41 responded to the invitation, indicating a willingness to participate, and 16 participants were selected from those 41 replies.

Based on my calculations, this translates into a cooperation rate of 0.6%, which would be a concern even if we could assume that self-selection happens randomly. But it doesn’t of course. And what BfR ended up with is a small, self-selected group of citizens from a specific region in Germany who are involved and interested enough in nanotechnology to attend a three-day meeting.

And all research in communication and political science shows that even if researchers invest significant amounts of money and go through multiple rounds of recruitment, it is difficult to get a representative group of participants in these meetings. I recommend Dan Merkle’s piece on Deliberative Polling in Public Opinion Quarterly a while back. He goes step by step through the problems and biases that have plagued similar efforts.

Merkle, D. M. (1996). The National Issues Convention deliberative poll. Public Opinion Quarterly, 60, 588-619.

But again, none of this would be an issue if media outlets had just stuck with what the BfR was trying to do in the first place:

“The consumer conference draws on the model of the consensus conference. This tool was developed and is used in Denmark. The subject matter and goal of this consumer participation procedure is to assess new technologies and scientific developments from the angle of informed lay persons (citizens or consumers).

The effectiveness of consumer conferences depends on whether and, if so, how they are integrated into political or social processes. Hence, their impact can vary greatly.”

What this whole story highlights, of course, is the need for sound social science in the area of emerging technologies. Public discourse about new technologies tends to focus more and more on the policy implications or the moral aspects of the issue rather than the scientific facts. An that is good, given the immense societal implications that many of these new technologies carry with them. The recent debates about stem cell research and GMOs are good examples. As a result, federal funding guidelines and other regulations of research are all directly tied to the public opinion dynamics surrounding these emerging technologies.

This also means, of course, that we are in desperate need of social science research that systematically tracks opinions, helps us understand the interplay of opinion formation and media coverage, and explores the dynamics between policy groups, media, and citizens. Consensus conferences and technology forums are important components of this research. But they are not surveys or tools of assessing public opinion. Journalists who make inferences from sentiments expressed at consensus conferences to public opinion among the larger public are therefore simply misleading their audiences.


Matthew C. Nisbet said...

Dietram and I have for a long time been in agreement over the limits of small scale "public engagement"-style town meetings. These settings might provide important mechanisms for decision-makers to receive input from stakeholders, while also serving to soften some of the polarization among the already intensely activated and involved, but they do little to reach a wider public. Study after study show that only the already opinion-intense and/or mobilized are the citizens most likely to show up at these meetings. We found this in a recent article we published in Political Behavior, where the citizens most willing to show up, participate, and speak out at a proposed local town meeting on stem cell research were the citizens who held intense opinions and who were already being mobilized by way of church and other recruitment networks.

Goidel, K. & Nisbet, M.C. (2006). Exploring the roots of public participation in the controversy over stem cell research and cloning. Political Behavior, 28 (2), 175-192.

Instead of town meetings as an engagement tool, I have been pushing framing as an alternative, "micro-targeting" segments of the public with intepretations of the issue that take a complex topic like nanotechnology and global warming and make it personally relevant. See this news write-up from a recent lecture I gave at AAAS HQ in DC as part of their Policy Seminar series.