Featured Posts

SitP: Heina Dadabhoy from Islam to Atheism. [caption id="attachment_2139" align="alignright" width="239"] A ninja warrior welcomes guests to Convergence/Skepchickcon[/caption]Boston Skeptics welcomes our January guest speaker, atheist feminist secular...

Read more

Book Club: The Emperor of All Maladies by Siddhartha... [caption id="attachment_2131" align="alignright" width="197"] The Emperor of All Maladies[/caption]This month's book is The Emperor of All Maladies: A Biography of Cancer by Siddhartha Mukherjee, an oncologist...

Read more

SitP: David Ropeik and the Risk Perception Gap Update! Thanks to Andrea and Francois, we now have a video of David Ropeik's talk available on our

Read more

SitP: Larry Gilbertson on GMOs and Biotech [caption id="attachment_2117" align="alignright" width="300"] Feeding the world[/caption]The population of the earth will exceed 9 billion people by 2050. Arable land is decreasing, dietary preferences...

Read more

  • Prev
  • Next

No Really, Just Trust Us: Science Reporting & Citations

Posted on : Aug-25-2009 | By : Joshua | In : Blog Post, skepticism

Tags: , , , ,


Maybe I should cut Boston Globe some slack on this, since their entire science and medicine department got sacked due to budget cuts… except that every single newspaper makes exactly this mistake, including the Boston Globe when it still had a fully-staffed science department. Wait, shit, I’m burying the lede, aren’t I? Damn, maybe those journalists are good for something after all.

You may have noticed that whenever you read a science article in a newspaper, even when they’re just quoting from a press release, journalists will almost never give the title of the study they’re talking about. If you’re lucky, they’ll give the lead author’s name or the name of the journal in which the study was published.

I can kind of see how they might justify this. After all, most academic journals are subscription-only, so most newspaper readers wouldn’t be able to look up the article if they wanted to. But, even for the pay journals, usually an abstract is available online, as it is in the case of the New England Journal of Medicine article that the linked Boston Globe story references. However, failure to give citations is less about whether Joe Average has a subscription to NEJM and more about respecting the process of science. As Ben Goldacre writes here about the media’s propensity to treat science as “absolute truth statements from arbitrary authority figures in white coats, rather than clear descriptions of studies, and the reasons why people draw conclusions from them”.

You can certainly see that at work in the Boston Globe article on vetebroplasty. The writer sets up a tired old “he said, she said” frame, pitting the arbitrary authority of “two recent studies” against the anecdotal wisdom of “many patients – and their doctors”. It’s as if the study authors just randomly decided (is that what they mean by a “randomised trial”?) one day that vertebroplasty is crap, the experience (read: anecdotes) of real-world doctors and patients be damned! Ivory tower! Arrogance!

Of course, that’s not how it works. In reality, the authors of the vertebroplasty studies wanted to figure out whether this popular procedure has any effect, so they designed an experiment to figure it out. (I just picked one study I found in NEJM that was published this month. I have no idea whether it’s either of the studies Boston Globe is talking about, since they didn’t give the article titles, so I just have to assume it is. This is why the titles are fucking important to know.) As per the academic gold standard of the Randomised Controlled Trial, they collected volunteers to undergo either a sham surgery (the controlled part) or the real deal, selected at random (the random part). It’s unclear whether the surgeons were blinded, but that’s usually difficult to do with surgery.

The results? “Improvements in pain and pain-related disability associated with osteoporotic compression fractures in patients treated with vertebroplasty were similar to the improvements in a control group.” This is something we typically see, a non-significant difference between the treatment and the placebo drug or sham procedure. The abstract even includes exact numbers, for those with the statistical knowledge to interpret them.

An obvious criticism leaps out at me: the study has a sample size of 131, which is too small to be really conclusive. But note that this isn’t a criticism anybody makes in the Globe article. The Globe’s not interested in what the study actually says or what its actual flaws might be. It’s only interested in pitting authority versus authority, because that’s easy and doesn’t require any knowledge of the subject area or tedious investigation.

But, hey, it’s not for us plebes to go questioning authorities like academic medical researchers or some doctors or newspaper writers. Just sit back, relax, and take their word for it.

Be Sociable, Share!

Comments (3)

I read the article when you linked to it in an email and thought “well… I’ve seen worse, so what’s so bad about this one?” But your post makes some really good points and the fact that you do it on an article that’s not even the worst of offenders really helps solidify your case. It really is a valid argument to be made that science reporting without the science included is just a mislabled op ed and no better than tabloidism.

Actually, a 131 is a large enough sample that this sort of thing should be regarded as reasonably reliable. Completely conclusive is never obtained but this looks pretty strong.

Good point, but the actual worth of the study was kind of tangential. I’ll admit I don’t know a whole lot about sample sizes and power calculations. I was just trying to give an example of the kind of thing the Globe article’s author might have said rather than simply quoting a few people who insist that vetebroplasty “works”.

Write a comment

You must be logged in to post a comment.