Bafflement, frustration, annoyance: anyone who’s been around the e-cigarette sector for a while is familiar with the emotions prompted by much of the science published in this area.
There is plenty of useful and well-constructed research drawing reasonable conclusions too, of course – the problem isn’t a complete absence of decent scientific work, it’s the way that much of it is crowded out by more questionable papers that start from dubious assumptions, or leap to conclusions, or simply don’t provide clear enough data to let us know exactly what they are doing.
Media reporting, and the media departments of academic institutions anxious to obtain coverage for their scientists, can make matters worse (indeed, my colleague Aidan Semmens blogs this week about one such case).
Of course, clearer-headed observers – some scientists, some not – do publish their own analyses of what’s wrong with such research. Occasionally, corrections get published alongside the original work. Very very occasionally, papers get withdrawn. Yet the damage is already done, in an area where policy is led much more by non-scientists’ perceptions of “the science” than by actual scientific consensus (as indeed is public opinion).
But is chipping away at a flawed edifice the only solution? The psychologist Stuart Ritchie of King’s College London has a more radical idea: do away with the current system of peer-reviewed papers in edited journals altogether.
Today’s journal-led norm for publishing science “comes with big problems”, argues Ritchie in The Guardian: “Paradoxically, the sacred status of a published, peer-reviewed paper makes it harder to get the contents of those papers right.”
The biggest issue, he says, is that of publication bias, the scientific world’s equivalent of media sensationalism: “Reviewers and editors are more likely to give a scientific paper a good write-up and publish it in their journal if it reports positive or exciting results. So scientists go to great lengths to hype up their studies, lean on their analyses so they produce ‘better’ results, and sometimes even commit fraud in order to impress those all-important gatekeepers”, the editors and peer-reviewers.
Identifying and countering bad research
What are his alternatives? “Maybe the decision to publish could be made based only on the methodology of a study, rather than on its results” – something which, in the tobacco world, would focus attention on thoughtful experimental design (where it is often lacking) at the same time as reducing the incentive to provide Big Answers at all costs.
Or going even further, “maybe scientists could just publish all their research by default, and journals would curate, rather than decide, which results get out into the world”. In an ideal scenario, suggests Ritchie, instead of predigesting extensive work for relatively short papers published in journals, scientists would self-publish “notebooks” – websites containing all the data and analysis for a specific project.
This, too, would be welcome in the tobacco-alternatives world. Not only might it draw more attention to problematic issues such as methodology and definitional conventions, the availability of full data sets would enable third parties to conduct their own analyses – even where the “wrong” approach was taken by the original researchers. It might not eliminate bad tobacco research, but it could make it easier to identify and argue against.
Ritchie’s vision may be a dramatic one, but he is far from alone in seeing systemic shortcomings in journal-dominated science publishing, and looking for alternatives. The online research journal eLife, for example, retains some editorial control over content – it doesn’t accept everything that’s submitted – but it strives for a less competitive, more open model.
For example, it publicly distances itself from the “impact factor” concept (which gives journals a ranking of importance based on how often they are cited, and is widely regarded as putting pressure on scientists to get their work into the highest-ranked journals). It says “we do not artificially limit the number of articles we publish or have a set acceptance rate”, again to minimise the temptation to hype up. And, importantly, it invites public comments as part of peer review.
Journal publishing is not going to disappear overnight, if ever. It’s a very big business with many vested interests and, of course, a great track record in publishing important science as well as questionable science – it certainly shouldn’t be dismissed or demonised. But particularly in areas like tobacco, where there is so much political and media interest (and most of it headed in one direction), its weaknesses can be all too visible.
– Barnaby Page ECigIntelligence staff
Artwork: Mohamed Hassan
- If you are interested in how science and the media can affect the vaping market, you can now download for free our full briefing “Warning on youth vaping ‘a desperate effort to maintain a crisis narrative’” using the form at the top of this page.
