Monday, February 10, 2014

Fads and fashions in science can kill

From Peering into Peer Reviewpicture by  ERIC PALMA
A cluster of recent papers in Science and Nature have caught my eye and collectively draw attention to some of the problematic aspects of how science currently works.

The first paper is called "Modelling the effects of subjective and objective decision making in scientific peer review" by In-Uck Park, Mike W. Peacey & Marcus R. Munafò and done in Bristol and Bath, published in Nature.

The abstract is telling:
The objective of science is to advance knowledge, primarily in two interlinked ways: circulating ideas, and defending or criticizing the ideas of others. Peer review acts as the gatekeeper to these mechanisms. Given the increasing concern surrounding the reproducibility of much published research1, it is critical to understand whether peer review is intrinsically susceptible to failure, or whether other extrinsic factors are responsible that distort scientists’ decisions. Here we show that even when scientists are motivated to promote the truth, their behaviour may be influenced, and even dominated, by information gleaned from their peers’ behaviour, rather than by their personal dispositions. This phenomenon, known as herding, subjects the scientific community to an inherent risk of converging on an incorrect answer and raises the possibility that, under certain conditions, science may not be self-correcting. We further demonstrate that exercising some subjectivity in reviewer decisions, which serves to curb the herding process, can be beneficial for the scientific community in processing available information to estimate truth more accurately. By examining the impact of different models of reviewer decisions on the dynamic process of publication, and thereby on eventual aggregation of knowledge, we provide a new perspective on the ongoing discussion of how the peer-review process may be improved.
The first paras give a flavour:
"Current incentive structures in science promote attempts to publish in prestigious journals, which frequently prioritize new, exciting findings. One consequence of this may be the emergence of fads and fashions ... leading to convergence on a particular paradigm or methodology. This may not matter if this convergence is on the truth...However, there is increasing concern that many published research findings are in fact false1. It is common for early findings to be refuted by subsequent evidence, often leading to the formation of groups that interpret the same evidence in notably different ways2, and this phenomenon is observed across many scientific disciplines3, 4. There are a number of relatively recent examples of convergence on false hypotheses...Once established, these can become surprisingly difficult to refute6... Science may therefore not be as self-correcting as is commonly believed8, and the selective reporting of results can produce literatures that “consist in substantial part of false conclusions”9.
They then demonstrate that in a simple model, the consensus can converge on a false hypothesis. I'm not wholly convinced by the details of their model, but the phenomenon they are studying is certainly important.

The seriousness of this problem is underlined by two reports in Science: one reporting a controversy over a paper suggesting that defective research may have led to the deaths of 800,000 people in Europe due to faulty guidelines about the use of Beta Blockers.  The other showing that the NIH Peer Review process does a very poor job of predicting which studies are likely to have a high impact on practice.

In some respects this is a sub-problem of the bigger problem which is that people with different background worldviews can look at the same evidence and never converge. Atheists and Christians may be an example.

No comments: