REPRODUCIBILITY IN THERAPEUTIC RESEARCH

Ideally, medical practitioners and health policy analysts feel more confident when the results and conclusions of scientific experiments can be replicated. However, it has been found that many scientific studies are difficult or impossible to replicate or reproduce. In response, scientists have launched various replication initiatives to assess the robustness of published research. For example, in psychology, numerous labs have volunteered to re-run studies, with the methods used by the original researchers : in one examination of 100 experiments only 39 were successfully replicated. The same concerns have been expressed about biomedical research. But while attempts to replicate psychological studies are often easy to implement and feasible, attempts to replicate medical studies (pre-clinical or clinical) are often costly and difficult to run. This is especially true for randomized controlled trials (RCTs) although these studies are typically of major importance, and expense, in informing health practices and health policies. 

 

It is nonetheless possible to explore, in a first approach, whether qualitatively similar results and conclusions can be drawn from an independent re-analysis of RCT data. An empirical analysis suggests that only a small number of re-analyses of RCTs have been published to date; of these, only a minority were conducted by entirely independent authors. Thirty-five per cent of published re-analyses yielded changes in findings that implied conclusions different from those of the original article as to which patients should be treated. Most re-analyses have been conducted for selected studies, including some very disputed studies, for example study 329, a well known study on paroxetine in adolescent depression presenting the drug as safe and effective while the re-analysis demonstrated a lack of efficacy and some serious safety issues. 

 

Data sharing is increasing in medicine and will enable re-analyses of this sort to be run. Numerous pharmaceutical companies have created mechanisms for investigators to access IPD. Although more than 6000 trials are already available to investigators through open data platforms, only 15.5% had been requested by a small number of researchers. Most proposals have focused on non-pre-specified subgroups or predictors of response rather than on validation of study results. 

 

To date, no independent, systematic and large scale re-analysis project has been published in the field of RCTs. We propose to conduct a project of this sort, which is needed 1/ to explore the existence and the extent of a ‘reproducibility crisis’ in biomedicine and 2/ to better inform editors, funders and other stakeholders as to whether their new policies really make science better, more transparent and reproducible.