Systematic review protocols and registration

Why? How? What (needs improving)?

Systematic reviews are a key tool in the practice of evidence based medicine. Their findings and conclusions aim to support decision-making in health care practice and policy, but are also a valuable basis for future research. Systematic reviews synthesise and critically appraise all relevant evidence available regarding a specific research question using systematic methods (hence the name).

But how to make the methods systematic?

The answer is simple: By prospectively specifying them. Not only does this provide a roadmap for the review team, it also reduces the risk that the methods are being modified during the systematic review process depending on the results, thus introducing bias. For example, if I decided to report only certain outcomes and analyses because they have significant results, I would bias my systematic review’s results and conclusions.

Two transparent ways of prospectively specifying one’s methods are to publish a systematic review protocol as a peer-reviewed article and/or to register the systematic review in PROSPERO, the international prospective register of systematic reviews (1).

Publishing a systematic review protocol as a peer-reviewed article has the advantage that independent reviewers will critically appraise the proposed methods. While publishing protocols has long been the standard for Cochrane reviews, protocols for most other systematic reviews have only regularly been published from 2012 onwards. Nowadays several systematic review protocols are published each day, e.g. in the journals Systematic Reviews or BMJ Open.

PROSPERO records have the advantage that they can be updated throughout the review process to report important changes and to reflect the current status of the review (ongoing, completed, published or discontinued). Prospective registration of systematic reviews is possible since February 2011, when PROSPERO was launched. Since then, over 50,000 reviews have been registered (as of July 1, 2019)*.

These numbers demonstrate the success of past efforts, which also include the publication of a reporting guideline for systematic review protocols, the PRISMA-P checklist (2). Nevertheless, there are still various issues surrounding systematic review protocols and registration, some of which I have investigated in my research:

1. The time between submission of a manuscript for a systematic review protocol and its publication has increased over time (3). In recent years, it was disproportionately high, with no improvement in sight. This might discourage authors from publishing protocols as peer-reviewed articles.
2. Systematic reviews often differed from their protocols in one or more aspects of the methods, but the changes were seldom reported and explained by the authors (4). Changing pre-specified methods is not bad per se – it might even be a good idea. However, the process must be made transparent for protocols to fulfil their purpose.
3. Few systematic reviews updated their PROSPERO records’ status following publication. Most were still marked as ongoing (more details on that at EBMLive 2019). So, if someone was to search PROSPERO for published reviews on a specific research question, they would not find what they are looking for. In turn, we don’t know if systematic reviews whose status is ongoing are actually ongoing, or whether they have been published or discontinued.

Therefore, I believe that we need further innovation in the field of systematic review protocols and registration. We should discuss alternative models of peer review for systematic review protocols, agree on guidance on the reporting of discrepancies between review and protocol, and find new ways to keep PROSPERO up-to-date. I am looking forward to discussing my ideas with you at EBMLive 2019!

Short bio

Tanja Rombey is a 2019 Doug Altman Scholarship recipient and a doctoral student (Theoretical Medicine) at Witten/Herdecke University, Germany. She has a Master’s degree in Public Health from the University of Sheffield and a Bachelor’s degree in Health Economics from the University of Cologne. Tanja has a keen interest in systematic reviews and their methods as a part of evidence based medicine. She currently works on a systematic review and non-linear dose-response meta-analysis. Tanja has no conflicts of interest to declare.

 References

1. Booth A, Clarke M, Ghersi D, Moher D, Petticrew M, Stewart L. An international registry of systematic-review protocols. Lancet (London, England). 2011;377(9760):108-9.
2. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic reviews. 2015;4:1.
3. Rombey T, Allers K, Mathes T, Hoffmann F, Pieper D. A descriptive analysis of the characteristics and the peer review process of systematic review protocols published in an open peer review journal from 2012 to 2017. BMC Med Res Methodol. 2019;19(1):57.
4. Koensgen N, Rombey T, Allers K, Mathes T, Hoffmann F, Pieper D. Comparison of non-Cochrane systematic reviews and their published protocols: differences occurred frequently but were seldom explained. Journal of clinical epidemiology. 2019;110:34-41.