English
 
Privacy Policy Disclaimer
  Advanced SearchBrowse

Item

ITEM ACTIONSEXPORT
  Cochrane evaluation of (semi-)automated review methods: protocol for an adaptive platform study within reviews

Gartlehner, G., Banda, S., Callaghan, M., Chase, J.-A., Dobrescu, A., Eisele-Metzger, A., Flemyng, E., Gardner, S., Griebler, U., Helfer, B., Jemiolo, P., Macura, B., Minx, J. C., Noel-Storr, A., Tahmasebi, N. R., Sharifan, A., Meerpohl, J. J., Thomas, J. (2026): Cochrane evaluation of (semi-)automated review methods: protocol for an adaptive platform study within reviews. - Journal of Clinical Epidemiology, 198, 112390.
https://doi.org/10.1016/j.jclinepi.2026.112390

Item is

Files

show Files
hide Files
:
1-s2.0-S0895435626002660-main.pdf (Publisher version), 2MB
Name:
1-s2.0-S0895435626002660-main.pdf
Description:
-
OA-Status:
Hybrid
Visibility:
Public
MIME-Type / Checksum:
application/pdf / [MD5]
Technical Metadata:
Copyright Date:
-
Copyright Info:
-

Locators

show

Creators

show
hide
 Creators:
Gartlehner, Gerald1, Author
Banda, Susan1, Author
Callaghan, Max2, Author                 
Chase, Jo-Ana1, Author
Dobrescu, Andreea1, Author
Eisele-Metzger, Angelika1, Author
Flemyng, Ella1, Author
Gardner, Sean1, Author
Griebler, Ursula1, Author
Helfer, Bartosz1, Author
Jemiolo, Pawel1, Author
Macura, Biljana1, Author
Minx, Jan C.2, Author                 
Noel-Storr, Anna1, Author
Tahmasebi, Noosheen Rajabzadeh1, Author
Sharifan, Amin1, Author
Meerpohl, Joerg J.1, Author
Thomas, James1, Author
Affiliations:
1External Organizations, ou_persistent22              
2Potsdam Institute for Climate Impact Research, Potsdam, ou_persistent13              

Content

show
hide
Free keywords: Study protocol; Evidence synthesis; Artificial intelligence; Workflow validation; Study within reviews; Systematic reviews
 Abstract: Background and Objectives: Artificial intelligence (AI) has the potential to improve the efficiency of evidence synthesis and reduce human error. However, robust methods for evaluating rapidly evolving AI tools within the practical workflows of evidence synthesis remain underdeveloped. This protocol describes a study design for assessing the effectiveness, efficiency, and usability of AI tools in comparison to traditional human-only workflows in the context of Cochrane systematic reviews.
Methods: Members of the Cochrane Evaluation of (Semi-)Automated Review Methods (CESAR) project developed an adaptive platform study-within-a-review design, modeled after clinical platform trials. This design employs a master protocol to concurrently evaluate multiple AI tools (interventions) against a standard human-only process (control) across 3 key review tasks: title and abstract screening,
full-text screening, and data extraction. The adaptive framework allows for the addition or removal of AI tools based on interim performance analyses without necessitating a restart of the study. Performance will be assessed using metrics such as accuracy (sensitivity, specificity, precision), efficiency (time on task), response stability, impact of errors, and usability, in alignment with Responsible use of AI in
evidence SynthEsis principles.
Results: The study will generate comparative data about the performance and usability of specific AI tools used in a semiautomated or fully automated manner relative to standard human effort. The protocol provides a flexible framework for the assessment of AI tools in evidence synthesis, addressing the limitations of static, one-time evaluations.
Conclusion: This study protocol presents a novel methodological approach to addressing the challenges of evaluating AI tools for evidence syntheses. By validating entire workflows rather than individual technologies, the findings will establish an evidence base for determining the viability of integrating AI into evidence synthesis workflows. The adaptive design of this study is flexible and can be adopted by other investigators, ensuring that the evaluation framework remains relevant as new tools emerge.

Details

show
hide
Language(s): eng - English
 Dates: 2026-07-152026-10-01
 Publication Status: Finally published
 Pages: 10
 Publishing info: -
 Table of Contents: -
 Rev. Type: Peer
 Identifiers: DOI: 10.1016/j.jclinepi.2026.112390
PIKDOMAIN: RD5 - Climate Economics and Policy - MCC Berlin
Organisational keyword: RD5 - Climate Economics and Policy - MCC Berlin
Working Group: Evidence for Climate Solutions
MDB-ID: No data to archive
Regional keyword: Global
Model / method: Quantitative Methods
Model / method: Research Synthesis
OATYPE: Hybrid Open Access
 Degree: -

Event

show

Legal Case

show

Project information

show

Source 1

show
hide
Title: Journal of Clinical Epidemiology
Source Genre: Journal
 Creator(s):
Affiliations:
Publ. Info: -
Pages: - Volume / Issue: 198 Sequence Number: 112390 Start / End Page: - Identifier: Publisher: Elsevier
CoNE: https://publications.pik-potsdam.de/cone/journals/resource/1878-5921