Location
Ngee Ann Kongsi Auditorium (NAKA)
Start Date
4-6-2026 11:30 AM
End Date
4-6-2026 12:00 PM
Description
Systematic reviews are essential to evidence-based research but are often time-consuming and labor-intensive, for researchers to conduct, and for librarians to participate in, or teach. The rapid advances in Generative AI and AI-powered research tools, have created a growing interest in whether these technologies can assist—or even automate—parts of the systematic review process. However, questions remain about their effectiveness, reproducibility, and potential bias. This study explores the role of GenAI tools in the early stages of a systematic review: from literature discovery and screening, up to the identification of included studies.
We propose a comparative case study of one systematic review topic, conducted using three distinct methods. Method 1 (Gold Standard) is a fully manual review following PRISMA guidelines, using traditional searches and non-GenAI tools like ResearchRabbit and ASReview. Method 2 (Human-in-the-Loop) integrates GenAI tools such as ChatGPT, Gemini, and Perplexity deep search, along with research tools like Scite, Elicit, Consensus, Undermind, and AI2 Asta to assist in searching and screening, with human oversight at each step. Method 3 (AI-Dominant) explores an automated end-to-end workflow built with n8n and available LLM APIs (e.g. ChatGPT, Gemini), with minimal human intervention.
The three approaches will be compared on key metrics: efficiency (time and cost), effectiveness (recall, precision, and accuracy of included studies), reproducibility (consistency of outputs and replicability of workflows), and bias (e.g. geographic or citation bias in included studies). The goal is to produce a practical framework for librarians and researchers to evaluate AI tools for use in systematic reviews and provide guidance on which tasks are currently suitable for AI augmentation and which still require human expertise.
Included in
Can AI Help in Systematic Reviews? A Comparative Study of Manual, AI-Assisted, and AI-Dominant Workflows for Librarians and Researchers
Ngee Ann Kongsi Auditorium (NAKA)
Systematic reviews are essential to evidence-based research but are often time-consuming and labor-intensive, for researchers to conduct, and for librarians to participate in, or teach. The rapid advances in Generative AI and AI-powered research tools, have created a growing interest in whether these technologies can assist—or even automate—parts of the systematic review process. However, questions remain about their effectiveness, reproducibility, and potential bias. This study explores the role of GenAI tools in the early stages of a systematic review: from literature discovery and screening, up to the identification of included studies.
We propose a comparative case study of one systematic review topic, conducted using three distinct methods. Method 1 (Gold Standard) is a fully manual review following PRISMA guidelines, using traditional searches and non-GenAI tools like ResearchRabbit and ASReview. Method 2 (Human-in-the-Loop) integrates GenAI tools such as ChatGPT, Gemini, and Perplexity deep search, along with research tools like Scite, Elicit, Consensus, Undermind, and AI2 Asta to assist in searching and screening, with human oversight at each step. Method 3 (AI-Dominant) explores an automated end-to-end workflow built with n8n and available LLM APIs (e.g. ChatGPT, Gemini), with minimal human intervention.
The three approaches will be compared on key metrics: efficiency (time and cost), effectiveness (recall, precision, and accuracy of included studies), reproducibility (consistency of outputs and replicability of workflows), and bias (e.g. geographic or citation bias in included studies). The goal is to produce a practical framework for librarians and researchers to evaluate AI tools for use in systematic reviews and provide guidance on which tasks are currently suitable for AI augmentation and which still require human expertise.