Capability of chatbots powered by large language models to support the screening process of scoping reviews: a feasibility study

Abstract

Objectives

The surge in publications increases screening time required to maintain high-quality literature reviews. One of the most time-consuming phases is title and abstract screening. Machine learning tools have semi-automated this process for systematic reviews, with limited success for scoping reviews. ChatGPT, a chatbot based on a large language model, might support scoping review screening by identifying key concepts and themes. We hypothesize that ChatGPT outperforms the semi-automated tool Rayyan, increasing efficiency at acceptable costs while maintaining a low type II error.

Materials and Methods

We conducted a retrospective study using human screening decisions on a scoping review of 15 307 abstracts as a benchmark. A training set of 100 abstracts was used for prompt engineering for ChatGPT and training Rayyan. Screening decisions for all abstracts were obtained via an application programming interface for ChatGPT and manually for Rayyan. We calculated performance metrics, including accuracy, sensitivity, and specificity with Stata.

Results

ChatGPT 4.0 decided upon 15 306 abstracts, vastly outperforming Rayyan. ChatGPT 4.0 demonstrated an accuracy of 68%, specificity of 67%, sensitivity of 88%-89%, a negative predictive value of 99%, and an 11% false negative rate when compared to human researchers’ decisions. The workload savings were at 64% reasonable costs.

Discussion and Conclusion

This study demonstrated ChatGPT’s potential to be applied in the first phase of the literature appraisal process for scoping reviews. However, human oversight remains paramount. Additional research on ChatGPT’s parameters, the prompts and screening scenarios is necessary in order to validate these results and to develop a standardized approach.

Introduction

Given the exponential increase in scientific publications in recent years, there is a need for timely literature reviews to summarize the available evidence and inform healthcare guidelines.¹ However, these reviews are labor-intensive and time-consuming, with systematic reviews often taking over a year to publish.^1–8 This has led to a growing interest in leveraging artificial intelligence tools to support the review process.^4,5,8 A particularly time-consuming effort lies in the screening of titles and abstracts, typically involving thousands of abstracts reviewed by two independent researchers.^7,9 This phase is crucial for the quality and validity of the review, as results are dependent on a comprehensive database search and a thorough, unbiased screening process.^10,11 The abstract screening phase averages 33 days, with the yield rate of relevant articles ranging on average from 1% to 2.9%.^2,3,8,12 This highlights the potential benefits of artificial intelligence support in this area.^2,4

Text mining, an artificial intelligence technique, has supported automation in systematic reviews from 2006 on.⁴ Since then, the underlying artificial intelligence techniques have been refined with review tools such as Rayyan, Covidence, and EPPI-Reviewer employing semi-automated machine learning algorithms to rank references by relevance.^7,13,14 These tools learn from human decisions to develop an initial ranking and improve it with subsequent human input.⁵ Natural language processing and text mining techniques—including word tokenization, stop word removal, stemming, and data-mining—extract features to structure the textual data and infer implicit knowledge.^2,4,12 These features are then used to train classifiers to predict the inclusion likelihood of an unclassified abstract and to rank accordingly.^2,4,7,13,14 Learning algorithms vary, with most tools using support vector machine and some others relying on Naive Bayes and k-nearest neighbours.^2,4,15 Several studies estimate that machine learning tools can halve the screening workload, while detecting 95% of relevant articles.^5,12,14

While automation tools have proven effective for systematic reviews, they have not proven equally beneficial for scoping reviews.¹² Scoping reviews are increasingly utilized to synthesize evidence and identify evidence gaps.¹⁶ Unlike systematic reviews, which address narrow, well-defined research questions, scoping reviews tackle broader questions with wider search and screening criteria.^17,18 Their inclusion criteria typically focus on broad concepts and contexts rather than specific interventions and comparators.¹⁶ For instance, a scoping review on digital tools for interprofessional communication in healthcare encompasses a variety of digital solutions across diverse professional contexts including physicians and nurses.¹⁹ Scoping reviews further incorporate evidence from various sources, including primary research and non-empirical evidence.²⁰

Given that chatbots employing large language models have demonstrated potential to identify key concepts and themes within texts and have shown promising results when screening abstracts for systematic reviews, they might also enhance screening processes for scoping reviews.^6,10,11,21 Large language models are trained on extensive datasets to learn patterns and understand context, followed by fine-tuning for expected outcomes.²² They use deep neural networks to process information through multiple interconnected layers.²³ In the case of Open AI’s ChatGPT, the input layer tokenizes text. The tokens are then converted into high-dimensional vectors in the embedding layer to capture semantic meaning. This information is then passed to several transformer blocks, with the output being processed through several layers to generate the final probability distribution over the vocabulary.²⁴ This structure enables ChatGPT to respond to queries without prior training from end-users.^11,25

Currently, semi-automated tools such as Rayyan, a review tool designed for systematic reviews, have also been used to support scoping reviews. We hypothesize, however, that using ChatGPT in the first screening phase of scoping reviews is superior to using Rayyan, and increases efficiency at acceptable costs while maintaining a low type II error (ie, missing relevant articles). To test this hypothesis, we compared the decisions made by ChatGPT and Rayyan to those of human researchers during the first screening phase of a sample scoping review and assessed the performance.

Methodology

This retrospective study used human researchers’ final decisions at the first screening phase as a benchmark.

Scoping review description

In December 2022, we conducted a comprehensive search across five scientific databases for publications related to digitally supported interprofessional communication and collaboration in healthcare. We aimed to identify studies that involved a digital tool facilitating interaction between (1) healthcare workers from different professions or (2) healthcare workers with the same background employed in different organizations. We included all types of primary research from any healthcare setting and location, written in English, French, German, Portuguese, or Spanish. We excluded abstracts, book chapters, and literature focused on interactions with students or patients. Details on search strategy, criteria and research process are published elsewhere.¹⁹ After removing 11 767 duplicates in Covidence (Veritas Health Innovation), 15 307 unique records were exported to Microsoft Excel (Microsoft Corporation). Nearly all abstracts were written in English, with seven being uniquely in German, four each in French and Spanish and one in Portuguese. Overall, 3852 abstract were initially missing in the dataset, of which 3612 were retrieved and added manually. This refined dataset formed the basis for analysis.

Selection of chatbot and comparator

We selected ChatGPT 4.0, as a representative of large language models due to its widespread use and strong performance across various tasks.^23,25 Its previous version, ChatGPT 3.5, was also included due to its lower cost. As the comparator we chose Rayyan (Qatar Computing Research Institute), a free, well-established web-based screening tool using artificial intelligence that provided the best user experience in a comparative study of various screening tools.²⁶ Unlike other literature review tools, Rayyan provides quantifiable rating of the inclusion likelihood of abstracts.¹³ Rayyan uses text mining methods, extracting words from titles and abstracts and converting them into numerical features (n-grams) to structure the data. A support vector machine classifier analyses these features and creates a predictive model based on the patterns identified in reviewers’ decisions, updating its predictions as reviewers proceed with screening. An abstract’s inclusion likelihood is displayed in the form of a 5-star rating.^12,13 In our study, a rating of < 2.5 out of 5 stars was deemed to be an exclusion; a rating of >2.5 stars an inclusion and a rating of 2.5 stars was “not rated.”

Training the tools

To create a training set of 100 abstracts, we randomly selected 50 excluded and 50 included abstracts according to the screening decisions from two independent researchers, with a third researcher resolving discrepancies. Of the 50 included abstracts, 30 were excluded after full text screening. This training set served (1) to train Rayyan, which requires at least 50 decisions including five inclusions and five exclusions and (2) to engineer the prompt for ChatGPT.²⁷ All abstracts were uploaded to Rayyan and training abstracts labelled according to the human screening results. Rayyan’s ratings were computed and recorded in a Microsoft Excel sheet (Microsoft Corporation).

The first author (K.N.) conducted iterative prompt engineering, refining the prompt based on the decisions and explanations that ChatGPT provided for inclusion or exclusion rather than relying on formalized prompt engineering techniques. The final prompt, including both the instructions and the abstract (see Figure 1), resulted in a sensitivity of 69% and a specificity of 82%. Five days later, the training set was re-evaluated with the same prompt to test ChatGPT’s decision consistency, yielding 68% sensitivity and 79% specificity, displaying a moderate interrater reliability of 0.75.^28,29 The results of both trials were deemed acceptable and we proceeded with full dataset analysis on February 21, 2024, with a second analysis on February 27, 2024. We obtained ChatGPT’s decisions, including an explanation for the decision, through an application programming interface (API) from Excel, using version 6.9 of an Excel macro (ListenData, 2023).³⁰ The “AIAssistant” function was used without word count restriction, prompting ChatGPT to generate the decision and explanation without recalling the prior interactions. Temperature was set to 0.7 throughout all ratings to balance repeatability and creativity. The temperature, ranging from 0 to 2, controls the randomness of the output: a value of 0 selects the most probable words, leading to less creative but repeatable outputs, while a temperature of 1 and beyond allows for more creative but less repeatable outputs.³¹

Figure 1.

ChatGPT prompt with example responses for inclusion and exclusion. (A) ChatGPT prompt featuring a brief explanation of the overall task, followed by the inclusion criteria and a clarification on the abstract’s focus, as well as the exclusion criteria. The prompt concludes with the abstract, illustrated here with a placeholder text. (B) An example ChatGPT’s decision to include an abstract. (C) An example ChatGPT’s decision to exclude an abstract.

Data analysis

We conducted descriptive analysis with Stata version 16 (StataCorp LLC). Using the human decisions as a benchmark, we calculated for each tool and model the number of falsely included and excluded abstracts, as well as those correctly included and excluded. From these, we computed the accuracy (proportion of abstracts correctly judged by AI among all abstracts), sensitivity (proportion of abstracts included by AI among all included as determined by reference standard—also referred to as recall), specificity (proportion of irrelevant abstracts that were excluded by AI), precision (proportion of relevant abstracts determined by reference standard that were included by AI) and negative predictive value (proportion of irrelevant abstracts determined by reference standard that were excluded by AI) (see Figure 2).^10,11,32 Additionally, we derived the false negative rate (proportion of falsely excluded abstracts by AI among all included as determined by reference standard), the proportion missed (proportion of falsely excluded abstracts by AI among irrelevant articles as determined by reference standard) and the workload saving (proportion of abstracts correctly excluded by AI among all abstracts).^11,32 We then calculated the interrater reliability after excluding the not-rated abstracts for each interrater reliability calculation and the costs involved per screening method. As we conducted two sets of ratings for ChatGPT 4.0, we used the average cost. For ChatGPT 4.0 and 3.5, we also qualitatively analysed the output structure and performed a cost evaluation across all tools.

Figure 2.

Metrics of ChatGPT’s and Rayyan’s decisions: ChatGPT 4.0 (: rating 1, n = 15 306 and : rating 2, n = 15 306), ChatGPT 3.5 (, n = 15 213), and Rayyan (, n = 4). (A) Confusion matrix illustrating the AI tool’s decisions vs human decisions as true negatives and positives, as well as false negatives and positives, alongside other relevant metrics. (B) Additional metrics to complement the confusion matrix.

Study dissemination

Study results were presented at the European Conference of Public Health.³³

Results

Manual screening results

Manual title and abstract screening excluded 14 633 manuscripts (95.6%). The mean interrater reliability across reviewer pairs—calculated among the five researchers involved in the abstract screening process—was 0.29 with a standard error (SE) of 0.07, ranging from 0.62 (SE: 0.08) to 0.08 (SE: 0.14). Screening occurred between December 2022 and February 2023.

Description of the (semi-)automated screening results

Reasons for not-rated

After training on 100 abstracts, Rayyan rated only four abstracts differently than 2.5 stars: two with 1.5 stars (rated as “exclude”) and one each with 3.5 and 4.5 stars (rated as “include”). ChatGPT 4.0 did not rate one abstract due to exceeding the 8192 token limit. Conversely, ChatGPT 3.5 rated that abstract but did not rate 94 other abstracts; 93 of which had no abstract (39% of all missing abstracts). In all other no-abstract cases, ChatGPT 3.5 labelled the entry as “exclude.” ChatGPT 4.0 labelled all entries without abstract as “exclude.”

Errors and other indicators

After excluding not-rated abstracts, all ChatGPT models showed similar levels of accuracy (68%), precision (11%), negative predictive value (99%), and specificity (67%). The sensitivity for ChatGPT 3.5 was with 84% lower than for 4.0 (rating 1: 88% and rating 2: 89%). Rayyan’s 100% achievement across these indicators relied on 4 classified abstracts. ChatGPT 4.0 and 3.5 displayed false negative rates of 11% and 16%, respectively. Compared to final full-text decisions, 3% (ChatGPT 4.0) and 4% (ChatGPT 3.5) of abstracts were falsely excluded. Using ChatGPT for screening achieved a workload saving of 64% across all ChatGPT models.

Interrater reliability

Rayyan’s interrater reliability with human researchers was 1.00 (SE: 0.50) but based on four abstracts. The interrater reliability between humans and all ChatGPT ratings ranged from 0.12 to 0.13 (SE: 0.00) (see Figure 3). For 1648 abstracts, the evaluations of the two ChatGPT 4.0 ratings differed, resulting in an interrater reliability of 0.76 (SE: 0.01). The interrater reliability between ChatGPT 3.5 and ChatGPT 4.0 ratings was lower (0.41, SE: 0.01).

Figure 3.

Interrater reliability with SE. A score below 0 indicates a poor, 0.00-0.20 a slight, 0.21-0.40 a fair, 0.41-0.60 a moderate, 0.61-0.80 a substantial, and 0.81-1.00 an almost perfect interrater reliability.³⁴  N refers to the total number of abstracts evaluated by human raters and/or tools. (A) Average interrater reliability of human raters. (B) Interrater reliability between the human benchmark and the decisions of the different ChatGPT models and ratings. “4.0 (1.)” refers to the first rating of ChatGPT 4.0, “4.0 (2.)” to the second rating and “3.5” to the decisions of ChatGPT 3.5. (C) Interrater reliability between the different models and ratings of ChatGPT.

Output format

ChatGPT 4.0 consistently explained its ratings leading with the decision (Include or Exclude), followed by an explanation. In contrast, ChatGPT 3.5 used 3 different patterns:

1. Decision (eg, “Include”) followed by the explanation (n: 11 711),

2. Textual decision (eg, “This abstract should be excluded based on the inclusion and exclusion criteria provided.”) followed by the explanation (n: 1374) and

3. Explanation followed by the decision (n: 2129).

Cost evaluation

The costs incurred for screening differed according to the tool and model used:

1. Rayyan: free of charge.

2. ChatGPT 3.5: $9.06 (API usage fee only).

3. ChatGPT 4.0: $505.72 (base subscription $20 + average API usage fee $485.72).

Note: *The API usage fee represents the average of both ratings (rating 1: $491.91 and rating 2: $479.53), as in some cases the Excel file crashed, requiring reanalysis.

The API usage fee depends on the number of tokens per prompt and output. In English, a token is approximately equivalent to 4 characters.³⁵

Discussion

Summary of results

ChatGPT 4.0 screened and decided upon 15 306 abstracts as compared to four abstracts being decided upon by the semi-automated screening tool Rayyan. All ChatGPT models showed consistent levels of accuracy (68%), precision (11%), negative predictive value (99%), specificity (67%) and workload savings (64%). The sensitivity for ChatGPT 4.0 was at 88% and 89%, and at 84% for ChatGPT 3.5 and the false negative rate at 11% and 16%, respectively. ChatGPT’s high sensitivity demonstrated its strong ability to correctly identify relevant studies, while its lower specificity and accuracy reflected a tendency of over-inclusion, reducing precision. With a negative predictive value of 99%, ChatGPT correctly identified irrelevant studies. The false negative rate of 11% (ChatGPT 4.0) and 16% (ChatGPT 3.5) highlighted the necessity for human oversight to ensure inclusion of all relevant studies and to maintain the review’s quality. At a model temperature of 0.7, the interrater reliability for the ChatGPT 4.0 ratings was substantial, with a moderate reliability when compared with ChatGPT 3.5. Both models demonstrated only slight reliability when compared with human researchers’ decisions. While ChatGPT 4.0 consistently used the same output format, ChatGPT 3.5 produced different output patterns. The cost of deployment varied: Rayyan was free of charge, whereas using ChatGPT 3.5 cost $9.06 and ChatGPT 4.0 amounted to $505.72. The higher ability of ChatGPT 4.0 to correctly identify potentially relevant articles justified the higher cost.

Comparison with semi-automated tools

Our study demonstrates that chatbots are a more feasible alternative in scoping review screening compared to current semi-automated tools, which require substantial human interaction to train and refine their models.¹¹ Despite training with double the recommended sample, Rayyan only decided on four articles.²⁷ However, larger training samples and iterative retraining may improve Rayyan’s performance.³²

In systematic review screening, Rayyan achieved a sensitivity of 78% and a proportion of missed references of 0.5%.³⁶ Semi-automated tools such as Abstrackr and Distiller AI have shown similar sensitivities, with specificities ranging from 72% to 95% and for Distiller AI a precision of 16%.^32,37 Rayyan’s low performance in this study highlights the need for alternative tools in scoping reviews. Semi-automated tools also perform worse for systematic reviews with complex inclusion criteria or multiple research questions, with some studies suggesting limiting their use to reviews only including randomized controlled trials.^32,38,39

Efforts to reduce workload by using semi-automated tools often compromise recall (ie, missing relevant articles) due to three main reasons: (1) unclear stopping point, (2) imbalanced dataset, and (3) biased researchers.^1,12,40 Researchers use various metrics to determine when to discontinue manual screening, such as a certain number of consecutive articles excluded, a minimum prediction score, a pre-determined number of articles screened, or the time spent on screening.^1,12,32,41 The optimal stopping point likely varies by review and is only know in retrospect.^12,14,41 A heuristic estimate of 50% is widely accepted across tools, with studies reporting 95% of all abstracts to be included after screening 29.5%-47.1% of abstracts.^7,12 However, even low-ranked articles may be relevant.^12,14 Imbalanced datasets can bias algorithms towards exclusion.^13,39 Lastly, ranking the articles according to relevance might influence researchers’ decisions, potentially causing complacency and a tendency to underestimate the importance of articles presented at a later stage.¹²

Comparison with large language models

Chatbots like ChatGPT demonstrate advantages for screening processes compared to semi-automated tools.^12,42 First, as a zero-shot model, ChatGPT requires no prior training by the end-user, nor are seed articles needed.^10,43 Second, ChatGPT’s ability to analyse fuzzy or unstructured data enables understanding whether “review” refers to a literature or customer review.⁴² Similarly to human researchers, ChatGPT can provide explanations for its decisions, potentially highlighting errors in reasoning.¹⁰ Lastly, ChatGPT’s multilingual understanding enables researchers to extend their search to non-English research articles, often underrepresented in reviews.⁴⁴

Albeit the performance metrics being slightly lower than for semi-automated tools frequently used for systematic reviews, ChatGPT showed promising results: Several studies demonstrated high efficacy of ChatGPT in systematic review screening, which addresses narrower research questions compared to scoping reviews.^{10,11,21,32,37} One study using ChatGPT 4.0 for a scoping review screening reported an accuracy of 94%, a specificity of 94%, and a sensitivity of 100%.¹⁰ The sensitivity of our study was higher than the specificity, suggesting that ChatGPT was effective at including relevant abstracts but less efficient in excluding irrelevant ones.

Repeating the same prompt on different days resulted in a substantial ChatGPT 4.0 interrater reliability of 0.76, aligning with prior findings.²⁹ The slight interrater reliability with human researchers in our study was lower than in a study with systematic reviews but consistent with other scoping reviews.^10,11 This difference is likely due to scoping reviews demonstrating less clear inclusion and exclusion criteria than systematic reviews. Human researchers also demonstrated a lower interrater reliability in scoping reviews as compared to systematic reviews.^10,11 To foster trust, good alignment between the decisions of human researchers and ChatGPT is imperative.¹¹

Next steps

Different strategies for reducing screening workload have been proposed in the literature, such as using semi-automated tools or relying on a single reviewer. However, these approaches often affect recall: For instance, a single reviewer might miss around 13% of relevant studies.⁷ ChatGPT-supported screening, with its speedy analysis of large datasets, opens up new possibilities for supporting human researchers while ensuring accurate screening and a high recall.^10,45,46 Potential strategies include:

1. Sequential approach: This approach includes an initial screening using ChatGPT to exploit its speed and scalability, with human researchers reviewing only the abstracts included by ChatGPT.^11,32,42 This strategy, based on ChatGPT’s high negative predictive value, could effectively limit the number of abstracts to be screened by researchers, possibly improving researchers’ concentration and motivation.⁴¹

2. Hybrid approach: A hybrid approach combines decisions by one human researcher and the Chatbot, with conflicts being resolved by an additional researcher.^2,29 This might balance the high sensitivity of chatbots with the high specificity of human raters.^2,11

3. Multiple chatbot voting rounds: In this approach, ChatGPT conducts multiple screening rounds. Then, abstracts voted for inclusion at least once are included, or abstracts are included when they achieved a minimum number of votes for inclusion.⁴⁶ This approach can be combined with either the sequential approach or the hybrid approach.

Regardless of the chosen approach, the workload and cost savings possibly limit screening fatigue and enable researchers to use a broader search string, maximizing sensitivity and improving recall.^13,15,41,42

Several considerations are necessary when using ChatGPT: First, accurate outputs rely on complete, correct and non-biased inputs.²¹ Therefore, input data needs to be meticulously prepared, including the manual addition of abstracts. Second, clear and specific instructions (prompts) to the model improve the accuracy of the answers.^47,48 Using the PCC scheme (Population-Concept-Context), frequently applied in scoping reviews, for providing inclusion criteria may be beneficial. Initial trials with prompts using the PICOS scheme (Population-Intervention-Comparison-Outcome-Study design) for systematic review screening with ChatGPT demonstrated good results.¹¹ Providing more information, such as study type and year, might further enhance the chatbot’s performance.¹⁰ Iterative engineering of the prompts is necessary for a good human-chatbot interrater reliability.²⁹ Trials with different chatbot parameters, such as temperature, affecting the randomness of the output generated might further be beneficial.³¹ Lastly, human oversight and trust are crucial. The uptake of ChatGPT might be slow, as researchers tend to be hesitant to use tools not yet widely accepted in the scientific community.³⁸ While automatic deduplication of references is an accepted standard procedure, especially proprietary large language models are perceived as a black box due to the complexity and lack of transparency in their output generation process and because system parameters in terms of both technology and input by researchers are not commonly shared.⁴² Continuous updates of the tool might yield different results, complicating replication efforts.¹¹ To gain researchers’ trust and acceptance in the scientific community, human oversight and a standardized and transparent approach is needed, alongside studies demonstrating the AI tool’s non-inferiority to human researchers in specific phases of a literature review.^7,10,37,38 Furthermore, allowing researchers to set the AI tools’ decision threshold might reduce risk and increase trust in the tool.³⁷

Future research

To deepen our understanding of ChatGPT’s decisions and improve performance, future research is needed to qualitatively analyze ChatGPT’s explanations and compare its reasoning to that of human researchers. Special attention should be given to decisions where ChatGPT diverged from human judgment or from its own judgment in another rating.¹¹ Additionally, we recommend prospectively evaluating the performance and workload savings of ChatGPT when used alongside a researcher, compared against a researcher pair and investigating the different screening approaches discussed above.¹³

Beyond abstract screening, ChatGPT offers potential for implementation in various stages of the review process, including search strategy derivation, full text screening and data extraction.^5,11,21,42 ChatGPT could generate search terms and adapt them to different databases.²¹ As ChatGPT’s token restriction has been expanded, it could become a viable tool for the full text screening phase, as well.¹⁰ Additionally, ChatGPT’s ability to understand context suggests its usefulness in data extraction.¹⁴

Limitations

Key strengths of our study are to elicit reasons for ChatGPT’s decisions, to repeat the rating of ChatGPT 4.0 and to use the current gold standard (final decisions of 2 independent researchers with another researcher settling differences) as reference.³² However, this study also has some limitations. First, despite being the gold standard, research has shown that human decisions are not flawless, being dependent upon their expertise, experience and language proficiency.^5,11,36 Reviewers are trained to be over-inclusive, as evidenced by a 70% exclusion rate during full text screening, yet missing 3% of relevant studies.^37,46,49 Second, our results are based on a single scoping review with a well-defined scope. Further research is needed to investigate the generalizability to other disciplines and broader topics.^10–12 Third, we did not apply a systematic approach to prompt engineering during the refinement process. Utilizing a more structured method for prompt engineering could have potentially improved ChatGPT’s performance. Lastly, due to practical constraints, we compared only one chatbot (two models) and one semi-automated tool whose core technology (supported vector machine) may not reflect the most advanced technology.¹³

Conclusions

With an exponentially growing body of research, maintaining quality in reviews will likely require increased screening time. Our study demonstrates ChatGPT’s potential to be applied in the first screening phase of scoping reviews, demonstrating good levels of accuracy, specificity, and sensitivity and vastly outperforming the semi-automated machine learning tool Rayyan. ChatGPT also demonstrated a negative predictive value of 99% and workload savings of 64%.

Despite the promising results, caution is warranted in solely relying on ChatGPT, as its decisions resulted in a false negative rate of 11% for ChatGPT 4.0 and 16% for ChatGPT 3.5. Human oversight remains paramount. Further research on ChatGPT’s parameters, the prompt, screening scenarios and fields of research is necessary in order to validate these results and develop a standardized approach.

Author contributions

Kim Nordmann: Conceptualization, Methodology, Formal analysis, Investigation, Resources, Data curation, Visualization, Project administration, Writing—original draft. Stefanie Sauter: Writing—review & editing. Michael Schaller: Methodology, Writing—review & editing. Florian Fischer: Supervision, Methodology, Writing—review & editing

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Conflicts of interest

The author(s) declare no competing interests.

Data availability

The data that support the findings of this study are available from the corresponding author, K.N., upon reasonable request.

Ethics approval

As the study did not involve sensitive data, no ethical clearance was necessary.

Declaration of generative AI in scientific writing

During the preparation of this work, the authors used GTP 4.0 in order to improve readability and language. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.