We read Gupta et al's manuscript on ChatGPT (OpenAI; San Francisco, CA) with anticipation and excitement.1 They built upon their previous article about utilizing ChatGPT to develop ideas for systematic reviews.2,3 This iteration had the chatbot generate 10 specific and 10 general systematic reviews for 12 different cosmetic surgery ideas. This resulted in a total of 240 systematic reviews designed for investigation purposes, and Gupta et al set out to determine their true novelty using popular search engines. The novelty was higher for the specific prompting scheme than for the general (75% specific vs 35% general).
As we read Gupta et al's work, we asked ourselves why should we limit the prompt to just systematic review ideas. Is there a potential limitation to ChatGPT regarding systematic reviews? What if chatbots were able to perform the entire systematic review process themselves? This prompted us to ask ChatGPT a series of inputs pertaining to plastic and reconstructive surgery systematic reviews, to emphasize its possible pitfalls and dangers:
Write me a systematic review on vaginoplasty with references that can be published in the Aesthetic Surgery Journal (Appendix A and Appendix B, available online at www.aestheticsurgeryjournal.com).
Write me a full manuscript systematic review on vaginoplasty with an abstract, introduction, methods, results, conclusions, and discussion with references that can be published in a plastic surgery journal (Appendix C, available online at www.aestheticsurgeryjournal.com).
Write me a 5000-word manuscript systematic review on vaginoplasty with an abstract, introduction, methods, results, conclusions, and discussion with references that can be published in a plastic surgery journal (Appendix D, available online at www.aestheticsurgeryjournal.com).
ChatGPT instantaneously performed the above tasks, which would take considerable time and resources if performed manually. It even “performed” a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) review along with its own database search and mimicked a viable abstract as typically seen in other reviews. However, upon taking a closer look, we found that ChatGPT made general claims and did not provide a full manuscript when tasked with doing so. Furthermore, it failed to generate a response and had to be reprompted because it did not fulfill the entirety of the task, leaving it incomplete. Additionally, ChatGPT provided a limited reference list, calling into question its supposed search strategy and the legitimacy of its PRISMA review. The program in its current form does not provide a PRISMA flow diagram or describe its strategy for including and excluding articles. These are potential areas that can be explored in the development of future academic-oriented chatbots. An ideal program in the future may make this possible. Nonetheless the capabilities of ChatGPT impress us.
We believe, for the purposes of a systematic review, ChatGPT in its current state is limited to generating ideas. There is a need for considerable improvements for it to be able to execute the entire systematic review process singlehandedly.
Supplemental Material
This article contains supplemental material located online at www.aestheticsurgeryjournal.com.
Supplementary Material
Acknowledgments
The authors would like to acknowledge OpenAI's ChatGPT for its contribution to the generated supplements.
Disclosures
Dr Dorafshar is entitled to a technology he invented with KLS Martin (Jacksonville, FL) (patent), is entitled to receive royalties on a book he published under an agreement with Elsevier (Amsterdam, the Netherlands), receives an honorarium for educational lectures from DePuy Synthes (Raynham, MA), receives indirect research support from KLS Martin and DePuy Synthes, and has received payment from KCI (San Antonio, TX) as a consultant. The other authors declared no potential conflicts of interest with respect to the research, authorship, and publication of this article.
Funding
The authors received no financial support for the research, authorship, and publication of this article.
REFERENCES
- 1. Gupta R, Park JB, Bisht C, et al. Expanding cosmetic plastic surgery research with ChatGPT. Aesthet Surg J. 2023;43(8):930–937. doi: 10.1093/asj/sjad069 [DOI] [PubMed] [Google Scholar]
- 2. Gupta R, Pande P, Herzog I, et al. Application of ChatGPT in cosmetic plastic surgery: ally or antagonist? Aesthet Surg J. 2023;43(7):NP587–NP590. doi: 10.1093/asj/sjad042 [DOI] [PubMed] [Google Scholar]
- 3. Najafali D, Reiche E, Camacho JM, Morrison SD, Dorafshar AH. Let's chat about chatbots: additional thoughts on ChatGPT and its role in plastic surgery along with its ability to perform systematic reviews. Aesthet Surg J. 2023;43(7):NP591–NP592. doi: 10.1093/asj/sjad056 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.