A mixed-methods study comparing human-led and ChatGPT-driven qualitative analysis in medical education research

ABSTRACT

Qualitative research, used to analyse non-numerical data including interview texts, is crucial in understanding medical education processes. However, it is often complex and time-consuming, leading to an interest in technology for streamlining the analysis. This study investigated the applicability of ChatGPT, a large language model, in thematic analysis for medical qualitative research. Previous research has used ChatGPT to explore the deductive process as a qualitative study. This study evaluated thematic analysis including the inductive process by ChatGPT with reference to human qualitative analysis. A convergent design mixed-methods study was used. Using a thematic analysis approach, ChatGPT (model: GPT-4) analysed some interview data from a previously published medical research article. The assessors evaluated the qualitative analysis of ChatGPT using human qualitative analysis as a benchmark. Three assessors compared the human-conducted and ChatGPT-driven qualitative analyses. ChatGPT scored higher in most aspects but showed variable transferability and mixed depth scores. In the integrated analysis including qualitative data, six themes were identified: superficial similarity of results with human analysis, good first impression, explicit association with data and process, contamination by directions in prompts, deficiency of thick descriptions based on context and research questions, and lack of theoretical derivation. ChatGPT excels at extracting key data points and summarising information; however, it is prone to prompt contamination, which necessitates careful scrutiny. To achieve deeper analysis, it is essential to supplement the research context with human input and explore the theoretical framework.

INTRODUCTION

Qualitative research plays a crucial role in the field of medicine by providing valuable insights for understanding and improving psychosocial aspects of medicine, medical education processes, and practices.^1,2 The history of qualitative research in medicine spans over 20 years, and various research methodologies have been employed.^3,4 Analytical methods, such as thematic and content analyses, have been widely used by researchers.^5-7 Thematic analysis, in particular, is a common method in qualitative research that allows researchers to identify meaningful patterns and themes within data. In addition, clear steps are proposed for thematic analysis, making it easy to validate the analysis process.⁷ Although there are supporting tools, such as NVivo, for these analyses, they have a fundamental requirement for humans to manually analyse the language, extract concepts, describe theories, and construct models. Consequently, mastering qualitative analysis requires extensive training, and the analysis process is time-consuming.⁸

In recent years, advancements in large-scale language models have enabled ground-breaking achievements in the field of natural language processing.^9,10 In particular, ChatGPT, based on the GPT-4 architecture developed by OpenAI and released on 14 March 2023, is equipped with advanced text generation and comprehension abilities, demonstrating performance comparable to that of top human scorers on descriptive test questions.^11,12 ChatGPT is also gaining attention in the medical field for its potential. It has been shown that it can generate draft responses to patients’ questions that are comparable to those of a doctor.¹³ Therefore, there are expectations for its potential to support healthcare services. Such linguistic capabilities of ChatGPT suggest the potential to streamline thematic analysis in qualitative research and alleviate the burden on researchers. Prior research suggests that ChatGPT (GPT-3.5) can assign predefined codes to text.¹⁴ However, no study has examined whether large language models (LLMs), such as ChatGPT, can derive codes and themes from a text on their own in a qualitative analysis process, such as thematic analysis, and whether such codes are equivalent to those generated by human qualitative analysis. Furthermore, there are concerns that the application of LLMs, such as ChatGPT, to academia could lead to misuse or ethical issues.¹⁵ It is also necessary to consider what problems might arise when conducting thematic analysis using ChatGPT.

In this study, we examined the effectiveness of using ChatGPT for thematic analysis in qualitative research and compared its analysis with human analysis. We hope to introduce new possibilities regarding research methods in medicine. Our research investigated to what extent the quality of thematic analysis conducted by ChatGPT is equivalent to that performed by humans. We explored the characteristics of any differences and their implications. Through this study, we aim to propose new methodologies for enhancing the reliability and efficiency of qualitative research in medicine, as well as to deepen the discussion on the potential of utilising LLMs for qualitative research. By elucidating the advantages and limitations of ChatGPT-based qualitative analysis compared with human analysis, we provide insights into the selection and application of qualitative research methods that may benefit future medical studies. Furthermore, the findings of this study advance qualitative research not only in medicine but also in social science and cultural anthropology fields, among others.

METHODS

Design

In this post-positivism-based mixed-methods research study, we used a convergent design to compare human-conducted qualitative analysis in previously published medical education research with qualitative analysis newly implemented using ChatGPT.^16,17 In the previous study, a human-performed thematic analysis was conducted on the transcriptions of interviews with supervisors, which were previously published in an academic paper.¹⁸ Both the full transcripts of the interviews and the process of the thematic analysis were made publicly available. In this research, a thematic analysis of the same data was performed using ChatGPT. The process of this analysis was then compared to the human-conducted thematic analysis from both qualitative and quantitative perspectives. We conducted a quantitative examination using the scales described below to gain a comparative perspective. To consider the meaning of the obtained scores and further examine the depth and creativity of ChatGPT’s analysis, assessors also qualitatively evaluated the analysis by ChatGPT and documented their evaluation as comments. Subsequently, analysis and integration were conducted using the quantitative and qualitative data. Using this approach, we thoroughly investigated the effectiveness of qualitative analysis using ChatGPT and the differences between it and human analysis. The overall flow of the research is illustrated in Fig. 1.

Fig. 1.

Overall research flow

The previous study refers to an academic paper that has already been published. Within this paper, the interviews with supervisors that were subjected to thematic analysis, as well as the process of thematic analysis itself, are publicly available. In the current research, a thematic analysis using ChatGPT was conducted on these publicly available interview records. The thematic analysis performed by humans in the previous study served as a reference for qualitatively and quantitatively evaluating the thematic analysis conducted by ChatGPT.

Thematic analysis using ChatGPT

We conducted a qualitative analysis of interview data using OpenAI’s ChatGPT (model: GPT-4). For the analysis, we selected the qualitative analysis section of the authors’ published paper ‘Locally adapting generic rubrics for the implementation of outcome-based medical education: a mixed-methods approach’.¹⁶ This study compared generic rubrics and localised evaluation forms used to assess medical residents. This paper was selected because transcripts of the interviews are publicly available, peer reviewers could be contacted for collaboration with the current study, ChatGPT does not know about the paper because it was published after ChatGPT’s data were collected, and the authors (TK, AJ, and HN) thoroughly understand the qualitative analysis process, which makes it easier to evaluate the differences between ChatGPT’s qualitative analysis and human analysis. Because ChatGPT currently cannot accept long inputs, the authors selected easily understandable sections from the published interviews, even if they were short texts.

Thematic analysis was conducted following Braun and Clarke’s six-step process for using an inductive thematic analysis approach: (1) familiarisation with the data, (2) generating initial codes, (3) searching for themes, (4) reviewing themes, (5) defining and naming themes, and (6) producing the final report.^7,19 The research question for the analysis was the same as in the original paper: ‘What is the effect of such local adaptation on supervising doctors as assessors?’ Analysis using ChatGPT was conducted via the following process.

First, ChatGPT was asked to act as an experienced qualitative researcher. We conducted a thematic analysis and confirmed that the answers aligned with the authors’ understanding. After briefly explaining the research background in a prompt, we requested that ChatGPT familiarise itself with the interview data and summarise them as part of Step 1. The interview data were input in two parts, and we asked for summaries at each stage to ensure that the generated summaries were consistent with the authors’ understanding. Next, as part of Step 2, we asked ChatGPT to generate the initial codes for the read text. After confirming that the codes were appropriately assigned, we requested that ChatGPT perform steps (3) searching for themes, (4) reviewing themes, and (5) defining and naming themes. Finally, considering the progress thus far, we asked ChatGPT to produce a final report (Step 6) regarding the differences between the two evaluation forms. Appendix 1 presents the prompts and generated responses. This process required 45 min to complete.

Assessors

The authors and peer reviewers of the analysed article participated in this study to assess the quality of the ChatGPT analysis. The authors and peer reviewers were experts involved in medical education research and well-versed in qualitative research. Furthermore, their involvement in writing and reviewing the paper gave them a deep understanding of the research data and content.

Quantitative data and scale for assessing qualitative analysis

To assess qualitative analysis with ChatGPT from a quantitative perspective, a custom scale was developed to compare the differences between human- and ChatGPT-driven qualitative analyses. Because existing scales for evaluating the quality of qualitative research primarily focus on the overall research design rather than the specific analysis process,^20-22 a new scale was created for this study by integrating relevant sections from textbooks on qualitative coding and articles on qualitative research quality assessment.^20-26 The assessors, either the individuals who conducted the thematic analysis in previous studies or peer reviewers who validated those evaluations, were chosen based on their in-depth understanding of the process involved in the human-led qualitative analysis. These individuals are suitable for the task as they can utilise human qualitative analysis as a benchmark when evaluating the analysis performed by ChatGPT. Therefore, all scores for human qualitative analysis were fixed at 4 as a reference when evaluating the analysis performed by ChatGPT. Furthermore, by placing the reference point mid-scale, it was possible to assign both higher and lower scores using the same scale. The qualitative analysis performed by ChatGPT was scored on independent perspectives on the scale, which ranged from 1 to 7 (Table 1).

Table 1.

Scale to evaluate quality of qualitative analysis

Name	Description	Main studies
Depth	Evaluate the depth and richness of the analyses. Consider if it goes beyond a mere description of the data and provides meaningful interpretations. On the scale, 1 should represent superficial and 7 in-depth.	Tong et al,22 2007
Confirmability	Evaluate the confirmability of the analysis, considering its accuracy in relation to the text. On a scale from 1 to 7, 1 represents no correlation and 7 represents a strong correlation.	Braun et al,19 2019; O'Brien et al,20 2014; Saldaña et al,23 2021
Credibility	Consider whether the analysis accurately aligns with the research question. On a scale ranging from 1 to 7, 1 indicates complete lack of credibility and 7 indicates high credibility.	Stenfors et al,24 2020
Transferability	Consider whether the analysis is applicable in other contexts. On a scale ranging from 1 to 7, 1 indicates complete lack of transferability and 7 indicates high transferability.	O'Brien et al,20 2014; Lumsden et al,25 2022
Dependability	Consider the extent to which transparency is ensured in the analysis and whether the process is clear. On a scale from 1 to 7, 1 represents low dependability and 7 represents high dependability.	O'Brien et al,20 2014; Saldaña et al,23 2021
Consistency	Consider whether the analysis is free from inconsistencies and remains consistent throughout. On a scale ranging from 1 to 7, 1 indicates complete lack of consistency and 7 indicates high consistency.	Frambach et al,21 2013

Three assessors, who had either authored or peer-reviewed the target paper and were thus familiar with human qualitative analysis, were asked to review a Word document containing the ChatGPT qualitative analysis process conducted via thematic analysis. They then assigned scores to the ChatGPT analysis using the developed scale. Owing to the limited sample size, no statistical analysis was performed, and the results are presented as graphs.

Qualitative data

Because of the limitations of quantitative evaluation solely based on a scale, which tends to bias towards content that can be objectively evaluated, it becomes challenging to consider the depth and creativity of the analysis. Therefore, we also qualitatively assessed the thematic analysis performed by ChatGPT. After evaluation based on the scale, the assessors were asked to provide free-text descriptions in the same form.

Data analysis and integration

In convergent design mixed-methods approaches, an integration analysis is performed to consolidate the quantitative and qualitative data that have been collected.²⁷ We integrated the quantitative and qualitative data through a process inspired by the analysis process of thematic analysis.

1. Familiarisation with the data: TK and JM independently and thoroughly reviewed the quantitative data and qualitative data, the process of human-driven qualitative analysis, and the ChatGPT-driven qualitative analysis.

2. Generating initial codes: TK and JM independently assigned codes to the qualitative data (comments).

3. Code refinement and theme development: TK and JM brought their codes together for discussion. Taking advantage of TK’s position as the party who conducted the human-driven qualitative analysis and JM’s expertise in qualitative analysis without being involved in the human-driven analysis, the codes were revised to incorporate richer perspectives through discussion. In addition, during the discussion, quantitative data derived from the scores were also compared and considered. Based on these discussions, TK generated themes.

4. Theme review and refinement: All researchers reviewed and refined the themes, ensuring they accurately represented the dataset and provided a comprehensive understanding of the differences between human-driven and ChatGPT-driven qualitative analyses. Discrepancies were resolved through discussion and consensus.

5. Reporting findings: The final themes and associated codes are presented in the results, highlighting the key differences between human-driven and ChatGPT-driven qualitative analyses.

Reflexivity. Reflexivity was considered and practiced throughout the research process to enhance the trustworthiness and validity of the findings. The research team, including the authors of the paper¹⁶ and the peer reviewers, comprised individuals with diverse backgrounds, experiences, and perspectives related to medical education research.

TK, the primary investigator responsible for the ChatGPT-driven qualitative analysis, is a physician in general practice and a medical education researcher. As the author of the analysed paper, TK is deeply familiar with its content and has prior experience in qualitative research. During the comparison of human- and ChatGPT-driven qualitative analyses, TK actively reflected on how his biases and assumptions potentially influenced the evaluation process. JM, a family physician with a master’s degree in medical anthropology and a PhD degree from conducting qualitative research in medical education research, provided a unique perspective to the team. Importantly, although JM was not directly involved in the original paper on which the comparison was based, he has expertise in the subject matter of that paper, having lectured on the topic of rubrics in domestic faculty development programmes. This positions him to provide additional and valuable insight. AJ, the peer reviewer of the analysed paper, offered an external and independent perspective on the quality of the qualitative analysis. TK and JM, who conducted the data analysis and integration, have received training in qualitative research in a master’s programme in medical education and a master’s programmes in medical anthropology and PhD, respectively. HN, a qualitative research expert in medical education, provided valuable insights into the research process and contributed to reflexivity discussions.

Throughout the research process, the team engaged in reflective discussions and continuously examined their roles, biases, and potential influences on the research findings. These reflective practices, combined with the diverse backgrounds of the research team members, contributed to generating a more robust and transparent analysis, ensuring the trustworthiness and validity of the results.

RESULTS

Two of the authors and one of the peer reviewers who had worked on the analysed paper agreed to participate in the ChatGPT qualitative analysis. Thus, three assessors conducted a comparative examination of the human and ChatGPT qualitative analysis.

Quantitative data

As mentioned in the Methods section, human qualitative analysis was assigned a fixed score of 4, whereas ChatGPT qualitative analysis was assessed on a scale of 1 (lowest) to 7 (highest) based on six aspects: depth, confirmability, credibility, transferability, dependability, and consistency. Three assessors conducted analyses based on these criteria. The evaluation was conducted using a web form from April 1st to April 6th, 2023.

The results of the evaluation are shown in Fig. 2. For confirmability, credibility, dependability, and consistency, all assessors assigned higher scores to the ChatGPT qualitative analysis than to the human qualitative analysis. Regarding transferability, one assessor assigned an equal score to both the human and ChatGPT analyses, while the others assigned higher scores to the ChatGPT analysis. However, the evaluation of depth varied more among assessors.

Fig. 2.

Qualitative analysis scores of ChatGPT compared with those of humans

Three assessors rated ChatGPT on six aspects, with the human qualitative analysis assigned a fixed rating of 4: depth, confirmability, credibility, transferability, dependability, and consistency. The minimum score was 1, and the maximum score was 7.

Qualitative data

The three assessors provided comments on the human and ChatGPT qualitative analyses via a web form, along with the above scores (Supplemental Table).

Data analysis and integration

Data analysis and integration were led by TK and JM as mentioned in Methods. The obtained codes were tabulated, and themes were identified (Table 2).

Table 2.

Codes derived from the thematic analysis

Code ID	Code	Text ID
1	Faithful to sentences in the text	A-1
2	Oversight of contamination	A-1,B-5
3	Lack of analysis of social situation and relationship beyond the text	A-2
4	Analysis of what is said not why and how it happens	A-2
5	Explicit association between interview data and research question	A-3
6	Ensuring third person verification of analysis process	A-4
7	Consistency	A-5,B-3
8	Good impression of ChatGPT	A-6,B-1
9	Ability to trace the steps of thematic analysis explicitly	B-2
10	Explicit process	B-3
11	Transparent process	B-3
12	Succinct report production	B-4
13	Lack of 'thick' description	B-4
14	Coherent and precise analytical steps	B-5
15	Better summary of the interview	B-6
16	Lack of explanatory analysis	B-7
17	Absence of theorisation	B-7
18	Good at inductive process	B-8
19	Questionable ability for deductive approach	B-8
20	Lack of consideration of larger context	B-8
21	Similarity in theme	C-2
22	Lack of context-sensitive meaning-making	C-3,C-5
23	Lack of consideration of local context	C-4
24	Sociocultural contextualisation of the data	C-7

Superficial similarity of results with human qualitative analysis (Related codes: 8, 9, 21). Similar processes were followed in the ChatGPT and human thematic analyses, and there were some similarities in the derived themes and their meanings. The theme ‘Invalid assessments’, derived by humans, suggests that abstract descriptions and difficult-to-assess items in the generic rubric lead to inappropriate assessments, which are improved by localised tools. This is similar to the meaning of the ‘streamlined assessment process’ derived by ChatGPT, which suggested that simple items in the localised tools make assessments easier. Despite these differences, the main finding was that the human qualitative and ChatGPT analyses were similar.

Good first impression (Related codes: 8, 18). ChatGPT’s ability to identify key points from interview data and provide summaries, along with its clear analytical process, generally led assessors to have a favourable impression of its analysis.

Explicit association with data and process (Related codes: 1, 5, 6, 9, 11, 14, 15). The ChatGPT analysis made it easier to understand the linkages to the interview data and research questions than the human analysis, and it was also clear which steps of the thematic analysis had been performed in which part of the process. In human thematic analysis, it is difficult to see how the six steps of thematic analysis were used; however, in the analysis by ChatGPT, it was clear which parts of the steps were used.

Contamination by directions in prompts (Related codes: 2). Analysis by ChatGPT included mixed results, with some results derived from prompt directions rather than interview data. In the sixth step of the thematic analysis using the ChatGPT, Theme 1 included the following text: ‘the localised tools seemed to better capture the specific context and situations of the hospital under study’. This does not appear in the ChatGPT analysis. ChatGPT was given the following direction outlining the background of the study: we localised the generic rubric and developed localised tools, taking into account the situation of the hospital under study and the situation, and we believe that this information constituted contamination of the analysis results. However, this contamination was not noticed by the assessors involved in this study.

Deficiency of thick descriptions based on context and research question (Related codes: 3, 4, 13, 16, 19, 22, 23, 24). While ChatGPT selected and summarised the main points of the interview results, it did not connect them to a thick description that considered the interviewee’s context and the background from which the research questions were derived. As pointed out in C-3 (in Supplemental Table), ChatGPT derived the theme ‘Enhanced clarity and specificity’, which is clear from reading the text alone, whereas humans derived the theme ‘Mismatch between the rubrics and clinical context’, which considers the context in which the interviewee was placed and the wording of the rubric.

Lack of theoretical derivation (Related codes: 17, 20). ChatGPT did not derive a theory or theoretical framework for textual content analysis. In the analysis by humans, cognitive load theory was devised as a theoretical framework based on the situation in which interviewees were placed and the interviewees’ comments on the number of questions asked; accordingly, the analysis was deepened. However, no such theory was derived or used in the analysis by ChatGPT.

DISCUSSION

In this study, we employed a large-scale language model, ChatGPT (model: GPT-4), to conduct thematic analysis of text data based solely on simple, pre-training-free instructions, and compared the results quantitatively and qualitatively with those obtained from human-performed thematic analysis. The qualitative analysis conducted by ChatGPT yielded higher scores than those of the human analysts in multiple domains. However, the qualitative assessment revealed certain challenges associated with the qualitative analysis performed by ChatGPT.

The qualitative analysis conducted by ChatGPT yielded results similar to those derived from the human analysis. In this context, the connection between the text and the analysis performed by ChatGPT was more explicit than that of the human qualitative analysis, and the relationship between the analytical steps in thematic analysis was also clear. This is likely to have contributed to the high scores achieved for numerous indicators during the quantitative assessment, as well as the positive impressions conveyed in the comments regarding the analysis. These findings indicate that ChatGPT accurately processed the main points of interview texts and effectively summarised and synthesised information. Previous studies have demonstrated that LLMs, such as ChatGPT, exhibit superior performance in summarising text content.²⁸ Meanwhile, such seemingly positive analytical outcomes may also make it more difficult to discern potential issues within the analysis.

This study revealed that one of the less noticeable problems with ChatGPT analysis was the contamination by directions in the prompts. This means that the directive used in interactions with ChatGPT inadvertently influenced the results of the data analysis. It is already known that LLMs, such as ChatGPT, exhibit a phenomenon called hallucination, wherein they generate plausible yet factually incorrect outputs.²⁹ The inclusion of instructional sentences does not necessarily diverge from the truth; however, it often goes unnoticed by researchers because of the assumptions embedded within instructions. In this study, assessors who authored or peer-reviewed the research papers targeted for analysis by ChatGPT did not detect the inclusion of instructional sentences. This inclusion is likely because a premise was not conveyed to ChatGPT, namely that the analysis should be based on interview data alone, without incorporating the content of the instructions. This discrepancy between the assumptions made by humans and the instructions conveyed to ChatGPT can be difficult to detect and necessitates vigilance. To ensure the reliability of analysis results, it can be argued that the results obtained from ChatGPT require careful and meticulous verification by humans. This point, along with the perspective of data protection, should be noted in future research activities involving ChatGPT. In this study, only already published data were inputted into ChatGPT, so there was no concern about data leakage. However, large-scale artificial intelligence systems, including ChatGPT, are often provided on the cloud. As qualitative research often deals with private matters of the participants, it is necessary to pay close attention to how data are handled on the cloud in future research. In other words, it is essential to be mindful of the characteristics of how artificial intelligence handles data and instructions and to what extent data are shared.

Furthermore, qualitative analysis requires more than merely summarising text; it also requires a deep analysis and application of theories that consider the background and context of the interviewees. ChatGPT’s qualitative analysis faces challenges in this regard. Both quantitative and qualitative assessments revealed that ChatGPT struggles with in-depth analysis. This is likely because ChatGPT has not been provided with the vast amount of information that humans consider during qualitative analysis, including non-verbal information obtained by immersing oneself in the interview and participants’ clinical and educational settings. Additionally, ChatGPT has not been provided with the theories discussed in previous studies. While making the qualitative analysis by ChatGPT more transparent, the absence of such implicit information serves as a limiting factor in its ability to derive valuable insights. These findings highlight the relative strengths of human analysts and ChatGPT.

Shaw et al explained the strategic approach to research analysis as three distinct logics, namely deduction, induction, and abduction, which connect data and theory.³⁰ In deduction logic, data are interpreted by applying existing theories, and theories are tested using data. In induction logic, data are described, and theories are generated from the data. In abduction logic, a recursive and iterative analysis is conducted, moving back and forth between data and theory.^30,31 With reference to the diagram of Shaw et al, the potential applications of ChatGPT in qualitative analysis based on the results of prior research and the present study are illustrated in Fig. 3.

Fig. 3.

Potential applications of ChatGPT in qualitative analysis

ChatGPT can utilise templates and assign codes, thus playing a part in the process of deduction. When humans provide context and background, ChatGPT can participate in the process of induction by generating summaries of qualitative data. For the iterative process of abduction, involving a back and forth between theory and data, dialogue between ChatGPT and humans may be required.

Prior research involving text coding based on codebooks has demonstrated the potential for utilising LLMs, including ChatGPT, for deductive analysis.²⁸ While both inductive and deductive approaches can be employed in thematic analysis,³² it has been suggested that, in fields such as medicine, in which analysis is conducted in conjunction with existing theories and frameworks, an abductive approach is often adopted.^30,31 The analysis conducted by ChatGPT in this study summarised the main points of the data more clearly than did humans through an inductive approach. However, it was unable to derive theories that considered the background and context of the data. ChatGPT was unable to extract particularly intriguing statements from the text and delve deeper into them. Additionally, merely instructing ChatGPT to follow the six steps of thematic analysis in sequence appears to be insufficient to facilitate an abductive approach in which the exploration of existing theories and frameworks is conducted concurrently with data comparison for a deeper analysis. Furthermore, for more creative approaches, such as the reflective thematic analysis by Braun and Clarke and the abductive thematic analysis by Thompson, it is essential to consider the context and the researcher’s position.^19,31 In this study, even when ChatGPT was instructed to proceed from Step 4 to Step 5 in the thematic analysis (Appendix 1), the analysis did not substantially deepen, largely involving only a slight rephrasing of the content. While this underscores a significant limitation of the current analytical method, these steps could potentially serve as a key to facilitating an abductive approach in the thematic analysis by engaging in dialogue with ChatGPT about the context and theoretical frameworks of the research participants and researchers.

Limitations

At present, detailed information on GPT-4 has not been sufficiently disclosed, which impacts the transparency of this study. More detailed data are anticipated to become available in the future. The data in this study were small-scale and the number of assessors was limited to three, which necessitates caution in interpretation. However, the chosen assessors had a deep understanding of the data, and analysis using prior research and theory enhanced the transferability of the findings.

Owing to the current input limitations of ChatGPT, it was only able to analyse a portion of the interview data in this study. This significant restriction, the inability to handle large amounts of text data, which is often required in qualitative analysis, is expected to be improved with future technological advancements.

In this study, we were unable to identify suitable quantitative measures to evaluate the process of thematic analysis for qualitative assessments by ChatGPT. Therefore, we created and evaluated our metrics using existing studies as guides. We hope that in the future, measures that have been examined for reliability and validity will be developed.

The assessors were either authors or reviewers of the papers analysed by ChatGPT, and one of the two individuals who conducted the thematic analysis of the assessors’ comments was the author of the analysed paper. While this may introduce bias into the results and analysis process, it also has the potential to lead to a deeper analysis owing to the assessors’ familiarity with the background and context.

In this study, ChatGPT was only given thematic analysis and text data without detailed information on the background of medical education research and theories. Consequently, its full potential may not have been accurately assessed. Future research that includes input regarding the research context and dialogue regarding theories may reveal capabilities not evident in this study.

CONCLUSIONS

In this study, we conducted a quantitative and qualitative examination of thematic analysis performed by humans and ChatGPT. ChatGPT demonstrated excellence in extracting and summarising data points, but there was the possibility of direction contamination, which requires careful verification. Furthermore, to achieve deeper analysis, it is necessary to supplement the research context with human input and examine the theoretical framework.

ARTICLE INFORMATION

Ethics statement

Ethical approval was not required.

Contributors

TK assumes responsibility for the overall study design, data collection and analysis, as well as for the composition of the manuscript. JM provided guidance on the research design, collaborated with TK in conducting thematic analysis, and contributed to writing the manuscript. AJ offered advice on the research design and worked alongside TK in the manuscript composition process. HN partnered with TK to devise the comprehensive study design and participated in writing the manuscript.

Competing interests

All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/disclosure-of-interest/ and declare: TK is supported by JSPS KAKENHI Grant Number 21K10372; no other relationships or activities that could appear to have influenced the submitted work exist.

Acknowledgements

This work was supported by JSPS KAKENHI Grant Number 21K10372. We would like to thank Editage (www.editage.com) for English language editing.

Role of the funding source

The JSPS KAKENHI Grant Number 21K10372 covered the English editing service expenses.

Transparency declaration

The lead author affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study have been explained.

APPENDIX

Appendix 1

Thematic analysis using ChatGPT

SUPPLEMENTARY INFORMATION

Supplemental Table

Assessors’ comments

Abbreviations

LLMs

large language models