Generative AI in Otolaryngology Residency Personal Statement Writing: A Mixed‐Methods Analysis

ABSTRACT

Objective

Generative Artificial Intelligence (GAI) interfaces have rapidly integrated into various societal domains. Widespread accessibility of GAI for drafting personal statements poses challenges for evaluators to gauge writing ability and personal insight. This study aims to compare the quality of GAI‐generated personal statements to those written by successful applicants in OHNS residency programs, via integration of statistical and qualitative thematic analyses.

Methods

Personal statements were collected from successful OHNS residency applicants. Characteristic extraction from submitted statements was used to generate GAI‐written personal statements using ChatGPT 4.0. All statements were blindly reviewed by 21 experienced evaluators on a 10‐point Likert scale of authenticity, readability, personability, and overall quality. Thematic analysis of qualitative reviewer comments was conducted to extract deeper insights into evaluators' perceptions. Quantitative results were compared using independent t‐tests, while thematic coding was performed inductively using NVivo software.

Results

GAI‐generated personal statements significantly outperformed applicant‐written statements in all assessed domains, including authenticity (7.67 vs. 7.05, p = 0.002), readability (8.03 vs. 7.49, p = 0.002), personability (7.33 vs. 6.72, p = 0.004), and overall score (7.49 vs. 6.90, p = 0.005). Thematic analysis revealed that GAI statements were seen as “well‐constructed but generic,” while applicant statements were often “verbose and lacked focus.” Additionally, reviewers noted concerns regarding personal insight and engagement in AI‐generated statements.

Conclusion

GAI‐generated personal statements were rated more favorably across all domains, raising critical questions about the future of personal statements in the residency application process. While AI in medical education continues to evolve, clear guidelines on its ethical use in residency applications are essential.

Level of Evidence

N/A

This study compares personal statements generated by ChatGPT 4.0 with those written by applicants to otolaryngology residency programs. ChatGPT‐generated statements scored higher in authenticity, readability, personability, and overall assessment by reviewers. The findings highlight the potential impact of generative AI on residency applications and emphasize the need for clear guidelines to ensure ethical and equitable use of AI tools in academic settings.

1. Introduction

Since introduction to public use in 2022, generative artificial intelligence (GAI) has demonstrated a rapidly evolving role across many aspects of society [1, 2, 3]. Increased availability of graphic user interfaces, such as ChatGPT (OpenAI), has transformed approaches to qualitative assessment processes, including personal statement composition for surgical residency applications [4]. The otolaryngology–head and neck surgery (OHNS) residency application process is highly competitive, incorporating both quantitative and qualitative metrics. The personal statement traditionally functions as a key tool for demonstrating creativity, originality, academic rigor, interest, and passion for otolaryngology [5]. However, with the rise of GAI, evaluators face new challenges in assessing genuine applicant insight and writing proficiency [6].

Prior research has investigated the utility of GAI in medical and educational contexts. In a similar design to the current study, Patel et al. [7] demonstrated that ChatGPT‐generated personal statements for plastic surgery residency applications were indistinguishable from those written by applicants when comparing readability, originality, authenticity, and overall quality. Similarly, a recent review of the role of artificial intelligence within clinical physiatry practice concluded that “future conversations will be needed within [Physiatry], and more broadly within medical education, regarding innovation and the appropriate use of AI” [8]. However, when specifically examining residency application, no study to date has performed a blind review of dual‐sourced statements, with qualitative analysis of assessor comments.

The current study employs a mixed‐methods approach, integrating both quantitative scoring and qualitative thematic analysis, to comprehensively assess personal statements for application to residency in OHNS drafted by GAI against statements written by applicants who already successfully matched to an OHNS program through the Canadian Residency Matching Service (CaRMS). We hypothesized that there would be no statistical difference in objective ratings but anticipated unique insights from qualitative reviewer comments.

2. Materials and Methods

2.1. Study Design

This study follows a mixed‐methods design, incorporating both quantitative analysis of Likert‐scale reviewer ratings and qualitative thematic analysis of free‐text reviewer comments. Thematic analysis was conducted using an inductive approach, following Braun and Clarke's six‐step thematic analysis methodology, and analyzed using NVivo 14 (2023) software [9, 10, 11].

2.2. Ethics

Study approval was obtained as a Human Participant Ethics Protocol from the University of Toronto Health Sciences Research and Ethics Board (Human Protocol Number: 46701). Participant anonymity was maintained through a two‐stage anonymization process.

2.3. Participant Selection and Primary Anonymization

Participants were recruited in two stages. “Applicant” participants were recruited from a cross‐section of successful applicants to any OHNS residency program in Ontario, Canada, from 2017 to 2022 (prior to the broad public availability of GAI interfaces). Only personal statements that met the 1000‐word criteria for length and recommended content were accepted, and applicants were asked to anonymize their submissions to within their comfort level (Table 1). “Rater” participants were then recruited from an e‐mail directory of global otolaryngology program directors, academic faculty, senior residents, clinical fellows, and surgeon‐scientist trainees with inclusion criteria of at least one prior year of involvement in a residency and/or specialist registrar selection process, including as an interviewer, committee member, program director, and/or file reviewer.

TABLE 1.

Sample anonymization of applicant characteristics following dual anonymization process, encompassing (A) writer anonymization to their relevant comfort level and (B) post‐submission anonymization by the authorship team.

Original statement content	Stage 1 (applicant) anonymized version	Stage 2 (study) anonymized version
“I was employed as an assistant team equipment manager for the Junior AAA [City] Vikings hockey team, working closely with 10–12‐year‐old athletes.”	“I was employed as an assistant team equipment manager for a Junior AAA hockey team, working closely with adolescent athletes.”	“I was employed as an assistant manager for a highly competitive lacrosse team within my home city, working closely with adolescent athletes.”
“I then accepted a role as a Kindergarten reading instructor volunteer at [Name] Heights Public Elementary School in [City, Province].”	“I then accepted a role as a Kindergarten reading instructor volunteer at a local elementary school in Orillia, Ontario.”	“I then accepted a role as a junior teaching assistant volunteer at a local elementary school in my hometown.”
“My interest in Otolaryngology was drawn from my Uncle [Name] who was diagnosed with laryngeal carcinoma in 2003. It forever changed his life.”	“My interest in Otolaryngology was drawn from my Uncle who was diagnosed with laryngeal carcinoma in his early life. It forever changed him.”	“My interest in Otolaryngology was drawn from a close relative who was diagnosed with head and neck cancer in his early life. It forever changed him.”
“I received the gold medal in my mechanical engineering undergraduate program at [University Name] University, which taught me the spatial reasoning and problem solving required to succeed in Otolaryngology—Head and Neck Surgery.”	“I received top honors in my engineering undergraduate program at University, which taught me the spatial reasoning and problem solving required to succeed in Otolaryngology—Head and Neck Surgery.”	“I received top honors in my complex physics‐heavy University program, which taught me the spatial reasoning and problem solving required to succeed in Otolaryngology—Head and Neck Surgery.”

2.4. Secondary Anonymization

Where required, identifying information was further removed by one of two study authors with care to not disrupt the authenticity of the underlying statement. All edits were reviewed by two authors and were considered “dual anonymized” (once by the applicant and once by the reviewers) (Table 1).

2.5. Characteristic Extraction

To create GAI‐generated statements, an unmatched case–control design was used. From each applicant personal statement, three characteristics pertaining to the applicant (termed “characteristics”) were extracted and submitted to a common pool of all extracted characteristics. From this common pool, each characteristic was translated to an equivalent experience for each other applicant (i.e., “competitive hockey” referenced by one applicant was translated to “competitive lacrosse” for the characteristic pool; “I then was employed as a speech‐language pathologist” was translated to “I then was employed as an audiologist”). Samples of extracted characteristics are outlined in Table 2.

TABLE 2.

Sample extracted characteristics, used to form unmatched case samples in the GAI‐generated personal statement group.

Sample statement content	Extracted characteristic	Translated characteristic
“While employed as a pediatric audiologist…”	Employed as Pediatric Audiologist	Employed as Head and Neck Oncology Clinic Speech and Language Pathologist
“I was elected captain of my junior AAA hockey team.”	Captain of Competitive Hockey Team	Assistant Captain of Competitive Lacrosse Team
“My favorite poem is The Raven by Edgar Allan Poe.”	Favorite Poem is The Raven	Favorite Novel is Catcher in the Rye
“Then my grandfather was diagnosed with oral cavity cancer.”	Grandfather Diagnosed with Oral Cavity Cancer	Grandmother Diagnosed with Thyroid Cancer
“I received a Gold Medal in my third‐year ecobiology class”	Gold Medal in Third‐Year Ecobiology Class	Top Score in Fourth Year Organic Chemistry Class

2.6. Prompt and Large Language Model Strategy

To systematically generate GAI‐drafted personal statements, ChatGPT 4.0 (OpenAI) was employed, utilizing a standardized prompt structure to ensure consistency across all generated statements. A zero‐shot prompting approach was chosen to maximize the model's adaptability without introducing training biases from prior examples. The prompt explicitly outlined the necessary structural components of a competitive otolaryngology residency personal statement while incorporating extracted applicant characteristics to simulate a personalized writing style. Full prompt input is outlined in Table 3. A zero‐shot prompting approach was used in this study to assess the model's capacity to independently generate structured and high‐quality personal statements without iterative refinements. This strategy ensured that the output was not influenced by predefined templates or existing human‐written examples, allowing an unbiased comparison between GAI‐generated and applicant‐written statements. The temperature parameter was set to 0.7 to balance creativity and coherence. A lower temperature (e.g., 0.3–0.5) would have prioritized deterministic responses, potentially limiting the variation and adaptability required for a compelling personal statement. Conversely, a higher temperature (e.g., > 0.8) could have introduced excessive randomness, leading to statements with incoherent and/or exaggerated narratives. The selected value of 0.7 facilitated the generation of logically structured yet engaging content while mitigating the risk of factual inaccuracies and artificial “hallucinations.”

TABLE 3.

Prompt table outlining input criteria for ChatGPT generation of personal statements.

Context	You are an advanced language model trained to generate high‐quality personal statements for medical residency applications. Your task is to compose a compelling and authentic personal statement for an applicant applying to an Otolaryngology—Head and Neck Surgery (OHNS) residency program.
Purpose statement	The personal statement should reflect the applicant's journey, motivation, and suitability for a career in OHNS.
Structure	Follow a logical flow, typically: Introduction: Personal motivation and unique background; Clinical and Research Experiences: Relevant medical training, research, or clinical exposure; Passion for OHNS: Specific encounters or influences leading to interest in the field; Conclusion: Reflection on career aspirations and fit within the specialty.
Tone and style	The writing should be engaging, articulate, and professional while maintaining a personal and human‐like touch. Avoid cliches and generic phrases; ensure specificity and depth in personal reflections. Showcase authenticity and personal insight, avoiding exaggerated claims or unrealistic experiences.
Applicant‐specific details	Incorporate the following applicant characteristics, ensuring they are seamlessly integrated into the narrative: (Extracted characteristic 1); (Extracted characteristic 2); (Extracted characteristic 3).
Ethical compliance	Ensure all content adheres to ethical guidelines, avoiding fabrications or misleading information.
Word limit	Maintain a length between 750 and 1000 words, similar to standard residency application requirements.

2.7. GAI Personal Statement Generation

Once applicant‐drafted personal statements were collected and anonymized, and characteristic extraction performed, an equivalent number of GAI‐drafted personal statements were generated using ChatGPT 4.0 (OpenAI). Upon completion, seven statements from each source were included, for a total of 14 personal statements. The generation prompt (Table 3) outlined the necessary requirements for inclusion per the CaRMS website, components of a strong personal statement (per common resources for personal statement writing), and three translated characteristics from the common bank. GAI personal statements were then reviewed by two authors for egregious inaccuracies, which were corrected using ChatGPT4.0. Final applicant statements were not manually edited by the authorship.

2.8. Rating Collection

Applicant and GAI‐generated personal statements were anonymized and randomly numbered from 1 to 14 in the same digital folder, which was distributed to reviewers who were blinded to the original source of each statement and the study's intent.

Reviewers were instructed to rate personal statements on a 10‐point Likert scale on four domains, with one representing a “low” or “disagree” rating and 10 representing a “high” or “agree” rating. The four domains were: Authenticity (defined as “providing strong insight into the applicant”), personability (defined as “feeling connected, and the ability to personally relate to the applicant”), readability (defined as “having logical flow and few errors”), and an overall score (defined as the rater's “willingness to offer the applicant an interview to the residency program”). Space was provided for open‐ended comments alongside each domain, which were collated for thematic analysis.

2.9. Quantitative Analysis

For ranking comparison, statistical analyses were performed using SPSS. Independent T‐test for equality of means was used to compare aggregate data and reported as mean value (out of 10) alongside the 95% confidence interval of the difference in means (out of 10) for each reviewed domain. Two‐sided p‐values were reported, with equal variances not assumed.

2.10. Qualitative and Thematic Analysis

The qualitative component of this study employed thematic analysis following the methodological framework outlined by Braun and Clarke to identify patterns and themes in evaluator feedback regarding GAI‐generated and applicant‐written personal statements [9, 11]. This analytical approach was chosen for its flexibility and rigor in systematically identifying and interpreting patterns within textual data. All open‐ended comments from reviewers were collated into a common dataset. The research team engaged in repeated reading of the comments to ensure immersion in the data and an understanding of its depth and breadth. Initial reflections and observations were noted to guide subsequent coding.

A systematic coding process was undertaken using NVivo software (2023). Individual units of meaning, including key phrases and sentences reflecting evaluators' perceptions of statement quality, authenticity, readability, and personability, were assigned descriptive codes. Codes were generated inductively, ensuring that data rather than preconceived categories guided the coding process. The research team examined relationships between codes and aggregated them into potential themes. Codes capturing similar conceptual elements were clustered together, forming overarching themes that encapsulated key evaluative perspectives on GAI‐ and applicant‐written statements. Candidate themes were iteratively reviewed against the dataset to ensure coherence and consistency. This process involved verifying themes accurately represented the coded data and refining theme boundaries where necessary. Thematic maps were constructed to visualize relationships among themes. Each theme was clearly defined to articulate its central concept and relevance to the study's research objectives. Sub‐themes were identified where applicable to capture nuanced variations within broader thematic categories. A structured narrative was developed, illustrating the identified themes with representative verbatim excerpts from reviewer comments. The final thematic framework provided insights into how evaluators perceived and differentiated between GAI‐generated and applicant‐written personal statements. Positive and negative valences were assigned upon agreement of two authors, according to preset template criteria (Table 4).

TABLE 4.

Sample phrases assigned positive and negative valence values for each coded domain.

Theme	Sample phrases
	Positive valence	Negative valence
Writing quality	“Well‐structured and flows naturally, making it easy to read and follow.”	“Feels formulaic and lacks originality, as if following a generic template.”
Insight	“Provides a compelling and reflective account of personal growth and motivation.”	“Lacks depth in personal reflection and does not provide meaningful insight into the applicant.”
Engagement	“Creates a strong connection with the reader, making the applicant feel relatable and personable.”	“Feels detached and impersonal, making it difficult to connect with the applicant.”
Specificity and focus	“Clearly articulates motivations for otolaryngology with specific, relevant experiences.”	“Overly vague and does not provide concrete examples to support claims.”
Structure and flow	“Ideas progress logically, with smooth transitions between sections.”	“Choppy and disjointed, making it difficult to follow the narrative.”
Language and grammar	“Polished and professionally written with strong command of language.”	“Contains awkward phrasing and minor grammatical errors that detract from readability.”

3. Results

3.1. Data Collection

Seven applicant‐written personal statements were collected and seven were created using GAI, for a total of 14 personal statements. Twenty‐one reviewers consented, met inclusion criteria, and completed study participation for a total of 294 reviews. No reviewer attrition was observed. Sixty applicant statement comments and 59 GAI statement comments were included.

3.2. Quantitative

Personal statements generated by ChatGPT 4.0 significantly outscored applicant‐written personal statements in all evaluated categories (Figures 1 and 2): Authenticity (7.67 vs. 7.05, p = 0.002, 95% CI: 0.229–1.017), Readability (8.03 vs. 7.49, p = 0.002, 95% CI: 0.208–0.881), Personability (7.33 vs. 6.72, p = 0.004, 95% CI: 0.197–1.026), and Overall Score (7.49 vs. 6.90, p = 0.005, 95% CI: 0.182–0.998).

FIGURE 1.

Summary graphical comparison of findings between domains. Significantly higher scores were observed in all assessed domains (* denotes significance). [Color figure can be viewed in the online issue, which is available at www.laryngoscope.com.]

FIGURE 2.

Graphical depiction of rating frequencies compared between GAI and applicant‐written statement, as determined to be significantly higher in authenticity (A), personability (B), readability (C), or overall score (D). [Color figure can be viewed in the online issue, which is available at www.laryngoscope.com.]

3.3. Thematic Analysis

Thematic analysis of coded comments revealed differences in how applicant‐ and GAI‐generated personal statements were perceived across six domains: Writing Quality, Insight, Engagement, Specificity and Focus, Structure and Flow, and Language and Grammar (Table 5). Writing Quality received the highest number of positive comments for GAI‐generated statements (36%), with reviewers frequently describing them as well‐written and coherent. In contrast, applicant statements received fewer positive comments (24%) and more frequent criticism for verbosity (18%). In the Insight domain, both applicant‐ and GAI‐generated statements received similar proportions of positive comments (25%), though applicant statements were more often criticized for superficial descriptions of achievements (30%) compared to GAI statements (20%). Engagement was the most positively rated domain overall, with both applicant (33%) and GAI (27%) statements praised for being authentic and interesting to read. For Specificity and Focus, applicant statements received more negative comments (41%), often for lacking breadth or concrete examples. GAI statements were also criticized for vagueness (29%) but were positively noted for their focused interest in otolaryngology (12%). In Structure and Flow, both applicant and GAI statements received similar positive feedback (18%). However, GAI statements were more frequently described as disjointed or choppy (29%) compared to applicant statements (35%). In Language and Grammar, comments were evenly distributed across both groups, with similar proportions of positive and negative feedback.

TABLE 5.

Qualitative thematic analysis results for coded occurrence counts for each statement source.

Theme	Applicant‐generated count (% total)		GAI‐generated count (% total)
	Positive	Negative	Positive	Negative
Writing quality	8 (24%)	6 (18%)	12 (36%)	7 (21%)
Insight	5 (25%)	6 (30%)	5 (25%)	4 (20%)
Engagement	5 (33%)	2 (13%)	4 (27%)	4 (27%)
Specificity and focus	3 (18%)	7 (41%)	2 (12%)	5 (29%)
Structure and flow	3 (18%)	6 (35%)	3 (18%)	5 (29%)
Language and grammar	3 (33%)	2 (22%)	2 (22%)	2 (22%)

4. Discussion

In the present study, personal statements generated by ChatGPT 4.0 were ubiquitously rated higher by experienced reviewers than those written by successful applicants to Canadian OHNS residency programs. Thematic analysis revealed that the writing quality of GAI‐generated statements was observed to be more rigid, whereas those written by applicants were verbose or overly lengthy. However, thematic analysis of other domains was similar across groups. These findings suggest that GAI may be capable of producing content that exceeds standards traditionally expected from applicants in the otolaryngology residency selection process.

ChatGPT‐generated personal statements for admission to American plastic surgery residency programs have been found to be indistinguishable from those written by human applicants across similar domains of assessment to the current study [7]. This trend, now demonstrated in plastic surgery and OHNS, reinforces the potential of GAI to disrupt current residency application evaluation processes where personal statements have historically represented a critical component of applicant assessment [12]. There is ongoing debate surrounding the academic integrity of using AI‐generated content throughout its potential uses in academia [13, 14, 15, 16, 17, 18]. Previous reviews on policy creation regarding the use of AI in residency applications have concluded that as GAI becomes more prevalent, residency programs must develop transparent guidelines regarding how AI‐generated content will be accepted [8, 18]. Without clear policy and high‐fidelity detection strategies, there is a high risk of inconsistency and inherent bias in applicant evaluation, as programs may vary widely in their acceptance and assessment of AI‐generated content.

Success of GAI‐generated written samples raise concerns about the authenticity and individuality of submissions. One main critique of AI‐generated content is a tendency to produce “artificial hallucinations,” which are superficially plausible yet factually fabricated content created to appease the user command [19, 20]. Although the current protocol did not assess the factual accuracy of all generated personal statements, thematic analysis revealed that GAI‐generated statements were routinely criticized for being “generic” and “lacking in unique personal insights.” During the GAI drafting phase, one GAI‐generated statement outlined the writer's extensive experience as a chief resident, which is contradictory for an applicant desiring entry into a residency program. This aligns with numerous previous concerns that while AI‐generated content can superficially appear coherent and well‐structured, it may include misrepresentations of the topic [3, 21].

Despite associated ethical concerns, an overall neutral‐to‐positive comparative thematic assessment of GAI‐generated statements suggests that artificial intelligence‐based tools may be harnessed to enhance the application process. Specifically, residency applicants may employ GAI assistance as an adjunct to refine previously drafted statements, ensuring the submitted product is well‐organized while reflecting the applicant's authentic experience and voice [22, 23, 24, 25]. However, this will require careful oversight to prevent over‐reliance on GAI and ensure final submissions are an accurate reflection of the applicant. One previous critique has argued that residency programs should discourage the use of GAI to the same degree as paid editing services, ensuring a level playing field for all applicants regardless of socioeconomic background [26]. This perspective is imperative in light of recent efforts endeavoring to eliminate historic discrimination and bias in the surgical residency selection process [27, 28, 29]. Conversely, GAI may function to level the playing field of historic inequities in writing, and the availability of GAI as a writing aid may function to reduce reliance upon paid services.

The traditional personal statement may become less important as a differentiating factor in the evaluation of residency applications. Residency programs may consider alternative methods for assessing applicant traits that have traditionally been evaluated [8]. This may include an increased importance placed on supervised interviews or situational judgment tests: forms of character assessment less susceptible to AI manipulation and more likely to aid in holistic applicant assessment. Alternatively, programs may require applicants to sign attestation statements about the authenticity of their content [7]. However, rapid advancement in GAI technology has been shown to likely outpace detection software, resulting in challenging enforcement.

This study has several limitations. Personal statements were exclusively generated using ChatGPT 4.0, which may limit generalizability to other GAI models or future versions. The study did not assess downstream outcomes such as residency match success or clinical performance. Although evaluators were blinded to statement source and study intent, the high‐profile nature of GAI may have influenced their perceptions. Additionally, qualitative research is inherently subjective, and thematic analysis relies on the interpretation of coders, which introduces the potential for bias despite efforts to standardize coding. The study did not formally assess intercoder reliability, which may limit reproducibility of the findings. Evaluator perceptions may also have been shaped by their own backgrounds, experiences, or preconceptions about AI‐generated content. Moreover, the study did not explore how applicant demographic characteristics—such as gender, race, or ethnicity—may interact with AI content to influence evaluations, a known source of bias in residency selection [5]. It is possible that similar biases may have affected evaluations in the current study, particularly if the evaluators held preconceived notions about the quality or authenticity of AI‐generated content. Future research with a more diverse pool of evaluators, applicants, and GAI models is needed to explore how AI may affect equity, transparency, and bias in the residency application process.

4.1. Reflexivity Statement

Our research team acknowledges that our roles in residency selection may shape our perspectives on GAI in personal statement evaluation. To reduce bias, we used anonymization, blinded evaluations, and inductive thematic analysis. While our findings highlight the strengths of AI‐generated statements, we remain mindful of concerns regarding authenticity and fairness. By maintaining reflexivity, we aim to offer balanced insights that inform responsible AI use in residency applications and ethical discussions.

5. Conclusion

Personal statements generated by ChatGPT 4.0 outperformed applicant‐written statements in authenticity, readability, personability, and overall score. These findings raise important questions about the role of personal statements in the residency application and the broader use of GAI in academia. Clear, ethical guidelines are needed as GAI evolves, and further research is essential.

Conflicts of Interest

The authors declare no conflicts of interest.

Wihlidal J. G. J., Wolter N. E., Propst E. J., et al., “Generative AI in Otolaryngology Residency Personal Statement Writing: A Mixed‐Methods Analysis,” The Laryngoscope 135, no. 10 (2025): 3570–3578, 10.1002/lary.32188.