Automating inductive thematic analyses of health content using large language models: a proof-of-concept study using social media data

Abstract

Objectives

Large language models (LLMs) face challenges in inductive thematic analysis, a task requiring deep interpretive, domain-specific expertise. We evaluated the feasibility of using LLMs to replicate expert-driven thematic analysis of social media data.

Materials and Methods

Using 2 temporally nonintersecting Reddit datasets on xylazine (n = 286 and 686, for model optimization and validation, respectively) with 12 expert-derived themes, we evaluated 5 LLMs against expert coding. We modeled the task as a series of binary classifications, rather than a single, multilabel classification, employing zero-, single-, and few-shot prompting strategies and measuring performance via accuracy, precision, recall, and F₁ score.

Results

On the validation set, GPT-4o with 2-shot prompting performed best (accuracy: 90.9%; F₁ score: 0.71). For high-prevalence themes, model-derived thematic distributions closely mirrored expert classifications (eg, xylazine: 13.6% vs 17.8%; medications for opioid use disorders: 16.5% vs 17.8%).

Conclusion

Our findings suggest that few-shot LLM-based approaches can automate thematic analyses, offering a scalable supplement for qualitative research.

Background

The emergence of generative large language models (LLMs) has spurred the development of natural language processing applications in medicine.^1,2 Current LLMs demonstrate strong performance in qualitative tasks, such as synthesis and summarization, when applied to domain-specific texts within medicine and other fields.^3–6 These capabilities suggest that LLMs may help scale qualitative research methodologies, which are traditionally constrained by the practical limitations of manual, resource-intensive analysis. However, it remains methodologically challenging to deploy LLMs for specific qualitative tasks, such as inductive thematic analysis. Thematic analysis is a method for identifying and interpreting patterns of themes within data. It is typically an inductive, bottom-up process, where themes emerge directly from the data through manual analysis and interpretation.⁷ Thematic analyses are often performed by experts with specialized knowledge of target domains, resulting in characterization of data through subjective, domain-specific interpretations. Automation of thematic analysis remains a challenge due to the need to incorporate subjective judgment, context, and domain-specific knowledge. Additional limitations of LLMs for performing unsupervised thematic analyses include the tendency to perpetuate or amplify societal biases present in their training data, inability to capture nuances of unseen data concepts, and generation of variable results on repeat runs with identical inputs.^8–10 The latter is particularly critical for modeling thematic analysis experiments with LLMs, as large datasets, such as thousands of posts from social media corpora, cannot be processed within single context windows, and themes generated from distinct subsets of data may lack consistency.

We aim to explore, develop, and validate LLM-based strategies for replicating expert-driven thematic analyses. Our data source, Reddit, is a popular social network that facilitates candid, anonymous discussions on sensitive topics, including substance use.^11,12 We specifically focus on the substance xylazine, which is an emerging concern in the ongoing opioid crisis in the United States.¹³ Xylazine, an α-2 adrenergic agonist used as a sedative in veterinary care for various animal species, has increasingly been found in the illicit drug supply, rapidly expanding since 2019,¹⁴ and has been associated with necrotizing skin wounds and lesions.¹⁵ We detail our approach to conducting and evaluating the effectiveness of LLMs for thematic analysis of xylazine-related Reddit data via comparison against an expert-curated thematic categorization.

Materials and methods

Data collection and preparation

We collected data from Reddit via the Python Reddit API Wrapper (PRAW), which provides access to publicly available posts from chosen subreddits. To collect xylazine wound-relevant data, we employed keywords related to xylazine, as well as wound-, ulcer-, and necrosis-related terms, including common variants (eg, tranq) and misspellings. We divided the data into 2 sets based on timestamps: 2014-2023 (5376 posts; 961 subreddits), and 2024-March 2025 (13 928 posts; 4272 subreddits). For manual thematic categorization, we drew random samples from each set: 286 from set 1 (DS1) and 236 from set 2 (DS2), illustrated in Figure 1 (we initially sampled 300 and 250 posts from the 2 sets, respectively, based on our assessment of feasibility for manual analysis, and then removed posts that were duplicative, blank, or poorly formatted posts). We also retained the full filtered set 2 dataset (DS2F, 686 posts) for thematic distribution analysis. The temporal separation between datasets was intentional and used to assess generalizations across different periods, specifically because we anticipated evolution in discussions mirroring that of the opioid crisis.

Figure 1.

Reddit data extraction and thematic analyses workflow.

Thematic analysis and manual classification

Medical toxicology-trained authors initially analyzed a random subset (n = 15) of posts from DS1 to identify relevant themes based on individual post content (Previously conducted work is detailed in Spadaro et al.¹⁶). Following theme identification and consolidation by domain experts, the larger samples from DS1 (n = 286) and DS2 (n = 236) were manually annotated. Twelve xylazine-relevant themes were identified as representative of the social media chatter dataset, DS1 (full details of themes are available in Appendix A2). Of the classified posts that included xylazine wound-relevant content, G: Pathophysiology of xylazine, C: Location of wounds, and L: Geography and locale were the most prominent themes, found in 29.4%, 19.6%, and 12.9% of posts, respectively.¹⁶

Automated thematic analysis

Dataset 1 (DS1): Prompt development and model comparison

Preliminary testing on DS1 revealed that LLMs performed better when the task was modeled as multiple binary classification tasks (eg, for a given theme, 1 = relevant, 0 = not relevant) rather than a single multilabel classification across all 13 categories. This finding guided our adoption of the multiple binary classification framework for all subsequent experiments. We used the same manually coded subset (n = 15) for iterative fine-tuning. Specifically, we adjusted prompt phrasing, task instructions, and exemplar formatting in multiple rounds of testing to align model predictions more closely with expert-coded labels and maximize F₁ score (full prompts are provided in Appendix A1). Initial zero-shot experiments revealed signs of overfitting, characterized by inflated classification distributions relative to manual coding. To address this, we implemented single-shot and multishot prompting strategies, providing the model with representative examples of each theme.

Following prompt optimization, we scaled our evaluation to the full DS1 (n = 286) using 5 LLMs: GPT-3.5-turbo, GPT-4o, DeepSeekV3 (deepseek-v3-0324-ud-q2_k_xl), Gemma3 (gemma-3-27b-it), and Llama3 (llama3-70b). These open- and closed-source models were chosen based on their reported success in recent literature (model outlines are provided in Appendix A4).^17–21 Each model was evaluated using identical prompt structures and few-shot examples, based on precision, recall, and F₁ scores.

Dataset 2 (DS2): Validation

We deployed the 2 best-performing model-prompt combinations to evaluate scalability and generalizability on DS2 (n = 236). Finally, we applied the best-performing model–prompt combination to the complete DS2 filtered (DS2F) dataset (n = 686). This analysis was conducted to evaluate the model’s output on the larger collection of unclassified posts and to assess whether the thematic distribution patterns observed in the validation subset were consistent across the dataset.

Results

Prompt development and model comparison with DS1

Table 1 summarizes the performance metrics for all model–prompt combinations evaluated on DS1. Of the zero-shot approaches, DS1_0Shot_deepseekV3 achieved the highest accuracy at 89.9%. However, its mean precision (0.611; SD = 0.022) and mean recall (0.599; SD = 0.022) were modest, resulting in F₁ score of 0.605 (SD = 0.018). The other 4 models achieved similar results, with the closest being DS1_0Shot_gpt-4o (accuracy: 89.1%; F₁ score: 0.567, 95% CI: [0.528-0.604]). Misclassification patterns are depicted as a confusion matrix in Appendix A3.

Table 1.

DS1 model evaluations (recall, precision, F₁ score, and accuracy).

Model	Precision	Recall	F1 score (95% CI)	Accuracy (95% CI)	Avg rank
DS1_2shot_deepseekV3	0.614	0.612	0.613 (0.576-0.648)	0.9 (0.891-0.910)	1.625
DS1_2shot_gpt-4o	0.626	0.566	0.594 (0.555-0.631)	0.9 (0.890-0.910)	2.375
DS1_0shot_deepseekV3	0.611	0.599	0.605 (0.570-0.640)	0.899 (0.889-0.909)	2.75
DS1_1shot gpt-4o	0.619	0.559	0.588 (0.549-0.625)	0.899 (0.889-0.909)	3.75
DS1_0shot_gpt-4o	0.584	0.551	0.567 (0.528-0.604)	0.892 (0.881-0.902)	5.25
DS1_0shot llama3	0.574	0.478	0.522 (0.484-0.561)	0.887 (0.877-0.897)	7.25
DS1_lshot_gemma3	0.522	0.53	0.526 (0.491-0.562)]	0.877 (0.866-0.888)	8.0
DS1_2shot_gemma3	0.515	0.568	0.54 (0.507-0.577)	0.875 (0.864-0.887)	8.0
DS1_1shot_llama3	0.527	0.455	0.488 (0.447-0.527)	0.877 (0.866-0.888)	9.0
DS1_1shot_gpt-3.5-turbo	0.524	0.441	0.478 (0.437-0.516)	0.876 (0.866-0.886)	10.25
DS1_0shot_gpt-3.5-turbo	0.518	0.47	0.493 (0.452-0.530)	0.875 (0.865-0.885)	10.75
DS1_2shot_gpt-3.5-turbo	0.524	0.395	0.45 (0.408-0.492)	0.876 (0.866-0.886)	11.25
DS1_0shot_gemma3	0.469	0.524	0.495 (0.459-0.531)	0.862 (0.851-0.873)	11.25
DS1_2shot_llama3	0511	039	0.443 (0.401-0.481)	0.873 (0.863-0.884)	13.5
DS1_1shot_deepseekV3	0.354	0.315	0.333 (0.296-0.372)	0.838 (0.825-0.850)	15.0

Avg rank represents the average ranking among all models across tasks.

The most prominent themes identified in the DS1_0Shot_deepseekV3 automation combination were G: Pathophysiology of xylazine (28.7%, 82 posts), L: Geography and locale (18.5%, 53), and B: Other drug use habits (16.8%, 48). These results align well with the top relevant categories classified by subject matter experts; the LLM gold standard classes share 2 of their 3 most commonly coded themes, G and L.

In both the single- and multishot approaches, the GPT-4o models, DS1_1Shot_gpt-4o and DS1_2Shot_gpt-4o, were identified as the best performing, by accuracy. The single-shot approach achieved accuracy: 89.9%, 95% CI: [88.9-90.9]; F₁ score: 0.588, 95% CI: [0.549-0.625]. The multishot approach achieved accuracy of 90.0%, 95% CI: [89.0-91.0]; F₁ score: 0.594, 95% CI: [0.555-0.631].

The complete theme distribution is available in Figure 2. The top relevant themes identified by DS1_1Shot_gpt-4o were G (27.6%, 79 posts), B (21%, 60), and L (18.5%, 53). The top relevant themes identified by DS1_2Shot_gpt-4o were also G (31.1%, 89 posts), B (19.9%, 57), and L (18.2%, 52). Similar to zero-shot approaches, these combinations shared 2 of the 3 most prevalent manually identified themes (G and L) of the gold standard subset. The DS1_2Shot_deepseekV3 model–prompt combination was a close second to DS1_2Shot_gpt-4o, with the deepseekV3 combination achieving an accuracy of 90.0%, 95% CI: [89.1-91.0]; F₁ score: 0.613, 95% CI: [0.576-0.648].

Figure 2.

DS1 top-performing models’ theme distribution compared to the gold standard (out of 286 posts).

Validation on DS2

The 2 top-performing model–prompt combinations (DS1_2Shot_deepseekV3 and DS1_2Shot_gpt-4o) were used to perform the automatic thematic analysis of DS2. A DS2 subset of 236 posts was manually classified as the gold standard, revealing the top themes as B: Other drugs use habits (40.7%, 96 posts), H: Specific xylazine withdrawals (26.7%, 63 posts), and a tie between A: Xylazine use habits and I: Posts about MOUDs (17.8% each, 42 posts). GPT-4o outperformed DeepSeek-V3 across all metrics, achieving 90.9% accuracy (95% CI: [89.9-91.9]) vs 89.5% (95% CI: [88.3-90.6]), with superior F₁ score (0.708 vs 0.671).

The top themes identified by DS2_gpt-4o were B (53%, 125), I (16.5%, 39), L (14.8%, 35), and A (13.5%, 32); for DS2_deepseekV3, they were B (43.2%, 102), I (19%, 45), A (16.1%, 38), and L (12.3%, 29). Three of the 4 top themes identified by the models matched the gold standard, and both models showed stable results across runs, with low variability. Their ∼90% accuracy and balanced precision–recall indicate strong potential for scalable automated thematic analysis of social media content. Detailed metrics and theme distributions are presented in Table 2 and Figure 3, respectively.

Table 2.

DS2 model evaluations (recall, precision, F₁ score, and accuracy).

Model	Precision	Recall	F1 score (95% CI)	Accuracy
DS2_gpt-4o	0.76	0.663	0.708 (0.675-0.739)	0.909 (0.899-0.919)
DS2_deepseekV3	0.702	0.642	0.671 (0.635-0.704)	0.895 (0.883-0.906)

Avg rank represents the average ranking among all models across tasks.

Figure 3.

DS2 models’ distribution compared to the gold standard (out of 236 posts).

Validation on unlabeled data (DS2F)

The top-performing model, DS2_gpt-4o, was applied to the full filtered dataset, DS2F, of 686 posts. The distribution of this model’s classified output highlighted themes B (65.1%, 446 posts), I (23.4%, 160), L (20.4%, 140), and A (14.3%, 98) as the most relevant to the posts. The xylazine wound-related themes identified as least relevant to the DS2F posts were E (0.73%, 5), F (1.02%, 7), and H (7.45%, 51).

Discussion

Our findings demonstrate that, with appropriate task framing (eg, decomposing into multiple binary classification tasks rather than a single multi-label classification), systematic prompt engineering, and the inclusion of representative few-shot examples, LLMs can approximate human performance in thematic analysis. This suggests that thematic analyses can be effectively automated for large-scale data, reducing need for expert time commitment. GPT-4o outperformed DeepSeek-V3 across all evaluation metrics, in precision (0.76 vs 0.702), recall (0.663 vs 0.642), and F₁ scores (0.708 vs 0.671), demonstrating superiority of closed-source GPT models over open-source DeepSeek models. However, refined prompt engineering may improve performance of current open-source models, enabling them to close the relatively small performance gap. While using Application Programming Interface (API)-based models, such as GPT-4o, may be costly, they can still be leveraged to reduce expert-related annotation costs. Open-source models may be particularly appealing as they can be hosted locally, enabling the processing of large-scale data without a linear increase in cost while complying with institutional data security protocols.

Our results indicate that LLM-generated themes can track shifts in discourse over time. From DS1 to DS2, xylazine-related post volume increased, with focus shifting from wound care (DS1) to concerns about drug impurities, analogs, and fentanyl–xylazine combinations (DS2). Temporal analysis also validated our method, as many posts from 2012 to 2019 were correctly flagged as unrelated to xylazine wounds, aligning with epidemiological data showing xylazine’s emergence after 2019. This demonstrates the method’s ability to capture evolving discussions about substance use on social media. This capability has broader implications for drug surveillance, as it could enable rapid detection of emerging substance use patterns and identify new trends in established substances, such as nonmedical ketamine use or buprenorphine precipitated withdrawal. Such automated monitoring could provide public health officials with early warning systems for emerging drug threats by distinguishing meaningful shifts in substance use discourse from background chatter.

Our experiments revealed significant differences between zero-shot and few-shot approaches. Initial zero-shot prompts failed to effectively capture many themes, with specific categories, such as “Stigma related to xylazine wounds,” remaining completely undetected. Single-shot and multi-shot prompt configurations improved thematic detection for most models, with the final prompts producing classification distributions that closely aligned with manual coding patterns. This suggests that while LLMs are capable of identifying themes with reliable performance, inductive thematic analysis still requires human initiation and specification of themes to facilitate LLM-driven inference. In our experiments, we attempted to test the feasibility of full automation by injecting, via the prompts, contextual information surrounding each post, including post content and title. However, the expanded prompts reduced performance.

Limitations

Fully unsupervised thematic analysis remains a challenge for LLMs. Moderate precision and recall scores (60%-76%) indicate that automated classification, while promising, still produces false positives/negatives that may adversely impact downstream analyses. Our keyword-based filtering approach may have excluded relevant content that uses alternative terminology or discusses xylazine indirectly.

Future work

Future work should focus on improving the performance of low-shot methods for thematic analysis, improving LLM agreement with human experts through iterative feedback mechanisms, designing frameworks for conducting purely unsupervised categorization that can identify emergent themes without predetermined categories, and performance thresholds at which operational deployment in public health settings is feasible. The latter can be particularly impactful, as current unsupervised approaches, such as topic modeling, are not capable of performing expert-like thematic analyses.

Conclusion

Our study demonstrated that LLMs can be leveraged to conduct thematic analyses when guided by few-shot examples. Future work should explore strategies for improving LLM agreement with human experts and designing frameworks for conducting purely unsupervised categorization.

Author contributions

JaMor Hairston (Conceptualization, Data curation, Methodology, Software, Validation, Writing—original draft), Ritvik Ranjan (Validation, Writing—review & editing), Sahithi Lakamana (Data curation, Software, Validation, Writing—review & editing), Anthony Spadaro (Conceptualization, Methodology, Writing—review & editing), Selen Bozkurt (Supervision, Writing—review & editing), Jeanmarie Perrone (Conceptualization, Methodology, Supervision, Writing—review & editing), and Abeed Sarker (Conceptualization, Funding acquisition, Methodology, Supervision, Writing—original draft, Writing—review & editing)

Supplementary material

Supplementary material is available at JAMIA Open online.

Funding

Research reported in this publication was supported by National Institute on Drug Abuse (NIDA) of the National Institutes of Health (NIH) under award number R01DA057599 and R01DA057599-01S1. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Conflicts of interest

None declared.

Data availability

All data included in the study were publicly available at the time of collection. Specific Reddit posts can be collected via the described API. Thematically coded texts will be shared upon reasonable request to the corresponding author, subject to a data use agreement in accordance with Emory University policy.