Performance Evaluation of Popular Open-Source Large Language Models in Healthcare

Abstract

This paper evaluated user preferences and performance metrics for two widely used open-source large language models (LLMs), Llama 3.1 8B and Mistral 3 Small 24B (AWQ), compared to the proprietary model GPT-4o, in the context of serving as a user-oriented healthcare assistant. The study highlighted the advantages of open-source LLMs, including transparency, cost-effectiveness, and customization potential for specific applications. A dual approach was used: first, ten participants ranked model-generated responses to various healthcare questions; second, computational performance metrics like response time, throughput, and time-to-first-token were benchmarked under different user loads. Results indicated that the majority of participants preferred GPT-4o responses; however, both open-source LLMs had relatively similar ratings. Furthermore, the benchmarking results underscored the efficiency and reliability of the models under load, showcasing their capabilities for real-world applications. This research contributed to understanding how open-source LLMs could meet the needs of diverse users across different domains, encouraging further exploration and adoption in various industries.

1. Introduction

Large Language Models (LLMs) have proven to offer significant benefits in healthcare delivery and research. However, the current landscape is dominated by commercial proprietary LLMs, such as GPT4o or Gemini 2.5 Pro, which have significant implications related to privacy and transparency in healthcare settings. Consequently, there is a growing interest in open-weight non-proprietary LLMs. LLMs such as Llama and Mistral are trained on healthcare data making them attractive alternative options.

Despite a lack of state-of-the-art performance, open-weight LLMs offer significant benefits, such as transparency, customizability, cost-effectiveness, and privacy. Researchers and developers can freely download and experiment with open-weight LLMs; this transparency fosters trust and aids in identifying bias or errors. Additionally, organizations can fine-tune these models for specific needs, such as healthcare or finance, enhancing performance in niche areas, as seen with models like BioMistral tailored for biomedical applications [1]. Retrieval-augmented generation (RAG) techniques have also improved LLM performance in clinical settings, ensuring advancements are shared widely for faster development of robust models [2,3]. Open-weight LLMs have no scaling cost other than the cost of setup, reducing financial burdens for smaller enterprises and academic institutions [2], meanwhile having strong privacy guarantees by storing sensitive patient data locally. These advantages show great potential for open-weight LLMs to improve the efficiency and effectiveness of healthcare delivery such as AI-powered telehealth services.

This study examined two research questions: (1) Can open-weight model provide comparable responses to frontier proprietary LLMs in a patient-chatbot conversational setting? (2) Do different local models exhibit significant performance differences on computational benchmarks (response time, throughput, and time-to-first-token)?

2. Methods

We evaluated two open-weight large language models, Llama 3.1 8B Instruct and Mistral 3 Small 24B Instruct (AWQ), in terms of both response quality and computational performance. For conciseness, these two models will be referred to as Llama and Mistral in the following text. The study consisted of two main phases: a user preference assessment, where human evaluators ranked model-generated responses, and a systematic benchmarking of latency and throughput under varying workloads. For additional context, we included GPT-4o in the user study as a high-performance closed-source baseline. This combined approach—pairing human evaluations with systematic benchmarking—enabled us to assess both the practical usability and technical capabilities of these models in real-world applications.

Ten participants were recruited to review and rank responses from each of the 3 models. Participants were first assigned a healthcare domain, selected from Primary Care, Nutrition, Lifestyle, Cancer, and Pediatrics. They are then instructed to generate a set of questions pertaining to it, mimicking a patient-chatbot conversation. This step resulted in 53 total questions. For each question, we obtained responses from GPT-4o, Llama, and Mistral. The models’ outputs were presented in randomized order with the model names hidden, and evaluators assigned rankings (1st, 2nd, 3rd) based on perceived quality, coherence, and usefulness. These rankings were aggregated into an overall preference score, where a lower cumulative score indicated stronger performance (with 53 being the ideal minimum). This approach allowed us to assess whether users exhibited a clear preference for any model, and whether proprietary models exhibit significant quality improvements over open-weight models.

Next, to measure computational efficiency, we conducted a series of controlled tests assessing response time, throughput, and time-to-first-token (TTFT) under increasing concurrent user loads. We measured each statistic for each question in the 53-question corpus and calculated the average and standard deviation. Response time was defined as the total duration (in seconds) required for a full response to be generated, while throughput measured the average number of tokens produced per second. These two metrics measure processing efficiency. TTFT, which reflects perceived responsiveness, was recorded as the delay between input submission and the first token appearing in the output stream. We did not benchmark GPT-4o due to the interference from internet traffic and OpenAI’s load-balancing.

All experiments were run using vLLM, an optimized inference framework, to ensure consistent concurrency handling. We selected this framework due to its ability to maintain stable performance under multiple simultaneous requests, minimizing artificial bottlenecks. Both open-weight models were deployed on identical hardware (a single Nvidia RTX A6000) to eliminate variance from system-specific factors. Due to resource limitations, we cannot perform user preference annotation across a large variety of models. Hence, we limit ourselves to three models, representing a closed-source frontier LLM, a medium-sized open-weight LLM, and a small open-weight LLM. GPT-4o is used as a baseline due to it being a frontier general-purpose LLM that is both performant and cost-efficient. We selected Llama and Mistral as they are established models representing distinct model size categories. We used the AWQ quantized version of Mistral 3 Small 24B Instruct to fit into our hardware. These two models provide insights into whether a smaller model produces comparable results to a larger quantized model.

3. Results

Of the ten participants, eight (80%) were female, and two (20%) were male. The most common age group was 36–40 years (4, 40%), followed by 31–35 years (2, 20%), 41–45 years (2, 20%), and 22–30 years (2, 20%). Regarding experience of using AI tools such as large language models (LLMs), seven (70%) participants reported using AI tools for 2–3 years, while three (30%) had 1 year of experience. In terms of estimated usage of LLMs per week, six (60%) participants reported 0–10 hours, two (20%) reported 11–20 hours, and one (10%) reported 21–30 hours, Table 1.

Table 1.

Study participant characteristics

Characteristic	Frequency (n)	Percentage (%)
Gender
Female	8	80
Male	2	20
Occupation
Healthcare Professional (MD, RN)	4	40
General User	6	60
Age Group (years)
22–30	2	20
31–35	2	20
36–40	4	40
41–45	2	20
Years Using AI Tools
1 year	3	30
2–3 years	7	70
Estimated LLM Usage (hours/week)
0–10	6	60
11–20	2	20
21–30	1	10

The accumulated rankings from ten participants across 53 questions revealed nuanced differences in perceived response quality between the two open-weight models and GPT-4o (Figure 1, top left). GPT-4o achieved the lowest (best) aggregate score of 117, slightly outperforming Mistral (score: 129) and Llama (score: 132). These results show that frontier closed-source LLMs do not offer significant quality gains in a general patient-chatbot conversation setting.

Figure 1.

We measure the user preference score, time-to-first-token, response time, and throughput for different LLMs. A lower score is preferred for throughput (bottom-right), while a higher score is preferred for all other metrics. For all computational metrics (response time, throughput, TTFT), each bar displays the average and standard deviation across the 53 questions.

The evaluation of response times under varying concurrent user loads revealed clear differences between the two open-weight models (Figure 1, bottom left). Llama demonstrated consistently faster response times, averaging 1.29–1.43 seconds across all tested concurrency levels (1–20 users). In contrast, Mistral was significantly slower, with response times averaging 2.71 seconds, more than double that of Llama. However, both Llama and Mistral deliver near-identical TTFT performance (Figure 1, top right), averaging 0.04–0.05 seconds across all concurrency levels, with no statistically significant difference between models. This parity persists even under high concurrent loads (20+ users), demonstrating that neither model’s architecture introduces meaningful latency in initial token generation. Both models demonstrate remarkably similar throughput performance across all tested concurrency levels, with neither model showing a statistically significant advantage in tokens processed per second, even at high concurrency loads to 20+ users (Figure 1, bottom-right). Both systems show stability under increasing load, showcasing the efficiency of the shared vLLM inference framework. However, Mistral has slightly higher throughput variance (±3–5%) during stress tests, whereas Llama displays more consistent token generation rates.

4. Discussion and Conclusions

This study systematically evaluated the performance characteristics of two prominent open-source large language models, Llama 3.1 8B Instruct and Mistral 3 Small 24B Instruct (AWQ), through user preference assessments and computational benchmarking. Our mixed-methods study design incorporated subjective human evaluations and objective performance metrics, including response time, throughput, and time-to-first-token measurements, with GPT-4o as a reference point for comparative analysis. We report that both open-weight models achieved competitive performance relative to the proprietary benchmark, exhibiting distinct strengths across different evaluation dimensions. This suggests that open-source LLMs can be used instead of proprietary LLMs to improve telehealth-related services such as consultation or coaching.

The findings revealed several significant insights regarding current open-source LLM capabilities. First, the models’ comparable performance to GPT-4o in user preference rankings challenges prevailing assumptions about the necessity of proprietary architectures for high-quality text generation. Second, our computational benchmarks identified clear tradeoffs between model architectures: Llama demonstrated superior response latency characteristics and more stable throughput under increasing concurrent loads. Mistral’s larger parameter count and quantization showed no significant penalty in initial response latency despite its greater computational complexity. These results suggest that careful architectural and optimization choices can effectively mitigate the performance penalties traditionally associated with larger model sizes.

This study had limitations. The user evaluation cohort may not represent broader population preferences, and our throughput testing did not assess scaling beyond 20 concurrent users, which raises concerns about performance in enterprise-level deployments. Future research should involve larger studies with diverse demographics and more rigorous stress testing. Further investigation is needed to understand how model scale, quantization methods, and response quality interact and if Mistral’s advantages apply to various tasks. These findings enhance our understanding of open-source LLMs and aid in model selection for practical applications.

In conclusion, the evaluation of the open-source large language models Llama 3.1 8B Instruct and Mistral 3 Small 24B Instruct (AWQ) shows that they align well with user preferences and perform strongly compared to the proprietary GPT-4. These models serve as viable alternatives and may surpass proprietary user satisfaction and adoption options. Their transparency and collaborative improvement make them appealing for various applications. As organizations seek cost-effective and ethical AI solutions, this study supports the continued exploration and adoption of open-source models, ensuring a future where innovation and accessibility in language technologies coexist.