Graphical abstract

Introduction

Background/rationale

Large language models (LLMs) have demonstrated remarkable capabilities in medical question-answering (QA) tasks, including interpreting complex terminology, clinical reasoning, and analyzing multi-modal data [1,2]. These models excel in standardized medical examinations such as the United States Medical Licensing Examination (USMLE), achieving accuracies exceeding 85% on Step 1 questions [3-5]. Their proficiency extends globally, with validated performance across diverse languages and healthcare systems, including those of Germany, Japan, Chile, and South Korea [6-9]. Recent advances in multi-modal LLMs (e.g., OpenAI’s ChatGPT, Google’s Gemini, Anthropic’s Claude) further enhance their utility by integrating visual and textual inputs—critical features for image-based medical assessments [3,10]. Despite these advancements, applications in low-resource settings remain underexplored, particularly in regions lacking standardized training materials.

In Thailand, medical students preparing for the Thai National Licensing Examination (ThaiNLE) face significant challenges due to the absence of publicly available, expert-validated study resources. Current preparation relies primarily on informally shared recollections of past exam questions, disproportionately disadvantaging students from rural institutions with limited peer networks [11]. This inequity highlights the urgent need for scalable, high-quality educational tools.

Objectives

We hypothesize that state-of-the-art LLMs can achieve passing scores on ThaiNLE Step 1 questions and demonstrate robust multi-modal capabilities, thereby serving as equitable preparatory resources that mitigate disparities in medical education access. To test this hypothesis, we evaluate GPT, Gemini, and Claude models using a comprehensive ThaiNLE mock examination dataset, assessing their accuracy, domain-specific performance, and image-text integration efficacy.

Methods

Ethics statement

Given that no human participants were involved, the research was exempt from institutional review board.

Study design

A cross-sectional study was conducted.

Data sources

We utilized a mock examination dataset consisting of 304 multiple-choice questions in English. The dataset includes “general principles” (G) and “organ systems” (S) questions, organized into 20 groups, such as biochemistry and molecular biology, biology of cells, and human development and genetics (Table 1). This mock examination is widely used and has a difficulty level comparable to actual examinations, although somewhat simpler than Step 1 of the USMLE. Additionally, 10.2% of the questions included images. There was a class imbalance among questions within each topic, reflecting the relative emphasis typically found in real examinations, though not exactly matching the official syllabus. A total of 19 board-certified physicians from various specialties—including internal medicine, pediatrics, and pathology—verified the answers. Any discrepancies in responses were resolved through a consensus majority approach, whereby the final answer reflected the response most frequently selected by the reviewers.

Passing scores

The original examination consists of 300 questions covering topics aligned with the Medical Competency Assessment Criteria for National License 2012. For clarity and to avoid confusion, we renamed the topic identifiers. The original naming convention used “B1.x” for general principles topics and “B2–B11” for systems topics. These categories were renamed to “G” for general principles and “S” for systems topics.

We used the national passing scores from the Center for Medical Competency Assessment as benchmarks for evaluating LLM performance. Table 2 presents the passing scores and national averages for the main (summer) examinations from 2019 to 2024. The mean passing score for the main examination rounds from 2019 to 2024 was 52.3%, while the national average was 56.1%. The passing scores showed fluctuations over this 6-year period, with the highest passing percentage recorded in 2021 (53%) and the lowest in 2024 (51%). National average scores also exhibited variability, notably increasing to 63.05% in 2024—significantly higher than in previous years. The standard deviation ranged from 12.60 in 2020 to 16.08 in 2024, indicating varying degrees of score dispersion across these years.

Models and inference

We evaluated 4 multi-modal LLMs (GPT-4, Claude 3 Opus, Gemini 1.0 Pro, Gemini 1.5 Pro) using their official Application Programming Interfaces (APIs) between March and June 2024, following established benchmarking protocols for medical QA systems [2]. All models were configured with zero temperature to ensure deterministic outputs, suppressing stochastic generation while maintaining clinical response fidelity. Token limits followed manufacturer specifications (32,768 tokens for GPT-4/Claude 3 Opus; undisclosed for Gemini models), sufficient for processing exam questions with integrated images (<5 MB/file). Model versions remained fixed during the evaluation period (Supplement 1), with API requests programmatically managed via Python’s requests library. No prompt engineering or fine-tuning was applied, maintaining consistency with prior medical benchmarking studies [3].

Each model predicted answers to the entire dataset over 5 inference runs. Questions were inputted individually to avoid confusing the model and to simulate how a student might submit questions one at a time. Answers from each run were compared to the ground truth and scored accordingly. Scores from each run were then averaged, and a 95% confidence interval (CI) was calculated.

A zero-shot prompt served as the foundational instruction outlining the specific question and task for the LLMs. The prompt outputted a single letter to clearly articulate the directive for the QA task. The exact prompt used was: “Given the following medical multiple-choice question, select the best answer from the provided options. Your response should be in the form of a single letter (A, B, C, D, E). Do not provide any explanation or additional information—only the letter or digit corresponding to the correct answer. Answer the following question.”

Statistical methods

To evaluate the performance of the LLMs, we employed micro-average and macro-average metrics, along with their 95% CIs. Using both metrics provides a comprehensive evaluation of the LLMs’ performance. While micro-average gives an overall performance view, macro-average ensures performance in each subtopic is equally considered, preventing any topic from being neglected.

The micro-average is determined by the percentage of correct answers, as shown in Equation 1. It emphasizes overall performance by giving equal weight to each individual question.

(Equation 1)

Micro-average=Number of correct answersTotal number of questions

While the micro-average can be influenced by the most frequent classes, we also utilized the macro-average (Equation 2) because it provides equal weight to each subtopic, irrespective of the number of questions per topic, ensuring performance in all subtopics is equally evaluated.

(Equation 2)

Macro-average=1N∑i=1NNumber of correct answers in subtopic iTotal number of questions in subtopic i

Results

Main results

Table 3 shows the overall and group accuracies for GPT, Claude, and Gemini models (Dataset 1). The overall accuracy demonstrated a clear hierarchical trend, with GPT-4o achieving superior performance across all models at 88.9% (95% CI, 88.7%–89.1%). GPT-4 followed with an accuracy of 83.3%, though it showed a wider CI (79.6%–87.0%), indicating variability in its results. Claude-3.5-Sonnet achieved 80.1% (79.6%–80.6%), notably higher than its predecessor, Claude-3-Opus, which reached 77.8% (77.4%–78.2%). The Gemini models showed the lowest performance, with Gemini-1.5-Pro obtaining 72.8% (72.2%–73.4%) and Gemini-1.0-Pro reaching 61.4% (61.3%–61.5%). In terms of macro-average accuracy, the results were consistent, reflecting similar hierarchical trends across the models. The high balanced accuracy of GPT-4o (89.1%; 95% CI, 88.8%–89.4%) indicates robust performance across diverse topics, confirming its reliability in handling varied medical content.

Comparison of model performance

To evaluate whether the performance differences were statistically significant, we conducted pairwise z-tests comparing the micro-average accuracies of all 6 models over the 304-question dataset. GPT-4o significantly outperformed all other models, including GPT-4 (P<0.001), Claude-3.5-Sonnet (P=0.0028), and Gemini-1.5-Pro (P<0.0001), with Cohen’s h values indicating moderate to large effect sizes. Claude-3.5-Sonnet also significantly outperformed Gemini-1.5-Pro (P=0.034) and Gemini-1.0-Pro (P<0.00001). These results confirmed that the observed performance differences were both statistically significant and practically meaningful.

Results by topic

Fig. 1 presents the accuracy results categorized by general principles and systems. All models demonstrated higher accuracy on “general principle” questions compared to “systems” questions. Accuracy for general principles was relatively consistent, with Gemini-1.0-Pro achieving 64.1%, and Gemini-1.5-Pro significantly improving to 84.6%. Claude-3-Opus and Claude-3.5-Sonnet further increased accuracy, reaching 80.2% and 84.7%, respectively. GPT-4 achieved 83.5%, while GPT-4o led with an accuracy of 90.0%. In contrast, performance in the systems category varied notably across models: Gemini-1.0-Pro scored 62.3%, Gemini-1.5-Pro improved to 67.9%, and Claude-3-Opus and Claude-3.5-Sonnet reached 74.5% and 79.1%, respectively. GPT-4 achieved 82.2%, and GPT-4o again outperformed all others with an accuracy of 88.3%. Overall, GPT-4o consistently delivered the highest accuracy across both general principles and systems. Table 4 illustrates the micro-accuracy of each model by topic.

Comparing questions with and without images

The comparative analysis of model accuracy for questions with and without images is detailed in this section, showcasing variations in performance across models.

Questions with images

Micro-accuracy varied, with Gemini-1.0-Pro achieving 70.3% (69.0%–71.6%), and Gemini-1.5-Pro slightly lower at 68.4% (66.0%–70.8%). Claude-3-Opus and Claude-3.5-Sonnet improved accuracy to 72.9% (68.2%–77.6%) and 74.8% (73.5%–76.1%), respectively. GPT-4 had higher accuracy at 78.4% (75.0%–81.8%), with GPT-4o reaching the highest accuracy of 82.6% (81.1%–84.1%) (Table 5).

Most models performed better on text-only questions compared to image-based questions (Fig. 2). Notably, Gemini-1.0-Pro performed better on image questions, showing an increase of 9.9% points compared to text-only questions. GPT-4 and GPT-4o consistently outperformed other models in both categories, with GPT-4o achieving the highest overall accuracy. This variation in performance between image and non-image questions suggests differing levels of robustness across models in handling visual inputs.

Discussion

Key results

This study aimed to assess the performance of LLMs on the ThaiNLE Step 1 examination by analyzing their overall and domain-specific performances in handling multimodal questions. All models surpassed the national passing thresholds, with GPT-4o achieving the highest overall accuracy and consistent domain-specific performance. Although the models demonstrated strong general capabilities, performance varied considerably across medical specialties and question types, especially for image-based questions.

Our evaluation of GPT, Gemini, and Claude models on the ThaiNLE mock examination revealed 3 key findings. First, all models exceeded historical passing thresholds, with GPT-4o demonstrating superior overall accuracy (88.9%; 95% CI, 88.7–89.1) and balanced performance across topics (macro-average 89.1%) (Table 3). GPT-4o also maintained dominance across both major question categories, achieving 90.0% accuracy in general principles and 88.3% in systems (Fig. 1), outperforming Claude-3.5-Sonnet (80.1% overall, 84.7% systems) and Gemini-1.5-Pro (72.8% overall, 67.9% systems).

Second, the performance variability across medical domains highlights model-specific limitations (Table 4). GPT-4’s weaker quantitative reasoning (74.0% in G9 vs. 99.2% in S4) aligns with its documented limitations regarding mathematical precision, while Claude-3.5-Sonnet’s inconsistent cardiovascular performance (96.7% in S1 vs. 58.3% in S5) might indicate insufficient exposure to procedural clinical scenarios. LLM variability across specialties is likely due to differences in training data and required clinical reasoning skills. For instance, the high accuracy in obstetrics and gynecology (84.6%) reflects standardized protocols and workflows in the specialty. Conversely, lower accuracy in psychiatry (78.0%–86.0%) may highlight challenges related to cultural factors (Thai-specific practices) and ethical ambiguities inherent in interpreting subjective symptoms. Third, most models showed decreased accuracy on image-based questions compared to text-only questions. An exception was Gemini-1.0-Pro, which unexpectedly achieved higher accuracy (+9.9%) on visual questions despite poor textual performance (61.4% micro-average). This finding suggests its vision module may compensate for language-processing deficiencies—a trade-off warranting further investigation.

Interpretation

The observed performance hierarchy (GPT-4o > GPT-4 > Claude > Gemini) suggests architectural advantages in OpenAI’s models for medical QA tasks, especially their ability to maintain high macro-average accuracy across diverse topics. GPT-4o’s superior performance across 17 out of 20 topics (Table 4), including perfect scores in biochemistry (G2) and neurology (S4), demonstrates its robust reasoning capabilities. This advanced performance is likely attributable to its enhanced multimodal transformer architecture and larger effective context window, enabling richer contextual understanding and reliable clinical reasoning. Pairwise statistical comparisons confirmed that GPT-4o significantly outperformed all other models (P<0.001), with large effect sizes (Cohen’s h >0.8) observed in most comparisons. Meanwhile, differences between Claude-3.5-Sonnet and GPT-4, as well as between Gemini-1.0 and Gemini-1.5, were also statistically significant, albeit with smaller effect sizes. Persistent knowledge gaps—such as GPT-4o’s genetics accuracy (80.0% in G3) and Claude-3.5-Sonnet’s musculoskeletal performance (58.3% in S5)—appear unrelated to overall model size or training recency. Although GPT-4o consistently achieved the highest accuracy, its advantage was smaller in certain domains, likely due to topic complexity, uneven training data, or domain-specific terminology. Gemini-1.0-Pro’s superior image-based accuracy, despite poor textual performance, further highlights that vision-module benefits are highly model-specific and do not uniformly translate into superior multimodal performance.

Comparison with previous studies

Our findings align with previous studies evaluating multimodal LLM performance on medical examinations. Research conducted in Chile [8] and Japan [7] indicates that integrating visual data does not always enhance accuracy, whereas studies in Germany [6] and Taiwan [12] found overall improvements in newer models do not necessarily translate into consistent gains across all medical domains. Similarly, USMLE-based evaluations confirm that, despite improvements in overall scores by newer models, significant variability remains in specific medical domains [4]. Our observation that the evaluated models passed the examination aligns closely with studies from Germany, Chile, and Taiwan [6,8,12], but contrasts with findings from Japan [7], where models failed to achieve passing scores.

Limitations

This study has 3 primary limitations: (1) restricted dataset access (304 questions) limits reproducibility and generalizability; (2) a small subset of image-based questions (n=31) precludes definitive conclusions regarding multimodal capabilities; and (3) the exclusion of fine-tuned models obscures potential performance enhancements achievable through domain adaptation. Additionally, focusing exclusively on multiple-choice questions neglects the complexity inherent in open-ended questions, where subjective interpretation and cultural nuances play substantial roles. The lack of real-world testing conditions further limits direct comparability to human performance.

Generalizability

Our findings suggest that high-performing LLMs have significant potential as globally scalable solutions for medical education, especially in resource-limited settings. Although the evaluation used Thai-language and region-specific content, the consistency of domain-specific performance variability parallels patterns seen internationally [4,6,12], supporting broader applicability. Nonetheless, generalizability may be limited by language nuances, cultural contexts, and curricular differences, highlighting the necessity for localized fine-tuning. Considerations such as cost-effectiveness (e.g., affordability of artificial intelligence [AI]-driven tutorials), technical accessibility (e.g., stable internet access), and instructor-student ratios should also be taken into account, as these factors could mitigate or exacerbate educational disparities in low-resource settings [13].

Suggestions

Future research should explore domain-specific fine-tuning and multilingual optimization to enhance LLM reliability, particularly in complex or underperforming domains. Additionally, comparative studies evaluating real-world deployment—such as classroom integration or personalized tutoring—may further elucidate their practical educational utility.

Conclusion

Our results confirm the hypothesis that LLMs can effectively address inequities in ThaiNLE preparation, as all models exceeded passing thresholds. GPT-4o’s narrow confidence intervals (88.7%–89.1%) suggest reliable performance, particularly beneficial to rural students lacking traditional resources. Nonetheless, the observed domain-specific weaknesses necessitate targeted improvements before widespread clinical deployment. By integrating these findings into medical education frameworks, institutions can deliver equitable, AI-driven support while maintaining rigorous competency standards.

Supplementary materials

Fig. 1.

Macro-accuracy in general principles and systems. CI, confidence interval.

Fig. 2.

Comparison between questions with and without images.

References

• 1. Sumbal A, Sumbal R, Amir A. Can ChatGPT-3.5 Pass a medical exam?: a systematic review of ChatGPT’s performance in academic testing. J Med Educ Curric Dev 2024;11:23821205241238641. https://doi.org/10.1177/23821205241238641 ArticlePubMedPMC
• 2. Lievin V, Hother CE, Motzfeldt AG, Winther O. Can large language models reason about medical questions? Patterns (N Y) 2024;5:100943. https://doi.org/10.1016/j.patter.2024.100943 ArticlePubMedPMC
• 3. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepano C, Madriaga M, Aggabao R, Diaz-Candido G, Maningo J, Tseng V. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health 2023;2:e0000198. https://doi.org/10.1371/journal.pdig.0000198 ArticlePubMedPMC
• 4. Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)?: the implications of large language models for medical education and knowledge assessment. JMIR Med Educ 2023;9:e45312. https://doi.org/10.2196/45312 ArticlePubMedPMC
• 5. Koga S. The potential of ChatGPT in medical education: focusing on USMLE preparation. Ann Biomed Eng 2023;51:2123-2124. https://doi.org/10.1007/s10439-023-03253-7 ArticlePubMed
• 6. Meyer A, Riese J, Streichert T. Comparison of the performance of GPT-3.5 and GPT-4 with that of medical students on the Written German Medical Licensing Examination: observational study. JMIR Med Educ 2024;10:e50965. https://doi.org/10.2196/50965 ArticlePubMedPMC
• 7. Nakao T, Miki S, Nakamura Y, Kikuchi T, Nomura Y, Hanaoka S, Yoshikawa T, Abe O. Capability of GPT-4V(ision) in the Japanese National Medical Licensing Examination: evaluation study. JMIR Med Educ 2024;10:e54393. https://doi.org/10.2196/54393 ArticlePubMedPMC
• 8. Rojas M, Rojas M, Burgess V, Toro-Perez J, Salehi S. Exploring the performance of ChatGPT versions 3.5, 4, and 4 with vision in the Chilean Medical Licensing Examination: observational study. JMIR Med Educ 2024;10:e55048. https://doi.org/10.2196/55048 ArticlePubMedPMC
• 9. Jang D, Yun TR, Lee CY, Kwon YK, Kim CE. GPT-4 can pass the Korean National Licensing Examination for Korean medicine doctors. PLOS Digit Health 2023;2:e0000416. https://doi.org/10.1371/journal.pdig.0000416 ArticlePubMedPMC
• 10. Schubert MC, Wick W, Venkataramani V. Performance of large language models on a Neurology Board-Style Examination. JAMA Netw Open 2023;6:e2346721. https://doi.org/10.1001/jamanetworkopen.2023.46721 ArticlePubMedPMC
• 11. Apornvirat S, Namboonlue C, Laohawetwanit T. Comparative analysis of ChatGPT and Bard in answering pathology examination questions requiring image interpretation. Am J Clin Pathol 2024;162:252-260. https://doi.org/10.1093/ajcp/aqae036 ArticlePubMed
• 12. Huang CH, Hsiao HJ, Yeh PC, Wu KC, Kao CH. Performance of ChatGPT on Stage 1 of the Taiwanese medical licensing exam. Digit Health 2024;10:20552076241233144. https://doi.org/10.1177/20552076241233144 ArticlePubMedPMC
• 13. Zhang J, Fenton SH. Preparing healthcare education for an AI-augmented future. Npj Health Syst 2024;1:4. https://doi.org/10.1038/s44401-024-00006-z ArticlePubMedPMC

Figure & Data

References

Citations

Citations to this article as recorded by