O2072: Comparing Performance and Variability of Large Language Models in Answering Image-Based Neonatology Questions

Background: ChatGPT, Claude, and Gemini are publicly available large language models (LLMs) that excel at recognizing and generating human-like language. Some LLMs have achieved passing scores on multiple text-based medical licensing exams. Their performance on image-based, board-style questions in neonatology remains underexplored.

Methods: This comparative study assessed ChatGPT4o, Claude 3.5 Sonnet, and Gemini 1.5 Flash on 66 image-based questions from Neonatology Review: Q&A (3rd edition). Each model received a simple prompt across three trials, spaced two weeks apart, to assess accuracy and variability. ChatGPT4o was further tested with advanced prompts using role specification and chain-of-thought reasoning. Accuracy, consistency, and response variance were evaluated using descriptive statistics and chi-square tests.

Results: On average, ChatGPT4o answered 59.6% (± 3.2%) of the 66 questions correctly, followed by Claude at 51% (± 3.2%) and Gemini at 23.7% (± 5.3%). ChatGPT4o and Claude demonstrated equal consistency in correct responses (variance = 4.3), whereas Gemini showed greater variability (variance = 12.3). ChatGPT4o repeated its answers across the three trials for 46 of 66 questions (70%), Claude for 44 of 66 (67%), and Gemini for 48 of 66 (73%), indicating relative stability in responses regardless of accuracy. ChatGPT 4 and Claude provided unique explanations for each response, while only 3% of Gemini’s explanations were identical. In radiology-related questions, ChatGPT4o answered 73% correctly, Claude 26.7% and Gemini 0%. Advanced prompting increased ChatGPT4o’s average correct response rate to 66.7% (± 5.3%), though this was not statistically significant (χ² = 1.83, p = 0.116). ChatGPT4o demonstrated similar response consistency with advanced prompting, repeating its answers for 47 of 66 questions (71%).

Conclusion: This study reveals that LLMs demonstrate lower accuracy on image-based questions compared to previously reported performances on text-based neonatology questions. Both ChatGPT4o and Claude achieved over 50% accuracy and outperformed Gemini. Refined prompting techniques provided only modest performance gains for ChatGPT4o. Further research and prompting approaches will be essential to fully assess the potential of these models as they continue to evolve.