O3345: OpenAI Model Comparison: GPT-4.0 vs. 01 – Harnessing Verified AI for Future Quality Improvement in Pediatric Urology

Background: Large language models (LLMs) have shown promise in various medical applications, including pediatric urology. However, their accuracy and reliability still warrant close scrutiny. In this project, we compared two models: a standard GPT-4.0 and a separate OpenAI 01 model specifically designed for advanced reasoning. Our aim was to evaluate whether 01 could outperform GPT-4.0 on pediatric urology Self-Assessment Study Program (SASP) questions, offering a potential avenue for quality improvement if validated appropriately.

Methods: We selected 52 multiple-choice pediatric urology focused SASP questions from 2021–2023, each with one correct response and four distractors. GPT-4.0 and OpenAI 01 independently answered every question. We recorded each answer as correct or incorrect, then built 2×2 confusion matrices to calculate accuracy, precision, recall, F1 score, specificity, balanced accuracy, and Cohen’s Kappa. Both models’ results were compared to identify significant differences in performance.

Results: GPT-4.0 demonstrated moderate accuracy, aligning with earlier findings for GPT-based solutions. In contrast, OpenAI 01 correctly answered 34 of 52 questions, yielding an accuracy of 86.15%, precision of 65.38%, and a Cohen’s Kappa of 0.57. Specificity was 91.35%, indicative of fewer false positives. These figures surpassed GPT-4.0’s performance on the same dataset, suggesting that advanced reasoning features within 01 can provide a more reliable platform for question-answering in pediatric urology.

Conclusion: OpenAI 01 delivered notably higher accuracy and reliability than GPT-4.0 for pediatric urology questions. This improvement underscores the potential of advanced reasoning LLMs to support quality improvement initiatives. Nevertheless, rigorous validation and human oversight remain essential to ensure safety and correctness before employing these tools more widely in education or clinical practice.