P3.274: Grading Smarter, Not Harder: Harnessing LLMs to Optimize Rubric-Based Evaluation of Medical Student Write-Ups

Background: The written History and Physical (H&P) is a fundamental assessment in medical education of students’ ability to document and synthesize a patient encounter. During our institution’s pediatrics clerkship, students across 12 sites complete an H&P. Currently, site directors utilize the Pediatrics History and Physical Exam Evaluation (P-HAPEE) rubric to grade students’ submissions. However, inconsistent and escalating scores across sites has raised concerns about the P-HAPEE as a reliable tool for military pediatricians, especially given the unique challenges such as high turnover and limited time for training and precepting. The growing application of Large Language Models (LLMs) in academia presents a compelling opportunity to optimize grading of H&Ps, potentially leading to more standardized evaluations and timely feedback to medical students. Preliminary testing of LLMs to apply the P-HAPEE rubric highlighted that it lacked the detail required for the LLM to accurately grade submissions. The objectives are to: 1) design and validate a new rubric, and 2) determine the feasibility of LLMs to apply the rubric.

Methods: This educational quality improvement project will be completed in three phases:
Phase 1: Systematic design of a new analytic rubric: “Rubric V1”. Content validity from seven experts, resulting in “Rubric V2”.
Phase 2: Establish reliability of Rubric V2 via trained graders (military pediatricians) who apply it to 10 de-identified H&Ps (confirmed if agreement ≥ 90% and 𝞪 ≥ 0.70). Graders complete a “Rubric Utility Survey” with additional feedback. Concurrently, researchers train a LLM (ie, ChatGPT) to use Rubric V2. Once trained, comparison between LLM and human grades to determine reliability (confirmed if agreement ≥ 85% and 𝞪 ≥ 0.70).
Phase 3: Feasibility testing with the implementation of the new rubric in a real-world educational setting by comparing LLM to site director grades.

Results: This is an ongoing project. Experts feedback on Rubric V1 suggested analytic rubric design, variable weighting of categories, and clarifying vague grading terms such as “mostly complete”. Rubric V2 consists of 15 categories and a total of 51 points (see Figure 1). Phase 2 is currently underway and will be complete by June 2025. Phase 3 will be implemented in the calendar year 2026.

Conclusion: During Phase 1, a single point rubric was rejected due to limitations of an Honors/Pass/Fail grading schema. In initial testing, ChatGPT outperformed Gemini Advanced in applying the rubric to H&Ps. In preliminary survey results from Phase 2, graders found the rubric aligned with learning objectives but had mixed views on the level of detail and weighting. Further revisions may be necessary before Phase 3 implementation. If successful, this project could be a model to expand the use of LLM platforms to decrease subjectivity and increase efficiency within military medical education.