P3.056: Use of Large Language Models Applied to Clinical Narratives from the Electronic Medical Record Can Improve Child Abuse Detection

Background: Child maltreatment is a leading cause of trauma-related death in the United States. Nearly half of all child maltreatment fatalities are due to physical abuse, which often goes underrecognized, resulting in missed opportunities for preventing repeat injury. More robust methods to identify physical abuse victims are needed. While physical exam findings have been increasingly used to risk stratify abuse, valuable insights may also lie within narrative histories. Large language models (LLMs) offer a novel approach to synthesizing clinical notes for enhanced risk stratification, potentially enabling earlier detection and intervention.
The objective of this study was to evaluate the performance of an LLM for extracting historical features in a sample of children with documented physical injuries.

Methods: We performed a secondary analysis of data from 2 multicenter prospective studies of 2,964 children < 4 years of age evaluated in the emergency department (ED) with documented bruising (n=2161) and/or long bone fractures (n=803), collected across 7 urban children’s hospitals. Injuries for children in the data set represented both accidental and non-accidental injury, determined by an expert panel. Data included narrative clinical notes abstracted from the electronic health record (EHR) at key care points: triage, ED resident, ED attending, and social work and child abuse pediatrics consultations. We evaluated the performance of the instruction-tuned Gemma 2 LLM (gemma-2-9b-it) for extracting answers to 12 questions related to child injury mechanism, development, temperament, trauma evaluation, and caregiver characteristics. The LLM was given the persona: “You are a child abuse expert. Please review the clinical note(s) below from an Emergency Department admission and answer the question that follows.” Answer choices were limited to: “Yes”, “No”, “insufficient information”, and “non-applicable”. We iteratively refined question wording and reported LLM performance using manually coded interpretation as the gold standard.

Results: For all questions across all note types and encounters, the LLM successfully offered a prediction that aligned with the authors’ pre-determined answer choices. The LLM exhibited variable performance across the 12 questions. (Table 1) It showed excellent discrimination for identifying injury mechanism in relation to patient age and developmental stage (AUROC ranged from 0.901–0.984) and performed well in identifying abnormal trauma labs (AUROC 0.873). The LLM exhibited inferior performance on questions related to caregiver presence (AUROC 0.536–0.76).

Conclusion: The ability of LLMs to interpret narrative features in the EHR marks a significant advancement in leveraging unstructured data for suspected physical abuse, where both history and physical exam are critical. Future work in prompt engineering is needed to maximize LLM performance, particularly in questions related to caregiver relationships. This study supports the feasibility of LLMs for historical risk stratification in child maltreatment cases.