O3346: Classifying Urinary Tract Dilation and Vesicoureteral Reflux: de novo Natural Language Processing Model versus ChatGPT

Background: The urinary tract dilation (UTD) and Vesicoureteral reflux (VUR) classification system provides objective assessment relevant to management for children. However, the lack of uniform language in radiology reports leads to significant difficulty in both clinical management and research. We seek to develop a unified multi-task and multi-class model that can effectively extract UTD components and classifications from early postnatal ultrasound (US) reports using cutting-edge natural language processing (NLP) deep learning algorithms. We also developed a similar NLP model extracting VUR grades from voiding cystourethrogram (VCUG). We compared the performance of our model to the popular large language model ChatGPT.

Methods: We identified infants aged 0-90 days who had US for antenatal UTD (n = 2479). Reports and images were examined to establish UTD classification. Variables included central calyceal dilation, parenchymal appearance, abnormal parenchymal thickness, peripheral calyceal dilation, abnormal ureter, and bladder abnormality. VCUG reports (n = 2687) were labeled with no VUR or with grade 1-5. We developed a Bidirectional Encoder Representations from Transformers (BERT) encoder NLP model, chosen for its relative simplicity and wide adoption in clinical NLP. We used biomedical text-trained Bio_ClinicalBERT to gain domain-specific knowledge. We used stratified sampling in five-fold cross-validation to balance data for robust generalization. The model was iteratively trained and relabeled. Finally, our algorithm was compared to the GPT-4, in "3-shot (3s)” setting that included three random task-specific examples from each outcome category. The same five data splits were used for all experiments.

Results: Our model demonstrated higher accuracy compared to GPT-4o in assessing UTD components and formulating classification. The average F1 score for BERT was 0.942, substantially higher than the average GPT-4o score (0.868). For VUR grading, GPT-4o outperformed our model, with an average F1 score for BERT of 0.920 compared with 0.948 for GPT-4o.
Table 1 outlines the performance of our multi-task and multi-class predictive model and GPT-4o for UTD. Our model, referred to as BERT, demonstrated remarkably higher F1 scores compared to GPT-4 in "3-shot" settings across all tasks. The high performance across all tasks suggests that the model can reliably extract a variety of UTD components and classifications, ranging from central calyceal dilation to bladder status. Table 2 outlines the performance of our predictive model and GPT-4o for VUR.

Conclusion: Our BERT NLP model demonstrated outstanding performance in extracting UTD components and formulating classifications from ultrasound reports. Its high F1 scores, precision, recall, and AUC values make it a promising tool for automating the formulation of UTD classifications from diverse natural language descriptions. For the VUR grading, our BERT model was outperformed by GPT4o, which suggests simpler grading tasks can be sufficiently performed by exsiting, pre-trained large language models like ChatGPT.