P4.016: Can AI Replace Patient Reported Outcome Measures? A Pilot Study

Background: Patient reported outcome measures (PROMs) are increasingly utilized to in research and quality assurance/improvement and represent a way to quantify an assess health from the patient’s perspective. However, traditional PROMs are time consuming, limited in their scope and subject to floor and ceiling effects. Artificial intelligence (AI)-powered large language models (LLMs) are now adept at transcribing natural speech during standard clinical encounters, interpreting the data and generating comprehensive clinic notes from the perspective of the healthcare provider. This pilot aims to assess the potential of a LLM to predict PROMs through analysis of transcripts of standard clinic interactions. AI estimations of PROMs were compared to actual PROMS obtained during routine clinic visit.

Methods: 47 outpatient clinical visits from a pediatric orthopaedic spine clinic were analyzed. Patients completed computer adaptive Patient-Reported Outcomes Measurement Information System (PROMIS) surveys in multiple domains as standard of care. Audio recordings were made of the visits after obtaining verbal consent and assent from the patients/ guardians in accordance with institutional policy. Transcriptions of the visits were generated using publicly available AI tool. De-identified transcripts were introduced to a secure partition LLM which was prompted to extrapolate likely patient responses from PROMIS short forms. Patients were stratified based on new versus follow-up visits. Paired t- tests were used to compare t-scores across 8 domains from patient completed surveys and AI generated responses based on transcriptions. A non-significant outcome (p>0.05) indicated similar t-scores between AI and patients.

Results: Among new patient visits, AI estimated PROMIS scores were not significantly different (p>.05) from the patient reported scores in the domains of peer relationships, physical stress, physical activity, pain behavior, mobility, and pain interference.  AI estimated significantly better PROMIS scores in the domains of anxiety and depressive symptoms. Among follow-up patient visits, AI estimated PROMIS scores were not significantly different (p>.05) from the patient reported scores in the domains of physical stress, anxiety, and pain behavior. AI estimated significantly worse scores in the domains of peer relationships. physical activity, mobility, and significantly better scores the domains of depressive symptoms and pain interference.

Conclusion: AI demonstrated remarkable ability to assess patient reported outcomes through analysis of natural speech during a standard clinic visit. Overall, AI was more consistently able to estimate PROMIS scores based on new patient visits versus follow-up visits but appeared to underestimate disability related to depression and anxiety among new patients. AI and LLMs have the potential to facilitate the quantification outcomes through analysis of routine patient encounters, potentially avoiding the cumbersome processes associated with obtaining PROMs. Further research and customization of AI may enhance the capabilities hinted in this pilot.