P2.075: Variability in Probabilistic Word Interpretation in Large Language Models

Background: Imprecision remains a barrier to effective communication, especially in clinical settings where terms such as “likely” or “possibly” are used without ensuring that we understand these words consistently. When asked to convert these words into raw percentages, previous surveys have shown broad distributions in the interpretations of these terms (Mosteller 1990, Mauboussin 2018). With the introduction of large language models (LLMs), word selection becomes even more important. We wanted to understand how current LLMs interpreted these probabilistic words.

Methods: We submitted the following prompt to LLMs using each system’s default settings: ChatGPT4o, Copilot (ChatGPT4), Gemini 1.5, and Llama 3.1. Each prompt was performed five times in a new session each time.
"I am going to give you a list of probabilistic words. I want you to give me your interpretation of each word as a single numeric value from 0-100 on a basis of how likely you think that word means an event will occur with 100 being 100 percent of the time, and 0 being never occurring.
Almost certainly / Almost always / Always / Certainly / Frequently / Likely / Maybe / Might happen / More often than not / Never / Not often / Often / Possibly / Probably / Rarely / Real possibility / Serious possibility / Slam dunk / Unlikely / Usually / With high probability / With moderate probability / With low probability"

To compare with human interpretation, we performed an anonymous survey asking the same questions (Microsoft Forms). The survey was distributed to a sample of convenience of informaticists without collection of identifying factors: the Nationwide Children's Hospital clinical informatics division, the AMIA Clinical Informatics Fellows group-chat, and the Pediatric CDS Collaborative listserv.

Results: LLMs were most consistent at extremes, interpreting terms such as always and never and phrases with the word "almost" nearly identically upon repeated queries. The next most consistently interpreted phrase was Slam dunk. The least consistent words included often, probably, unlikely and with low probability. Comparing LLMs, Gemini had the most variability, while ChatGPT 4o had the least, with many terms resulting identical results repeatedly.

We received 40 responses from human participants. We found much greater variation in these responses, with greater standard deviations for every term in the human responses. Comparing LLMs to humans, we found LLMs returned higher average values for most terms. Although a limited sample, visualization of the human data showed the tendency to select numbers ending in 5 or 0. We had not run enough iterations of LLM tests to evaluate if this peculiarity was seen.

Conclusion: LLMs, reflective of the human language, demonstrate variable interpretations of probabilistic terms, highlighting the need for caution in clinical contexts. However, this variability is less than seen in human populations.