P2.069: Bilingual Barrier: ChatGPT as a Spanish Language Translation and Information Tool for Parents

Background: Roughly 43 million Americans use Spanish as their first language. As a result, many are comfortable discussing topics in their native language, such as their child’s health care. Artificial intelligence (AI) tools like ChatGPT-4 (Chat) have expanded options beyond standard health information translation and can generate health information in a user’s native language. Previous studies have shown that Chat can respond accurately to parental questions in English, but it is unknown how Chat responds to parents’ questions in Spanish.

Methods: General pediatrics questions and pediatrician responses (n=25) were extracted from the American Academy of Pediatrics’ (AAP) www.healthychildren.org. The “En Español” feature on the website provided the first 25 responses, while a second set of 25 responses were derived by inputting these AAP questions in Spanish into Chat. Two physicians fluent in Spanish rated the medical correctness (MC) and completeness of Spanish Chat’s answers and Spanish AAP pediatrician’s answers on a 5-point Likert scale. Results were compared to previous work that examined Chat responses in English to the same set of questions. Analysis was conducted in R.

Results: 50 responses (25 by Spanish Chat; 25 by Spanish AAP pediatricians) were analyzed. The MC of Spanish Chat’s responses (4.16) was significantly different (p=0.013) from the MC of Spanish AAP responses (4.56). Completeness of Spanish Chat’s responses (3.98) was significantly different (p=0.046) from completeness of Spanish AAP responses (4.36). Comparison to a previous abstract of physician-raters examining English Chat responses to this same bank of questions indicated significantly different responses to Spanish Chat in terms of MC (4.8; p < 0.001) and completeness (4.8; p < 0.001). See Table 1. Readability was evaluated using average word count, Fernández Huerta score, and Inflesz Ease Scale (Table 2).

Conclusion: An apparent disparity in Chat’s English and Spanish capabilities may pose a barrier in accessing reliable pediatric healthcare information online for Spanish-speaking parents. Requiring a higher reading level than AAP, Chat may not be able to provide the same level of understanding as professional online healthcare information. Difference in MC and Completeness between Chat’s Spanish and English indicates that users should be wary of entering Spanish Chat prompts for fear of sub-optimal results. Though AI could bridge healthcare language barriers, these results reiterate calls to improve multilingual models to ensure equitable access to accurate and complete information for Spanish speakers.