O3317: Validity Evidence for a Transport Medical Control Assessment Tool for Pediatric Subspecialty Fellows Using Simulated Interfacility Transport Calls

Background: The role of transport medical control (TMC) for interfacility transports requires adequate information gathering, appropriate medical decision making, and professionalism in communication. Pediatric subspecialty fellows often serve as TMC, but no published tools exist to assess fellow TMC performance. In this study, we sought to generate validity evidence for a previously developed TMC assessment tool using simulated transport calls.

Methods: We conducted a cross-sectional study among pediatric subspecialty attendings who supervise fellows serving as TMC in critical care medicine, emergency medicine, neonatology, and hospital medicine. The TMC assessment tool included 21 items within three domains: information gathering, medical knowledge/decision-making, and professionalism/communication. Each item had three possible responses (“yes”, “no”, “N/A”), and the score for the TMC assessment tool was defined as the proportion of “yes” items over the total number of items, excluding “N/A” items. We used Messick’s framework to evaluate validity evidence of content, response process, internal structure, and relationship to other variables. Participants listened to four subspecialty-specific simulated transport calls and completed the TMC assessment tool, a 4-point supervision level scale, and a survey on the relevance, ease of use and clarity of the TMC assessment tool.

Results: A total of 101 faculty from 54 institutions participated (Table 1). Most participants (82%) reported that no items should be removed from the TMC assessment tool, and only 20% reported that items should be added. Nearly all participants (99%) indicated that the TMC assessment tool captured the relevant skills and knowledge required for TMC very/somewhat well. Nearly all participants (99%) also indicated the TMC assessment tool was very/somewhat easy to use and 99% of participants also reported the items were very/somewhat clear. Reliability of the TMC assessment tool was high, with a Cronbach’s alpha of 0.86 (95% CI 0.84-0.87). The overall generalizability coefficient was also excellent at 0.92 with most of the variance (86%) related to the simulated call and none (0%) related to the subspecialty. As the TMC assessment tool score increased, faculty were more likely to indicate that the fellow required less supervision. With each additional point on the TMC assessment tool score, the odds that the supervision level score moved to the next-higher category was 10.3 times higher (95% CI 8.97 to 11.62, p < 0.05). TMC assessment tool scores are shown at each supervision level (Figure 2). The TMC assessment tool demonstrated outstanding ability to predict if a fellow could perform the role of TMC independently, with an area under the curve of 0.91.

Conclusion: This study provides multiple sources of validity evidence for a pediatric subspecialty fellow TMC assessment tool using simulated interfacility transport calls. This tool could be implemented by fellowship programs to improve TMC training and assessment of performance in the role of TMC.