O2083: Establishing Evidence of Validity for a Novel Pediatric Airway Management Assessment Tool in Pediatric Trainees

Background: Airway management is a core skill for pediatric critical care medicine (PCCM) and pediatric emergency medicine (PEM) physicians. Adverse events occur more often when airway management is performed by those with less experience. Previously, we used modified Delphi methods to define competency measures in airway management for pediatric trainees. These were used to develop Pediatric Airway Skills Tool (PediAir Skills), a novel assessment tool using behaviorally anchored rating scales in 8 domains. PediAir Skills rates individuals along a scale from novice to advanced beginner to competent to proficient to expert. Our objective is to provide evidence supporting the validity and reliability of PediAir Skills. We hypothesize that PediAir Skills will differentiate between novice and expert skills in airway management in simulations.

Methods: Participants were recruited from pediatric residents, PCCM and PEM fellows, PCCM and PEM faculty. Demographic data and experience with airway management was collected via survey. Participants completed 3 airway simulations. Simulations were recorded and reviewed by 2 study members. We evaluated criterion-related validity by examining the relationship of PediAir Skills scores with current level of training and prior airway experience. Inter-rater reliability was evaluated by Cohen’s kappa (for Likert items) and AIC (for mean scores). Internal consistency reliability was assessed using Cronbach’s alpha, and generalizability theory was used to examine relative contributors to measurement error and quantify overall reliability.

Results: Twenty-three participants completed simulations. Participants were pediatric residents (7/23; 30%), PCCM or PEM fellows (9/23; 39.1%), PCCM or PEM faculty (7/23; 30%) (Table 1). Participants with a higher level of training performed better on PediAir Skills (Case 1:p=0.002; Case 2:p=0.001; Case 3:p=0.004). Participants with higher number of intubations performed better on PediAir Skills (Case 1:p=0.006; Case 2:p=0.006; Case 3:p=0.035). We observed high internal consistency with Cronbach’s alpha >0.89 for each rater in each case. For overall scores, inter-rater agreement was high [Case 1: 0.902 (95%CI 0.786-0.957); Case 2: 0.871 (95%CI 0.716-0.944); Case 3: 0.847 (95% CI 0.642-0.935)]. Inter-rater reliability by domain was moderate to high (Table 2). The results of the generalizability study including all measurement facets (participants, raters, items, cases) demonstrated high reliability with a G-coefficient=0.948.

Conclusion: Physicians with more experience and higher levels of training consistently performed better using PediAir Skills, providing evidence of validity. PediAir Skills exhibits high levels of internal and external reliability. The next steps will include enhancement of PediAir Skills and additional testing in clinical scenarios with more rigorous difficulty.