O2091: Developing Organ Dysfunction Diagnostic Criteria for Children with Cancer and Post-Hematopoietic Cell Transplantation: A Novel Approach for Large Systematic Reviews

Background: Defining organ failure in children with cancer is essential for diagnosing and treating critical illness. The Pediatric Organ Dysfunction Information Update Mandate (PODIUM) provided evidence-informed, consensus definitions of 11 single-organ dysfunction in children. These criteria do not perform well in children with cancer and those post-hematopoietic cell transplantation. To address this challenge, we developed the PODIUM-Onc study to update the PODIUM systematic reviews focused specifically on organ dysfunction in children with cancer and leverage findings to adapt the original PODIUM organ dysfunction criteria for children with cancer. This study, however, encountered challenges common to large systematic reviews. We describe these challenges and solutions developed by our team relevant to both this study and future research.

Methods: With consultation from research librarians and multidisciplinary experts, we expanded the original PODIUM search criteria to include multi-organ dysfunction, more inclusive search terms, and removed the language filter. A total of 76,429 abstracts were identified across 10 of the 11 reviews (Table 1). Through this work we encountered several challenges: 1) Inter-rater reliability: We recruited 67 reviewers from 24 countries. To maintain a high interrater reliability (agreement between reviewers) a structured training protocol was developed. Four required group training sessions are held per systematic review: before beginning title and abstract, midway through title and abstract, before beginning full text, and midway through full text (Figure 1). Individual training sessions were held for reviewers with frequent errors and those “promoted” to a second or conflict resolver roles. 2) Duplicate articles: Covidence’s automatic deduplication software misses many duplicates. We created Excel formulas to identify and eliminate duplicates, reducing redundant work. 3) Including all languages: Much literature on this topic is not written in English. Excluding these articles can bias our criteria, limiting its generalizability outside of English-speaking settings. Leveraging our international collaborative network, our team chose to review all articles, regardless of language. Once the described solutions were implemented, chi squared was used to compare results before and after interventions described.

Results: To date, we held 27 group and 50 individual training sessions. Before individual training, 7.7% (n=3048/36395) of articles had disagreement among reviewers. After individual training, disagreement decreased to 6.5% (n=532/8077, p< 0.0001). Prior to implementing a systematic deduplication process, our team identified 57 duplicates among 15151 articles (0.3%). After implementing this process, we identified 510/24292 (2.1%, p< 0.0001). To date, we identified and screened 91 papers in languages other than English, including Spanish, Portuguese, German, Chinese, Korean, and others.

Conclusion: This study describes barriers and identified solutions to running large systematic reviews. Our structured training protocols, systematic deduplication, and language inclusion optimizes workflow, increases reviewer productivity, and maximizes applicability of results. This approach can improve conduct of large systematic reviews to facilitate creation of high-quality generalizable findings.