Improving Infectious Adverse Event Reporting for Children and Adolescents Enrolled in Clinical Trials for Acute Lymphoblastic Leukemia: A Report from the Children’s Oncology Group

Abstract

Infections cause substantial morbidity for children with acute lymphoblastic leukemia (ALL). Therefore, accurate characterization of infectious adverse events (AEs) reported on clinical trials is imperative to defining, comparing, and managing safety and toxicity. Here we describe key processes implemented to improve reporting of infectious AEs on two active phase III Children’s Oncology Group (COG) ALL trials. Processes include: (1) identifying infections as a targeted toxicity, (2) incorporation of infection-specific case report form questions, and (3) physician review of AEs with real-time data cleaning. Preliminary assessment of these processes suggests improved reporting, as well as opportunities for further improvement.

Background

Infections are a frequent and leading cause of morbidity and mortality in children receiving intensive therapy for acute lymphoblastic leukemia (ALL).^1–5 Rates of microbiologically documented infection are as high as 60% over the course of therapy, and the cumulative incidence of infection-related mortality ranges from 1–3%.^5,6 Given this burden, comprehensive and accurate capture of infectious adverse events (AEs) on clinical trials is critical for identifying the relative safety and tolerability of treatment regimens, thereby informing medical decision making by physicians, patients, and families.

Currently, adverse events (AEs) are ascertained and reported using manual processes that vary by site, but in general include chart review by a clinical research associate or research nurse (CRA/RN) to identify AEs, confirmation of AEs with site investigators, and then entry of confirmed AEs into case report forms (CRFs) within the clinical trial electronic data capture (EDC) system. On most Children’s Oncology Group (COG) clinical trials, AE names and grades are guided by the National Cancer Institute Common Terminology Criteria for Adverse Events (CTCAE). Despite this significant manual effort, the current process for AE evaluation and reporting leads to unsatisfactory results.^7–9

Several barriers to comprehensive and accurate AE reporting have been identified. When surveyed, more than half of CRA/RNs additionally report challenges matching clinical documentation to CTCAE terminology.¹⁴ In light of this, Miller et al. found that only two thirds of AEs could be graded according to CTCAE definitions by a CRA/RN without guidance from an attending physician. However, 53% of CRA/RNs reported challenges in engaging treating clinicians or site primary investigators to assist with AE reporting, and these clinicians often have not received training in AE reporting.¹⁴ As a result, trials underestimate^10,11 and misreport^12,13 toxicity leading to an inaccurate understanding of the risks conferred by the agents or chemotherapy regimens under investigation.

This report describes a system that has been developed to improve the accuracy and completeness of infectious AE data on two currently-enrolling frontline COG phase III trials for B-cell ALL: AALL1731- “A Study to Investigate Blinatumomab in Combination with Chemotherapy in Patients with Newly Diagnosed B-Lymphoblastic Leukemia (NCT03914625)” and AALL1732- “Inotuzumab Ozogamicin and Post-Induction Chemotherapy in Treating Patients With High-Risk B-ALL, Mixed Phenotype Acute Leukemia, and B-LLy (NCT03959085)”. This process incorporates the use of “targeted toxicities,” enhanced detailed CRF questions for infectious AEs, and twice monthly physician review of reported infectious AEs with real-time data cleaning in partnership with site CRA/RNs.

Processes to improve infectious AE capture on ALL trials

Infectious AE capture as a “targeted toxicity”

The COG ALL Toxicity Reporting Task Force developed a list of targeted toxicities that should be collected on all COG ALL trials, based on international consensus definitions of informative toxicities for patients with ALL.¹⁷ Employing a finite list of reportable AEs is consistent with guidance from the American Society of Clinical Oncology (ASCO) that recommends streamlining AE reporting to focus time and effort on serious and clinically relevant AEs and to limit heterogeneity across trials.¹⁸ This approach also increases the yield of toxicity reporting. Prior studies have shown that specification of AEs as targeted toxicities increases the rates of reporting of those toxicities, even after administration of the same chemotherapy regimen.^10,19 Therefore, since 2010 a standard list of AEs has been incorporated into all ALL trials conducted through COG. Beginning in 2017, this list was appended to include grade 3 or higher infectious AEs.

Incorporation of infection-specific questions on the AE CRF

To assist with capturing granular data on infections experienced on the trials, beyond what is included in the CTCAE definitions,²⁰ the AALL1731 and AALL1732 CRFs require supplemental data elements related to infectious AEs (including AEs with a potentially infectious etiology such as febrile neutropenia). Similar CRF questions have been used on other leukemia trials within the COG leading to a positive impact on AE reporting. For example, when the AAML0531 trial (NCT01407757) used a more detailed infectious CRF compared to the pilot study AAML03P1 (NCT00070174) that tested the identical chemotherapy regimen without infection-specific CRF questions, there was a substantial difference in rates of bloodstream infections reported between the two trials.¹⁰ For the current ALL trials, the supplemental questions have been revised from past trials that queried the presence of a “sterile-site infection” to asking whether there was a causative organism and, if so, the organism’s name and the source (e.g. blood, soft tissue, cerebrospinal fluid) for ease of interpretation by CRA/RNs.

Monitoring of infectious AEs by a central review committee with real-time data cleaning

In alignment with the ASCO policy statement that promotes central review as an effective measure for improving AE monitoring,²¹ the AALL1731 and AALL1732 trials have established an infection review committee composed of four pediatric hematology/oncology physicians. COG statisticians from each study generate reports twice monthly of infectious AEs that include subject ID numbers, site information, lead CRA/RN contact information, report entry date, CTCAE term, date of AE, grade of AE and answers to the infection specific CRF questions. AE entries are individually reviewed for completeness and clinical plausibility by one member of the infection review committee. Clarifying queries or prompts for corrections are emailed to the site’s lead CRA/RN with a discussion of relevant CTCAE definitions/criteria. These emails include references to reporting guidelines and internal standardized algorithms designed by the COG Toxicity Task Force to guide reporting that are pertinent to the clinical event (example email text is provided in Table 1). Subsequent reports are reviewed to track AEs that were updated in response to prompts, with a second email sent if queries are unanswered one month later. Tracking logs are maintained of all AE queries and their outcomes.

Table 1:

Common errors in adverse event reporting identified through prospective real time review and example query sent by infection review committee

Theme	Example AE reporting error	Demonstrative query email text
Incomplete CRF	Infection-specific CRF questions not completed or partially completed	Your site recently submitted an adverse event report on [DATE] for adverse event of soft tissue infection on [DATE] for [COG ID] ([REPORTING PERIOD]).  We see that the question of if an organism was identified was answered as “not applicable.”  All infectious and febrile neutropenia adverse events should have this question answered.  Can you please update this and answer this question?
Clinically implausible	Organism and source do not match (e.g. causative organism of clostridium difficile with source listed as blood)  AE designation and source do not match (e.g. sinusitis with positive blood culture)  AE reported with specific name that requires an organism to be positive and listed that no organism was identified (i.e. bacteremia reported as having no organism identified)	Your site recently submitted an adverse event report on [DATE] for an adverse event of viremia occurring on [DATE] for [COG ID] ([REPORTING PERIOD]).  Thank you for reporting this event.  It is listed that the patient had a positive organism of Respiratory syncytial virus from a nasopharyngeal swab. Viremia is defined by the presence of a virus in the blood stream.”  Unless there was also a positive virus from the blood, we would recommend changing this AE from viremia to upper respiratory infection.
Malalignment with CTCAE definitions	Febrile neutropenia with positive blood culture  Catheter-related infection with a positive blood culture  Sepsis selected without a causative organism in the bloodstream1  “Infections and Infestations, Other – specify” chosen instead of specific AE that exists (i.e. “pneumonia” listed as the other when CTCAE option for lung infection exists	“Your site recently submitted an adverse event report on [DATE] for an adverse event of febrile neutropenia on [DATE] for [COG ID] ([REPORTING PERIOD]). We see that the patient had a blood culture that was positive.  Given the positive blood culture, we would recommend reporting this as sepsis rather than febrile neutropenia.  In CTCAE v5, Grade 3 sepsis is defined as “Blood culture positive with signs or symptoms; treatment indicated.” Fever is considered a sign/symptom and therefore this event fulfills the definition of sepsis. See the attached reporting algorithm from the COG toxicity task force.

¹The CTCAE definition of “sepsis” (MedDRA 10040047) is defined as “a disorder characterized by the presence of pathogenic microorganisms in the blood stream that cause a rapidly progressing systemic reaction that may lead to shock”

Primary responsibility for AE review falls to one review committee member each month; the physician reviews AEs reported during the assigned month and the previous month such that every infectious AE is reviewed independently by two physicians. The infection review committee convenes monthly teleconferences to review challenging AE entries and to align approaches to recurring scenarios. Additional ad hoc conversations occur by email to adjudicate reporting guidance as needed. In addition, this committee is available for proactive queries as to how to report specific clinical scenarios, providing an additional level of clinical support to CRA/RNs to supplement support from local clinicians.

Interim process assessment

Interim assessment of the central AE review and real-time data cleaning is shown in Figure 1. Adverse event entries from the time of study activation (June 28, 2019 for AALL1731 and October 28, 2019 for AALL1732) until September 1, 2021 were included and results of responses to clarifying questions were examined as of November 1, 2021. A total of 2,260 infectious AEs were reported over this period, and clarifying queries were sent for 613 (27.1%) infectious AEs. Overall, 488 (79.6%) queries resulted in an updated or corrected AE entry. Of those that were not updated, the infection review committee received no response to 59 (9.6%). Another 41 (6.7%) queries received a response containing additional clinical information and led to a consensus between the reporting site and infection review committee that no change was required. Congruent with increasing enrollment, Figure 1 demonstrates that the absolute number of infectious AEs reported per month has increased over the course of the two trials. The number of queries sent has also increased over time, but at a slower rate, indicating that the proportion of AEs that are reported correctly is increasing over time. There is month to month variability in the total number of AE reports that appear to correspond to quarterly monitoring dates.

Figure 1: Infectious adverse event reporting by month on frontline Children’s Oncology Group acute lymphoblastic leukemia trials (AALL1731 and AALL1732).

The start of infectious AE reporting and subsequent queries lagged two and three months behind study activation, respectively. The total number of infectious AEs reported varies by month but has increased over the study period (solid line), with corresponding increases in the number of AEs that receive a query from the infection committee (dashed, dark grey) and the number that are updated in response to those queries (dashed, light grey). Month to month variability in AE reports correspond to quarterly monitoring dates.

The length of time between the AE and its reporting, did not seem to impact engagement of sites in the review process. For adverse events reported within 30 days of the event, 89% of queries received a site response or update and for AEs reported more than 30 days from the event, 86% of queries were acknowledged.

We have identified several common themes in the reporting errors that are identified through this process including incorrect CRF completion, clinically implausible events, and malalignment with CTCAE definitions. Representative examples of errors in reporting and subsequent query emails from the infection review committee are shown in Table 1. The most common queries were related to the reporting of febrile neutropenia, bacteremia, sepsis and COVID-19 infections.

Discussion

Accurate capture of AEs on clinical trials is critical to defining the risks of toxicity associated with treatment regimens. In this report we describe the processes instituted to optimize reporting of infectious AEs on two large ongoing COG Phase III trials, which include naming infections as targeted AEs, incorporating an enhanced CRF form for infectious toxicities, and central review by physicians with real time clinically informed data cleaning using a standardized approach. The proportion of AEs updated each month demonstrates the impact that the iterative physician central review and communication with site CRA/RNs has had on the accuracy of reported infectious AEs. Further, the reduced proportion of AE reports requiring clarification relative to the number of infectious AEs reported over the course of the studies suggests that resources, clarifications, and practical education provided to site CRA/RNs through this process is leading to a sustained improvement in reporting. We are hopeful that this system will lead to reliable and detailed toxicity profiles that will be critical to interpreting the results of these trials, identifying patients at high risk for infectious complications, and establishing an accurate baseline of infectious AEs for future comparisons.

The process currently utilized on the AALL1731 and AALL1732 trials evolved from a longitudinal effort within the COG to improve infectious AE reporting on clinical trials. Historically, CRA/RNs have provided feedback that accurate AE reporting is hindered by the ability to match CRF responses to data that are readily identifiable in the clinical chart.¹⁴ The revised infection-specific CRF questions used on these trials are designed to better reflect clinical documentation and may facilitate improved reporting moving forward. Furthermore, leveraging lessons learned from prior studies has led to development of a system in which each infectious AE is reviewed in a semi-real time manner, usually within one month of the AE report. This timeframe engages the site CRA/RN and clinical team to address queries while the clinical event is still recent and details easier to recall. Moreover, this real-time approach ensures that these infectious AE data are cleaned proximally to their reporting, increasing the reliability of these data for interim safety analyses.²²

Implementation of this system has also underscored several ongoing challenges to accurate and comprehensive AE reporting on clinical trials. While the clinically informed data cleaning based on central physician review is feasible to perform, even on these large ALL trials, this endeavor does require identification of study committee members committed to this effort with dedicated time each month. A substantial proportion of queries are related to incomplete data entry and consist of a prompt to sites to answer the infection-related CRF questions. Although our interventions are largely effective at resolving these queries, leveraging data reporting software and technology solutions within the EDC system, rather than manual review, could more efficiently address these gaps and allow for scaled efforts across more clinical trials and additional AEs of interest.

These efforts have also highlighted shortcomings of the CTCAE as a reporting framework. Closely following the CTCAE definitions can be challenging when there is incomplete or conflicting documentation of clinical events within the patient medical record, the CTCAE lacks clinically recognized entities, such as culture-negative sepsis, or when its use of AE terms differs from how such terms are used in common clinical practice.^15,16 For example, grade 3 sepsis is defined in CTCAE as a positive blood culture in the presence of any systemic symptom or sign, whereas many clinicians reserve the use of the term “sepsis” to denote clinical instability.²⁰ Further, the number of AEs defined within CTCAE has markedly increased over successive versions. As CTCAE has become increasingly granular, selecting the most precise AE term among more than 800 is understandably challenging, particularly when CRA/RNs have limited time, are reporting uncommon infectious complications, or encounter definitions included in CTCAE which may be subjective or overlapping.^13,16 Nevertheless, a common terminology is critical to intra- and inter-study consistency and comparisons. Thus, despite its challenges, following closely the definitions and grading in CTCAE remains a foundation of accurate AE capture. To this end, the COG Toxicity Task Force has developed and disseminated job aids and trainings for CRC/RNs on CTCAE-concordant AE reporting. Infectious AE decision algorithms are available on the COG website that guide CRC/RNs to the most accurate and precise CTCAE term and grade for common clinical scenarios. Efforts are underway to expand this type of resource, provide additional CRC/RN training, and study the impact of these interventions on AE reporting. In addition, prospective monitoring by a central review committee as implemented on AALL1731 and AALL1732 can provide this additional support to CRA/RNs to aid in mapping a complex clinical scenario to the most precise and parsimonious CTCAE definition.

Even with these clarifying processes in place, reporting remains imperfect. For one, while this described process appears highly effective at clarifying and standardizing AE reporting, it does not increase the sensitivity of AE capture. Because the review process targets reported AEs but does not directly interface with patient charts, it does not address AEs that may be missed in the chart or not inputted into the EDC system. Ongoing efforts to develop “trigger” tools that leverage EHR laboratory or medication data,^12,23 natural language processing,^24,25 or administrative data-based algorithms²⁶ to identify potential AEs directly from the clinical chart will be complementary to efforts like these. In addition, it is important to acknowledge that this central review process is not completely objective and remains vulnerable to the interpretation of the review team. While we cannot eliminate this subjectivity entirely, the consensus approach adopted by the four members of the review committee reduces potential for subjectivity and will, at a minimum, help standardize how the CTCAE is interpreted and how clinical events are captured across individual CRA/RNs and study sites. Clear documentation around these decisions will allow for reproducibility on subsequent trials, enabling accurate comparisons of infectious toxicities across treatment regimens.

The reporting of other complex AEs could benefit from a process similar to the one developed for infectious AEs on AALL1731/1732. Leveraging automated solutions for efficient and reliable AE capture and further harmonizing reporting expectations across trials will provide additional study team capacity for efforts focused on improving reporting of more complex AEs. The specific complex AEs that could be targeted for a real time review process would be informed by trial aims and design and could include neurotoxicity, cytokine release syndrome and pain.

Despite these limitations, the processes in place on these two large COG trials should continue to improve infectious AE reporting and enhance our understanding of the infectious toxicities associated with the treatment for pediatric ALL. Building on the success of this process for AALL1731 and AALL1732, a parallel process is now in place for the active COG phase III trial for AML, AAML1831 (NCT04293562). Developing a sustainable system to expand these efforts to other COG trials will be important and is a key objective of the COG Toxicity Task Force. The steps outlined in this report can provide a roadmap for implementation of parallel processes to capture infectious AEs on other clinical trials within the COG or other clinical trials groups. Moreover, analogous processes could be developed to monitor or grade other complex AEs with the goal of comprehensively defining the risk profiles associated with the treatment of pediatric cancers.

Abbreviations

AE

Adverse Event

ALL

Acute Lymphoblastic Leukemia

ASCO

American Society of Clinical Oncology

COG

Children’s Oncology Group

CRA/RN

Clinical Research Associate or Research Nurse

CTCAE

Common Terminology Criteria for Adverse Events

EDC

Electronic Data Capture