Implementation of KIDMATCH: A Clinical Decision Support Tool for Diagnosing Pediatric Patients with Multisystem Inflammatory Syndrome and Kawasaki Disease

Abstract

Multisystem inflammatory syndrome in children (MIS-C) is a novel disease identified during the COVID-19 pandemic that may lead to cardiac dysfunction or death in pediatric patients. Early detection of MIS-C remains a challenge given the lack of a diagnostic test and its clinical similarities to Kawasaki disease (KD) and other acute childhood illnesses. We developed and validated the KawasakI Disease vs Multisystem InflAmmaTory syndrome in CHildren (KIDMATCH) clinical decision support tool for screening patients for MIS-C, KD, or other febrile illnesses. Here we describe the implementation and iterative refinement of KIDMATCH with provider feedback as a web calculator in the clinical workflow within Rady Children’s Hospital. Our findings demonstrate KIDMATCH and its underlying artificial intelligence model have clinical utility in aiding clinicians at the time of initial evaluation within the hospital setting to distinguish patients who have MIS-C, KD, or other febrile illnesses.

Introduction

Multisystem inflammatory syndrome in children (MIS-C) is a novel inflammatory disease identified during the COVID-19 pandemic that generally occurs in pediatric patients 2-6 weeks following SARS-CoV-2 infection^1–6. Consequences of MIS-C include multisystem inflammation, cardiac dysfunction, and death. Cases of MIS-C have occurred in patients who had prior infection with any of the variants of concern, suggesting that MIS-C will likely remain a potential serious complication for pediatric patients infected with SARS-CoV-2. Furthermore, MIS-C shares similar characteristics to other pediatric infectious and inflammatory diseases, especially Kawasaki disease (KD) with which it may share many of the characteristic five clinical signs of rash, conjunctival injection, changes in lips or oropharyngeal mucosa, cervical lymphadenopathy, and changes in peripheral extremities. Neither MIS-C nor KD have a laboratory diagnostic test, necessitating sole reliance on clinical judgment to make an accurate diagnosis.

As MIS-C and KD diagnosis presents a formidable challenge to frontline clinicians during the COVID-19 pandemic, we developed the KawasakI Disease vs Multisystem InflAmmaTory syndrome in Children System (KIDMATCH) to distinguish between MIS-C, KD, and other febrile illnesses with similar phenotypes⁷. KIDMATCH consists of a two-stage algorithm designed to first discriminate between non-MIS-C and MIS-C patients and then classify if the non-MIS-C patients have another similar febrile illness or KD. KIDMATCH’s inputs are age, the five characteristic KD clinical signs, and the results from 5 laboratory tests (complete blood count, comprehensive metabolic panel, C-reactive protein, erythrocyte sedimentation rate, and gamma-glutamyl transferase) that are commonly acquired for pediatric patients in both inpatient and outpatient settings. KIDMATCH also includes a conformal prediction framework⁸ designed to reduce false alarms and improve model confidence by flagging unfamiliar samples and marking them as indeterminate rather than make a prediction. To provide context to the risk scores calculated by KIDMATCH, we inserted a module to generate the Shapley values⁹ and return the importance of features to users. Together, these components were designed to support the intended use of KIDMATCH as a clinical decision support tool to aid in the diagnosis of MIS-C, KD, and other febrile illnesses at the time of hospital presentation by alerting clinicians to suspected, at-risk patients.

In this pilot feasibility and refinement study, we evaluated the implementation and use of KIDMATCH in the emergency department (ED) and intensive care unit (ICU) at Rady Children’s Hospital. This study integrated user-centered design and implementation science principles to optimize the fit between the KIDMATCH features and provider user preferences and current clinical workflows based on research demonstrating the value of clinical input in the development of AI-based tools^{10, 11}.

Methods

Study Population for KIDMATCH Development

As previously described7, we trained KIDMATCH on 1517 patients diagnosed with MIS-C (n=69), KD (n=775), or other febrile illnesses (n=673) at Rady Children’s Hospital. We added MIS-C patients from Children’s Hospital of Los Angeles (n=50) and Connecticut Children’s Hospital (n=16) for a total of 135 MIS-C patients during internal validation. MIS-C patients were enrolled from May 14, 2020 to June 18, 2021 and defined according to the CDC case definition¹². No MIS-C patients had received a SARS-CoV-2 vaccine and all MIS-C patients had positive antibody testing for either the nucleocapsid or spike protein of SARS-CoV-2. KD patients met the case definition of the American Heart Association¹³ for either complete or incomplete KD. All KD subjects were enrolled from 2009-2019 prior to the pandemic to avoid the potential for misclassification and were diagnosed and treated by one of two highly experienced KD clinicians (A.H.T. and J.C.B.). Subjects with other febrile illness (FC) were also enrolled from 2009-2019 and met the following case definition: previously healthy child with fever for at least 3 days plus at least one of the clinical criteria for KD. Over 50% of the FC had a clinical suspicion for KD and were referred for evaluation. The final diagnoses for the FC were adjudicated 2-3 months after enrollment by two experienced pediatric clinicians who reviewed the clinical outcomes in the medical record and all available test results. Written consent or assent as appropriate was obtained from parents and subjects, and the study was approved by the Institutional Review Boards (IRB) of the University of California San Diego, Children’s Hospital Los Angeles, and Connecticut Children’s Hospital.

KIDMATCH Development and Use

Age, the five characteristic KD clinical signs (rash, conjunctival injection, changes in lips or oropharyngeal mucosa, cervical lymphadenopathy, and changes in peripheral extremities), and 17 laboratory measurements were chosen as features for KIDMATCH based on clinical consultation. The laboratory measurements were as follows: white blood count, age-adjusted hemoglobin, platelets, percentages of neutrophils, bands, lymphocytes, atypical lymphocytes, monocytes, and eosinophils, absolute neutrophil count, absolute band count, erythrocyte sedimentation rate (ESR), C-reactive protein (CRP), alanine aminotransferase (ALT), gamma-glutamyl transferase (GGT), albumin, and sodium. Laboratory tests and clinical signs were recorded at the time of initial evaluation and prior to treatment for all patients.

KIDMATCH consists of two feedforward neural networks trained in TensorFlow, a conformal prediction framework to flag and reject out-of-distribution samples⁸, and a function to calculate the Shapley values⁹ for a test sample. All components were written in Python 3.7. We used Streamlit, an open-source framework for building web applications, to construct the KIDMATCH System which is a web calculator that incorporated the TensorFlow trained network components in the backend. The KIDMATCH System was deployed on a virtual machine (Single 64-bit CPU, CentOS Linux 7.9.2099 Operating System, 2GB RAM) hosted on the internal servers of Rady Children’s Hospital.

In the clinical workflow at Rady Children’s Hospital, clinicians proceed with the standard of care for pediatric patients by conducting a physical examination and ordering laboratory tests. If the clinician has a suspicion of MIS-C or KD in a patient, they access the web calculator via a private URL and manually enter the feature values. This workflow was chosen because of how input features to KIDMATCH are acquired and lack of integration with the electronic health record on initial deployment. Clinicians use the risk score and feature ranking from KIDMATCH and other elements from the patient’s full evaluation to make the final diagnosis. After using the calculator, clinicians submit feedback to continuously improve the user display and core functionalities of KIDMATCH.

Clinical Feedback Procedures

We recruited four practicing clinicians (M.A.G., J.T.K., A.H.T., J.C.B.) to use KIDMATCH in their evaluation of suspected patients admitted to Rady Children’s Hospital. M.A.G. and J.T.K. are emergency medicine physicians who conduct the initial evaluation of suspected MIS-C or KD patients. A.H.T. and J.C.B. are pediatric infectious disease clinicians with a 15 and 40-year track record of caring for KD patients, respectively. Together, these clinicians screen and confirm the diagnosis for the majority of MIS-C and KD patients. Feedback was collected from these clinicians through an email with an open-ended question about suggested improvements after 4 weeks of using KIDMATCH, a survey after every use of KIDMATCH, and a follow-up implementation survey 6 months post deployment.

Clinical implementation was guided by the Exploration, Preparation, Implementation, Sustainment (EPIS) framework, a commonly used implementation science framework suitable for planning and evaluation efforts across different health contexts, interventions, and conditions^{14, 15}. For this analysis, we focused on the relevant inner context (clinical providers) and innovation (KIDMATCH) characteristics within the Preparation and Implementation phases to assess implementation determinants and innovation usability. After each use of KIDMATCH, providers were asked to fill out a survey inquiring about the time to use KIDMATCH, difficulty of interpreting KIDMATCH results, and any challenges encountered during use. The implementation survey consisted of a combination of questions from the Feasibility of Intervention Measure (FIM), Acceptability of Intervention Measure (AIM), Perceived Characteristics of Innovation Scale (PCIS), and Technology Acceptability Measure (TAM) as well as open-ended questions about the most appealing and challenging features of KIDMATCH and suggested improvements. The FIM and AIM assess the extent to which an innovation can be successfully used within a specific setting and the perception among providers that the innovation is palatable, respectively. The PCIS assesses the views of users on an innovation being implemented¹⁶. The TAM assesses the relationship between behaviors and attitudes of stakeholders towards use of a technology17. Ratings for the implementation survey were as follows: 1 – Strongly disagree, 2 – Disagree, 3 – Neutral, 4 – Agree, 5 – Strongly agree.

Results

Clinical Implementation

The initial KIDMATCH prototype included numerical fields for age and the 17 laboratory measurements and radio buttons for the five clinical signs and the type of blood count differential (Figure 1).

Figure 1.

Initial prototype of the KIDMATCH web calculator user interface with random values. Users manually entered values into the (A) user interface and selected the “Predict” and “Determine Feature Importance” buttons to (B) generate a risk score and (C) view a ranked chart of the feature importances to the risk score.

All fields had to be manually entered by a clinician from the top of the screen to the bottom after the laboratory results appeared in the electronic health record (EHR) for suspected patients. Afterwards, the clinician selected the “Predict” button to generate the risk score and then selected the “Determine Feature Importance” button to generate the Shapley values and view a bar graph with the most important features having the highest impact on the model risk score. If the conformal prediction framework rejected the sample, a warning message appeared after “Predict” button was selected and no risk score was calculated.

Some of the challenging cases KIDMATCH encountered following its deployment in January 2022 at Rady Children’s Hospital and the outcomes are described below:

Case 1: A 3-year-old whose cousin had a history of MIS-C presented with four days of fever, vomiting, and reported bloodshot eyes approximately four weeks following a known COVID-19 infection. The patient was well appearing with no focal findings on examination (his bloodshot eyes had resolved). Given the history, the treating clinician had suspicion for MIS-C versus an unrelated viral illness and labs were obtained. The laboratory tests came back indeterminate with CRP elevated at 3.5 mg/dL, WBC 11.0 TH/uL with 15% bands, ESR 34mm, and SARS-CoV-2 IgG antibody positive. The ED provider ran KIDMATCH in real-time, which predicted FC, and the patient was sent home as the provider was able to confidently exclude MIS-C and assumed he was a febrile child with a viral illness. KIDMATCH eliminated MIS-C as a potential diagnosis and enabled earlier discharge. On chart review, patient did not report exacerbated symptoms and did not return for care to the ED or primary pediatrician over the following two weeks.

Case 2: A 2-year-old male presented to the ED with 3 days of fever and a large neck mass was evaluated for possible KD. The patient had clinical evidence of pharyngitis and laboratory testing revealed impressively elevated inflammatory markers including WBC 12.2 TH/uL with 23% bands, CRP 26.7 mg/dL, ESR 37 mm as well as anemia with hemoglobin of 9.3 g/dL, findings that can all be seen in KD and MIS-C. Imaging of his neck revealed pharyngitis with retropharyngeal edema and he was admitted to the hospital and diagnosed with bacterial adenotonsillitis and pharyngitis. The ED provider ran KIDMATCH in real-time which correctly identified the patient as FC based on initial ED laboratory values and clinical characteristics. This result led the ED provider to diagnose bacterial lymphadenitis that required antibiotic treatment. An ED provider less experienced in evaluating sick children may have had significant difficulty interpreting the clinical and laboratory data in this patient. KIDMATCH predicting FC in patients with abnormal labs and an unclear clinical history may help clinicians expand their differential diagnosis beyond KD to look for other treatable causes of the patient’s illness.

Case 3: Patient presented to the ED for a second visit in 2 days with five days of fever with vomiting, diarrhea, and respiratory symptoms. He had no KD-like features by history or on exam. The initial evaluating clinicians did not consider MIS-C in their differential diagnosis due to the lack of KD-like features and no reported COVID-19 illness. Laboratory values returned showing pancytopenia with WBC 3.1 TH/uL, hemoglobin 6.5 g/dL, and platelets of 132 TH/uL, and elevated CRP of 20.9 mg/dL. Given the pancytopenia, an evaluation for oncologic processes was undertaken and admission was planned. After 7.5 hours of evaluation in the ED, a second provider expanded the evaluation and elicited a history of COVID-19 exposure 4 weeks prior (though patient was not sick at that time). Given this, when SARS-CoV-2 IgG antibody returned positive, sodium returned at 131 mg/dL, and oncologic processes were excluded, the patient was admitted and treated for MIS-C. He subsequently developed cardiogenic shock related to MIS-C and was treated in the PICU with vasopressor support, with initial echocardiogram revealing significant cardiac dysfunction with left ventricular ejection fraction of 32%. Retrospective performance of KIDMATCH using the patient’s presenting lab values and clinical features accurately diagnosed MIS-C. If performed routinely in real-time, KIDMATCH may have been able to provide useful clinical prediction to allow for more rapid diagnosis and treatment of MIS-C, which may have prevented the cardiogenic shock.

KIDMATCH Refinement

After being deployed for 4 weeks, we reviewed the qualitative written feedback from clinician collaborators who were part of our initial usability testing activities. The qualitative feedback was collected after use of the web calculator in the ED and ICU of Rady Children’s Hospital and used to refine the features and usability of KIDMATCH. Through a thematic analysis of the written feedback, the following themes and clinical recommendations were derived using the following general themes in feedback from clinician collaborators at Rady Children’s Hospital (Table 1).

Table 1.

Primary themes from clinician usability testing

1) Perform all calculations automatically

2) Modify the design to reflect the clinical thought process (e.g., physical evaluation followed by laboratory tests)

3) Add details about the algorithm

4) Incorporate a method to save results for retrospective audit and analysis.

These clinical recommendations were implemented into the current working prototype (Figure 2).

Figure 2.

Working prototype of the KIDMATCH web calculator. Values are entered into the (A) user interface and (B) after selecting the “Predict” button the user has the option of determining the feature importances for either MIS-C or FC/KD patients, viewing the KIDMATCH facts sheet, and saving the KIDMATCH results.

Similar to the initial prototype, users manually enter the values for each patient before selecting the “Predict” button (Figure 2A). The other KIDMATCH components appear after the “Predict” button is selected in the working prototype (Figure 2B). Users can obtain the feature importances for a patient classified as MIS-C by selecting the “Determine MIS-C Feature Importance” button and do the same for FC and KD patients by selecting the “Determine KD Feature Importance” button. The following changes were made for the working prototype as a result of the clinical recommendations:

Recommendation 1: Every field can be directly entered from the EHR manually into the working prototype without the need for any calculations, reducing user fatigue. We added additional fields for the Medical Record Number and the date of birth to enable KIDMATCH to save the risk scores to a CSV file on the virtual machine and created a function to calculate age instead of having clinicians perform the calculations. We removed the fields for absolute neutrophil count and absolute band count as these values could be calculated automatically.

Recommendation 2: To make the web calculator more intuitive, the data entry fields should reflect the clinician decision making process. Clinicians typically conduct the physical exam first before assessing laboratory values, so we moved the radio buttons on top of the laboratory fields. We also changed the interface to accept empty fields for missing values instead of a placeholder.

Recommendations 3 and 4: We added a model fact sheet that details the intended use of the model and how the model was developed as a viewable PDF (Figure 3) and a button to save the results (Figure 2B).

Figure 3.

KIDMATCH Model Fact Sheet

The current version of the web calculator (Figure 2) still requires clinicians to manually enter lab values, but since the web calculator is utilized only once for suspected patients, the current implementation provides more benefit by calculating informative risk scores than overloading clinicians. Results from the post-use survey collected over 6 months indicated that most results generated by KIDMATCH were easy to interpret (21/28, 75%) compared to confusing (7/28, 25%). Despite manual data entry, most clinicians reported taking 1-3 minutes to use KIDMATCH (16/28, 57%) compared to 3-5 minutes (9/28, 32%). A few responses took longer than 5 minutes (3/28, 11%) due to technical difficulties with loading the interface. After 6 months of deployment, we conducted an implementation survey to assess the usability and utility of KIDMATCH (Table 2).

Table 2.

Ratings from clinician users (n=4) to an implementation survey conducted 6 months post deployment. Ratings were on a scale of 1 (strongly disagree) to 5 (strongly agree). SD: standard deviation

Question	Mean	SD
KIDMATCH is implementable.	4.5	0.58
KIDMATCH is easy to use.	4.5	0.58
KIDMATCH is appealing to me.	4.25	0.96
Using KIDMATCH fits well with the way I like to work.	4	0.82
KIDMATCH is aligned with my clinical judgment.	4.25	0.5
KIDMATCH is clear and understandable.	4.5	0.58
Using KIDMATCH improves the quality of work that I do.	4	0.82
Using KIDMATCH makes it easier to do my job.	3.5	1
I find KIDMATCH useful in my job.	4.25	0.96
Learning to operate KIDMATCH is easy for me.	4.75	0.5
I find it easy to get KIDMATCH to do what I want it to do.	4.25	0.5

All but one question had an average rating of 4 (agree) or above. The highest rating was the ease of learning how to operate KIDMATCH followed by the clarity and ease of using KIDMATCH as implemented. The lowest rating with an average of 3.5 addressed the question of whether KIDMATCH made it easier to conduct clinical duties. In addition to the numerical ratings, the survey included open-ended questions about the most appealing and challenging features of KIDMATCH. Broadly, clinicians found the use of AI to analyze objective clinical and lab data to form a probability for a single disease state with explanations appealing. The most challenging feature was accessing KIDMATCH. Since the URL for KIDMATCH was not available within the EHR, users had to access a separate non intuitive link that was difficult to remember and find. Suggestions for improvement reiterated the request for automatic data entry and emphasized the need for an easier method to access the interface.

Discussion

We present an overview of the development and refinement process of implementing KIDMATCH into the clinical workflow at Rady Children’s Hospital to aid in the diagnosis of MIS-C and KD patients. When KIDMATCH has been used in the clinical workflow for suspected patients, it has clinical utility for patients by enabling earlier discharge for febrile patients who do not require intensive monitoring as observed in the case reports. It also could lead to earlier treatment for MIS-C or KD patients and prevent serious complications like cardiogenic shock or coronary artery aneurysms. Based on clinician feedback through open-ended questions for suggested improvements, creating automated functions to remove the burden of manual data entry and error is a key priority for acceptance and usability within the clinical workflow. Another critical point is ease of access. Although KIDMATCH is deployed on an internal virtual machine, accessing it from within the EHR is currently not possible. These areas of improvement impact the perceived use of KIDMATCH in making the clinician’s job easier as assessed through clinician ratings in the 6-month post deployment implementation survey. However, this survey demonstrated that when KIDMATCH is accessed, users find it easy to learn how to operate and understand and agree with the statement that it aligns with clinical judgment. These findings suggest that if we can improve the accessibility and automation of KIDMATCH, clinicians will be more inclined to use the tool to support their decision making.

Our findings are limited by the application of KIDMATCH to patients where clinicians have a clinical suspicion of MIS-C or KD, are uncertain about the diagnosis, and remember to use KIDMATCH after the initial clinical evaluation. As a result, we logged 28 patients over 6 months. This will be expanded in the future by designing an automated trigger within the EHR to notify clinicians when KIDMATCH may be used deployed for a patient based on their values. Additional future work includes pre-populating input fields by using Fast Healthcare Interoperability Resources (FHIR) API calls directly to the EHR and embedding the KIDMATCH App into the EHR using a SMART-on-FHIR design. Since the EHR vendor for Rady Children’s Hospital is Epic, we can utilize Epic’s App Orchard as an alternative deployment method to other healthcare systems instead of distributing the virtual machine. We are also in the process of evaluating the generalizability of KIDMATCH at different external hospitals by collecting data from diagnosed MIS-C, KD, and FC patients and validating the algorithm. In summary, post-implementation feedback from clinicians revealed specific EPIS inner context and innovation factors (e.g., automated data entry solutions, accessible interface features) that can be addressed for optimization in subsequent KIDMATCH implementation efforts. Besides existing earlier changes that were made to improve the user interface, disseminating information about the intended use of KIDMATCH amongst pediatric emergency medicine providers and increasing automation will guide our efforts to improve patient outcomes and address provider concerns. Finally, our findings reinforced the importance of nimble and responsive interactions between clinicians and AI designers so that KIDMATCH adoption remains high and clinicians continue to find value in their practice and decision-making.

Conclusion

Our study provides preliminary evidence for the clinical and implementation benefits of KIDMATCH at Rady Children’s Hospital by aiding clinicians in distinguishing patients who have MIS-C, KD, or other febrile illnesses at the time of initial evaluation. KIDMATCH has resolved complex cases during the time of hospital presentation, reducing the length of hospital stays for febrile children and enabling appropriate treatments to be administered earlier for MIS-C and KD patients. Aligned with user-centered design and implementation science thinking, clinician input has proven valuable in the iterative refinement of KIDMATCH’s user interface to optimize usability and adoption. By implementing KIDMATCH within a virtual machine that is deployed within the hospital network firewalls, we maintain patient privacy while having the ability to scale by packaging the virtual machine for distribution. Ongoing work includes connecting KIDMATCH directly to the EHR to enable the automatic loading of required data elements and a prospective validation study to evaluate the generalizability and clinical utility of KIDMATCH.