The diagnosis of respiratory disease in children using a phone-based cough and symptom analysis algorithm: The Smartphone Recordings of Cough Sounds 2 (SMARTCOUGH-C 2) trial design

Abstract

The diagnosis of acute respiratory diseases in children can be challenging, and no single objective diagnostic test exists for common pediatric respiratory diseases. Previous research has demonstrated that ResAppDx, a cough sound and symptom-based analysis algorithm, can identify common respiratory diseases at the point of care. We present the study protocol for SMARTCOUGH-C 2, a prospective diagnostic accuracy trial of a cough and symptom-based algorithm in a cohort of children presenting with acute respiratory diseases. The objective of the study is to assess the performance characteristics of the ResAppDx algorithm in the diagnosis of common pediatric acute respiratory diseases.

Background and Rationale

Respiratory disorders—including both upper respiratory tract disease (URTD) and lower respiratory tract disease (LRTD)—are the most common reasons for pediatric emergency room visits in the United States.¹ LRTDs including pneumonia, bronchiolitis, and asthma are a major cause of childhood severe illness and death both in the United States and globally.² Pneumonia kills an estimated 800,000 children under the age of 5 years annually worldwide, with the vast majority occurring in low and middle income countries.³

The diagnosis of respiratory diseases—especially LTRDs—in the primary care or emergency department setting can be challenging, even with access to appropriate clinical, radiological and laboratory support. There is no single objective diagnostic test for common pediatric respiratory diseases; as a result, the diagnosis relies on a combination of history, physical examination, laboratory testing, imaging, and clinical judgment.⁴ Recognizing the limited resources available for evaluation in low and middle-income countries, the World Health Organization (WHO) has developed an algorithm to diagnose pneumonia based on clinical evaluation rather than diagnostic testing.⁵ A recent systematic review found that fever and tachypnea have high sensitivity (79–92%) but low specificity (47–54%) in diagnosing pneumonia, and hypoxemia has moderate sensitivity and specificity (64% and 77% respectively).⁶ A WHO-led analysis of 10 studies comprising 15,029 children for whom radiographic diagnoses were available found that the WHO criteria for tachypnea demonstrated a sensitivity of 92% and specificity of 22% for diagnosing radiographic pneumonia, while lower chest indrawing demonstrated a sensitivity of 74% and specificity of 15%.⁷

Inaccurate diagnosis of respiratory disease can lead to delayed or inappropriate treatment, with antimicrobial overuse driving antimicrobial resistance in both individual patients and in a population.^8,9 These challenges have been magnified in the setting of the present COVID-19 epidemic, which has forced many providers to transition to telemedicine and highlighted the dependence of the healthcare system on diagnostic testing. Remote evaluations in telemedicine encounters rely on patient or parent-reported symptoms rather than physical examination findings and testing; as a result, providers frequently are more likely to mitigate risk by prescribing potentially unnecessary antibiotics.¹⁰

Respiratory diseases can cause pathophysiologic auscultatory breath sounds by creating turbulent air flow, airway resonance, airway collapse, or alveolar filling.¹¹ While these sounds are traditionally obtained during tidal or deep breathing, the forced exhalation of a cough may provide additional data on respiratory diseases. Cough analytic technology using techniques similar to speech recognition may have the potential to diagnose respiratory tract diseases.¹² Previous research has demonstrated that ResAppDx, a cough-based sound analysis algorithm, can distinguish pneumonia and croup from no disease.^13–15 A subsequent clinical study (SMARTCOUGH-C) assessed the performance of ResAppDx in a US-based clinical setting. Because of a wide range in performance across the studied indications (positive percent agreement 47–89%, negative percent agreement 47–95%),¹⁶ improvements were made based on the findings of this initial study. These improvements included refinements to the ResAppDx algorithms, expanding the focus from pneumonia as a primary outcome to including all common acute respiratory illnesses, revised and expanded clinical case definitions (including detailed criteria for each diagnosis), and improvements in the cough recording procedures. A subsequent study in Australia (Breathe Easy) demonstrated good performance of the refined algorithm among children presenting with acute respiratory illnesses.¹⁷

We present the study protocol for SMARTCOUGH-C 2, a prospective diagnostic accuracy trial of an optimized cough- and symptom-based algorithm in a cohort of children presenting with acute respiratory diseases. The objective of the study is to assess the performance characteristics of the ResAppDx algorithm in the diagnosis of common pediatric acute respiratory diseases.

Methods

Study design overview and aims

The SMARTCOUGH-C 2 study is a multi-center prospective, double-blind study comparing the diagnosis of acute pediatric respiratory diseases using the ResAppDx algorithm to clinical diagnosis determined by a committee of independent adjudicators.

Recruitment and participants

The study enrolls children age 30 days to 12 years presenting to one of three study sites (Massachusetts General Hospital, Cleveland Clinic Foundation Hospital, and Texas Children’s Hospital) with signs or symptoms of respiratory disease. Enrollment criteria are detailed in Table 1. We selected children age 30 days to 12 years given the high prevalence of acute respiratory illnesses in this age group. Wherever possible, we use World Health Organization criteria for vital sign abnormalities (hypoxemia and tachypnea) in order to maximize generalizability of our findings.⁵ Because the enrollment sites are all < 1000 ft above sea level, we classify SpO₂ < 90% as hypoxemia.¹⁸ Bronchodilator responsiveness is defined as improvement in aeration, reduction in wheezing, or reduced work of breathing after administration of a short-acting bronchodilator. At each of the study sites, we enroll children in multiple locations including primary care outpatient facilities, urgent care centers, emergency departments, and inpatient hospital wards within 24 hours of admission. The study protocol is approved by each participating institution’s institutional review board and is registered at ClinicalTrials.gov (NCT03392363). Written informed consent is obtained from the patient’s parent or legal guardian prior to the completion of any study procedures.

Table 1:

Enrollment Criteria

Inclusion criteria

Any one of the following: Cough Wheezing (continuous whistling sound during breathing reflective of lower airways obstruction) Stridor (harsh, high-pitched sound during inspiration reflective of upper airways obstruction) Chest retractions (inward movement of chest wall during inspiration) Use of accessory muscles of respiration (e.g., abdominal,sternocleidomastoid, scalene) Difficulty breathing Tachypnea Age less than 2 months: respiratory rate greater than 60 breaths/min Age 2 to 12 months: respiratory rate greater than 50 breaths/min Age 13 to 59 months: respiratory rate greater than 40 breaths/min Age 5 to 12 years: respiratory rate greater than 30 breaths/min Rhinorrhea (runny nose) Abnormal respiratory exam per treating clinician (in addition to the signs above, other abnormalities including crackles, rhonchi, pleural rub) Cyanosis/hypoxemia (oxygen saturation <90%) AND Onset of symptoms within the past 14 days AND The ability to cough spontaneously or voluntarily

Exclusion criteria

Absolute Receiving mechanical ventilatory support (invasive or non-invasive) or high flow oxygen at the time of evaluation History of structural airways or vocal cord abnormalities Medical instability per treating clinician (e.g., requiring immediate resuscitation or airway support) Tracheostomy Relative Children with a medical contraindication to voluntary cough (severe respiratory distress, history of pneumothorax, recent eye/chest/abdominal surgery, or recent hemoptysis) are enrolled only if they are coughing spontaneously.

Cough analytic technology and study procedures

After playing a brief video explaining the study procedures, a proprietary smartphone-based application is used to collect the data analyzed by the ResAppDx algorithm, including the subject’s age, sex, and patient- or parent-reported symptoms (presence of fever, wheeze, runny nose, and duration of illness). Symptom reports are elicited by direct questioning of the child and parent. The built-in smartphone microphone is used to record five spontaneous or voluntary coughs from the subject over a maximum of a one-hour period. Each cough is automatically identified using the ResAppDx proprietary cough identification algorithm. The smartphone (an Apple iPhone 6s) is held by research staff with the screen facing up and the microphone 50 cm from the subject’s mouth at 45 degrees from the path of travel of the cough airflow (Figure 1). Environmental noise is minimized as much as possible.

Figure 1:

Positioning of the smartphone device relative to the subject

After five coughs are identified by ResAppDx, the audio data is saved to the smartphone with a unique Recording ID number assigned by the software. If the subject is unable to provide a minimum of five coughs recognized by the ResAppDx, the application does not progress past the audio capture screen and the subject is excluded from further analysis. All cough recordings undergo a quality review by study staff to ensure they are free from obvious background noise, coughs by other people in the room, and identifying information (i.e., the subject’s name).

In the intended commercial format, the cough sounds and the clinical symptoms will be analyzed in real time on the smartphone using the ResAppDx algorithm to indicate whether the patient has one or more of the targeted respiratory diseases. However, in the SMARTCOUGH C-2 study to ensure blinding during clinical care, data collection, and quality review, the ResAppDx algorithm is only applied to the cough recordings and symptom data by an independent statistical analysis team after enrollment is complete and the study database is locked.

Additional data is collected directly from the subject or parent, the medical record, and the treating team including inclusion/exclusion criteria, demographic data, baseline characteristics, history of present illness, past medical history, home respiratory medications, physical examination findings, laboratory testing, and imaging. No laboratory testing or imaging is performed as part of the study procedures, but all tests and imaging obtained by the clinical providers during routine care are made available to the adjudication committees. Research staff obtain and document physical exam findings (including auscultation) as reported by the clinical provider at the time of enrollment.

Adjudication procedures

The reference standard against which the ResAppDx diagnosis will be compared is provided by independent adjudication. All cases are reviewed by a Clinical Adjudication Committee, and all chest radiographs are reviewed by a Radiology Adjudication Committee. Both committees are blinded to the ResAppDx diagnosis, and radiologic adjudicators are also blinded to the clinical history.

Case definitions for common pediatric respiratory disorders have been created by the study investigators and representatives of the sponsor based on review of the literature and society guidelines (Table 2). The endpoints are considered independently for the purposes of adjudication (i.e., a subject can be positive for multiple endpoints), though several of the endpoints are related in a hierarchical fashion (Figure 2). It is anticipated that many subjects will have multiple diagnoses (e.g., Upper Respiratory Tract Disease, Lower Respiratory Tract Disease, and Bronchiolitis would be expected to co-exist). To improve consistency in interpretation of chest radiographs, radiologic adjudicators use the WHO protocol for standardized interpretation of chest radiography for pneumonia,¹⁹ which allows five possible radiologic conclusions: (1) Primary Endpoint Pneumonia (WHO PEP), (2) Other Infiltrate (OI), (3) WHO PEP and OI, (4) No consolidation, infiltrate or effusion, and (5) Uninterpretable.^20,21 Members of both committees have been trained in the relevant case definitions and tested on standardized test subjects/images in order to ensure consistency in applying the case definitions.

Table 2:

Case definitions for clinical adjudication

Disease	Required features
Upper Respiratory Tract Disease38,39	Must have one or more of the following: Nasal congestion Rhinorrhea Sore throat
Croup40,41	Must have seal-like barking cough based on adjudicator review of cough recording
Lower Respiratory Tract Disease41–43	Must have one or more of the following: Wheezing on auscultation or silent chest (no wheezing because of severe airway obstruction in status asthmaticus) at the time of recording Any chest findings on auscultation, including crackles, bronchial breath sounds (tubular, hollow sounds associated with surrounding consolidation), or focally/asymmetrically decreased breath sounds Increased work of breathing (e.g., using accessory muscles) unless this is purely associated with stridor or upper airway limitation Chest pain or tightness not associated with musculoskeletal, chest wall, or cardiac disease Productive cough > 5 days that is not explained by upper airway disease Hypoxemia in the absence of cyanotic heart disease while awake (<90%)
Asth ma/Reactive Airways Disease44,45	Must be wheezing on auscultation or silent chest (no wheezing because of severe airway obstruction in status asthmaticus) at the time of recording Must be responsive to bronchodilators either During this illness, or During prior illnesses If subject has wheeze but no bronchodilator test is available (during this illness or during prior illness) then must diagnose as UNSURE
Viral Lower Respiratory Tract Infection43,46	Must have both of the following present: Persistent cough Diffuse crackles and/or wheeze that does not resolve with bronchodilator If less than 2 years old, and if subject has wheeze but no bronchodilator test is available (during this illness or during prior illnesses) then may diagnose Viral LRTI as YES If 2 years old or older, and if subject has wheeze but no bronchodilator test is available (during this illness or during prior illness) then must diagnose Viral LRTI as UNSURE
Bronchiolitis46	Must be less than 2 years old Must have both of the following present: Persistent cough Diffuse crackles and/or wheeze that does not resolve with bronchodilator If subject has wheeze but no bronchodilator test is available (during this illness or during prior illnesses) then may still diagnose Bronchiolitis as YES
Pneumonia23,24,41	Clinical pneumonia: must have at least one finding in both of the following categories: Reported history of fever in prior 48 hours, or fever present at the time of exam; cough, dyspnea, or chest pain Focal chest findings, including crackles, bronchial breath sounds, or focally decreased breath sounds; OR chest radiograph with new consolidation Among subjects for whom a chest radiograph is being obtained by the clinical provider: WHO Primary Endpoint Pneumonia plus Clinical Pneumonia (Radiographic Pneumonia #1) WHO Primary Endpoint Pneumonia or Other Infiltrate plus Clinical Pneumonia (Radiographic Pneumonia #2)

Figure 2:

Relationship Between Study Endpoints

Since the identification of pneumonia relies on an imperfect diagnostic test (a chest radiograph),²² three reference standards are used to reflect a range of sensitivity and specificity: (1) clinical pneumonia, a high-sensitivity diagnosis that relies only on clinical signs and symptoms of pneumonia OR chest radiograph with new consolidation, (2) WHO Primary Endpoint Pneumonia plus clinical pneumonia, a high-specificity diagnosis requiring lobar radiographic consolidation plus clinical pneumonia, and (3) WHO Primary Endpoint Pneumonia or Other Infiltrate plus clinical pneumonia, an intermediate specificity diagnosis requiring either lobar consolidation or non-specific interstitial radiographic findings plus clinical pneumonia.^23,24 (Clinical adjudicators are asked to review the child’s medical history to determine whether a consolidation that was present on a radiograph obtained during this illness was previously noted to be present; if so, this would not be considered as part of the definition of clinical pneumonia).

Two clinical adjudicators independently review all available clinical data (including the clinical record, laboratory testing, imaging, data from the research case report forms, and an audio recording of the subject’s cough) for each subject to determine the presence or absence of each disease at the time of cough sound recording. Because physical examination data are not reliably documented in the clinical record, research staff obtain examination data by direct report from the clinical provider at the time of enrollment. Clinical adjudicators are also asked to determine which subjects have been treated with medication prior to the cough sound recording such that the signs and symptoms of their respiratory disease have likely been altered to the point that they no longer meet criteria for the disease. These pre-treated subjects are excluded from the main analysis for that disease. The possible adjudication outcomes include “Yes,” “No,” and “Unsure.” Similarly, two radiologic adjudicators will independently review each film. In case of disagreement between the first two adjudicators in each review, a third adjudicator is assigned to independently review the case. If there is persistent disagreement, the case is discussed by an adjudication committee until consensus is reached.

Statistical Methods and Analytic Approach

Mathematical features isolated from subject cough sounds and parent- and patient-reported symptoms are applied by the ResAppDx algorithm to determine the presence or absence of each of the study endpoints. Since a non-gold standard test (i.e., adjudication) is used as the comparator standard, sensitivity and specificity (termed Positive Percent Agreement [PPA] and Negative Percent Agreement [NPA], respectively, following FDA guidelines²⁵) will be calculated. We will also calculate positive and negative likelihood ratios, along with 95% confidence intervals for PPA, NPA, and likelihood ratios using the Clopper-Pearson method.²⁶

The overall analysis population will include subjects who meet eligibility criteria, complete the study visit and all study-specific tasks, undergo a physical assessment, produce the required five coughs identified by the ResAppDx algorithm, and pass the cough recording quality review. Because bronchiolitis is defined as a disease only occurring in subjects ≤ 24 months of age, subjects older than 24 months will be excluded from the analysis population for the bronchiolitis PPA and NPA calculations. Similarly, for the radiographic pneumonia endpoints, only subjects in whom a chest radiograph is available will be included in the analysis. In addition, subjects identified by the CAC as being pre-treated such that the signs or symptoms of the endpoint in question were likely to have been altered to the point that they no longer meet the criteria for the disease will be given a CAC final diagnosis of “Unsure” and excluded from the primary analysis. Two sensitivity analyses will be performed, assuming (1) all “Unsure” outcomes as positive, and all “Unsure” outcomes as negative.

While the primary analysis population will include only children who are able to complete study procedures, the study documents and reports all cases of screen failures (not meeting inclusion criteria or no cough attempted) and cough recording failures (subjects who did not provide the required minimum of 5 coughs, subjects whose cough sound recording contained Protected Health Information, subjects in whom the recording failed quality review, and/or the recording was not made as per the study protocol). We will document detailed reasons for both screen failures and cough recording failures, and we will compare the characteristics and outcomes of children who were able to complete the study vs. those were unable to do so. This analysis will aid in understanding the real-world applicability of this device, particularly in low-resource settings.

Sample size justification

Based on data generated in previous studies,²⁷ we estimate the PPA and NPA of the ResAppDx algorithm to be 85% and 80%, respectively. For an assumed N=1500 of evaluable subjects, Table 3 presents the 95% confidence intervals as a function of disease prevalence. Using historical clinical data, we anticipate the prevalence of croup at 5%, pneumonia at 10%, and asthma/RAD at 25% of the eligible study population. The resulting precision with the proposed sample size was determined to be sufficient to establish the proposed test as a useful tool in the aid of diagnosing respiratory disease in children.

Table 3:

Expected 95% Confidence Intervals (CI) for diseases assuming a range of prevalence for an assumed sample size of evaluable subjects of 1500

Disease prevalence	0.05	0.1	0.15	0.2	0.25
# Cases	75	150	225	300	375
PPA	0.85	0.85	0.85	0.85	0.85
95% CI Lower (PPA)	0.769	0.793	0.803	0.810	0.814
95% CI Upper (PPA)	0.931	0.907	0.897	0.890	0.886
NPA	0.8	0.8	0.8	0.8	0.8
95% CI Lower (NPA)	0.779	0.779	0.778	0.777	0.777
95% CI Upper (NPA)	0.821	0.821	0.822	0.823	0.823

Discussion

This paper presents the protocol used in the SMARTCOUGH-C 2 study to measure the diagnostic accuracy of the ResAppDx algorithm in detecting common pediatric respiratory diseases. If the study is successful, the ResAppDx algorithm will be implemented as a novel mobile software application for smartphones intended to be used under the direction of a physician to aid in the diagnosis of childhood respiratory illnesses.

There are several potential advantages to a smartphone-based diagnostic test in evaluation of children with acute respiratory diseases. The testing procedure is relatively straightforward, requires minimal expertise, poses minimal risk to a child, and takes advantage of naturally occurring coughs. Smartphones have built-in microphones that meet acoustic requirements for respiratory sound recording, are ubiquitous even in low-resource settings, possess substantial computing power permitting for on-device analyses, and enable non-contact assessments that minimize risk of cross-contamination between patients. The integration of these features allows for an all-in-one data acquisition, analysis, and decision-making device. Particularly relevant to a telemedicine application, a smartphone-based diagnostic test can also provide objective data to assist in the remote evaluation of a sick child. While initially we envision cough sounds would be recorded in the office/clinic/emergency department and a diagnosis would be made available to a provider at the point of care, cough sounds could also be recorded at home and transmitted to a physician as part of remote evaluation in a telehealth setting.

At the same time, the smartphone-based cough analysis platform entails several challenges that must be considered, both in the clinical research and implementation settings. Because the algorithm relies on analysis of audio data, background noise must be limited, which may be difficult in a busy clinical environment. To address this need, the most recent version of the recording software includes a “background noise meter” to warn users if there is excessive ambient noise. The operator must also be aware if other patients or family members are coughing during the test to ensure that the recording represents the patient being tested, and must instruct the child not to cover his/her mouth. Additionally, while the test can utilize either spontaneous or voluntary coughs, infants and young children may not able to provide voluntary coughs, so the test may require additional time to record spontaneous coughs.

Finally, because of the absence of a diagnostic gold standard or even commonly accepted case definitions for common pediatric respiratory illnesses, there are intrinsic limitations to the anticipated accuracy of any diagnostic test. Significant variability has been reported in the interpretation of physical examination findings,^6,28 chest radiographs,^29–31 and in the clinical definitions of common respiratory diseases.^32–34 We have attempted to mitigate the effects of these challenges by applying detailed case definitions, training adjudicators, and testing adjudicators using standardized cases. Nonetheless, adjudicators must still depend on data provided by treating providers, including the assessment of bronchodilator responsiveness, physical examination findings, and decisions regarding diagnostic testing and treatment.

Given the high burden of acute respiratory illnesses in low and middle-income countries, additional large-scale implementation studies beyond this trial will be needed to measure the impact of the ResAppDx algorithms in low-resource settings. These studies will further delineate the effect of recording environment, community respiratory illness epidemiology, and clinical setting on the utility of this application. Additional large-scale studies will also provide insights regarding the best approach to integration in clinical workflow, cost/benefit analyses, and opportunities to further improve the diagnostic algorithms. While the algorithms are being tested initially on an Apple iPhone device in this study, we are conducting additional testing using a variety of other common smartphone platforms, including Android devices, to ensure the generalizability of our findings. Finally, we anticipate that cough sound analysis will have other applications in the management of respiratory diseases, including grading severity of asthma, detecting airway obstruction, and in diagnosis of chronic obstructive pulmonary disease.^35–37