Big data: mind your Ps and Qs—the importance of precision and quality in data validation

CITATION

Stierer TL. Big data: mind your ps and qs—the importance of precision and quality in data validation. J Clin Sleep Med. 2020;16(2):161.–162.

Big data and artificial intelligence have virtually transformed the delivery and quality of United States health care over the past decade.¹ Integrated electronic health records, insurance claims, biobanks, and registries have allowed unprecedented sharing of extensive patient data across multiple clinical settings contributing insights into population health as well as driving precision medicine for the individual patient. While analytical real-world evidence and machine learning hold the promise of facilitating earlier diagnosis, improving outcomes, and enhancing patient satisfaction, large volume data comes with inherent challenges including data verification, population bias, and diagnostic coding accuracy.

In this issue of the Journal of Clinical Sleep Medicine, Keenan et al assess the performance of a simple electronic health record (EHR)-based case-identification algorithm using ICD-9 and ICD-10 codes to distinguish patients with diagnosed obstructive sleep apnea (OSA) from patients not diagnosed with the disorder across six health systems in the United States.² The authors found that in their multisite cohort, an algorithm defining OSA cases as individuals with at least two separate encounters associated with diagnostic codes for sleep apnea in the EHR and non-OSA cases as individuals with no codes yielded high positive and negative predictive values.

This investigation builds upon prior work performed by Laratta et al, who was able to validate a claims-based algorithm that identified true cases of OSA with a high degree of specificity in a cohort of Canadian patients referred for sleep testing as opposed to McIsaac et al, also from Canada, whose group found that no code or combination of codes provided a likelihood ratio high enough to adequately identify patients with OSA who underwent preoperative polysomnogram (PSG).^3,4

The appeal of a high performing simple algorithm that accurately classifies cases in a binary manner as having or not having a disease or condition is obvious. However, there are still questions that linger and indicate that the risk of case misclassification in this study may not be fully appreciated by the authors. This is exemplified by the chart review process used to determine true cases of which only 72% had objective data in the form of results from a nocturnal polysomnogram or home sleep test. The remainder of cases used alternate criteria which included the presence of OSA on the patient’s EHR problem list. Clinicians have long been aware that several components of the EHR, including current medication and problem lists are prone to errors that are carried over and perpetuated with each subsequent visit which has led to a national effort to promote improved medication reconciliation; however, it is still unclear who is responsible for updating and maintaining an accurate problem list.⁵ Primary care providers appear to be better than specialists in ensuring that the problem list reflects the patient’s current condition, nevertheless, Daskavitch in 2018 found that data derived from an EHR problem list did not have sufficient accuracy to be useful in risk adjustment.^6,7 Therefore, using the presence of a condition in the problem list as a surrogate for objective data may not be a valid method for case identification for individuals with missing data.

On the other hand, cases identified as non-OSA present an additional source of possible misclassification. Young et al showed that up to 90% of individuals with OSA in a general population had yet to receive a formal diagnosis.⁸ The validation of non-OSA case status utilizing manual chart review further introduces a factor of human error. Additionally, the absence of coding may be reflective of visits where screening for signs and symptoms of OSA was not performed. In a study of 44 randomly selected sites within a practice-based research network, only 20% of patients with sleep related symptoms spontaneously reported their symptoms to their primary care physician.⁹ The authors employed the use of the Symptomless Multivariable Apnea Prediction Index to estimate cases with a high risk of OSA among those classified as non-OSA. However, Lyon et al’s validation of this prediction tool was conducted in a surgical population restricted to only those patients who received general anesthesia and used case confirmation as those with a prior formal diagnosis of OSA with no objective data to confirm absence of the condition.¹⁰ When examined by the US Preventative Services Task Force, studies analyzing the performance of Multivariable Apnea Prediction Index, which includes symptoms associated with OSA, against home sleep apnea testing devices showed that there was oversampling of patients with high risk for OSA.¹¹ Furthermore, the accuracy of home sleep apnea testing used to validate the tool showed sensitivities and specificities ranging from 7% to 100% and 15% to 100% respectively using an apnea-hypopnea index (AHI) ≥ 15 events/h.

Finally, there is the question of the adequacy of the AHI to accurately characterize the different phenotypes of sleep disordered breathing or reflect the true heterogeneity of the diagnosis of OSA. This may be best illustrated by the patient with severe positional or REM sleep-related sleep-disordered breathing with an overall normal AHI or respiratory disturbance index due to the patient sleeping in the lateral or prone position for the majority of total sleep time during PSG or with only a small percent of total sleep time spent in REM sleep. Variability in interpretation of PSG data based solely upon overall AHI and respiratory disturbance index will continue to confound investigations assessing risk and outcome especially in the perioperative arena where patients are frequently required to recover in the supine position and whose sleep architecture may be dramatically altered by anesthesia and surgical procedure, 24-hour activity with interruption of sleep to obtain vital signs and administer medications, and exposure to artificial light.

The logarithmic increase in knowledge gained leveraging the vast information available using big data and artificial intelligence has the capacity to dramatically alter our approach to patient care. Each study like Keenan’s examining and validating the performance of algorithms used to categorize presence of disease status help to refine the appropriate application of data to identify risk and provide precision treatments to enhance population health. We must also remain cautious in our interpretation of these studies as we have not yet been able to completely eliminate a variety of biases and if not careful, risk missing the mark when utilizing this data to determine surveillance, treatment, and health policy decisions.

DISCLOSURE STATEMENT

The author reports no conflicts of interest.