The Reliability of Patient Self-reported Utilization in an Inflammatory Bowel Diseases Learning Health System

Abstract

Background

Inflammatory bowel disease (IBD) care is beset with substantial practice variation. Learning health systems (LHSs) aim to learn from this variation and improve quality of care by sharing feedback and improvement strategies within the LHS. Obtaining accurate information on outcomes and quality of care is a priority for LHS, which often includes patients’ self-reported data. While prior work has shown that patients can accurately report their diagnosis and surgical history, little is known about their ability to self-report recent healthcare utilization, medication use, and vaccination status.

Methods

We compared patient self-reported data within the IBD Qorus LHS regarding recent IBD-related emergency department (ED) visits, hospitalizations, computerized tomography (CT) scans, corticosteroid use, opioid use, influenza vaccinations, and pneumococcal vaccinations with electronic health record (EHR) data.

Results

We compared 328 patient self-reports to data extracted from the EHR. Sensitivity was moderate-to-high for ED visits, hospitalizations, and CT scans (76%, 87%, and 87%, respectively), sensitivity was lower for medication use with 71% sensitivity for corticosteroid use and only 50% sensitivity for self-reported use of opioids. Vaccinations were reported with high sensitivity, but overall agreement was low as many patients reported vaccinations that were not registered in the EHR.

Conclusions

Self-reported IBD-related ED visits, hospitalizations, and CT scans are reported with high sensitivity and accuracy. Medication use, and in particular opioid use, is less reliably reported. Vaccination self-report is likely more accurate than EHR data as many vaccinations are not accurately registered.

Background

The inflammatory bowel diseases (IBDs), which include ulcerative colitis and Crohn disease, are chronic inflammatory diseases of the intestines that affect approximately 3 million people in the United States.¹ Most patients are treated by one of 15,000 gastroenterologists in the United States² who practice care in different ways; studies have shown that there is wide practice variation in the United States both geographically and by practice setting.^3–7 Not surprisingly, there is variation in the associated outcomes of care as well, including mucosal healing and surgery rates.^6–8 Although some differences can be explained by differences in patient populations, variation in outcomes remains after accounting for patient-specific factors.^3–8

Learning health systems (LHSs) use clinical variation to collaboratively identify and implement strategies for care improvement.^9,10 An LHS can be a single hospital or a collaboration of multiple hospitals or clinics that collect data about practice patterns and patient outcomes. Data are compared across practitioners or practices, and practices and outcomes with substantial variation are identified. Conceptually, those with worse outcomes can learn from those with better outcomes in a collaborative fashion.^9,10 In IBD, 2 such models exist in the United States: The ImproveCareNow LHS, a network of >100 pediatric IBD centers,¹¹ has led to reductions in steroid use and a higher percentage of patients in remission.¹² The Crohn’s and Colitis Foundations’ IBD Qorus LHS,¹³ a network of >50 academic and private practices caring for adult patients with IBD, has resulted in reductions in emergency department (ED) visits, hospitalizations, and opioid use in participating clinics.¹⁴

LHSs rely on accurate data to inform their improvement efforts. Data can be collected through different methodologies, including administrative data, patient surveys, or provider input,^9,10,13,15 each with their own strengths and weaknesses. While administrative data are considered more objective, it can be hard to obtain, lack sufficient detail, and can be incomplete as electronic health records (EHRs) are mostly unable to communicate with each other. Obtaining data directly from patients might be a viable alternative in the absence of reliable administrative data.^16–18 In IBD, patients have been shown to accurately self-report their IBD diagnosis and IBD subtype,^19,20 recent hospitalizations,²¹ and surgical history.^19,20 However, some systematic under- and over-reporting has been observed in regard to the number of reported outpatient visits and reported length of stay in the hospital, respectively.²¹ Additionally, patients might find it challenging to discriminate between IBD-related and non-IBD-related encounters.²¹

In the IBD Qorus LHS, a set of quality measures are routinely collected from patients at participating sites. The collected measures were previously identified as priority outcome measures in IBD and include disease activity, healthcare utilization, medication use (steroids and opioids), and vaccination completion.²² IBD Qorus opted to collect this data directly from patients for the reasons discussed above. However, while IBD patients have been shown to reliably report hospitalizations using a survey,²¹ the reliability of patient self-reported ED visits, computerized tomography (CT) scans, medication use, and vaccination status has not been previously investigated.

Methods

Design

In this cross-sectional study, healthcare utilization, medication use, and vaccination status were extracted from the medical record and compared to patients’ self-report.

Setting

IBD Qorus is a LHS, which—at the time of the study—consisted of a mix of 26 academic and private IBD practices. In IBD Qorus, patient data are collected through an electronic platform that both patients and providers have access to. The participating gastroenterologist or care coordinator enters information about the patients’ diagnosis and disease phenotype and the patient is invited to complete a pre-visit survey (PVS) prior to their clinic visit. The PVS asks the patient about the reason for their visit, current symptoms, and healthcare utilization in the last 6 months including ED visits, hospitalizations, CT scans, corticosteroid and opioid use, and receipt of influenza and pneumococcal vaccinations (Table 1). Generally, this process is initiated prior to each clinic visit with patients receiving an email from their site to complete the PVS. In addition, many patients fill out a survey during enrollment in IBD Qorus and patients also have the option to independently start a PVS survey within the online system on demand. During the study period, the overall response rate to survey reminders was 49%, though 62% of surveys collected during this period were completed without a reminder (ie, either during enrollment or because the patient filled one out independently)

Table 1.

Survey questions to report self-reported utilization, medication use, and vaccination status in IBD Qorus

Question	Answer options
Have you been to an Emergency Department (ED) in the past 6 months due to your IBD?	Yes (if Yes: approximate date MM/DD/YY) No
Have you been hospitalized in the past 6 months due to your IBD?	Yes (if Yes: approximate date MM/DD/YY) No
Have you had a CT scan in the past 6 months for your IBD?	Yes (if Yes: approximate date MM/DD/YY) No
Are you currently taking steroids by mouth (prednisone) for your IBD?	Yes No
Are you currently taking narcotics (pain medications) for your IBD?	Yes No
Have you received an influenza (flu) vaccination within the past year?	Yes No Not sure
Have you ever received a pneumonia vaccination?	Yes No Not sure

Patients are invited to fill out this survey approximately 1 week prior to their visit.

For this study, a purposive sample of 4 Qorus sites was selected, including 2 university-based hospitals, 1 independent academic hospital, and 1 private practice site, which represented the broad diversity of participating sites in IBD Qorus.

Data Collection

PVS data and EHR data were extracted from the 4 participating Qorus sites between February 2016 and November 2018. Survey data were extracted through the IBD Qorus platform and included the patient’s name, medical record number, and age; the number of patient-reported IBD-related ED visits, hospitalizations, and CT scans in the last 6 months; use of corticosteroids or opioids at the time of the visit; receipt of influenza vaccination in the last year, and receipt of pneumococcal vaccination ever (Table 1).

EHR chart reviews were performed by trained data extractors who were familiar with IBD Qorus and its processes. Data were extracted using a standardized data-extraction form. Encounter summaries, visit notes, referral notes, test results, medication lists, and vaccination records were reviewed. The IBD diagnosis was confirmed and any evidence of ED visits, hospitalizations, and CT scans within the 6 months prior to the PVS completion were extracted, including the date of the encounter, reason for the encounter and whether the encounter was deemed IBD related or not. Corticosteroid and opioid use at the time of the survey was determined and the name of the medication was recorded. Similarly, evidence for influenza vaccination within the last year as well as prior pneumococcal vaccination were extracted. In case of uncertainty about the classification of the data, the chart was discussed with the first author and the treating physician.

Data Analysis

Patients with an unconfirmed IBD diagnosis were excluded from analysis and if a patient filled out multiple surveys, only the first one was included. Agreement between EHR and self-report was calculated, as well as sensitivity of the self-reported data. In addition, the percentage of cases in which patients reported an encounter that was not present in the EHR (1-specificity) was calculated. False negatives (ie, cases in which the patient did not report an encounter) and false positives (ie, cases in which the patient reported an encounter that was not present in the EHR), were evaluated in more detail. For the false negatives, we compared the time between the encounter and the PVS as we hypothesized that patients are more likely to report very recent encounters. For false positives, we assessed the percentage of patients with a non-IBD-related encounter in the EHR, as we hypothesized that patients might misreport non-IBD-related encounters as IBD-related encounters. Wilcoxon ranked sum tests and Fisher exact tests were performed to compare groups using continuous and categorical data, respectively. All analyses were performed in SAS 9.4 (Cary, NC).

Ethics

All patients consented to participate in IBD Qorus and to the use of their data for research purposes. The study was approved by the institutional review boards overseeing the participating sites.

Results

In total 328 IBD patients were included in the analysis, of which 133 (41%) were male and the median age was 42 [interquartile range (IQR) 32–58; Table 2]. Of those, 208 (63%) were diagnosed with Crohn disease, 108 (33%) with ulcerative colitis, and 12 (4%) with IBD unclassified. The median disease duration was 12 years (IQR 5–19).

Table 2.

Demographics and disease characteristics of included patients

No. of patients	328
Male gender, n (%)	133 (41)
Age, median (IQR)	42 (32–58)
Diagnosis, n (%)
Crohn disease	208 (63)
Ulcerative colitis	108 (33)
IBD unclassified	12 (4)
Disease duration in years, median (IQR)	12 (5–19)

The overall agreement between self-report and the EHR for IBD-related ED visits, hospitalizations, and CT scans within the last 6 months was, 92%, 96%, and 89%, respectively (Table 3). Of the 21 IBD-related ED visits in the EHR, 16 were reported by the patient on the survey (sensitivity of 76%). The reasons for the unreported ED visits included rectal or abdominal pain, (bloody) diarrhea, and a high ostomy output. The median amount of time since the encounter for unreported ED visits was numerically longer with higher maximum values (46 days, IQR 28–156) than for reported ED visits (43 days, IQR 28–75; P = 0.71; Table 4). Vice versa, of the people for whom no IBD-related ED visit was recorded in the EHR, 22 (7%) reported one on the survey. Of those, 3 (14%) did have a non IBD-related visit in their records, compared to 12 out of 279 patients (3%) who did not report an IBD-related visit (P = 0.087; Table 5). While these visits were deemed non-IBD related by the treating physicians, several were related to gastrointestinal complaints, including dizziness, diverticulitis, and hyperemesis due to cannabis use.

Table 3.

Numbers of IBD-related encounters reported on the PVS and in the EHR

Total (n = 328)		IBD-related ED visits	IBD-related hospitalizations	IBD-related CT scans	IBD-related steroid use	IBD-related opioid use	Influenza vaccinations	Pneumococcal vaccinations
Encounters reported in EHR		21	30	30	38	14	99	127
In EHR	On PVS	16	26	26	27	7	92	108
	Not on PVS	5	4	4	11	7	7	19
Not in EHR	On PVS	22	9	32	15	18	116	61
	Not on PVS	279	286	261	273	288	97	60
Agreement EHR—PVS		92%	96%	89%	92%	92%	61%	68%
Sensitivity		76%	87%	87%	71%	50%	93%	85%
1-Specificity		7%	3%	11%	5%	6%	54%	50%

We calculated overall agreement between the EHR and PVS, the sensitivity of patient-reported utilization (% of encounters in the EHR also reported by patients), and 1-specificity (% of patients for whom no encounter is recorded in the EHR for whom an encounter was reported on the PVS).

Table 4.

Duration between the encounter and the PVS compared between patients who accurately reported them on the PVS and those who did not

		n	Days since encounter (median, IQR)	P
IBD-related ED visits	In EHR but not reported by patient	5	46 (28–156)
	In EHR and reported by patient	16	43 (28–75)	0.71
IBD-related hospitalizations	In EHR but not reported by patient	4	152 (84–160)
	In EHR and reported by patient	26	66 (36–125)	0.20
IBD-related CT scans	In EHR but not reported by patient	4	84 (34–146)
	In EHR and reported by patient	26	89 (35–148)	1.00

Table 5.

Number and percentage of patients who did not have an IBD-related encounter, who did have an encounter not related to IBD, compared between patients who did (incorrectly) report an encounter on the PVS and those who did not report (correctly) an encounter on the PVS

		n	n (%) With a non-IBD-related encounter	P
IBD-related ED visits	None in EHR but reported by patient	22	3 (14)
	None in EHR and not reported by patient	279	12 (4)	0.087
IBD-related hospitalizations	None in EHR but reported by patient	9	1 (11)
	None in EHR and not reported by patient	286	2 (1)	0.089
IBD-related CT scans	None in EHR but reported by patient	32	1 (3)
	None in EHR and not reported by patient	261	1 (0.4)	0.21

Similarly, of the 30 IBD-related hospitalizations in the EHR, 26 were reported by the patient on the survey (sensitivity of 87%; Table 3). The reasons of unreported hospitalizations overlapped with the ED visit reasons and included abdominal pain and nausea, (bloody) diarrhea, high ostomy output, and anemia. The median amount of time since the hospitalization for unreported admissions was numerically longer (152 days, IQR 84–160) than for reported admissions (66 days, IQR 36–125; P = 0.20; Table 4). Of the patients for whom no IBD-related hospitalization was recorded in the EHR, 9 (3%) reported a hospitalization on the survey. Of those, 1 (11%) did have a non IBD-related hospitalization in the EHR (for a pneumonia), compared to 2 out of 286 who did not report an IBD-related hospitalization (1%; P = 0.089; Table 5). For IBD-related CT scans, the sensitivity was 87% (Table 3); the 4 CT scans that patients did not report were related to a postoperative evaluation, abdominal pain complaints, and placement of a nasogastric tube. The median amount of time since these scans was similar (84 days, IQR 34–146) compared to those that were correctly reported by patients (89 days, IQR 35–148; Table 4). Of the patients for whom no IBD-related CT scan was identified in the EHR, 32 (11%) reported one on the PVS. Of those, 1 (3%) did have a non IBD-related CT scan (for headaches), compared to 0.4% of those who did not report a CT scan (P = 0.21; Table 5).

Overall agreement between self-reported medication use and the EHR was 92% for both corticosteroid and opioid use. Of the 38 occasions in which systemic steroid use was documented in the EHR, 27 were also reported by the patient (71% sensitivity). Vice versa, of the patients for whom no steroid use was documented in the EHR, 15 (5%) reported steroid use on the PVS (Table 2). Of those, 5 (33%) did use a locally acting steroid (eg, budesonide), compared to 3% of patients who did not report steroids on the PVS (P = 0.087, data not shown). For opioids, only 7 out of 14 EHR-recorded occasions were reported by patients (50% sensitivity), and 18 patients (6%) reported opioid use that was not registered in the EHR.

Agreement between self-reported vaccination status and EHR-reported vaccination status was low, with 61% agreement for influenza vaccinations and 68% agreement for pneumococcal vaccinations. While vaccinations were usually reported by the patient if there was evidence in the EHR (93% and 85% sensitivity for influenza and pneumococcal vaccinations, respectively), many patients recorded receipt of a vaccine that was not recorded in the EHR (54% and 50%, for influenza and pneumococcal vaccinations, respectively).

Discussion

Here, we showed that IBD patients’ self-report of hospital utilization and medication use is highly accurate, with agreements between self-report and the EHR of ≥89% for all utilization and medication measures. Sensitivity of self-reported hospital utilization was high as well (≥76%), but we found preliminary signals that events that happened in the more distant past, might be less likely to be self-reported. We also found that non-IBD-related events might be reported as IBD-related events by patients. Sensitivity for steroid and opioid use was lower (71% and 50%, respectively), indicating these data need to be used with more caution. In the case of corticosteroids, the phrasing of the question, which asked for the use of “steroids by mouth (prednisone),” might not have been specific enough (Table 1) as demonstrated by the fact that several patients who used locally acting, oral steroids (eg, budesonide) reported steroid use on the survey, which was not intended by the question. For vaccinations, it appeared that patients were able to report these with high levels of sensitivity (≥86%), but that they were often not reported in the EHR: in about half of the cases in which no vaccinations were documented, the patient reported one, indicating that patient self-report might actually be more reliable for vaccination receipt than the EHR.

Prior work has demonstrated that IBD patients report their diagnosis, past surgical history, and recent hospitalizations with high accuracy.^19–21 Here, we show that IBD-related healthcare utilization including hospitalizations, ED visits, and CT scans, can also be accurately reported by patients within the context of an LHS. Consistent with prior data, we show that patient recall might be reduced when the event happened in the more distant past and that non IBD-related events might be labeled by patients as IBD related.²¹ Steroid use was also reported with reasonable accuracy. Because some of the over-reporting could be explained by the phrasing of the question, which did not clearly distinguish between locally and systemically acting corticosteroids, we have since revised the phrasing of this question on the PVS that is currently in use within IBD Qorus. The sensitivity of opioid self-report was only 50%, which is consistent with a prior study that found 44% sensitivity of opioid self-report.²³ This might be related to the stigma that patients experience²⁴ or to the fact that intermittent use is relatively common in IBD.²⁵ Lastly, we found that registration of vaccinations in the EHR is poor, which is consistent with prior work that found that direct patient report of influenza and pneumococcal vaccine status is likely more reliable.^26,27

The main limitation of this study is that the documentation in the EHR is likely incomplete. Utilization events encountered at outside hospitals might not be recorded, and medication and vaccination lists might not be updated. To counter this, we took a comprehensive approach in our EHR review and reviewed not only the encounter history, medication and vaccination lists, but also reviewed visit notes and outside records scanned into the EHR within the 6 months prior to PVS completion to get a comprehensive overview of the treatment plan and medical history. Additionally, as the data in this secondary data analysis was not collected for research purposes, we cannot assess the exact survey completion rate and the data likely suffer from selection bias. Lastly, our sampling included only data from 4 sites, and it is possible that this is not representative of the entire population within the LHS. However, we specifically sampled these sites with the intent to include diverse practice settings present within IBD Qorus.

Conclusions

We showed high levels of agreement between the EHR and patients’ self-reported use of IBD-related ED visits, hospitalizations, CT scans, steroid use, and opioid use. This is an important novel finding as access to reliable data within an LHS is vital, as LHSs—by definition—rely on accurate data to guide the design, implementation, and evaluation of quality improvement efforts. In addition, patient report allowed us to obtain insight into events that are not consistently recorded in the EHR, such as vaccination records. Therefore, patients’ self-reported data are a reliable source of information about healthcare utilization within the context of an LHS. Self-reported data can therefore reliably be used to inform research as well as quality improvement efforts.

Funding

This work was supported through a grant from the Crohn’s and Colitis Foundation (New York, NY).

Conflict of Interest Statement

W.K.D., D.J.C., C.A.S., and G.Y.M. receive funding from the Crohn’s and Colitis Foundation for work related to the IBD Qorus Learning Health System. S.A.W. and R.O. are employed by the Crohn’s and Colitis Foundation. N.Z.F. received consultancy fees from HealthPal. G.Y.M. consulted for AbbVie, Bristol-Meyers Squibb, Celgene, Janssen, Medtronic, Pfizer, Samsung Bioepis, Shionogi, Takeda, and Techlab. The other authors do not disclose any financial conflicts of interest.

Data Availability

Data are available upon reasonable request from the corresponding author.