Applying Patient-Reported Outcome Methodology to Capture Patient-Reported Health Data: Report From an NIH Collaboratory Roundtable

Abstract

Patient-reported health data provide information for pragmatic trials that may not be readily available from electronic health records or administrative claims data. In this report, we present key considerations for collecting patient-reported health information in pragmatic clinical trials, which are informed by best practices from patient-reported outcome research. We focus on question design and question administration via electronic data collection platforms with respect to 3 types of patient-reported health data: medication use, utilization of health care services, and comorbid conditions. We summarize key scientific literature on the accuracy of these patient-reported data compared with electronic health record data. We discuss question design in detail, specifically defining the concept to be measured, patient understanding of the concept, recall periods of the question, and patient willingness to report. In addition, we discuss approaches for question administration and data collection platforms, which are key aspects of successful patient-reported data collection.

INTRODUCTION

Electronic health records (EHRs), administrative claims data, and patient- or caregiver-reported data provide complementary information for health care research. In September 2017, the National Institutes of Health (NIH) Health Care Systems Research Collaboratory organized a roundtable discussion to address best practices for capturing patient-reported health (PRH) data in pragmatic studies and the optimal analytic approaches for integrating this information with other study data. In the first report from the roundtable, Rockhold et al [1] discuss the analysis of PRH data. In this paper, we present key considerations for collecting PRH information for pragmatic research, and focus on 4 key themes: definition of the concept to be measured, patients’ understanding of the concept, recall periods for the question, and patients’ willingness to report. We use 3 types of PRH data as examples—medication use, utilization of health care services, and comorbid conditions, and summarize key scientific literature on the accuracy of these PHR data compared to EHR data. Lastly, we discuss approaches for question administration and data collection platforms.

CONSIDERATIONS FOR QUESTION DESIGN

Medication Use

Medication use is a particularly complex element to assess. First, it can be defined in different ways, such as a having a prescription versus actually ingesting the pills. Questions that measure unique aspects of medication use include “Do you have a prescription for x drug?,” “Have you filled your prescription for x drug?,” or “Do you take x drug?.” The reference period (or recall period) of the question also changes the meaning substantially. For example, “Have you ever filled a prescription for x drug?” versus “Have you filled a prescription for x drug in the past 4 weeks?”

In answering questions about medication use, the setting and the number of medications asked about are important considerations. If the patient (or caregiver) is answering the questions from home, they may be able to refer to the pill bottles they have on hand to verify the names of their medications and the dates they were last filled. Studies of a large number of drugs are limited by patients’ ability to recognize the names (either generic or trade name) of the medications they are taking and by the practical issue of presenting lists of medication names to patients in a brief questionnaire. Studies of a limited set of drugs can provide an introduction to the question describing the drug(s) and include photos. If the clinic system or insurance provider is known and has accessible data, the EHR data can be used to verify the existence of a prescription and claims data can be used to verify that a prescription was filled. In contrast, whether a patient is taking the medication (ie, adherence) can be known primarily by asking the patient or caregiver or by information collected from pill bottles (eg, Medication Event Monitoring System caps). However, patients may have reasons for wanting to incorrectly report their medication use.

The first step in creating a measurement plan is to define the end point (ie, what exactly is being measured) and the population (ie, the characteristics of the study sample, with particular attention to health literacy and the setting in which patients will answer the questions). Next, the recall period and period of interest need to be reconciled. For example, a study may need information about medication use over the past 12 months, but it is determined that patients can only reliably recall the specific information in periods of up to 3 months. This means that medication use would need to be assessed 4 times, 3 months apart. If this is not feasible, the focus of the questions will need to be changed.

Most validation studies of self-reported medication use focus on a particular class of medications. Findings from 2613 adolescents in a Norwegian youth health survey that compared self-reported data with national claims data, indicated that sensitivity and specificity varied substantially across medication classes [2]. Contraceptive pills had the highest sensitivity, and painkillers had the lowest sensitivity. There is substantial literature regarding the agreement of self-reported medication adherence and medication monitoring devices [3,4].

Health Care Utilization

Discrepancies in PRH data vs EHR data or claims information may be due to patient perceptions or recall. If a patient spends 40 hours in an emergency department, they may consider this encounter to be a hospital admission. However, EHRs or claims data may not record a hospital stay because the patient was not admitted to the hospital and the length of stay was less than 48 hours. For patients who are seriously ill, the number of clinic visits or when they occurred may be difficult to recall accurately [5]. A systematic review by Leggett et al [6] found good agreement between self-reported and claims data regarding health care utilization, except for visits to general practitioners, outpatient days, and nurse visits.

Comorbid Conditions

A common barrier to asking patients about comorbid conditions is health literacy. Patients may not understand or recognize the terms used to describe comorbid conditions, especially technical terms (eg, “arrhythmia,” “cardiomyopathy”). Chronic conditions can be asked about in the present (eg, Do you have x?), while acute conditions may need to be asked about with a recall period (eg, Have you had x in the past 12 months?). The severity of the condition will likely dictate how easily it is recalled [5].

A large study from the California Breast Cancer Survivorship Consortium found the level of concordance between self-reported comorbidity and medical records varied by the type of comorbidity [7]. Women with a longer history of comorbidity or who took medications for a condition were more likely to report the condition. The sensitivities of self-report versus the medical record were similar by race/ethnicity and did not vary by age, neighborhood socioeconomic status, or education. Ye et al [8] conducted a large study among a population-based cohort of 881 prostate cancer patients. For most of the 20 medical conditions examined, there was agreement in the presence or absence of the condition between patient reports and medical records for more than 90% of patients. Race and education level were not significantly associated with the level of agreement in 18 out of 19 modeled conditions [8]. Investigators should be aware that neither patient self-report nor health records can be considered the “gold standard.” It is possible that patient report is more accurate than data abstracted from the medical record because of possible incomplete information regarding health history and comorbid conditions in the health record, especially if the patient has changed hospitals or health care systems.

General Considerations

Patient comprehension of any new questionnaire should be assessed prior to empirical validation and/or its use as a data collection tool. Cognitive interviewing [9] is a standard method for assessing patient comprehension and can reveal whether patients understand the questions in the way they are intended and whether patients feel they can accurately respond.

Sensitive topics, such as HIV status, depression, and addiction, should be asked about in a nonjudgmental way. The study team will need to take extra precautions to safeguard data privacy, and communicating these safeguards to patients may improve response rates and reporting accuracy. In addition to confidentiality, patients should be assured that their responses to these questions will not affect the quality of their care.

CONSIDERATIONS FOR QUESTION ADMINISTRATION

Ease of use and data security are primary considerations in the design of a patient-reported data collection plan. The timing (ie, schedule of assessments) and location of data collection also have a major impact on the design of the data collection process.

If data are collected during clinic visits, a member of the study team can give participants a tablet. A web-based system, instead of an app installed on the tablet, is preferable so that patient data are not stored on the tablet itself, which is a security risk. Paper forms should be available as an alternative.

When data are collected from patients outside of clinic, or “from home,” this expands the options for the frequency and schedule of the assessments. Data collection can be performed using web-based questionnaires that are completed on personal computers, tablets, or smartphones; app-based questionnaires completed on tablets or smartphones; automated telephone surveys (eg, interactive voice response systems); interviewer-administered surveys completed by phone; and paper surveys sent/returned by mail. The Pew Research Center has found that while the rate of technology adoption is increasing rapidly, one-third of adults 65 years and older never use the internet, and only 42% have smartphones [10]. The rates of internet use and having a smartphone are highly correlated with household income. Therefore, providing an alternative to internet-based data collection is essential for a representative study sample. Telephone-based survey methods enable data collection from patients who do not have internet access at home. Coons et al [11] provide an exhaustive discussion of issues related to mode of data collection in clinical trials.

Data collection using apps installed on patient-owned devices poses particular benefits and challenges compared with web-based data collection. The advantage of apps is that patients can enter information when they don’t have internet access. The data can later be synchronized with the study management database. However, apps are more complex to implement and maintain. For example, if notifications are disabled by the user or if the app is uninstalled, the patient may be unaware of the need to complete the questionnaire. Moreover, because the questionnaire data must be submitted via the app upon completion, the study team may not have real-time information about patients’ progress in completing the questionnaire.

There is substantial evidence supporting the equivalence of survey response data collected from screen-based questionnaires (eg, personal computer, tablet, or smartphone) and paper questionnaires [12,13]. Differences between visual formats and automated telephone-based (aural) questionnaires range from negligible to small [14–17]. Investigators should consider whether minor differences in survey responses are acceptable in order to collect data from a more generalizable study sample, and whether the analysis needs to control for survey mode. For questionnaire topics in which social desirability and privacy concerns may affect patient responses, there is evidence of under-reporting in interviewer-administered questionnaires versus questionnaire formats where the information is reported privately, such as through a web survey [18–20].

For each mode, user testing is essential prior to study data collection. It is also highly recommended that studies budget for the cost of making minor changes to the web-, app- or phone-based survey, not only during the testing phase but also after the study has started. We recommend the use of a single centralized PRH data collection platform that can be used by all study sites. Although some EHR systems can capture PRH data, often the platforms have limited ability for customization, and having the PRH assessment automatically triggered by an event newly recorded in the EHR database is ambitious.

A brief orientation about how and when to complete the questionnaire should be provided to the patient, along with a clear, simple instruction sheet to take home. Instructions can also be embedded in the app or included as part of the introduction to the web, telephone, or paper survey. Monitoring survey response rates (ie, compliance) on a frequent and regular basis is critical to successful data collection. The additional staff time required for training sites and orienting patients to the data collection, and for monitoring compliance, will result in better quality data. This level of effort may not be in keeping with the philosophy of a pragmatic study; however, it is essential in order for the data to have value.

CONCLUSION

Successful collection of PRH data in pragmatic trials requires careful consideration of a number of question design and administration issues. Best practices from the field of patient-reported outcome assessment can provide guidance on these issues, which we presented here with respect to the assessment of patient-reported medication use, health services utilization, and comorbid conditions.