Adapting Patient Experience Data Collection Processes for Lower Literacy Patient Populations Using Tablets at the Point of Care

Abstract

Background:

Patient experience surveys are widely used to capture patient-reported quality of care and are increasingly being used for formal reporting purposes. There is evidence that certain patient subgroups are less likely to respond to traditional CAHPS surveys. As patient-facing technologies become more common, it is important to examine whether tablet-based patient experience surveys have the potential to promote responses from more diverse populations.

Objectives:

To develop, gain perspectives about, and pilot an English and Spanish low-literacy adaptation of the CG-CAHPS administered on a tablet device at the point of care.

Research Design:

Cognitive testing and evaluation of a quality improvement pilot comparing a tablet-based adaptation and traditional paper-based versions of the CG-CAHPS survey.

Subjects:

English- and Spanish-speaking patients receiving primary care in an urban community clinic.

Measures:

To compare the acceptability of low-literacy tablet-based and traditional paper-based patient experience surveys, we examined the concordance of responses between survey modes and preferences for modality, as well as perspectives on usability and reporting care experiences. We examined demographic differences in responses to tablet-based versus mailed surveys from a quality improvement pilot.

Results:

The majority of cognitive interview participants preferred a low-literacy, tablet-based survey over a paper-based survey with traditional wording. In a quality improvement pilot comparing tablet-based administration at the point of care versus mailed surveys, respondents to the tablet-based survey were more likely to be younger and Latino.

Conclusions:

If designed with patient input, tablet-based surveys have the potential to improve the collection of patient experience data among diverse populations.

Introduction:

Patient experience surveys are widely used to collect information about patient experiences and satisfaction with care. In the United States, patient experience surveys are increasingly being used for formal reporting,¹ including certification of patient-centered medical homes,² online reporting and comparison,^3,4 and performance-based reimbursement programs.^5,6 The Consumer Assessment of Healthcare Providers & Systems Clinician & Group Survey (CG-CAHPS), the gold standard for collecting patient experience data from outpatient settings in the U.S., is used in multiple internal and external care improvement programs across healthcare settings,⁷ including pay-for-performance programs like the Medicare Shared Savings Program⁸ and Medicare Access and CHIP Reauthorization Act’s (MACRA) Merit-based Incentive Program.⁹

While the CAHPS is traditionally administered by mail, survey vendors offer alternate modes of administration, including phone- and web-based administration.¹⁰ Studies using CAHPS data have found response rates ranging from 34–61%^11–13 and response rates have been found to be lower for female, non-White, younger, and limited English proficient patients.^13–15 In particular, underserved patient populations facing greater challenges with literacy, numeracy, and/or English proficiency may face greater difficulty with existing CAHPS questionnaires,^13,15,16 underscoring the need for exploring new ways to engage diverse patients in reporting on their care.

Thus far there has been little discussion about whether CAHPS captures the true domains of patient healthcare experience among patients receiving care from safety net settings, or whether data collection via mobile technology might result in higher response rates from more representative populations. Therefore, we sought to evaluate the acceptability and usability of a low-literacy CG-CAHPS adaption administered via a tablet device among a diverse group of patients from a safety net health care setting.

Methods:

Study Setting and Participants:

The study took place at the San Francisco Health Network, which provides primary care to over 63,000 patients a year, of whom 35% are Latino, 25% are Asian, 17% are Black, and 17% are White. The majority of patients have Medicaid, Medicare, or are uninsured.

Survey Adaptation:

Low-literacy and Spanish Adaptation:

We first adapted the CAHPS (Clinician and Group version 3.0)¹⁷ to improve readability and shorten the survey length. We consulted strategies employed by previous researchers to improve readability¹⁸ and aimed for a 5^th grade reading level or less using the Flesch-Kincaid Grade Level test,^19,20 following recommendations for low-literacy audiences.²¹ Strategies to improve readability included replacing multisyllabic words, breaking up multiclause questions into multiple sentences, and reducing the number of words per question. To further improve readability, we removed the phrase “In the last 6 months” from each question and prefaced the entire survey with a screen that stated “These questions ask about your own health care, in the LAST 6 MONTHS.” We shortened the survey from 31 questions to 16 questions, based on recommendations from a validated study²² and feedback from patient advisory groups about the importance of specific questions to their clinical experience. For example, educational attainment was removed from the adapted survey due to feedback from patient advisors about the stigma of the question and perceived impertinence to their medical care. Based on feedback from patient advisors, we adjusted the response options for reporting gender and race/ethnicity, and added additional response options for reporting non-applicability of questions regarding test results and prescription medications (specific changes in Table 2). Based on guidance from a study validating a shorter CAHPS, we added a question to assess timeliness of care.²² In the remaining 16 questions, we improved readability from an average grade level of 7.0 to 4.6. To create a low-literacy Spanish adaptation, two members of the research team performed a translation and back-translation.

Table 2.

Survey content and concordance between responses

Q#	Domain	Original Question	Readability (Flesch-Kincaid grade level)	Adapted Question	Readability (Flesch-Kincaid grade level)	Concordance (%)
1	Access	In the last 6 months, how many times did you visit this provider to get care for yourself?	6.5	How many times did you go to this clinic to get care?	4.2	70.8
2	Access	In the last 6 months, when you contacted this provider’s office to get an appointment for care you needed right away, how often did you get an appointment as soon as you needed?	13.7	How often were you able to get an appointment when you needed it?	6.5	83.3
3	Access	In the last 6 months, how often did you see this provider within 15 min of your appointment time?	7.9	How often did you see your doctor or provider within 15 min of your appointment time?	8.3	N/A (Not on paper version)
4	Communication	In the last 6 months, how often did this provider listen carefully to you?	7.5	How often did your doctor listen carefully to you?	6.7	95.8
5	Communication	In the last 6 months, how often did this provider seem to know the important information about your medical history?	11.6	How often did your doctor know important things about your medical history?	9	79.2
6	Communication	In the last 6 months, how often did this provider spend enough time with you?	5.2	How often did your doctor spend enough time with you?	4.4	75.0
7	Communication	Using any number from 0 to 10, where 0 is the worst provider possible and 10 is the best provider possible, what number would you use to rate this provider?	14.2	What number would you use to rate your doctor? Use any number from 0 to 10, where 0 is the worst provider possible and 10 is the best provider possible.	7.5	87.5
8	Care Coordination	In the last 6 months, when this provider ordered a blood test, x-ray, or other test for you, how often did someone from this provider’s office follow up to give you those results? [Never/ Sometimes/ Usually/ Always]	13.3	How often were you able to get your test results if you wanted them? [Never/ Sometimes/ Usually/ Always/ I didn’t have any tests done]	6.1	41.7
9	Care Coordination	In the last 6 months, how often did you and someone from this provider’s office talk about all the prescription medicines you were taking? [Never/ Sometimes/ Usually/ Always]	11.4	How often did you talk to someone from the clinic about your medicine? [Never/ Sometimes/ Usually/ Always/ I don’t take any prescription medicine]	7.8	69.6
10	Office Staff	In the last 6 months, how often did clerks and receptionists at this provider’s office treat you with courtesy and respect?	10	How often did the staff at the front desk treat you with respect?	4.5	91.7
11	Proxy Respondent	Did someone help you complete this survey?	5.6	Did someone help you complete this survey?	5.6	N/A (Answers specific to each survey)
12	Proxy Respondent	How did that person help you? Mark one or more.	0	How did that person help you? Mark one or more.	0	N/A
13	Health Status	In general, how would you rate your overall health?	4.9	How would you rate your health?	0	70.8
14	Demographics	What is your age?	0	How old are you?	0	100.0
15	Demographics	Are you male or female? [Male/ Female]	0.5	What is your gender? [Male/ Female/ Trans-male/ Trans-female/ Not listed]	0.7	100.0
16	Demographics	Are you of Hispanic or Latino origin or descent? [Yes, Hispanic or Latino/ No, Not Hispanic or Latino]	8.8	What is your race or ethnicity? Mark one or more.	1.7	N/A (Questions combined and response options changed)
		What is your race? Mark one or more. [White/ Black or African American/ Asian/ Native Hawaiian or Other Pacific Islander/ American Indian or Alaska Native/ Other]	0

Note. Response options shown only for questions in which response options were adapted from the original CG-CAHPS 3.0 questions.

A total of 14 questions were deemed by the research team to retain the original structure of the CAHPS survey item, while 2 (Table 2; Questions 8, 9) were deemed to have been more substantially altered via the patient advisory board feedback process (Table 2). For example, the CAHPS sub-scale on care coordination involved questions about the frequency of follow-up about medications or test results, which patients found to be confusing if there was no expected standard process for those activities within the clinic (e.g., providers and staff did not follow up with test results if they were negative or deemed “in range”).

Technical Adaptation for Tablet Administration:

To develop a tablet-based application for administering the CAHPS survey, we partnered with Tickit Health to create an interface that was icon-based and simple to use. The final tablet-based platform displayed one survey question per screen and utilized icons to visualize the main topic of each question (e.g., an icon of a patient-provider interaction) and Likert scale response options (Figure 1). In addition, questions were broken up and arrayed on multiple lines each composed of logical phrases to promote readability.¹⁸ For questions with more than a few response options, a variety of displays for selecting response options (e.g., slider bar, scroll bar) were used to test usability. To test the feasibility of administering a tablet-based patient experience survey at the point of care, we piloted tablet-based survey administration in the exam rooms within one clinic in the network. During the pilot, changes were made to the interface of the survey (e.g., adding an always-on, self-explanatory homepage inviting patients to take the survey) and administrative practices (e.g., tethering tablets and cables to the exam room wall to prevent theft and ensure charging). The year-long pilot resulted in a steady 30–50 responses per month and higher representation from younger and Latino individuals (further details and data shown in Appendix A), supporting the feasibility of tablet-based administration at the point of care.

Figure 1.

Screenshot of tablet survey interface

Patient Advisory Board Feedback:

From October to December 2015, we elicited feedback from 3 active patient advisory councils at 3 respective clinics within the San Francisco Health Network to refine the tablet-based survey adaptation. We elicited feedback on survey content (focusing on improving readability and suitability with clinic processes), interface of the application, and workflow for administration in the clinics. We used this feedback to refine the survey before cognitive interviewing.

Patient Cognitive Interviews for Feedback & Validation:

Recruitment of Sample:

Once the adapted tablet-based CAHPS tool was created, we conducted cognitive interviews from April-July 2016. From an electronic query of patients who had visited the clinic in the past year, clinic providers and staff identified patients who were actively engaged in discussion about their care. Participants were eligible for the study if they (1) did not have severe cognitive impairment or visual impairment and (2) were comfortable speaking and reading English or Spanish. Study staff screened and recruited participants by phone.

Cognitive Interviews:

We completed 25 cognitive interviews (19 in English, 6 in Spanish) to (1) collect feedback about the interface and content of the tablet-based survey, (2) compare the tablet-based survey to the standard paper-based CG-CAHPS, and (3) elicit perspectives about reporting experience to providers and clinics. Participants reported demographic information (gender, age, race/ethnicity, and education), current health conditions, and current medication use. To assess health literacy status, we used a validated screening question, “How confident do you feel filling out medical forms by yourself?”,^23,24 categorizing participants noting quite a bit of confidence or less as having limited health literacy.^24,25 Participants reported how often they used a tablet or smartphone. We asked participants to report how interested they were in using the Internet to manage their healthcare (5-point Likert scale).

To gain perspectives about usability and acceptability during the cognitive interviews, we gave participants two versions of the survey (paper versus tablet-based) in a random order and followed a standard think aloud approach.^26,27 To provide a standardized, usual care comparison of the CG-CAHPS survey, we provided a paper version of the CG-CAHPS survey with the original wording of 16 questions from survey version 3.0. The tablet-based survey contained the same questions adapted for a low literacy audience. If participants asked for help at any step while using the tablet, the research team provided assistance to the next step of the process. Following the completion of both surveys, we used a retrospective approach to ask participants to reflect on their overall satisfaction, usability of each survey, and comparability of the 2 modalities.

We administered the tablet-based survey on an iPad Air 2. Interviews were video-recorded using Game Capture software. Interviews were transcribed and de-identified prior to analysis. Participants received $25 for participating in the study. The [Institution] Institutional Review Board approved the study.

Analysis:

We first summarize the major tablet application design changes made, including the patient advisory board feedback and the pre and post readability level of each survey item.

Next, within the cognitive interviews, we summarize participant characteristics and then present our qualitative findings related to acceptability, usability, and relevance of the shorter, lower literacy CAHPS items and the process of completing the survey on the tablet. To identify themes in perspectives about patient experience and preferences for survey administration, authors read the interview transcripts in their entirety before independently analyzing them. We used deductive analysis, informed by the interview guide to identify themes specific to the question domains and predetermined categories of patient experience and usability. We also used inductive coding to identify emerging themes, meeting regularly to discuss and establish consensus on themes.²⁸ Coding was done using Dedoose (Manhattan Beach, CA). During cognitive interviews, we also quantitatively assessed respondents’ answers on the written and tablet administration of the survey, to calculate the concordance of answers between the 2 modes of survey administration.

Results:

Participant Characteristics:

Half (13, 52.0%) of the interviewee sample was male, the majority (21, 84.0%) were non-White, and the mean age was 53. Over half (14, 56.0%) had at least one chronic condition, and 68% (17) reported limited health literacy. While the majority (18, 72.0%) used a smartphone daily, tablet use was less common (Table 1); 40% of participants had never used a tablet. Interest in using the internet to manage healthcare was mixed; one third (9, 36.0%) reported high interest.

Table 1.

Cognitive Interview Participant Characteristics

Characteristic	Overall (n=25)
Gender, n (%)
Male	13 (52.0)
Race/Ethnicity, n (%)
Black or African American	4 (16.0)
Hispanic or Latino	12 (48.0)
Asian or Pacific Islander	3 (12.0)
White or Caucasian	4 (16.0)
Other or Mixed	2 (8.0)
Age, mean (SD)	53 (10.6)
Highest Education Completed, n (%)
Less than High School Degree	6 (24.0)
High School Degree	6 (24.0)
Some college or 2-year degree	9 (36.0)
4-Year College or more	3 (12.0)
Chronic Conditions, n (%)
Hypertension	10 (40.0)
Diabetes	8 (32.0)
Asthma or COPD	3 (12.0)
Heart Disease	3 (12.0)
Number of Medications, mean (SD)	2.2 (2.7)
Health Literacy, n (%)a
Adequate Health Literacy	8 (32.0)
Limited Health Literacy	17 (68.0)
Very Limited Health Literacy	9 (36.0)
Tablet Use (Frequency), n (%)
Daily	7 (28.0)
Weekly	3 (12.0)
Every 2–3 weeks	2 (8.0)
Monthly or less	3 (12.0)
Never	10 (40.0)
Smartphone Use (Frequency), n (%)
Daily	18 (72.0)
Weekly or less	1 (4.0)
Never	6 (24.0)
Interest in Using Internet to Manage Healthcare, n (%)
High interest	9 (36.0)
Some interest	6 (24.0)
Neutral	3 (12.0)
Low or no interest	7 (28.0)

^aHealth literacy status measured using screening question “How confident are you filling out medical forms by yourself?”, categorized as adequate (extremely), limited (quite a bit, somewhat, a little bit, not at all), and very limited (somewhat, a little bit, not at all

Question Adaptations and Concordance:

Response concordance between the tablet- and paper-based survey questions ranged from 41.7 to 100.0%, with an average of 82.5% (Table 2). Concordance was lower for the questions in which meaning had been altered through adaptation or in which response options had been adapted to allow participants to note non-applicability. For example, concordance was lowest (41.7%) for the adapted question “How often were you able to get your test results if you wanted them?” (Table 2, Question 8). Some participants found it difficult to answer the original question since the need for follow-up is often variable based on the specific test, echoing perspectives from patient advisory board members who had informed survey adaptation. In addition, 4 participants noted that this question was not applicable by choosing the response option “I didn’t have any tests done” in the adapted question. Concordance was highest for questions for which the original CAHPS wording remained largely intact (Table 2, Questions 4, 10–11, 14–15).

Perspectives of Survey Administration and Usability:

Overall, there was a preference for the tablet vs. paper administration, and several major themes emerged about usability (Table 3). Of the 25 participants, 18 (72.0%) noted that they preferred the tablet-based survey and 4 (16.0%) preferred the paper-based version. For example, one participant remarked, “I liked it [the tablet version] because everyone will fill it out when it’s that easy. If it’s difficult, then people like me who don’t have a lot of education will have trouble” (Age 55–64, Female, Latina, Spanish speaker). Three participants (12.0%) stated no preference for either mode, one of whom noted that their preference would be situational: “Sometimes, it’s quicker when you use a tablet because you can just zip through, and sometimes if you want to take a little more time to write out your answers, then it’s good to use paper, so it depends on the situation” (Age 35–44, Male, Mixed Race/Ethnicity). Despite varying experience using tablets, all participants could complete the survey with light assistance. The most common usability barriers were lack of knowledge of how to use an onscreen scroll bar (6, 24.0%); general difficulties using a tablet, such as pressing too hard (5, 25%); and issues using an onscreen keyboard (3, 12.0%).

Table 3.

Perspectives of Survey Administration and Reporting Care Experiences

Theme	Subtheme	Examples Quote(s)	Proportion of Participants
Benefits of Tablet-Based Survey	Ease of Use	“This is quite easy to do, because everything is easy to read. You just literally have to point your finger at your answer.” (Age 35–44, Female, Mixed Race)	Most
	Modernity	“I think I prefer it – yes, definitely, probably a tablet. Isn’t that like [how] everything is done electronically these days?” (Age 35–44, Female, Hispanic or Latino)	Most
	“Fun” element	“But when you do it on a tablet, it’s more exciting…So people are going to like it. I liked it so I went ahead and did what I had to do and answered everything.” (Age 55–64, Male, Hispanic or Latino)	Some
	Speed	“It’s faster [than filling out the paper survey]…Maybe because it has figures and stuff like that.” (Age 55–64, Female, Hispanic or Latino)	Some
	Administration at the point of care	“It gives me something to do, right?…I would have more fun doing the tablet and I don’t mind sitting there for 15–20 minutes waiting for my doctor to come out.” (ID#14, Age 55–64, Male, Asian) “It’s better in the clinic because you’re in the same environment.” (Age 55–64, Female, Hispanic or Latino)	Some
Benefits of Paper-Based Survey	Lack of familiarity or experience with tablets	“I can’t push the little button. [Laughter] Is there something, somewhere I’m supposed to…?” Interviewer: “Try lightly tapping it. Does that work?” (Age 45–54, Female, Black or African American)	Some
	Ease of administration	“It’s not as pretty as the tablet one but it’s still easy to navigate through and fill it up faster.” (Age 45–54, Male, White)	Some
	Potential for mistakes using tablet	“Because if you’re not careful, you will have answered a question without realizing it and you’ll jump to the next question. This can mean that you submit a wrong answer.” (Age 35–44, Female, Hispanic or Latino, Spanish speaker)	Some
	Preference for traditional administration	“No, there are a lot of old people, people who are older than I am, and they are used to reading a piece of paper and they wouldn’t complete a questionnaire on a computer or a tablet.” (Age 55–64, Female, Hispanic or Latino, Spanish speaker)	Few
Care Experience	Importance of reporting care experiences	“I know how important it is for surveys, because if I don’t say anything, then nothing changes and they don’t keep up with it. So if they just assumed that people are getting the service they want or need, then they don’t change anything.” (Age 35–44, Male, Mixed Race) “I think it’s really important to know how – if somebody feels frustrated about something, there are things that people might not feel comfortable telling somebody…being able to put it into – anonymously maybe, is really a good idea.” (Age 45–54, Male, White)	Most
	Understanding that metrics don’t always reflect effort or quality of care	“It’s just that sometimes it’s a long wait. That’s why they told me, ‘You’re a very patient person. You’re just sitting there while other parents, others will just get really mad.’ I don’t know, something I learned to just be patient. They try to do their job and do it as fast as they can but they can commit a mistake. They commit a mistake if they’re just rushing to do things. You don’t want to do that, especially people’s health.” (Age 35–44, Female, Hispanic or Latino) “I want them to know that they’re doing a great job but if I put ‘sometimes’, it doesn’t necessarily translate that way from the way it looks on the survey. And if I’ve never spoken to them about a prescription in the last six months, I would put ‘never’ and that’s a negative against them. So this whole question is very difficult to answer.” (Age 45–54, Male, White)	Some
	Need for open-ended questions	“Maybe the survey could include a space for comments so people could write and the clinic could read about where they need to make improvements.” (Age 35–44, Female, Hispanic or Latino, Spanish-speaking)	Some
	Desire to report good feedback or gratefulness	“I like to thank them. At times that it’s necessary or any time that I get treatment or treated there. I’m grateful. They should know that it’s appreciated.” (Age 45–54, Male, Hispanic or Latino) “You know how people complain all the time but then it’s like, ‘What about all the good stuff?’” (Age 55–64, Female, White)	Some
	Feeling that survey would not resolve personal issues or improve care	“That’s something that you can talk to your doctor…If you have good communication with your doctor, you would let him know what happens. Or here in the survey, you will get 1,001 answers and it would get you nowhere…It will be good for the survey, but not for the clinic.” (Age 65–74, Male, Hispanic or Latino)	Few

Technical Advantages of Tablet-Based Survey:

Many participants embraced the idea of using a tablet-based survey as fitting in with the technology-driven modern world: “People are used to playing with their phones and it’s just kind of—it’s more familiar” (Age 55–64, Female, Latina). Some participants found the tablet survey more fun or exciting than completing the paper-based version: “It’s a new thing. I’m going to go home and tell my kids I touched on a tablet” (Age 45–54, Female, Black). Although both surveys contained the same number of questions, some participants reported that it was faster to complete the survey using the tablet, citing the use of figures, survey interface, and adaptations to questions to improve readability: “If each question has its own page…it feels like it will be faster. I don’t think there was a huge difference between this [tablet-based survey] and this [paper-based survey] except that this was a little clearer than the paper. This felt faster…probably a little faster because of the way the questions were asked” (Age 45–54, Male, White).

Administration at the Point of Care:

Participants also noted the value of completing a survey at the point of care, which would: (1) allow them to reflect on the quality of their care while in the same environment, (2) occupy their time while waiting for an appointment, and (3) eliminate the barriers to completing and returning the survey from home: “…When I’m at home—it’s like, ‘I’ll do it later.’ I might get around to it and I might not. So, if I’m already at the doctor’s office, it’s going to get done” (Age 55–64, Female, Latina). While administration at the point of care was a benefit for most participants, a couple of respondents noted the potential for breaches in privacy in a clinical setting: “I think that you have that at home and you tend to be a little bit more private, so you have more time to write it out because I think that tablets are a little more public…Somebody could be reading over your shoulder.” (Age 35–44, Male, Mixed Race/Ethnicity)

Familiarity with and Tradition of Paper-Based Survey:

For the small minority of participants who preferred the paper-based survey (n=4), lack of experience with mobile devices played a large role in determining their preference. One participant noted: “There are a lot of people, myself included, who… It’s not that we are against the tablet. It’s just that we are a little afraid to use it. We think if we don’t understand it, we’re better off not touching it” (Age 55–64, Female, Latina, Spanish speaker). While all participants completed the paper survey unaided, some expressed that they would require assistance in using a less familiar tablet-based survey, or that the introduction of technology increased the potential of making mistakes or submitting inaccurate answers.

Perspectives on Reporting Care Experiences:

Table 3 summarizes participant perspectives about whether and how the survey items captured their experiences with healthcare. The most highly prioritized domains by the participants were provider communication, access to care, and staff respect. While participants valued aspects of quality from their providers, they often emphasized that sharing information about their health was most important.

Most participants emphasized the importance of anonymously reporting care experiences, which empowered them to drive changes in their care. However, participants noted that the quality of their care was not always concordant with the metrics that are captured by patient experience surveys. In a few instances, the discordance between care experience and survey measures drove participants to intentionally choose responses that protected their clinic. In response to the timeliness of care question, one participant explained the discordance between her response of “usually” and her actual experience: “I lied on that one… I’ve never seen my doctor within 15 minutes when I go in the office…I’ve sat there for almost 45 minutes in that room. I’m not complaining because it’s the doctor. I mean, some people have more special needs” (Age 45–54, Female, Black or African American). In addition, some participants felt that the current survey did not contain enough opportunity to report gratitude or good quality of care. For a few participants, the survey was insufficient to enact meaningful change in the quality of their care or to resolve negative experiences, which they felt were more appropriate for in-person communication. To improve the capture of their true care experiences, some participants suggested the addition of free response questions to allow for more nuanced and personalized reports of care.

Discussion:

Despite varying levels of education, health literacy, and experience with tablets, we found that both English- and Spanish-speaking patients in a safety net healthcare setting strongly preferred a tablet-based delivery of CAHPS patient experience questions vs. traditional paper-based survey methods. Overall, acceptance and preference for tablets seemed to be linked to the streamlined design of the tablet application which presented only a small number of usability barriers. In addition, the shorter and lower literacy survey delivered via tablet appeared functional for most items, and there were clear priorities among patients in using the experience survey to report about the relationships/communication with their care team and access to care.

While there is evidence in hospital settings that differential response rates do not bias performance comparisons following adjustment for case mix and nonresponse bias,^29,30 improving response rates particularly for populations with traditionally low representation may improve the accuracy of reports of care in diverse settings. In a study comparing web-based to mailed CAHPS collection, web-based administration yielded lower response rates, but comparable results for the majority of CAHPS domains.¹⁰ However, other studies have found that evaluations of care may be influenced by mode of administration.^31,32 Moreover, studies examining the effects of readability have found that improved readability of CAHPS questions¹⁵ and survey instructions³³ may improve response rates. While formal quality reporting programs using CG-CAHPS questions require use of a mixed-methods protocol of mail and telephone-based administration,^8,9 this study provides exploratory results about the acceptability of survey administration at the point of care. As health systems increasingly explore customized administration of patient experience surveys, further research is needed to examine the effects of enhanced readability and survey mode on ratings of patient care.

Although the study was limited to a small sample, the use of cognitive interviews allowed us to gain in-depth perspectives about the design and content of a low-literacy tablet-based CAHPS adaptation. While the study was not a psychometric evaluation of the low-literacy survey items or tablet modality, we present exploratory results on the acceptability and reliability of the presented items among safety net patients. Because we adapted the CAHPS in content, presentation, and mode before cognitive interviewing, it is difficult to know what types of adaptations were most influential in directing preference for the tablet-based survey. Although we know of other efforts underway to create lower literacy³⁴ and tablet-delivered CAHPS survey items,¹⁸ this is one of the first studies to our knowledge that has officially reported on this process. Moreover, we feel that this study is unique in harnessing an in-depth collaboration with a digital health company (Tickit Health) to create a final product that maximized readability and usability, as opposed to putting verbatim text onto a mobile platform without as much attention to the design process. Of note, this company had previous experience in participatory mobile health design in multiple languages, which directly informed this work.^35–37

Moving forward, there is more work needed in several domains. First, there are multiple questions about the best strategies for reporting processes. Operational and practical work is required to determine the feasibility of mode of delivery (emailed link, texting, in-clinic, mailed). Although almost all participants in this study were open to tablet-based survey administration, providing options for completing the survey will be critical for matching patient preferences. Moreover, integrating survey administration into clinic operations will require work to optimize clinical workflows, protect patient privacy, and ensure that point-of-care data collection does not compromise data quality and validity. Finally, while employing customized surveys and collecting open-ended feedback would allow patients to report experiences that may be more personal,³⁸ identifying and addressing issues may require significant time, may be less comparable across patients, and may be less applicable for formal reporting purposes.¹¹ As we move forward with federal policy supporting patient experience data collection and reporting, this study provides next steps to ensure underrepresented and vulnerable patient perspectives are engaged and represented in this process. If designed with patient input, tablet-based surveys may be a feasible and effective method for collecting patient experience data at the point of care.

Supplementary Material

Appendix A

Respondents to tablet-based (n=334) and paper-based (n=134) patient experience surveys by race/ethnicity and age during a 1-year QI pilot (September 2015 to July 2016)