Abstract
Policy Points.
Narratives about patients’ experiences with outpatient care are essential for quality improvement because they convey ample actionable information that both elaborates on existing domains within patient experience surveys and describes multiple additional domains that are important to patients.
The content of narrative feedback from patients can potentially be translated to improved quality in multiple ways: clinicians can learn from their own patients, groups of clinicians can learn from the experience of their peers’ patients, and health system administrators can identify and respond to patterns in patients’ accounts that reflect systemic challenges to quality.
Consistent investment by payers and providers is required to ensure that patient narratives are rigorously collected, analyzed fully, and effectively used for quality improvement.
Context
For the past 25 years, health care providers and health system administrators have sought to improve care by surveying patients about their experiences. More recently, policymakers have acted to promote this learning by deploying financial incentives tied to survey scores. This article explores the potential of systematically elicited narratives about experiences with outpatient care to enrich quality improvement.
Methods
Narratives were collected from 348 patients recruited from a nationally representative Internet panel. Drawing from the literature on health services innovation, we developed a two‐part coding schema that categorized narrative content in terms of (a) the aspects of care being described, and (b) the actionability of this information for clinicians, quality improvement staff, and health system administrators. Narratives were coded using this schema, with high levels of reliability among the coders.
Findings
The scope of outpatient narratives divides evenly among aspects of care currently measured by patient experience surveys (35% of content), aspects related to measured domains but not captured by existing survey questions (31%), and aspects of care that are omitted from surveys entirely (34%). Overall, the narrative data focused heavily on relational aspects of care (43%), elaborating on this aspect of experience well beyond what is captured with communication‐related questions on existing surveys. Three‐quarters of elicited narratives had some actionable content, and almost a third contained three or more separate actionable elements.
Conclusions
In a health policy environment that incentivizes attention to patient experience, rigorously elicited narratives hold substantial promise for improving quality in general and patients’ experiences with care in particular. They do so in two ways: by making concrete what went wrong or right in domains covered by existing surveys, and by expanding our view of what aspects of care matter to patients as articulated in their own words and thus how care can be made more patient‐centered. Most narratives convey experiences that are potentially actionable by those committed to improving health care quality in outpatient settings.
Keywords: narratives, patient experience, patient surveys, patient‐centered, quality improvement
Calls to deliver health care in a way that better aligns with patients’ needs and priorities are increasingly prevalent.1, 2, 3 Patient‐centered care, shared decision making, patient activation, and patient engagement are just a few of the terms commonly used to characterize a system in which patients’ values, expertise, and experiences are prioritized. In the words of a 2017 National Academy of Medicine (NAM) report on patient and family engaged care (PFEC), “In a culture of PFEC, patients are not merely subjects of their care; they are active participants whose voices are honored.… Individuals’ (and their families’) expertise about their bodies, lifestyles, and priorities is incorporated into care planning and their care experience is valued and incorporated into [quality] improvement efforts”4 (p1) (italics added).
Patients’ care experiences must be systematically assessed before they can be consistently incorporated into improvement efforts. The shift in survey approaches during the 1990s from measuring patient satisfaction to measuring patients’ experiences constituted a crucial first step in this direction.5 Over the past two decades, patients have been able to report on aspects of their care experiences on standardized surveys. In the United States, the Consumer Assessment of Healthcare Providers and Systems (CAHPS) initiative, supported by the Agency for Healthcare Research and Quality (AHRQ) since 1995, illustrates this approach and has played an increasingly central role in national quality improvement (QI) efforts.6 Over the past decade, comparable surveys have been deployed across Europe and the Commonwealth countries.7
Because health care in the United States operates as a marketplace within which (at least some) patients can leave their current clinician if care does not meet their needs, providers have long had some incentive to use patient experience surveys to understand consumers’ perspectives.8 With the inclusion of CAHPS measures in the Centers for Medicare and Medicaid Services’ (CMS) reimbursement formulae, patients’ experiences—as represented by their responses to cognitively tested survey questions—became an even more substantial part of the mainstream value equation in health care. As summarized by the NAM report, “Recognizing the importance of not only what care is provided to patients but also how effectively it is provided from the patient perspective has literally changed the conversation of health care leaders today.”4 (p21)
Along with efforts to systematically assess patient experience came changes such as staff‐development initiatives, new organizational structures, and the establishment of high‐level administrative positions (often called “chief experience officers”) in many health systems. By 2017, 70% of US hospitals and 60% of US practices reported having a specified senior‐level staff person in such a role, and investments in staff to focus on patient experience are also on the rise.9 Although survey scores for hospitalized patients were the initial focus for these new leaders because of their early inclusion in CMS reimbursement formulae, surveys of patients’ ambulatory care are increasingly seen as crucial, and Congress has mandated that outpatient surveys be used in Medicare payment formulae for ambulatory care and comparative reporting websites.10
As patient experience surveys and professionals proliferated, another trend emerged: users of health services increasingly gave voice to their own experience in narrative form. In the past decade, narratives have been increasingly elicited by advocacy organizations, clinicians, and health systems.11, 12, 13 Patients also began to volunteer their own narrative comments about experiences with care on the Internet via social media, in blogs, and on rating sites such as Yelp, Angie's List, and HealthGrades.14, 15, 16 Some patients added comments to CAHPS or other patient surveys, offering qualitative data in response to questions like “Do you have anything to add?” or just handwriting descriptions of their experiences in survey margins.17 As patient experience surveys added more ways for patients to incorporate open‐ended responses, the percentage of respondents leaving comments increased from 15%‐20% to 60%‐75%.16, 18, 19, 20, 21
As use of the Internet grew by leaps and bounds, additional evidence that patients are eager to describe experiences with care in their own words mounted. By 2015, narrative comments had become the form of patient‐reported experience that the American public was most likely to encounter when searching for information about clinicians on the Internet.22AHRQ responded by investing in the development of a validated set of open‐ended questions able to reliably capture meaningful, complete, and balanced narratives from diverse patients.23 A beta version of that instrument was released by the agency in 2016 as a supplemental item set for the CAHPS Clinician & Group (CG‐CAHPS) survey.
The benefits of leveraging scores constructed from closed‐ended questions on patient experience surveys for QI in hospital settings have been well documented.24 Although the impact of patient comments on these surveys has been less carefully studied, some evidence indicates the inclusion of open‐ended feedback from patients is well received by clinicians in inpatient settings.17, 18, 25 Researchers note, for example, that comments play “a central role in follow‐up activities” related to QI by providing concrete information about the events being reported by patients.18 (p514)
In outpatient settings, however, the potential for leveraging patient feedback has so far been less evident. Later implementation of patient experience surveys for ambulatory care relative to hospital care may be one reason for the lack of evidence, and limited sample sizes for individual clinicians may be another.26 A recent review of studies from multiple countries concluded that data from patient experience surveys, even when explicitly deployed for QI, “has not been found so far to result in significant improvements in future patients’ experiences at a practice or practitioner level.”27
Research suggests that there is potential for patients' narrative input to enhance outpatient care. For example, early exposure of clinicians to patient feedback about ambulatory care did reveal that many practitioners see value in patient narratives.28, 29 As one study summarized, “Inclusion of qualitative patient feedback, providing explanation for ratings, presented at the practitioner level could also enhance the attention paid to it in practice.”25 However, other clinicians remain skeptical about the accuracy of patients’ comments characterizing their practice12 and are uncertain about the actionability of this feedback for improving quality of care.6 Challenges interpreting the meaning of free‐text comments have also been documented.6 A small body of research has examined the content of patient comments about ambulatory care, but no studies have explored their value for improving quality or reshaping clinical practices.30, 31, 32 It seems clear from the existing literature that for narratives’ potential to enhance QI to be fully realized, innovative approaches to enhance their diffusion will be necessary.
Existing research has not yet answered some critical questions about the innovative potential of qualitative feedback for QI in outpatient settings. How might the plethora of patients’ comments about ambulatory care, and the newly developed capacity to elicit comments about outpatient treatment in a more rigorous manner, be deployed to enhance the quality of outpatient care? How can patients’ stories about care experiences with clinicians enhance improvement processes above and beyond what is possible using standardized survey metrics alone? Can narrative descriptions add to our understanding of the experience of care and of what specific aspects of care did and did not work well? In rapidly changing institutions and practice settings, where “the keys to improving patients’ experience of care are neither obvious nor effortless,”33 what promise do stories hold? In brief, what can patients’ own words convey to clinicians, QI teams, and organizational leaders seeking to understand and improve patients’ experiences with care?
This study lays the foundation for addressing these questions. It does so by creating a conceptual framework for anticipating what facilitates or impedes the diffusion of narratives into quality improvement; by defining and rigorously measuring the attributes of narratives that make them meaningful and actionable; and by assessing the prevalence of those attributes in a representative sample of 348 elicited patient accounts about outpatient care. We focus on the sorts of narratives that can be elicited through existing large‐scale patient experience surveys, thereby leveraging a representative sample of patient feedback for insights about clinical practice.23 The paper concludes by considering how such narratives can be situated within broader QI initiatives and what distinctive contributions they might make given that many different kinds of narratives have value.
Conceptual Foundations
The rapid proliferation of comments being offered by patients online has evoked considerable speculation about their potential to enhance QI and promote more patient‐centered care.12, 26, 27, 34, 35, 36 But learning from and responding to narrative feedback are complex tasks. Indeed, they may require innovative shifts in how clinicians, their clinical and administrative staff, practice leaders, and health system administrators conceptualize, understand, and derive practical meaning from patient experience data.
To conceptualize how narratives from patients can be useful and used for quality improvement, we situate adoption of this emerging form of feedback within the burgeoning literature on diffusion of health care innovations. We approach the development of such a foundation with a hybrid of deduction and induction. First, we draw on past theories of health services innovation and diffusion to identify attributes of narratives that might impede or facilitate their adoption into practice. Second, using the narrative data itself, we employ qualitative analytic techniques to inductively identify patterns of narrative attributes that seemed to be plausible prerequisites to their optimal use for quality improvement.
Narrative Feedback as Innovation
Improving quality of care based on narrative feedback requires learning from patients’ experiences in new ways and developing analytic approaches quite different from those requisite for working with quantitative survey data.
Because narratives are conveyed in text that is sometimes garbled, fragmented, or laden with multiple plausible inferences, extracting meaning can call for interpretative skills that do not apply to data obtained from closed‐ended survey scales.15, 37
Because narrative feedback takes the form of words rather than numbers, identifying broad patterns in the content and variability of that feedback requires a reliable way to categorize reported experiences, rendering them countable and comparable over time and across sites.38
Because narratives convey what matters most to patients, their focus often extends beyond the domains of experience assessed by conventional closed‐ended survey questions.19, 32, 39 Learning from the content of these new domains is most likely to occur if there are individuals within organizations who both see these new topics as important and have the expertise to respond to the concerns.
Because narratives convey pathos, gratitude, personal priorities, irritation, and a range of other complex dimensions of personality and attributions, their messages are often multivalent and cannot be simplistically categorized as “positive” or “negative” assessments. Learning from narratives may therefore require an open mind, emotional intelligence, and the understanding that patients vary in terms of what they want and need.40
Learning from narrative feedback thus involves several distinct forms of innovation: developing new interpretive skills; creating an analytic infrastructure for identifying patterns within aggregated narratives; broadening the range of issues considered promising for QI; and attending to the emotional connotations of clinician‐patient relationships. In any given context, some of these capacities might already be in place, while others are still nascent. But all are essential to extract the fullest meaning from narrative feedback and to use such feedback to improve quality in the various ways we explore in this article.
Uptake of Narrative Feedback as Diffusion of Innovation
Attention to narrative feedback among health care providers has emerged relatively recently, gradually expanding over the past decade.12, 13, 15, 37 Within the United States, spread of attention to narratives remains incomplete: a recent assessment of the availability of patient comments on publicly accessible websites found that narratives are available for less than half of all clinicians.26 The substantial literature on the diffusion of health services innovations—now encompassing more than 500 studies—provides insight about factors that might facilitate or inhibit the willingness of key decision makers in the delivery system to embrace narratives as a tool for quality improvement.41, 42
The implementation of innovations in health care settings often depends crucially on buy‐in from frontline decision makers—clinicians, administrative staff, and practice leaders27,40,41—which is powerfully shaped by the contexts in which they practice. Achieving this buy‐in depends on the perceived value and relevance of the innovation among key actors, the perceived complexity of the innovation, and its compatibility with users’ existing aspirations and prevailing practices. Adoption and impact are fostered by higher perceived value, greater compatibility and relevance, and more generous external support and inducements, but are diminished by complexity.
The discussion section of this paper revisits these contextual factors in greater detail. The focus of our analysis here, however, is on perceived value. More specifically, we examine how the content of narratives themselves may influence their perceived usefulness and use as a source of innovation for QI—beginning with identifying two critical attributes on which usability and impact are likely to depend.
The scope of experience being reported. The degree to which the data in narratives overlap with existing incentivized patient experience surveys will potentially influence their perceived value among future users, the degree of perceived compatibility with existing QI practices based on survey results, and the financial incentives encouraging key actors to use them.
The actionability of such feedback. Only sufficiently concrete narratives are truly actionable. If they can inform QI efforts by supplying specific details about at least some aspects of when, what, where, who, and how, then they have potential to be used to modify problematic practices and emphasize effective ones.
Scope of Experiences
As policymakers around the world strive to make health care more responsive to patients and large‐scale patient experience surveys have become a favored tool, considering how patient narratives inform domains both within and outside the scope of those surveys provides a starting point for effectively incorporating narratives into QI efforts. These surveys focus on a consistent set of core domains, most commonly including patients’ assessments of communication with clinicians, access to services, and overall experiences with care.7, 43 In the United States, the CAHPS family of surveys, sponsored by AHRQ, has become the standard for assessing patient experience, in large part because the surveys are used by CMS to incentivize improvement and by many health systems as internal metrics of performance.6, 44 These surveys aggregate responses to individual questions into four to six composite domains of patient experience.
We hypothesize that narrative feedback that “fits” with these composites will feel most relevant to and compatible with existing improvement efforts because it can explain why patients rated experiences in these domains as they did. Past research demonstrates that assessments conveyed in comments predicted patients’ overall assessment of their health care experiences, above and beyond their responses to closed‐ended questions in the same domain.17, 45, 46 Because these are the domains incentivized for both clinicians and health care organizations, narrative content pertaining to these domains might be more readily adopted into existing QI initiatives.
Conversely, for aspects of patient experience not covered by conventional surveys, patient narratives introduce dimensions of quality or patient‐centeredness that will feel less familiar to those involved in ongoing QI efforts. This may create short‐term barriers to narrative uptake to the extent that novelty introduces uncertainty and reluctance to engage with new and ill‐defined domains of experience. More than half of all patient narratives related to hospitalization focus exclusively on experiences that fall outside the scope of conventional patient experience surveys.18, 39 These new domains include emotional support or comfort from clinicians,17, 39 perceived technical competence of clinicians,17, 39, 47 safety risks and medical errors,17, 25 coverage and billing issues,17, 39 and communication to patients from practice or hospital administrators rather than clinicians.17
Although the unfamiliarity of this broader scope may initially impede use of narrative feedback, we anticipate that in the longer run these expanded domains might be seen as particularly valuable for quality improvement precisely because they have not previously been a focus of QI. This leaves more opportunities for “low‐hanging fruit” in terms of future improvement.39, 47
For purposes of this paper, we constructed a tripartite categorization of narrative content: (a) integral to CG‐CAHPS closed‐ended questions, ie, elaborating on aspects of patient experience captured by closed‐ended questions, (b) proximal to closed‐ended questions, ie, covering aspects of care within the domains captured by the survey's composite measures but not directly matching the wording of closed‐ended questions, and (c) distal, ie, involving aspects of patient experience excluded or omitted from current closed‐ended survey questions.
Actionability of Narrative Content
Though patient narratives are often presumed to be “actionable,”12, 26, 27, 35, 36 there has been little careful definition of actionability nor assessment of its prevalence in patients’ descriptions. Past studies of health care innovation and the literature on the use of narratives in performance feedback for service providers suggest four key dimensions of actionability in the content of narratives.
The first is the specificity of the narrative content. Patients are not always able to accurately assess all aspects of their care, nor can they necessarily make reliable attributions about why an experience unfolded as it did.48, 49 But accounts that are more specific reduce the degree of inference required for those who read them; in short, they are less complex to interpret and thus more readily acted upon.17, 25, 50 The specificity of narrative feedback facilitates a constructive response. In the words of one research team that evaluated the impact of narrative feedback in hospital settings, “free text helps to identify specific actions … that can be acted on and that can feel ‘owned’ by staff.”32
Second, the emotional valence of the narrative matters. Positive and negative feedback will typically prove to be actionable in different ways, each subject to distinct constraints. Negative depictions will generally be actionable, as long as the patient experiences are conveyed with sufficient clarity and detail. Narrative feedback offers a particularly effective way of identifying negative experiences that could otherwise have been masked by patients’ generally positive assessment of their health care encounters. A substantial portion of patients (18%‐20%) who rate their experience with clinical staff in hospitals to be “top‐box” (most positive) on closed‐ended questions still have something negative to express in their comments,39, 51, 52 thereby identifying specific quality shortfalls even for clinicians or practices with generally high quantitative ratings.
But identification of correctable errors is not the only value of narrative content. Positive feedback can help with sustaining good practices or bumping clinicians up a notch, as long as the comments are specific and make clinicians feel seen and appreciated as individuals.53 For QI staff or practice‐level leaders, positive feedback about one clinician can be leveraged to encourage other clinicians to improve practices.54, 55 This can provide a particularly powerful inducement for change when colleagues in a practice can serve as exemplars for actions or attitudes that extend beyond the realm of conventional norms of practice.
Third, actionability depends in part on the prevalence of the actions or practices described by patients. Numerous clinical practices have been identified as elements of high‐quality or patient‐centered care yet remain unevenly pursued by clinicians. Examples include shared decision making56, 57 and various aspects of coordinated care.58 Narratives that single out examples of positive or negative experiences in these domains may help QI staff to encourage greater consistency simply by establishing that patients notice whether or not they occur and value consistent attention to them.
By contrast, some practices are rare, yet so clearly improve upon conventional provider behavior that they can catalyze broader transformative change. Examples include clinicians who proactively communicate with patients and families outside of office visits or clinicians who take responsibility for helping patients resolve problems with billing or insurance coverage. Clinicians who implement such practices are more the exception than the rule, but they can nonetheless serve as exemplars for “raising the bar” on clinical practice. Narrative data about exemplary practices may be most effectively used by clinicians willing to act as “local champions” for change55, 59 who encourage their peers to explore how to improve aspects of their own work.
Fourth, the actionability of patient narratives depends on the degree to which key roles and/or actors are mentioned. Actionability by no means requires that patients explicitly name those who acted in problematic or exemplary ways. Rather, narratives can be adequately actionable when they identify roles played in the health care setting—for example, whether miscommunication occurred because the treating clinician failed to explain matters clearly or whether other staff introduced confusion with garbled or conflicting guidance. The more clearly narratives identify the specific roles most responsible for particular outcomes, good or bad—especially in instances where multiple clinicians and staff were involved in care—the more actionable they are likely to be.
These four dimensions of actionability can appear in various combinations in any given patient comment. A review of past studies examining how providers interpret and learn from patient feedback suggests that it is useful to distinguish among three specific combinations, because each identifies a different pathway for learning and thus a different approach for incorporating narrative feedback into QI efforts.25, 27, 28, 29 Each of the three combinations also highlights different ways in which clinicians and their staff might engage with the content of the patient narratives and draw from them motivation to adapt existing clinical practices.
Negative and preventable care experiences. Comments that identify something problematic in a patient's recent health care experience are categorized as actionable by clinicians if and only if (a) the feedback is sufficiently concrete to determine what had transpired from the patient's perspective, and (b) the problem in question is clearly related to the actions or attributes of clinicians or their affiliated staff or some changeable aspect of the practice itself. Having the narrative itself identify the cause of problems is not a prerequisite for actionability, because investigating why difficulties occur is itself a meaningful form of action.
Positive and exemplary care experiences. Comments that identify actions or practices of clinicians or their staff that are clearly extraordinary are always actionable because they have the potential to elevate general norms of practice. If the lauded behaviors were sporadically evident for a particular clinician, actionability could involve making them more consistent or extending the positive practices to other clinicians. If the behaviors were always present, actionability would focus exclusively at the QI level, aiming to increase the prevalence of such practices among other clinicians or their staff.
Positive but not universal care experiences. Comments may include praise for actions that ought to be expected from any competent clinician, such as writing timely prescriptions or referring patients to an appropriate specialist. But some activities that represent well‐established norms of patient‐centered care, for example, helping to set up referral appointments or following up on treatment or test results, remain inconsistent in practice. QI teams might benefit from leveraging positive feedback about these activities for clinicians who engage in them to improve care provided by clinicians in the same practice who fall short.
Integrating Scope of Content With Actionability
Taken in combination with our tripartite categorization of narrative content in terms of its relation to closed‐ended CAHPS questions (integral, proximal, and distal), these three forms of actionability yield a 3‐by‐3 matrix (Figure 1). Each cell in the matrix identifies different opportunities for harnessing narrative content to improve ambulatory care quality. Each also identifies different pathways through which change would need to be actualized.
Figure 1.
Mapping the Potential Value of Narrative Content for Quality Improvement in Health Care
Abbreviations: CAHPS, Consumer Assessment of Healthcare Providers and Systems; QI, quality improvement.
“Composite” refers to the quantitative metrics constructed from the aggregation of survey responses to several different questions that are all in a particular domain of practice, such as provider‐patient communication or coordination of care. “Composite score” refers to the performance rating for a clinician or clinicians using that composite. “Overall ratings” refers to the quantitative assessment of the complete patient care experience, reported on CAHPS using a 1‐10 scale.
We hypothesized that the resulting nine categories would each be evident in the patient narrative comments we set out to analyze. However, we had no prior assumptions about which categories would prove most prevalent or demonstrate the most compelling evidence for encouraging change in clinical practices. The coding schemes and their applications described in this section constitute the first tools we know of for determining the prevalence of patient feedback in each of these nine categories. The conceptual schema presented in Figure 1 also highlights how the diffusion of narrative feedback, as a form of innovation, is mediated by the intersection of content and actionability. Consider two examples, drawn from opposite extremes of the schema.
When narratives document negative experiences that are integral to CG‐CAHPS composites (the upper left cell in the matrix of Figure 1), they refer to familiar domains for QI that are already monitored and incentivized by payers and health systems. We therefore anticipate that clinicians will be strongly motivated to learn from this feedback, provided the accounts are concrete enough to be actionable. In short, these are conditions in which the uptake of narrative feedback should be most widespread and readily encouraged simply by distributing comments to clinicians on a regular basis. Conversely, when narratives capture exemplary practices in domains distal from CAHPS composites, uptake will likely be more limited and promoting the use of narrative feedback most challenging. To be successful, QI initiatives based on these data may require the active involvement of clinician champions—ideally from the same setting—who have already adopted such practices.57, 60
Despite the persisting relevance of organizational and contextual factors,56 we anticipate that narrative content itself will strongly influence both perceived and actual utility of narratives, and thus their diffusion. This core hypothesis motivates the analysis presented in the following pages.
Research Methods
To address gaps in the existing literature and translate conceptual insights from the previous section into measurable attributes of narratives, we needed an approach that could meet several challenges. It needed to reliably assess how much the content of rigorously elicited narratives overlaps with conventional closed‐ended questions for outpatient care. Further, it was essential to determine the degree to which narrative feedback is actionable, as well as the relationship between the content areas of the narratives and how actionable they are.
The Narrative Elicitation Protocol
The 348 narrative comments used in this study were collected using the Narrative Elicitation Protocol (NEP) designated by AHRQ as a supplemental question set for CG‐CAHPS.14, 61 The NEP is composed of a sequence of five open‐ended questions (Figure 2). To replicate two of the primary modes currently used to collect patient experience surveys, narratives analyzed here were collected by both telephone (20%) and web‐based written (80%) modes.
Figure 2.
Question Wording for the CG‐CAHPS Narrative Elicitation Protocol
Because the NEP was designed to elicit fulsome reporting of patient narratives, it encourages more complete and detailed comments than are typically observed on websites reporting patient comments about clinicians.60, 62 This can be most simply conveyed by comparing word counts. On average, patient comments regarding clinicians that can be found on Internet websites run between 35 and 45 words.14 By contrast, the average word count for the narratives elicited by the NEP is about 130 words.
The Sample Used for Collecting Patient Narratives
Data were collected from a standing Internet panel of more than 50,000 households recruited and maintained by the research firm Gesellschaft für Konsumforschung (GfK). This panel is representative of the American public in terms of sociodemographics, Internet usage prior to joining the panel, and health status.63 A random sample of panelists was invited to participate, and those who agreed were screened for having contact with a clinician in the prior year.
Table 1 presents the characteristics of participants compared to the general American public. The sample was older and had more frequently used health care in the past year, but was only slightly more likely to be in long‐established relationships with the clinician about whom they were reporting. Because the sample included only those who regularly participated on an Internet panel, people with limited literacy or English proficiency were certainly underrepresented, compared to the general public.
Table 1.
Elicitation Sample Compared to US Adult Population
| Individual | Elicitation | US |
|---|---|---|
| Characteristics | Samplea (n = 348) | Population |
| Sociodemographics b | ||
| Age | ||
| <30 | 11% | 21% |
| 30‐44 | 17% | 26% |
| 45‐60 | 28% | 27% |
| >60 | 44% | 26% |
| Race/ethnicity | ||
| White | 78% | 66% |
| African American | 10% | 12% |
| Latino | 7% | 15% |
| Other | 5% | 8% |
| Education | ||
| High school or less | 39% | 42% |
| Some college | 30% | 29% |
| College graduate | 31% | 29% |
| Health Status and Utilization c | ||
| Chronic health problems | ||
| Yes | 37% | 50% |
| No | 63% | 50% |
| Physician visits in previous year | ||
| 1 | 21% | 31% |
| 2‐3 | 43% | 43% |
| 4‐9 | 29% | 22% |
| >9 | 7% | 4% |
| Time with current physician | ||
| 1 year or less | 24% | 37% |
| 2‐3 years | 20% | 19% |
| 3‐4 years | 17% | 12% |
| 5+ years | 39% | 32% |
Elicitation sample characteristics calculated by authors; data from GfK Knowledge Panel.
Sociodemographic data are from the 2012 US Census.
Population information on chronic health problems and self‐reported health calculated from the National Health Interview Survey.
Coding the Elicited Narratives
We used qualitative content analysis to code the narratives in two distinct yet interrelated ways: for domains of experience, and for actionability. In both instances, a three‐person team (including the first two authors as senior investigators) coded an initial subsample of narratives and then iteratively adjusted the coding scheme and associated coding conventions. Coding was then completed by two coauthors. For actionability, two clinicians (fourth and fifth authors) brought both their methodological expertise and their applied QI and clinical experience to the coding process, working with a subteam of other authors to develop, refine, and implement the coding process. The detailed norms and conventions we developed for both experiential domains and for actionability coding are available online as technical appendices (Online Appendix B and C).
Coding for Domains of Experience
Following the norms of qualitative content analysis, codes for organizing the narrative data into “domains of experience” by compressing “a large number of texts into fewer content related categories” were inductively identified from the elicited patient narratives.64 Some domains corresponded directly to with the existing CAHPS questions and were thus labeled “integral” to the survey. Other domains were in the CG‐CAHPS areas of access, communication, care coordination, and office staff, but not related to the survey questions themselves; these we called “proximal.” Finally, we identified clusters of codes outside the four composite areas of the CG‐CAHPS survey and labeled these “distal.” The final axial coding schema comprised 20 clusters of codes. Because we coded only segments of text long and coherent enough to be understood in coding reports, some segments were assigned codes corresponding to two coding areas (eg, both integral and proximal) because the segment contained one or more descriptors corresponding to each area.
Coding for Actionability
Our definition of actionability yielded five specific coding categories (each with subcodes) particularly relevant for QI. These can be most easily distinguished as the who, what, when, where, and how of the experiences described in the narrative. “Who” in the practice was identified by the narrative was coded using eight subcodes for titles such as “doctor,” “staff,” and so on. “What” occurred was captured first through exclusion/inclusion criteria for identifying actionable passages in the narratives, and then by coding for whether the action described was implicit or explicit (four codes) and derived from an attribute of the clinician or an action/practice (eight codes). “When” was coded according to whether the described events were sporadic or constant (three codes), and with respect to proximity to the office‐based clinical interaction (eight codes, including “before,” “during,” and “after,” and various combinations of these). “Where” was captured by four codes for setting (eg, in or out of the office). “How” was coded with four valence codes assigned according to how the patient contributing the narrative felt about the events described.
Finally, in our coding for actionability, we sought to distinguish positive experiences that were common in health care but not yet universally practiced (QI taking the form of ensuring that these become universal in their scope) from those that might be considered exemplary but were not yet a clearly established norm. Because norms of practice vary across specialties, we took as our baseline those standard in primary care, as identified by professional associations and federal agencies striving to improve quality (AHRQ and CMS) and interpreted during coding by clinician coauthors on this paper.
Reliability of Coding
Interrater reliability, assessed using the kappa statistic, was high for coding domains of both care and actionability (Table 2). The most reliable domain codes were for integral and distal codes (kappa = 0.89), the least for proximal codes (kappa = 0.81). The aggregate kappa for axial coding was 0.86. Reliability of the actionability coding ranged from a high of 0.95 for valence to a low of 0.72 for prevalence of established norms for practice. The aggregate kappa for actionability coding was 0.89. Although the coding was highly reliable, it should be recognized that for some narratives, not all the attributes of actionability could be fully identified. The most frequently unidentifiable attributes involved timing (17% of all actionable episodes), setting (11%), and the key actor or actors (10%).
Table 2.
Intercoder Reliability for Axial and Actionability Coding
| Axial Coding | Actionability Coding | ||
|---|---|---|---|
| Characteristic | Kappa | Characteristic | Kappa |
| Integral codesa | 0.89 | Valence | 0.95 |
| Proximal codes | 0.81 | Prevalence/norms | 0.72 |
| Distal codes | 0.89 | Reported behavior [What] | 0.86 |
| Periodicity [When] | 0.92 | ||
| Relevant actor [Who] | 0.91 | ||
| Identified actions [What] | 0.88 | ||
| Setting [Where] | 0.92 | ||
| Timing [When] | 0.92 | ||
Average kappa across categories in this set of domains.
Results
We convey our findings in 3 stages. We begin by presenting the scope of experiences described in the narrative data. Building on our conceptual model regarding the potential utility for narratives to drive QI (when collected in conjunction with CG‐CAHPS surveys; see Figure 1), we summarize findings that (a) provide detail about already‐measured aspects of experience (integral data), (b) describe aspects of care covered by the survey's composites but not by its specific questions (proximal data), and (c) focus on dimensions of experience not currently included on the survey at all (distal data). We then address the actionability of the elicited data—that is, the extent to which it can inform QI efforts by supplying specific details about when, what, where, who, and how, which can then be used to improve patients’ experiences with care by modifying problematic practices and emphasizing effective ones. We summarize the overall prevalence of actionability in the data as well as the prevalence of specific descriptors, and we report the balance of negative and positive events described. Finally, we consider the intersection of content and actionability to provide an integrated picture of the promise narrative data hold for enriching QI. Examples of narratives included directly in the paper's text are supplemented by five figures (Figures 4, 5, 6, 7, and 10), which present more extended quotes. Positive and negative examples used throughout the paper overall reflect the same balance found in the aggregated data (80% positive, 20% negative).
Figure 4.
Illustrative Examples of Concrete Narrative Feedback for the Communication Composite
Figure 5.
Illustrative Examples of Concrete Narrative Feedback for the Office Staff, Access, and Care Coordination Composites
Figure 6.
Illustrative Examples of Concrete Narrative Feedback for the New Domains Not in CG‐CAHPS
Figure 7.
Illustrating Extent of Actionability
Figure 10.
Intersection of Axial and Actionability From Domains Related to Communication (by Type of Actor)
Abbreviation: QI, quality improvement.
a “Composite scores” refers to the quantitative metrics constructed from the aggregation of survey responses in a particular domain of practice, for example, provider‐patient communication, coordination of care, and so on.
Scope of Experiences
The scope of experiences conveyed in the narratives was spread evenly among the integral (34.7%), proximal (31.1%), and distal (34.2%) categories (Table 3). Integral and proximal feedback—cumulatively nearly 66% of the total content—map to the CG‐CAHPS survey's 4 composite areas: communication, office staff, access, and care coordination. The remaining 34% of content falls into 11 categories of experience not covered on the survey.
Table 3.
Distribution of the Axial Codes and Prevalence of Narrative Comments
| Integral Codes | Proximal Codes | Distal Codes | |||
|---|---|---|---|---|---|
| Category | Frequency | Category | Frequency | Category | Frequency |
| 34.7% | 31.1% | 34.2% | |||
| Access to Care (CAHPS) | 4.8% | Access to Care (non‐CAHPS) | 4.2% | ||
| Ability to get immediate care | 2.4% | Electronic interactions (unrelated to questions) | 0.6% | Efficient interactions in the office | 1.3% |
| Timely checkups | 1.4% | Wait time in office | 1.1% | ||
| Timely answers to questions | 0.9% | Physical accessibility of office | 0.3% | ||
| Prescriptions apart from office visits | 0.6% | ||||
| Perceived accessibility (of physician) | 0.3% | ||||
| Flexibility with appointments | 0.8% | ||||
| Communication (CAHPS) | 18.6% | Communication (non‐CAHPS) | 15.8% | ||
| MD explains things clearly | 4.0% | Friendly/warm communication | 4.9% | Emotional rapport | 7.4% |
| MD listens carefully | 7.8% | Communication as discussion | 1.1% | Patient empowerment | 0.4% |
| MD respects what patient says | 2.4% | Uses humor to foster interactions | 0.8% | Mutuality | 0.5% |
| MD spends enough time | 4.3% | Personalized relationship | 3.5% | MD use of computer in office | 0.4% |
| Makes comfortable | 3.4% | ||||
| Treats as equal | 1.0% | ||||
| Inspires trust | 0.4% | ||||
| Proactive communication | 0.4% | ||||
| Care Coordination (CAHPS) | 3.3% | Care Coordination (non‐CAHPS) | 3.1% | ||
| MD knows medical history | 1.2% | Exchanges information with other providers | 0.4% | Care approach | 1.6% |
| Office follows up on test results | 1.7% | Facilitates referrals to specialists | 2.2% | Coverage/cost | 1.2% |
| Discusses all prescriptions | 0.3% | Discusses care received from other providers | 0.3% | ||
| Office Staff (CAHPS) | 8.0% | Office Staff (non‐CAHPS) | 7.9% | ||
| Clerks/receptionists (staff) helpful | 3.5% | Friendliness/niceness of staff | 2.8% | Non‐MD clinical staff | 3.9% |
| Staff courteous and respectful | 4.4% | Billing/financial/insurance assistance | 0.7% | Attributes of the practice | 1.6% |
| Competence of office staff | 2.5% | ||||
| Personalized approach by office staff | 0.8% | ||||
| Staff professionalism | 1.0% | ||||
| Perceived technical quality | 12.6% | ||||
| Perceived thoroughness | 3.7% | ||||
Integral and Proximal Content for the Communication Domain
Communication between patients and clinicians accounts for more than half of the narrative feedback related to the 4 CG‐CAHPS composites (Figure 3). Because data on communication are so prevalent relative to the other 3 composite domains, and because the disproportionate importance of communication both to patients and for QI is well documented,50, 65 we have elected to focus the bulk of our detailed description of narrative content related to integral and proximal codes through the lens of this single domain. (We briefly return to the other three CG‐CAHPS composites later in the paper.)
Figure 3.
Frequency of Domains of Patient Experience Reported in Narrative Accounts [Color figure can be viewed at wileyonlinelibrary.com]
In the communication domain, roughly half of the narrative descriptions illustrated aspects of existing CG‐CAHPS questions, thus offering details about why survey respondents scored their clinicians low or high and/or disclosing the specific dimensions of care that were most important to the patient. Respondents illuminated dimensions of the CG‐CAHPS question about whether “your physician explains things clearly” through almost entirely positive examples of clinicians being thorough, “mak[ing] sure that I have a complete understanding,” and talking in understandable “layman” or “simple” terms. Patients also offered details about what was or was not explained well (eg, medications, procedures, visit summaries) and consequences of various courses of action.
The second question in the communication composite regarding “listening carefully” was fleshed out via (again largely positive) descriptions of how patients assessed clinicians’ skills in this area—for example, by whether they “ask additional questions to help determine all symptoms,” “seem bored or bothered” rather than attentive, or provide patients adequate opportunity to “react and process.” The narratives also summarize perceptions of actions associated with attentive listening: one person noted that her doctor “listened, [and] changed meds right away,” another that the clinician “does listen and take action” by communicating with staff to problem‐solve.
Narrative responses to the third CG‐CAHPS communication question, “how often did this provider show respect for what you had to say,” emphasized feeling trusted, believed, and valued. As one patient put it, “he basically takes my word for things and he understands that I know exactly what I'm talking about.” Other patients specified being “included … in my care” or asked whether they agree with proposed treatments. On the negative side, patients described feeling pushed “through like cattle” or having doctors unwilling to accept patients’ treatment choices or take what they said at face value.
Responses to the final question in the communications composite—whether physicians spend enough time with patients during visits—highlight concepts such as clinicians being hurried or rushed on the one hand, or on the other hand, making time for important processes like reviewing treatment options, explaining procedures, or asking questions. Some narratives described feelings associated with clinicians’ time allocation practices; for example, one person noted that getting too little of the doctor's time made them “feel like a cheque and not a person,” and another that she or he felt “disappointed and insignificant.”
The other half of the narrative material related to physician‐patient communication clustered into 8 distinct proximal categories (frequencies for each are presented in Table 3; illustrative comments in Figure 4). The most prevalent of these categories describes the “feel” of communication from patients’ perspectives. Comments include descriptions of clinicians being open, honest, or personable and contain terms like “talks openly,” “greets warmly,” “interacts well,” or “we are cool.” Personalized relationships is another large category; it includes many highly descriptive narratives focused on, for example, finding things in common, being remembered “personally,” and discussing issues besides physical health, such as family members, pets, vacations, and “how things are going in life.” Patients noted their clinician “knows my demeanor,” is “quick to give you a hug,” or “consider[s] me a person who needs help and not another insured person they can bill the maximum amount.” The third most frequent category—patients comfortable being open with the doctor—is similar to personalized relationships, but here the narratives emphasize being relieved of embarrassment or nervousness, or being able to “ask anything without being made to feel foolish.” Patients also talked about providers relieving their anxiety, refraining from “judging me,” and being open or easy to talk to.
Less prevalent proximal categories highlight specific dimensions of communication. Narratives about humor, for example, highlight communication patterns such as telling jokes, laughing together, and “bantering back and forth.” Those focused on dialogue refer to communication where the clinician “doesn't make me feel like he is an authority figure or an expert giving me advice but rather [creates] a discussion” and to decisions arrived at together after talking through options. A distinct but related category is clinicians whose patients feel they are “treated as equals.” Comments here describe communication that explicitly levels the playing field such that “we almost interact like colleagues,” mutual respect prevails, and there is “none of the old ‘I am the doctor and you are the patient and you don't have anything valuable to add.’” Comments that explicitly emphasize trust, such as “he's very honest” or “I put my trust in him,” were coded as “doctor inspires trust.” Finally, the “proactive communication” category captures narratives about being consistently asked “do you have any questions”; clinicians going out of their way to talk with family members about, for example, the need for surgery; or, on the negative side, doctors neglecting to tell their patients they were going on vacation or what to do in their absence.
Integral and Proximal Narrative Content for Access, Staff, and Coordination of Care
Comments related to the other three CG‐CAHPS composites are, in aggregate, about as common as narrative feedback related to clinician‐patient communication. As is true for the communication domain, there are about as many comments in proximal aspects of these domains as in the integral domains closely linked with CAHPS closed‐ended questions. Figure 5 offers selected illustrations of the most prevalent of these integral and proximal aspects of access, care coordination, and office staff. A more complete set of examples, comparable to the illustrations of communication contained in Figure 4, can be found in Online Appendix A.
Distal Categories
Our qualitative analysis revealed 11 clusters of comments that fall outside the four CG‐CAHPS composites, yet are potentially important for QI because they provide an inductively derived view of what additional issues matter to patients. Figure 6 contains illustrative excerpts for each distal category. Here, we summarize our syntheses of the qualitative data to provide a more textured view of what the narratives convey and the promise they hold for QI.
Nearly a third of the comments coded as distal describe aspects of clinician‐patient relationships that could not be subsumed under the communication composite. “Emotional rapport” is the largest subcategory; comments here generally convey positive emotional valence but few specific details. Patients talk about providers who are warm, concerned, compassionate, caring, friendly, pleasant, easy to get along with, and empathic. They also describe relationships with clinicians characterized by professionalism, respect, trust, and the feeling of being “in capable hands” or “having confidence” in the clinician.
Other relational aspects of care described under distal codes are more specific and granular. Patients described holistic or personalized care approaches, emphasizing feeling providers are able to find “a middle ground that allowed me to be how I am and yet maintained her role and authority” or to focus not just on a specific disease but on “any other thing that's happening to me … [through] attention to detail.” Negative comments noted feeling treated in a “mechanical way” or with an “assembly line attitude.” Narratives about mutuality emphasize shared respect for one another, working well together, or examples of mutual connection such as praying together. Comments about empowerment focus on what it means to “take part in my care”: being encouraged by the clinician, not feeling judged, or finding that clinicians offer alternatives that inform patients’ decision‐making process.
A second cluster of narratives focus on perceived quality of care. These account for roughly half of the distally coded data and fall into two categories: observations related to clinicians’ perceived knowledge or expert capacity, and observations about the perceived thoroughness with which care is delivered. Narratives often focus on distinct aspects of the care encounter such as follow‐up, including how test results are communicated and acted on, post‐visit instructions, or medication adjustments. Some patients speak about not only what happened (eg, “they called my house almost every day just to check up on how things were progressing”), but also how they felt about what had occurred (eg, medication tracking and adjustments “helped alleviate my anxiety”). Many negative comments highlighted mistakes experienced during care, such as receiving a letter intended for a different patient or having to have blood redrawn because of faulty lab slips.
Distal data also highlighted some specific aspects of the care team and of care practices not captured within existing CG‐CAHPS composites. The most prevalent of these was experiences with clinical staff—mostly nurses, but also physician assistants or laboratory technicians who provide care alongside the physician. Notably, many narratives describe relational aspects of experience with other clinicians, a topic not directly addressed by existing CAHPS questions. Some patients highlight the comforting presence of clinical staff, for example, after a difficult procedure or hard‐to‐hear news, or voice objections to feeling slighted, ignored, “spoken down to,” or needlessly lectured. Other comments focus on procedures: a number of people described experiences (both good and bad) having blood drawn or undergoing other forms of examination. “A new nurse of his took my blood pressure,” said one patient, “and was unable to get the proper reading the first couple of times she tried it. Instead of telling me she was new and that it might have been her problem, she told me I was too thin and that I made it difficult for her to get the reading.”
Another narrative thread that extends beyond the CAHPS composites focuses on the efficiency of the office visit. Are things promptly ready for patients when they need to check out? Are issues dealt with in a timely way? Are test results received and processed quickly? Many patients’ descriptions about efficiency capture their values and priorities. Shorter visits facilitate access for those with limited free time or inflexible work schedules (“they took care of everything and it was in and out and no hassles”). Concise interactions are seen by some as reflecting clinical skills (“If I go in to see Dr. W he is on time, quickly gets to the heart of the matter and has a solution right at hand”), and by others as a sign of respect for their own busy lives (“I had to wait … I guess that the office does not value my time”).
A small but highly specific set of narratives focus on general attributes of the practice itself rather than any particular clinician or staff member. One such attribute is confidentiality: an objectionable “breach in privacy” that can occur in the lobby or waiting area. Another is the size and structure of the practice, which can result in being seen by “a floater MD” (with well‐ or poorly received results), or in confusion and high turnover among staff. Several patients also made observations about the way staff and providers work together, objecting to the way staff are treated or observing that “the doctor has know [sic] idea what is going on with the staff.”
Finally, several categories in the distal codes illustrate phenomena in the care setting not captured in the survey's four composites but that may emerge into increased prominence over time. Some of these are already captured in supplemental CG‐CAHPS item sets. For example, narratives highlight the impact of health technology, providing feedback on the perceived benefits and costs of clinicians using computerized systems. Some patients described enhanced coordination of care, ease and timeliness of information access during visits, or how “finding out the results [of my tests] immediately is wonderful.” Others noted feeling overlooked when clinicians attended to computers instead of them, or how glad they were that their doctor “sat down and listened to me and took notes, rather than entering information on the computer.”
Another emergent aspect of care sporadically highlighted in the narratives is clinicians’ practices vis‐à‐vis tests and treatments that have limited or inconsistent clinical value, for example, diagnostic scans as a quick response to reported pain, or antibiotics for symptoms that may well resolve without them. Patients assessed this aspect of care in varied ways. Some voiced appreciation for clinicians who recommended delaying invasive, expensive, or potentially unnecessary procedures. As one person put it, “I am a practical person and he does not require unneeded tests and things. I like that and respect him for it.” Others noted that their clinician is “not a pill pusher” or “gave options” before recommending surgery. Yet others described clinicians who order tests they don't believe are necessary, fail to act quickly enough, or make patients “beg and plead their case” before prescribing. Patients also opined that more preventive care (eg, vaccines) would have been useful, or they voiced appreciation for “preventive medicine.”
Comments about cost and insurance coverage issues with providers or at practices (as distinct from with office staff) also stood out in the narratives. The majority of these described negative experiences: providers who do not understand coverage or who fail to facilitate it through precertification (eg, of colonoscopies), appropriate coding of visits, or coordination among relevant parties. Comments about problems with cost included inaccurate billing, erroneous charges, and having cost issues disregarded by a provider who “doesn't hear me when I tell her what I can and cannot afford.” Other narratives highlighted good practices: providers who discuss cost openly and promote shared decisions about which expenses to incur; examples of seamlessly facilitated paperwork or coverage‐related referral forms; and direct phone calls or letters from providers to insurers.
Actionable Content in Patient Narratives
Narrative accounts identify actionable patient experiences by conveying in sufficient detail the who, what, when, and where of the event, as well as how the experience felt to the patient. To report on frequencies, we identified passages in each narrative that demarcated distinct events for the patients: many such passages were assembled by consolidating portions of a given patient's responses to multiple questions in the NEP. Each such event was then judged to be actionable or not based on the specificity of its description (Figure 7). We calculated and report here the prevalence of these actionable passages and their attributes: the who, what, when, where, and how across the 348 elicited narratives.
The Prevalence of Actionable Events
Overall, 80% of narratives contained actionable content. Fifty‐six percent had multiple actionable events, and 17% contained four or more distinct actionable elements (see Figure 8). Some narratives concisely characterized a well‐defined episode: “I was dizzy,” one patient said, “…it was horrible I would say it lasted for at least 30 min they {sic} nurse staff was wonderful stay right by my side until I could leave.” Another patient reported, via fragments dispersed throughout his narrative, being disappointed in his continuity of care, first noting that the clinic “has not been able to keep a steady staff of doctors. They all are quite able but they are always changing.” He acknowledged that the clinic's approach assured that “I could always find some physician who was working at the clinic to see me,” but since there was “always a new doctor,” he concluded at the end of the narrative that it was “hard to get the doctor acquainted with a patient.”
Figure 8.
Prevalence of Actionable Passages in Patient Narratives [Color figure can be viewed at wileyonlinelibrary.com]
The “What” of Actionable Events
Roughly 5% of reported events were associated exclusively with personal attributes of the clinician or staff, rather than their actions or practices. For example, some clinicians were characterized as “calm,” others as “pushy.”
The vast majority of actionable events (95%), however, were linked by patients either to particular actions (26%) or to consistent practices (69%) on the part of clinicians and staff. The following example illustrates both a discrete event (assistance offered when the patient's spouse died) and a consistent aspect of care (regular dialogue about personal experiences):
I have known and been served by my physician for over 25 years. He knows me and my family very well. He and I have similar interest outside of our professional relationship and often share experiences with each other. He was especially helpful when my wife passed away because he had known and treated us both as friends as well as patients for many years.
The “Who” of Actionable Events
Fifty‐five percent of the actionable events focused exclusively on actions of and interactions with physicians (Figure 9). Another 20% involved the primary clinician in conjunction with administrative staff, clinical staff, or both.
Figure 9.
Prevalence of Identified Actors in Actionable Elements of Narratives [Color figure can be viewed at wileyonlinelibrary.com]
Roughly 15% of actionable events centered entirely on actors other than the primary clinician. These sometimes involved clerical staff, in other cases clinical staff (eg, nurses, physician assistants, medical technicians). Ten percent had no clearly identified actor, referring instead to an undefined “they.”
The “Where” of Actionable Feedback
Seventy‐two percent of actionable events took place in the office setting. The remainder involved interactions via various forms of electronic communication. These ranged from the scheduling of appointments to the use of web‐based portals for communication with either physicians or their clinical staff. Interactions related to follow‐up after testing or treatment represented the largest focus of these out‐of‐office communications. For example, patients extolled practices where “Whenever I ask a question, they usually listen to me and call me if they don't have the answer with the answers” or clinicians who personally reached out, for example, by phone.
The “When” of Actionable Events
Timing of an event involves both its sequencing in a particular episode of care and the extent to which it repeats across episodes of care. Forty‐nine percent of actionable events were reported as taking place during the medical exam itself. Another 23% referred to interactions that occurred during the office visit either before the exam, after the exam, or both. Yet another 16% occurred outside of the office.
For almost all actionable passages (99%), it was possible to identify the frequency of the actions or events on which the patient was reporting. Three‐quarters were described as repeating or frequently observed; the remainder either were one‐off events or occurred sporadically.
How It Felt: The Valence of Actionable Feedback
Consistent with past studies about patient feedback on hospital care, the outpatient feedback from negative experiences tended to be more specific, thereby making it more actionable. Roughly 20% of the overall narrative content carries a negative valence, yet 32% of the actionable feedback involves either negative or mixed experiences. In other words, negative experiences are 50% more likely to be actionable than positive experiences. Because of the overall tendency toward positive valence, however, more than two‐thirds of feedback coded as actionable is positive. Much of this feedback (93%) describes practices that should be the norm for primary care but are inconsistently implemented by clinicians. A far smaller proportion points to exemplary behavior that could shift norms of practice for primary care.
Of the 274 respondents who provided actionable feedback, 45% described at least one actionable event based on a negative experience, and 13% described more than one. Of those who reported a single negative actionable event, 61% also reported a positive or mixed experience. Of those who reported multiple negative events, half also reported a positive or mixed experience. In short, most negative feedback was offered by patients who also had positive things to say about their medical care.
Data That Are Not Actionable
Twenty percent of the narratives lacked actionable content. Most often, this reflected a depiction of patient experiences that was either too cryptic or too vague to be considered actionable (Figure 7). This was most often the case when patients were describing general attributes, for example, “capable doctors,” “good staff,” or “pleasant visits.” In other cases, the depiction was more concrete, but some crucial element was missing, as when particular events or outcomes were attributed to a nonspecific “they” at the practice (Figure 9).
Other depictions were not actionable because the patients represented their own actions or circumstances as being primarily the cause—for example, when patients attribute attenuated relationships with clinicians to their own busy work schedules. In other instances, problematic experiences were attributed to clinicians’ choices that inherently involve trade‐offs among priorities—for example, with respect to the practices’ location (close for some patients, far for others) or the choice to spend extra time with certain patients and cause others to wait.
The Intersection of Substantive Scope and Actionability
Analyses connecting scope (axial mapping of content) with actionability reveal a straightforward pattern at the highest level of aggregation: a consistent 60% of feedback from patients meets our criteria for actionability regardless of whether the content of what is being described is integral to, proximal to, or distal from the CG‐CAHPS composites (Table 4). The proportion of narrative that describes negative experiences or exemplary practices is also quite consistent across the three foci.
Table 4.
Percentage of Patient Experience Comments That Are Actionable, by Domain
| Substantive Domaina | Any Form of Actionable | Actionable & Negative | Positive & Not Universal | Actionable & Exemplary |
|---|---|---|---|---|
| Integral to CG‐CAHPS | 63.1% | 18.0% | 42.6% | 2.5% |
| Proximal to CG‐CAHPS | 62.7% | 16.3% | 44.6% | 1.8% |
| Distal from CG‐CAHPS | 62.0% | 19.6% | 38.1% | 4.3% |
aIntegral: Same patient experiences covered by CAHPS close‐ended survey questions; Proximal: In same domains as CAHPS close‐ended survey questions, but address aspects of patient experience not covered by CAHPS questions; Distal: Involve domains of patient experience not addressed in the CG‐CAHPs surveys.
For 17 of the 20 content subdomains we introduced earlier (four each for integral and proximal codes, a dozen for distal codes), more than half of the coded comments were identified as actionable. There was little actionable feedback, however, that mapped integrally to the core CG‐CAHPS office staff questions on helpfulness, courtesy, and respect, although the proximal comments about clerical staff were 63% actionable, right on par for other domains of patient experience. Comments focusing on clinicians’ emotional rapport with patients also provided less actionable content (23%).
Feedback was disproportionately negative for a few content areas: access to care, office staff (clerical and clinical), coverage and billing, and general attributes of the practice. Negative feedback comprised less than a quarter of the actionable content for 14 of the 20 subdomains.
Discussion and Conclusion
The findings summarized here suggest that patients’ narrative descriptions about experiences with outpatient care are a robust source of feedback for enhancing the quality and patient‐centeredness of health care. They offer content that illuminates, in equal proportion, experiences that correspond to existing CG‐CAHPS questions, those that extend the scope of CAHPS domains, and those not included on the current CG‐CAHPS survey. Further, nearly two‐thirds of elicited narratives in each of these three content metacategories include actionable passages containing details about when, what, where, who, and how that can plausibly be used to modify problematic practices and emphasize effective ones (see Table 4, column 1), thus demonstrating the relevance of such patient‐reported information for QI innovations.
There will, however, undoubtedly be significant challenges associated with harnessing the potential of narratives for QI.27 In the final section of this paper, we explore some of these, further clarifying how survey‐based narratives might influence clinical practice and situating this particular qualitative approach within a larger set of strategies for improving health care using narrative feedback. We also describe some of the methodological limitations of our analyses and identify opportunities for additional research.
The Diffusion of Narrative Feedback Into Quality Improvement Efforts
The conceptual model we developed for this analysis operationalized three specific forms of actionability, each defined in terms of potential use of qualitative data by people motivated to improve care, ranging from individual clinicians to QI teams (Figure 1). Nearly 80% of actionable data contained in the 348 narratives we analyzed offered praise for actions that are not extraordinary per se, yet are notable because they are both meaningful to patients and not universally practiced (see Figure 10, column 2). QI staff can leverage such positive data to reinforce good practices where they are already occurring and to motivate clinicians and staff in need of improvement to up their game. Another 18% of actionable data offered concrete examples of problematic experiences arguably within the power of either individual providers or QI teams to alter (Figure 10, column 1). The remaining 3% described experiences that were not only positive but also exemplary in nature—inspiring examples capable of raising the proverbial bar for an entire practice if appropriately leveraged by local champions, QI experts, or other leaders (Figure 10, column 3).
Because two‐thirds of narrative data are either integrally or proximally related to CG‐CAHPS domains, the use of narratives as a QI innovation is likely to have immediate perceived value among ambulatory care clinicians, practices, and systems. Such users may also view narratives as compatible with existing QI practices, and, perhaps most importantly, as a potentially cost‐effective strategy for the almost universally relevant task of improving CAHPS scores and thus, in the longer run, reimbursement rates. Without understanding why patients may be generating lower than optimal or satisfyingly high survey scores, those seeking to increase or even maintain their scores operate with a decisive handicap. Equipped with concrete descriptions of what is going well and what needs improvement, those charged with improving quality are arguably better positioned to design and implement interventions informed by the collective experience of the very patients who iteratively generate CAHPS scores.
Feedback describing experiences outside the current CG‐CAHPS domains offers what might be seen as longer‐term promise for QI. Our analysis suggests that patients care deeply not only about communication but also about relational aspects of care such as emotional rapport, empowerment, and mutuality. Perceptions of technical quality and thoroughness—most of which are heavily mediated (both for better and for worse) by whether the patient trusts her or his clinician—also significantly influence patients’ experiences, as does the role of clinicians such as nurses and physician assistants.
Organizations that learn to pay attention to these dimensions of care may find experiences improve for both providers and patients. If patterns mirror what has already been found in hospital settings,51 they may also see an increase—one that has in the past proven elusive—in their overall CG‐CAHPS scores. QI leaders who begin collecting and analyzing patients’ narratives may also wish to expand the scope of their patient experience surveys over time to better capture what their populations care about—for example, by using appropriate existing supplementary CAHPS measure sets. In the longer run, measure developers may be motivated by analysis of patterns in proximal and distal narrative data to create new patient‐driven measures where adequate ones do not already exist.
Taking Narrative Feedback to Scale
Our findings present both qualitative and quantitative representations of patient narratives. We intend this mixture to convey the variety of ways in which narratives can be used to improve quality and promote clinical practices that are responsive to patients. For individual clinicians, we anticipate that the value of narrative feedback will come primarily from having each clinician read through the narratives submitted by his or her patients. Given current sampling norms for patient experience surveys, this interpretive task should involve no more than 20 narratives each month—not an overly daunting number to review. Since much of the actionable content centers on clinician‐patient interaction, a close reading by the clinicians most involved with these patients seems an important linchpin for responding to this feedback. That said, clinicians may find it challenging to make good use of narrative feedback. Practices will likely need to invest in interpretive analysis in order to help clinicians understand, respond appropriately, and ensure maximum benefit.6
At higher levels of aggregation (ie, the clinic or health system level), the value of narrative feedback may come more from categorizing and counting patient comments. Because this happens at a much larger scale, complexity is inevitable and remains a substantial challenge to the potential diffusion of narratives as a source for QI innovation. We have demonstrated here a set of analytic strategies for organizing narrative content so that its value can be understood and made use of for practice improvement. Nonetheless, narrative data will always be more complex than survey scores, and this may inhibit uptake. Further, the process of reducing complexity through coding and labeling of narrative content is labor‐intensive if done by human coders.
One alternative might be greater use of natural language processing (NLP) programs to categorize narrative content. A number of companies are already competing to apply NLP to narrative data to simplify it for QI use. However, we know of no analyses in the health care context comparing the efficacy of human and machine coding, let alone the capacity of NLP to capture the nuances of actionability introduced in this paper. Future research in this area will be important and should be designed so as to capture fully the skill and subtlety people trained to code qualitative data bring to the task—for example, the capacity to identify emerging patterns, cross‐cutting connections, and nuances of language.
The aforementioned companies are actively selling narrative analysis as a service to QI leaders, chief experience officers, and C‐suite executives, some of the very actors we identified as plausibly motivated to use narrative data for QI. However, rates of interest and uptake remain speculative, despite the growing enthusiasm about narrative feedback evident in health services and clinical journals. We also cannot ascertain whether these initiatives are more attuned to negative or positive feedback from patients, more focused on CAHPS domains than other aspects of patient accounts, or more attentive to some aspects of actionability than to others.
Situating Survey‐Based Narratives
Embedding qualitative questions in patient experience surveys is just one way of generating narrative data for QI. There is a growing literature on various approaches—many of them more intensive—to learning from patients’ qualitative experiences. Some initiatives have originated with researchers, advocates, or policymakers interested in patient stories as a catalyst for health system improvement. The rise of narrative medicine, for example, has led to training programs designed to teach clinicians to listen carefully to their patients’ stories and bridge the divide that can dehumanize medicine.66 The US Department of Veterans Affairs has implemented a suite of narrative projects to improve doctor‐patient communication.67 Other initiatives are even more firmly situated within the realm of quality improvement. Most notably, approaches based on experience‐based codesign use patient experience interviews as a catalyst for engaging patients with clinicians, staff, and administrators in projects designed to improve quality and render care more patient‐centered.68, 69, 70
Learning from in‐depth narratives is important in several ways. Such narratives convey a nuanced view of the patients’ lifeworld.71 They provide greater detail and coherence than those that can be elicited in large‐scale surveys.23 In the case of experience‐based codesign, they leverage rich narratives to engage patients in quality improvement.69 By contrast, embedding narrative elicitation within existing large‐scale patient experience surveys has the virtue of potentially “democratizing” patient feedback by eliciting responses from a wide range of patients, including those who might otherwise be reticent to discuss their experiences directly with clinicians.6 And it more closely connects narrative feedback with patient experience scores derived from closed‐ended questions, adding to the perceived relevance of patients’ own narratives by actors in health care who might not otherwise be practiced at valuing or using them.
We believe that narratives will continue to play an increasingly prominent role in quality improvement. As they do so, perhaps narrative elicitation through large‐scale surveys will prove most useful for identifying key opportunities and persisting problems. By contrast, we anticipate that experience‐based codesign and other patient‐engaged initiatives will play the essential role of field‐testing new interventions and assuring they continue to be informed by what patients know, value, and are able to contribute.
Limitations and Need for Future Research
The findings presented here are subject to a number of methodological limitations and point to a number of opportunities for future research. First, the elicited narratives we analyzed in this paper are richer and broader than naturally occurring comments on the Internet. Consequently, our results document the potential for narratives to enrich our understanding of ambulatory care but do not predict what might be gained from harvesting comments currently found on websites that publicly report quality metrics.
Second, identifying narrative content that could be actionable does not ensure that it always will be actionable in every clinical setting; substantial buy‐in from practice leadership is an essential prerequisite for constructive responses. We anticipate that narratives can act as important catalysts for improving ambulatory care, but whether constructive changes are made or not depends on whether people on the ground in any particular setting make them happen. Experience in outpatient settings responding to quantitative forms of patient feedback has been checkered, both in the United States and abroad.27, 28, 29 Examining the pathways through which narrative feedback is most readily adopted and the interventions that might encourage this adoption are important arenas for additional research.
It is also possible that the widespread collection of patient narratives might unduly raise patient expectations, leading them to anticipate a health care system that is more responsive to their feedback than will actually be the case. We anticipate, however, that as patients’ comments are increasingly posted on public websites, inadequate response to narrative feedback will itself, become a topic of commentary, creating additional pressures to respond in constructive ways.13, 26
Finally, the concept of actionability we operationalized here is not the only plausible standard, especially with respect to positive feedback. The reinforcement that affirmative narratives provide to clinicians may be an important buffer against burnout and other hardships of clinical practice; however, such benefits may be more about improving quality by preventing problems or strengthening workplace culture rather than “taking action” to address a problem. Future research might usefully compare preventive versus remedial approaches to QI using narratives.
Despite obvious challenges associated with use of qualitative feedback for QI, we believe this innovation holds substantial promise. Patient experience surveys paved the way for patients’ experiences to be routinely measured and valued, not only for improvement efforts, but also as an integral part of reimbursement formulae in health care. Patients’ words, however, convey far more nuance, detail, and emotional content than do their survey scores. As a result, their collective narratives may fuel quality innovations that are far more patient‐centered than those that rely on quantitative measures alone.
Heterogeneity of Experiences and the Future of Patient‐Centered Care
Quantitative data from patient experience surveys are routinely aggregated into means and other summary statistics. Narratives can similarly be aggregated through analysis designed to identify themes, as demonstrated in the preceding pages. Qualitative feedback also provides insight about variation in patients’ experiences, however, and part of its power lies in exploring this heterogeneity. “Patient voice” is often conceived of as a unitary phenomenon, but narratives illustrate the commonsense insight that patients’ experiences are diverse and we must listen for a chorus of voices if we want to understand patients' experiences and render health care more responsive to their needs.
Although we have focused in this paper primarily on the most prevalent themes that emerged in the course of our analysis, it is essential that QI efforts also take into account the reports of rare events and descriptions of less common, subjective reactions that abound in narratives. This may be particularly important when consequences are dire, professional behavior is exemplary, or the perspectives illustrated come from patients who are structurally disadvantaged or less well positioned to make their voices heard.
The term personalized medicine has become a catchphrase for using genomic profiles to predict health risks and target interventions.72 Narratives offer an alternative definition and an alternative route to enriching clinical practice. Accustoming clinicians, QI leaders, and health system administrators to valuing what patients have to say is an essential way to reframe which forms of evidence are considered valuable and whose perspectives are taken seriously. Attuning them to what patients describe about diverse experiences with care can inspire more nuanced, effective approaches to quality improvement. In short, listening closely to what patients’ own words convey may be the shortest and most meaningful path to truly personalized medicine.
Supporting information
Online Appendix A: In‐depth Analysis of Narratives Integral and Proximal to CG‐CAHPS Survey for Office Staff, Access to Care, and Coordination of Care
Online Appendix B: Approach to Axial Mapping of Narrative Content into CG‐CAHPS Domains
Online Appendix C: Coding Narrative Content for Actionability Related to Quality Improvement
Funding/Support
This research was supported by cooperative agreements between AHRQ and Yale University (1U18HS016978) and RAND (2U18HS016980‐06) and a grant from AHRQ to Yale University (1R21HS021858). Dr. Grob was also supported by the Clinical and Translational Science Award (CTSA) program, through the NIH National Center for Advancing Translational Sciences (NCATS), grant UL1TR002372, and the UW SMPH Wisconsin Partnership Program grant #3086, though the content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. Dr. Lagu was supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health under Award Number K01HL114745.
Conflict of Interest Disclosures: All authors have completed the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr. Lagu reported consulting fees outside the scope of this work from the Institute for Healthcare Improvement, under contract to CMS, for her work on a project to help health systems achieve disability competence. No other disclosures were reported.
Acknowledgments: The project on which this paper is based was facilitated by helpful feedback from a variety of sources, including the broader CAHPS research team (principal investigators Paul Cleary, Susan Edgman‐Levitan, Ron Hays, Marc Elliott, and Lise Rybowski); our project officers at AHRQ (Chris Crofton and Caren Ginsberg); and attendees at several web‐based conferences with staff and leadership from AHRQ and CMS. Very preliminary versions of these findings were also presented at an invitational stakeholder meeting on advancing the science of patient narratives sponsored by AHRQ in Rockville, MD, in September 2016 and at Grand Rounds at the University of California– San Francisco in January 2017. Molly Jones, Emily Warne, and Cecie Culp all provided skillful assistance preparing the manuscript.
References
- 1. Batalden M, Batalden P, Margolis P, et al. Coproduction of healthcare services. BMJ Qual Saf. 2015;0:1‐9. 10.1136/bmjqs-2015-004315. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Grob R. The heart of patient‐centered care. J Health Polit Policy Law. 2013;38(2):457‐465. [DOI] [PubMed] [Google Scholar]
- 3. Tanenbaum SJ. What is patient‐centered care? A typology of models and missions. Health Care Anal. 2015;23(3):272‐287. [DOI] [PubMed] [Google Scholar]
- 4. Frampton SB, Guastello S, Hoy L, Naylor M, Sheridan S, Johnston‐Fleece M. Harnessing evidence and experience to change culture: a guiding framework for patient and family engaged care. Discussion Paper (January 31, 2017), National Academy of Medicine, Washington, DC. https://nam.edu/wp-content/uploads/2017/01/Harnessing-Evidence-and-Experience-to-Change-Culture-A-Guiding-Framework-for-Patient-and-Family-Engaged-Care.pdf. Accessed February 12, 2019.
- 5. Crofton C, Lubalin JS, Darby C. Foreword. Consumer Assessment of Health Plans Study (CAHPS). Med Care. 1999;37(Suppl. 3):MS1‐9. [DOI] [PubMed] [Google Scholar]
- 6. Schlesinger M, Shaller D, Grob R. Using patient‐reported information to improve clinical practice. Health Serv Res. 2015;50(S2):2116‐2154. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Schlesinger M. Choice cuts: parsing policymakers’ pursuit of patient empowerment from an individual perspective. Health Econ Policy Law. 2010;5(3):365‐387. [DOI] [PubMed] [Google Scholar]
- 8. Schlesinger M, Druss B, Thomas R. No exit? The effect of health status on dissatisfaction and disenrollment from health plans. Health Serv Res. 1999;34(2):547‐576. [PMC free article] [PubMed] [Google Scholar]
- 9. Manary M, Staelin R, Kosel K, Schulman KA, Glickman SW. Organizational characteristics and patient experiences with hospital care: a survey study of hospital chief patient experience officers. Am J Med Qual. 2015;30(5):432‐440. [DOI] [PubMed] [Google Scholar]
- 10. Looking for more ways to learn about the Quality Payment Program? Centers for Medicare and Medicaid Services website. www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Value-Based-Programs/MACRA-MIPS-and-APMs/Quality-Payment-Program-Events.html. Updated February 27, 2018. Accessed March 3, 2018.
- 11. Pedersen AR. The role of patient narratives in healthcare innovation: supporting translation and meaning making. J Health Organ Manag. 2016;30(2):244‐257. [DOI] [PubMed] [Google Scholar]
- 12. Lee V. Transparency and trust—online patient reviews of physicians. N Engl J Med. 2017;376(3):197‐199. [DOI] [PubMed] [Google Scholar]
- 13. Holliday AM, Kachalia A, Meyer GS, Sequist TD. Physician and patient views on public physician rating websites: a cross‐sectional study. J Gen Intern Med. 2017;32(6):626‐631. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Sick B, Abraham JM. Seek and ye shall find: consumer search for objective health care cost and quality information. Am J Med Qual. 2011;26(6):433‐440. [DOI] [PubMed] [Google Scholar]
- 15. Findlay SD. Consumers’ interest in provider ratings grows, and improved report cards and other steps could accelerate their use. Health Aff (Millwood). 2016;35(4):688‐696. [DOI] [PubMed] [Google Scholar]
- 16. Schlesinger M, Grob R, Shaller D, et al. Taking patients’ narratives about clinicians from anecdote to science. N Engl J Med. 2015;373(7):675‐679. [DOI] [PubMed] [Google Scholar]
- 17. Huppertz JW, Smith R, Bombard C. The value of patients’ handwritten comments on HCAHPS surveys. J Healthc Manag. 2014;59(1):31‐48. [PubMed] [Google Scholar]
- 18. Riiskjaer E, Ammentorp J, Kofoed PE. The value of open‐ended questions in surveys on patient experience: number of comments and perceived usefulness from a hospital perspective. Int J Qual Health Care. 2012;24:509‐516. 10.1093/intqhc/mzs039. [DOI] [PubMed] [Google Scholar]
- 19. Bardach NS, Lyndon A, Asteria‐Peñaloza R, Goldman LE, Lin GA, Dudley RA. From the closest observers of patient care: a thematic analysis of online narrative reviews of hospitals. BMJ Qual Saf. 2015;25(11):889‐897. 10.1136/bmjqs-2015-004515. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Schneider EC, Ridgely MS, Quigley DD, et al. Developing and testing the health care safety hotline: a prototype consumer reporting system for patient safety events. Rand Health Q. 2017;6(3):1. [PMC free article] [PubMed] [Google Scholar]
- 21. Wiseman T, Lucas G, Sangha A, et al. Insights into the experiences of patients with cancer in London: framework analysis of free‐text data from the National Cancer Patient Experience Survey 2012/2013 from the two London Integrated Cancer Systems. BMJ Open Qual. 2015;5(10):e007792 10.1136/bmjopen-2015-007792. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Schlesinger M, Rybowski L, Rabson B, Shaller D. Introducing a Protocol to Obtain Patient Comments Using the CAHPS Clinician & Group Survey [webcast]. Agency for Healthcare Research and Quality. January 26, 2017. www.ahrq.gov/cahps/news-and-events/events/webinar-012617.html. Accessed February 27, 2018.
- 23. Grob R, Schlesinger M, Shaller D, et al. Breaking narrative ground: innovative methods for rigorously eliciting and assessing patient narratives. Health Serv Res. 2016;51(3):1248‐1272. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Elliott MN, Cohea CW, Lehrman WG, et al. Accelerating improvement and narrowing gaps: trends in patients’ experiences with hospital care reflected in HCAHPS public reporting. Health Serv Res. 2015;50(6):1850‐1867. 10.1111/1475-6773-12305. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Tsianakas V, Maben J, Wiseman T, et al. Using patients’ experiences to identify priorities for quality improvement in breast cancer care: patient narratives, surveys or both? BMC Health Serv Res. 2012;12:271. doi:10.1186-1472-6963-12-271. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Lagu T, Metayer K, Moran M, et al. Website characteristics and physician reviews on commercial physician‐rating websites. JAMA. 2017;317(7):766‐768. 10.1001/jama.2016.18553. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Baldie DJ, Guthrie B, Entwistle V, Kroll T. Exploring the impact and use of patients’ feedback about their care experiences in general practice settings—a realist synthesis. Fam Pract. 2018;35(1):13‐21. 10.1093/fampra/cmx067. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Friedberg MW, SteelFisher GK, Karp M, Schneider EC. Physician groups’ use of data from patient experience surveys. J Gen Intern Med. 2011;26(5):498‐504. 10.1007/s11606-010-1597-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Geissler KH, Friedberg MW, SteelFisher GK, Schneider EC. Motivators and barriers to using patient experience reports for performance improvement. Med Care Res Rev. 2013;70(6):621‐635. 10.1177/1077558713496319. [DOI] [PubMed] [Google Scholar]
- 30. López A, Detz A, Ratanawongsa N, Sarkar U. What patients say about their doctors online: a qualitative content analysis. J Gen Intern Med. 2012;27(6):685‐692. 10.1007/s11606-011-1958-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Alemi F, Torii M, Clementz L, Aron DC. Feasibility of real‐time satisfaction surveys through automated analysis of patients’ unstructured comments and sentiments. Qual Manag Health Care. 2012;21(1):9‐19. 10.1097/QMH.0b013e3182417fc4. [DOI] [PubMed] [Google Scholar]
- 32. Kadry B, Chu LF, Kadry B, Gammas D, Macario A. Analysis of 4999 online physician ratings indicates that most patients give physicians a favorable rating. J Med Internet Res. 2011;13(4):e95 10.2196/jmir.1960. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Press I. Concern for the patient's experience comes of age. Patient Experience Journal. 2014;1(1):4‐6. [Google Scholar]
- 34. Merchant RA, Volpp KG, Asch DA. Learning by listening—improving health care in the era of Yelp. JAMA. 2016;316(23):2483‐2484. 10.1001/jama.2016.16754. [DOI] [PubMed] [Google Scholar]
- 35. Dohan D, Garrett SB, Rendle KA, Halley M, Abramson C. The importance of integrating narrative into health care decision making. Health Aff (Millwood). 2016;35(4):720‐725. 10.1377/hlthaff.2015.1373. [DOI] [PubMed] [Google Scholar]
- 36. Trigg L. Patients’ opinions of health care providers for supporting choice and quality improvement. J Health Serv Res Policy. 2011;16(2):102‐107. 10.1258/jhsrp.2010.010010. [DOI] [PubMed] [Google Scholar]
- 37. Brutus S. Words versus numbers: a theoretical exploration of giving and receiving narrative comments in performance appraisal. Hum Resource Manage R. 2010;20(2):144‐157. doi: 10.1016/j.hrmr.2009.06.003. [DOI] [Google Scholar]
- 38. Cognetta‐Rieke C, Guney S. Analytical insights from patient narratives: the next step for better patient experience. J Patient Exp. 2014;1(1):20‐22. 10.1177/237437431400100105. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39. Ranard BL, Werner RM, Antanavicius T, et al. Yelp reviews of hospital care can supplement and inform traditional surveys of the patient experience of care. Health Aff (Millwood). 2016;35(4);697‐705. 10.1377/hlthaff.2015.1030. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40. Denniston C, Molloy E, Rees CE. “I will never ever go back”: patients’ written narratives of health care communication. Med Educ. 2018;52(7):757‐771. 10.1111/medu.13612. [DOI] [PubMed] [Google Scholar]
- 41. Greenhalgh R, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2014;82(4):581‐629. 10.1111/j.0887-378X.2004.00325.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50 10.1186/1748-5908-4-50. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43. Entwistle VA, Firnigl D, Ryan M, Francis J, Kinghorn P. Which experiences of healthcare delivery matter to service users and why? A critical interpretive synthesis and conceptual map. J Health Serv Res Policy. 2012;17(2):70‐78. 10.1258/jhsrp.2011.011029. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44. Elliott MN, Beckett MK, Lehrman WG, et al. Understanding the role played by Medicare's patient experience point system in hospital reimbursement. Health Aff (Millwood). 2016;35(9):1673‐1680. 10.1377/hlthaff.2015.0691. [DOI] [PubMed] [Google Scholar]
- 45. Glover M, Khalilzadeh O, Choy G, Prabhakar AM, Pandharipande PV, Gazelle GS. Hospital evaluations by social media: a comparative analysis of Facebook ratings among performance outliers. J Gen Intern Med. 2015;30(10):1440‐1446. 10.1007/s11606-015-3236-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46. Martino SC, Shaller D, Schlesinger M, et al. CAHPS and comments: how closed‐ended survey questions and narrative accounts interact in the assessment of patient experience. J Patient Exp. 2017;4(1):37‐45. 10.1177/2374373516685940. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47. Lagu T, Goff SL, Hannon NS, Shatz A, Lindenauer PK. A mixed‐methods analysis of patient reviews of hospital care in England: implications for public reporting of health care quality data in the United States. Jt Comm J Qual Patient Saf. 2013;39(1):7‐15. 10.1016/S1553-7250(13)39003-5. [DOI] [PubMed] [Google Scholar]
- 48. Matthias MS, Krebs EE, Bergman AA, Coffing JM, Bair MJ. Communicating about opioids for chronic pain: a qualitative study of patient attributions and the influence of the patient‐physician relationship. Eur J Pain. 2014;18(6):835‐843. [DOI] [PubMed] [Google Scholar]
- 49. Rosenthal M, Schlesinger M. Not afraid to blame: the neglected role of blame attribution in medical consumerism and some implications for health policy. Milbank Q. 2002;80(1):41‐95. 10.1111/1468-0009.00003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50. David EM. Examining the role of narrative performance appraisal comments on performance. Human Performance. 2013;26:430‐450. 10.1080/08959285.2013.836197. [DOI] [Google Scholar]
- 51. Jenkinson C, Coulter A, Bruster S, Richards N, Chandola T. Patients’ experiences and satisfaction with health care: results of a questionnaire study of specific aspects of care. Qual Saf Health Care. 2002;11:335‐339. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52. Gallan AS, Girju M, Girju R. Perfect ratings with negative comments: learning from contradictory patient survey responses. Patient Experience Journal. 2017;4(3):15‐28. [Google Scholar]
- 53. Bickman L, Kelley SD, Breda C, de Andrade AR, Riemer M. Effects of routine feedback to clinicians on mental health outcomes of youths: results of a randomized trial. Psychiatr Serv. 2011;62(12):1423‐1429. 10.1176/appi.ps.002052011. [DOI] [PubMed] [Google Scholar]
- 54. Locock L, Dopson S, Chambers D, Gabbay J. Understanding the role of opinion leaders in improving clinical effectiveness. Soc Sci Med. 2001;53(6):745‐757. 10.1016/S0277-9536(00)00387-7. [DOI] [PubMed] [Google Scholar]
- 55. Valente TW, Davis RL. Accelerating the diffusion of innovations using opinion leaders. Ann Am Acad Pol Soc Sci. 1999;566(1):55‐67. 10.1177/000271629956600105. [DOI] [Google Scholar]
- 56. Thompson‐Leduc P, Turcotte S, Labrecque M, Legare F. Prevalence of clinically significant decisional conflict: an analysis of five studies on decision‐making in primary care. BMJ Open. 2016;6(6):e011490 10.1136/bmjopen-2016-011490. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57. Lindly OJ, Zuckerman KE, Mistry KB. Clarifying the predictive value of family‐centered care and shared decision making for pediatric healthcare outcomes using the Medical Expenditure Panel Survey. Health Serv Res. 2017;52(1):313‐345. 10.1111/1475-6773.12488. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58. Schoen C, Osborn R, How SK, Doty MM, Peugh J. In chronic condition: experiences of patients with complex health care needs, in eight countries. Health Aff (Millwood). 2008;28(1):w1‐w16. 10.1377/hlthaff.28.1.w1. [DOI] [PubMed] [Google Scholar]
- 59. Ash JS, Stavri PZ, Dykstra R, Fournier L. Implementing computerized physician order entry: the importance of special people. Int J Med Inform. 2003;69(2‐3):235‐250. [DOI] [PubMed] [Google Scholar]
- 60. Kanouse DE, Schlesinger M, Shaller D, Martino SC, Rybowski L. How patient comments affect consumers’ use of physician performance measures. Med Care. 2016;54(1):24‐31. 10.1097/MLR.0000000000000443. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 61. CAHPS Patient Narrative Elicitation Protocol . Agency for Healthcare Research and Quality website. www.ahrq.gov/cahps/surveys-guidance/item-sets/elicitation/index.html. Updated August 2017. Accessed February 27, 2018.
- 62. Schlesinger M, Grob R, Shaller D, et al. A rigorous approach to large‐scale elicitation and analysis of patient narratives. Med Care Res Rev. 2018. [e‐published ahead of print] 10.1177/1077558718803859. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 63. Chang L, Krosnick JA. National surveys via RDD telephone interviewing versus the internet: comparing sample representativeness and response quality. Public Opinion Quarterly. 2009;73(4):641‐678. 10.1093/poq/nfp075. [DOI] [Google Scholar]
- 64. Cho JY, Lee EH. Reducing confusion about grounded theory and qualitative content analysis: similarities and differences. The Qualitative Report. 2014;19(32):1‐20. [Google Scholar]
- 65. Elliott MN, Kanouse DE, Edwards CA, Hilborne LH. Components of care vary in importance for overall patient‐reported experience by type of hospitalization. Med Care. 2009;47(8):842‐849. 10.1097/MLR.0b013e318197b22a. [DOI] [PubMed] [Google Scholar]
- 66. Charon R. Narrative medicine: a model for empathy, reflection, profession, and trust. JAMA. 2001;286(15):1897‐1902. 10.1001/jama.286.15.1897. [DOI] [PubMed] [Google Scholar]
- 67. Ash JS, Cottrell E, Saxton L, Newman L, Gebhardt E, Helfand M. Patient narratives representing patient voices to inform research: a pilot qualitative study. Stud Health Technol Inform. 2015;208:55‐60. [PMC free article] [PubMed] [Google Scholar]
- 68. Tsianakas V, Robert G, Maben J, et al. Implementing patient‐centred cancer care: using experience‐based co‐design to improve patient experience in breast and lung cancer services. Support Care Cancer. 2012;20(11):2639‐2647. 10.1007/s00520-012-1470-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 69. Donetto S, Pierri P, Tsianakas V, Robert G. Experience‐based co‐design and healthcare improvement: realizing participatory design in the public sector. The Design Journal. 2015;18(2):227‐248. . [DOI] [Google Scholar]
- 70. Locock L, Robert G, Boaz A, et al. Testing accelerated experience‐based co‐design: a qualitative study of using a national archive of patient experience narrative interviews to promote rapid patient‐centred service improvement. Health Services and Delivery Research. 2014;2(4). 10.3310/hsdr02040. [DOI] [PubMed] [Google Scholar]
- 71. Frank A. The Wounded Storyteller. Chicago, IL: University of Chicago Press; 1995. [Google Scholar]
- 72. Juengst E, McGowan M, Fishman J, Settersten RA. From “personalized” to “precision” medicine: the ethical and social implications of rhetorical reform in genomic medicine. Hastings Cent Rep. 2016;46(5):21‐23. 10.1002/hast.614. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Online Appendix A: In‐depth Analysis of Narratives Integral and Proximal to CG‐CAHPS Survey for Office Staff, Access to Care, and Coordination of Care
Online Appendix B: Approach to Axial Mapping of Narrative Content into CG‐CAHPS Domains
Online Appendix C: Coding Narrative Content for Actionability Related to Quality Improvement