A survey to evaluate facilitators and barriers to quality measurement and improvement: Adapting tools for implementation research in palliative care programs

Abstract

Context

Though critical for improving patient outcomes, palliative care quality indicators are not yet widely used. Better understanding of facilitators and barriers to palliative care quality measurement and improvement might improve their use and program quality.

Objectives

Development of a survey tool to assess palliative care team perspectives on facilitators and barriers to quality measurement and improvement in palliative care programs.

Methods

We used the adapted Consolidated Framework for Implementation Research to define domains and constructs to select instruments. We assembled a draft survey and assessed content validity through pilot testing and cognitive interviews with experts and front-line practitioners for key items. We analyzed responses using a constant comparative process to assess survey item issues and potential solutions. We developed a final survey using these results.

Results

The survey includes five published instruments and two additional item sets. Domains include organizational characteristics, individual and team characteristics, intervention characteristics and process of implementation. Survey modules include Quality Improvement in Palliative Care, Implementing Quality Improvement in the Palliative Care Program, Teamwork and Communication, Measuring the Quality of Palliative Care, and Palliative Care Quality in Your Program. Key refinements from cognitive interviews included item wording on palliative care team members, programs and quality issues.

Conclusion

This novel, adaptable instrument assesses palliative care team perspectives on barriers and facilitators for quality measurement and improvement in palliative care programs. Next steps include evaluation of the survey’s construct validity and how survey results correlate with findings from program quality initiatives.

Introduction

Measuring and improving the quality of care provided by palliative care programs is critical for improving outcomes for patients with serious and advanced illnesses and their families, from diagnosis with an advanced illness through the end of life.¹ Measuring processes of care, or what occurs in interactions between palliative care team members, patients and their families, can be applied through the entire spectrum of care.² Palliative care process quality indicators have now been developed for key measurable domains, such as communication and pain management, and for settings ranging from hospital to outpatient and home,^3–8 and are endorsed by national organizations.^{9, 10}

However, palliative care programs have been slow to adopt quality measurement activities. Extracting information on quality indicators from existing systems is time-consuming and often yields incomplete data. Although additional methods for collecting data for quality indicators are being developed, such as structured electronic record templates, point-of-care data collection and patient-reported outcomes,¹¹ these methods can be burdensome to both palliative care team members and patients. Palliative care team members may feel that measuring quality of care could lead to “cookbook medicine” and reduce the patient-centeredness of care, or that the aspects most key to quality of palliative care, particularly excellent communication, cannot be accurately measured. Barriers in data collection in the context of clinical care can also lead to incomplete or lower-quality data.¹¹ Similar issues may exist in quality improvement, where palliative care team members may feel that there is insufficient program support for improvement initiatives or that existing initiatives do not address the issues most important to patients.

Better understanding of key facilitators and barriers to implementing palliative care quality indicators could improve the completeness and quality of data collection through engaging palliative care team members more effectively. Understanding team members’ perceptions about quality could elucidate why performance on some quality indicators is lower. Understanding team members’ perceptions of program teamwork and leadership support could also help programs understand what changes may be needed to improve quality of care and better outcomes for patients.¹²

This paper outlines development of a survey to explore facilitators and barriers to quality measurement and improvement in palliative care programs. Specifically, we describe the process we used to select, adapt and supplement existing instruments and evaluate their content validity. The intended use of the survey is for palliative care teams interested in understanding how to use quality measurement in their programs, how to improve existing quality measurement initiatives, and how to implement or enhance quality improvement activities.

Methods

Existing validated instruments for health care professionals for concepts such as patient safety culture¹³ or contextual issues for quality improvement¹⁴ address key barriers and facilitators to quality measurement and improvement. However, these instruments do not address characteristics specific to palliative care programs or quality indicators for serious illness and end of life care. When instruments are adapted or applied for different purposes and populations, they should be revalidated. This requires consideration of content validity, the extent to which a measurement instrument (e.g., a survey) represents all facets of a given construct; construct validity, the degree to which the instrument actually measures the concept or construct that it is intended to measure; and criterion validity, or relationships of an instrument with other constructs to which it is conceptually related.¹⁵

Evaluating the validity of instruments adapted for a new area usually involves four key steps.^{16, 17} First is defining focal construct(s) and a conceptual model outlining how they relate to other key constructs. Second is operationalizing the focal constructs to select and adapt relevant instruments and then evaluating the items’ validity for the new use (e.g., evaluating how relevant and representative the items are to the construct(s) of interest). Based on this evaluation, an adapted working version of the instrument is developed. Third is evaluating the convergent and discriminant construct validity of the instruments (i.e., evaluating their relationships with conceptually similar and dissimilar constructs). The final step is evaluating criterion validity, the relationships of the instruments with other constructs to which they are conceptually related (i.e., do the instruments relate to measures of other constructs as anticipated). In practice, since steps three and four require significant sample sizes, these steps are generally conducted with the implementation of the survey. During steps three and four, researchers generally also recalculate instruments’ internal consistency reliability (or how individual items relate to each other within instruments’ subscales) to determine if and where the original analysis approach can still be used. If not, researchers rerun factor analysis to determine subscales for analyzing individual items together.

This paper outlines the evaluation of content validity for the survey.

Step 1: Conceptual framework and definition of domains and focal constructs

Conceptual framework

Our conceptual framework was the consolidated framework for implementation research (CFIR)¹⁸ with additional concepts from a recent adaptation for complex systems interventions.¹⁹ The CFIR synthesized relevant theories into an integrated framework for implementation research and has been widely applied for implementation science.²⁰ We selected four domains from the adapted CFIR as most relevant to the survey: (1) intervention characteristics (e.g., of the quality measurement program), (2) organizational characteristics; (3) characteristics and roles of providers, and (4) process of implementation. Each domain includes multiple constructs. We selected potential constructs from the four domains based on clinical practice guidelines for the conduct of palliative care programs²¹ and reviews of palliative care quality indicators⁹ and quality improvement interventions.^22–24 (Table 2)

Table 2.

Summary of survey modules and items with corresponding Consolidated Framework for Implementation Research (CFIR) domains and constructs

Survey Module	Scale description – as applied to palliative care program quality of care	Adapted consolidated framework for implementation research*			Source
		Domain	Construct(s)	Sub-construct
Module A: Quality Improvement (QI) in Palliative Care – improving the quality of care provided by the palliative care program	Team and leaders value QI and understand and effectively use quality improvement methods	Palliative care program characteristics	Implementation climate	Tension for change	Change process capability questionnaire (CPCQ)30,31
	Program has established process for QI	Program characteristics	Implementation climate	Relative priority	CPCQ
	Individual preparation to participate in and lead QI	Characteristics and roles of team members	Self-efficacy		CPCQ
	Program focus on educational support, recognition and rewards, selection of staff for experience and openness to QI	Program characteristics	Implementation climate		Implementation Climate Scale28
Module B: Implementing Quality Improvement in the Palliative Care Program	Implementation of or participation in any program QI initiatives	Intervention characteristics	Description of intervention		CPCQ
	Strategies used for QI (information, skills training, encouragement by leadership)	Process of implementation	Engaging		CPCQ
	Creating systems, reducing barriers, organizing, setting goals and benchmarking, customizing for QI	Process of implementation	Executing		CPCQ3
	Methods to reduce or evaluate negative consequences of QI	Process of implementation	Reflecting and evaluating		CPCQ3
	Relative priority of program leaders for QI	Program characteristics	Implementation climate	Relative priority	CPCQ3
	Individual participation in QI	Characteristics and roles of team members	Role		N/A
	Individual accountability for quality of care	Characteristics and roles of team members	Individual accountability		Unpublished
Module C: Teamwork & Communication	Openness to ideas, viewpoints, questions and disagreement among palliative care team	Characteristics and roles of team members	Communication		MO-SOPS Communication openness scale27
	Team members help each other out, work well together, treat each other with respect, emphasize teamwork	Characteristics and roles of team members	Teamwork		MO-SOPS Teamwork scale27
	Collective accountability for quality of care	Characteristics and roles of team members	Collective accountability		Unpublished
Module D: Measuring the Quality of Palliative Care	Team and leaders value quality measurement and understand and effectively use quality measurement methods	Program characteristics	Implementation climate	Tension for change	CPCQ
	Whether leadership is proactive, knowledgeable, supportive and perseverant about quality measurement	Program characteristics	Readiness for Implementation	Leadership engagement	Implementation Leadership Scale29
	What indicators are collected and how they are used	Intervention characteristics	Description of intervention		N/A
	Supported by scientific evidence and experts	Intervention characteristics	Evidence strength and quality		Organizational Readiness to Change Assessment (ORCA) 32
	Appropriate, comprehensive, respectful of needs and preferences, acceptable to patients and families	Intervention characteristics	Patient/family centeredness		ORCA
	Advantages and disadvantages	Intervention characteristics	Relative advantage		ORCA
	Ease and feasibility of collection, understandability	Intervention characteristics	Complexity, Health information technology usability		ORCA
	Responsive to change, trustworthy, useful	Intervention characteristics	Usefulness		N/A
	Focus on educational support, recognition and rewards for quality measurement in palliative care	Program characteristics	Implementation climate		Implementation Climate Scale
Module E: Palliative care quality in your program	Perceptions of quality related to quality indicators	N/A	N/A		Unpublished palliative care surveys

^*Note that, given feedback from the cognitive interviews, we reframed the “organizational characteristics” domain from the adapted CFIR as “palliative care program characteristics”, in order to provide better focus on the program, and the “Characteristics and roles of providers” as “Characteristics and roles of palliative care team members” to be inclusive of all members of the team. “Intervention characteristics” refers to the quality indicator data collection or quality improvement

N/A = Not applicable; MO-SOPS= Medical Office Survey on Patient Safety Culture; ORCA =Organizational Readiness to Change Assessment

Step 2: Operationalization and content validity

We searched for instruments relevant to the constructs in databases compiling validated instruments linked to CFIR constructs, the National Cancer Institute’s GEM-D&I (Grid-Enabled Measures Database – Dissemination and Implementation)²⁵ and the SIRC (Society for Implementation Research Collaboration) Systematic Review and Synthesis of Implementation Science Instruments,²⁶ and consultation with experts.

Based on Step 1, we selected five published instruments and two additional item sets, chosen based on best fit for the key relevant constructs and palliative care (Table 2). These included the Agency for Health Care Research and Quality’s Medical Office Survey on Patient Safety Culture (MO-SOPS),²⁷ which addresses communication and teamwork and includes a database for national benchmarking. For implementation climate, we adapted the Implementation Climate Scale,²⁸ developed and originally tested to evaluate evidence-based practice implementation. For readiness for implementation, we adapted the Implementation Leadership Scale,²⁹ developed and tested for aspects of unit-level leadership for implementation of evidence-based practice. For these and other constructs, we also adapted subscales from Solberg’s Change Process Capability Questionnaire (CPCQ), developed and tested for assessing medical group capability for quality improvement.^{30, 31} We also adapted subscales from the Organizational Readiness to Change Assessment (ORCA) instrument,³² which addresses strength of evidence for the intervention, quality of the organizational context to support the intervention and organizational capacity to facilitate the intervention. For collective accountability, we used newly developed unpublished items. For palliative care team members’ perceptions of how quality indicators are being applied and the quality of palliative care, we adapted items from unpublished surveys.

As a next step, palliative care team members with expertise in organizational culture, palliative care and survey methods adapted the items for palliative care team quality measurement and improvement. We organized instrument subscales into modules organized for these applications.³³ We then pilot tested the survey with six palliative care team members and refined the adapted item pool based on their feedback. Next, we conducted cognitive testing on 18 key adapted items that were unclear to these palliative care team members.^{34, 35}

We conducted six cognitive interviews with three palliative care quality measurement experts (oncology and internal medicine physician from different institutions) and three frontline care team members (a nurse case manager, nurse practitioner, and social worker from academic and community settings). Cognitive interviews are a specific type of qualitative data collection that uses cognitive theory to understand how respondents perceive and interpret the items in a survey. The goal of cognitive interviews is to produce “a coherent set of results and recommendations” regarding specific items, in order to refine item wording to meet the survey’s goals.³³ Three issues were addressed in the interviews: (1) Item-specific recommendations for changes to wording, including specificity and clarity; (2) Issues of overall instrument length or burden, such as items that are duplicative, unclear or lack relevance, and (3) Limitations in how the intended respondents can provide valid information through a survey.³³

At least two investigators participated in each cognitive interview, with initial analysis following each interview. We probed each question by following the initial query with a subset of both spoken responses and observations and assessed the ease with which respondents could answer each item. For example, for each question we noted if respondents needed any part of the question repeated, had any difficulty using the provided response options, or asked for clarification/qualified their answer. Respondents were then given the opportunity to state why they answered the way in which they did, how sure they were of their answer, and how easy or difficult it was to provide an answer to the question. For several items, we also asked what a certain term in the item meant to them. (Table 1)

Table 1.

Draft item and questions used for cognitive interviewing^*

Draft item	Thinking about the organization in which you work, please indicate the extent to which you agree with this statement: One of this organization’s goals is to improve the quality of palliative care we provide.*
Draft response options	[ ] Not at all (1) [ ] Slight extent (2) [ ] Moderate extent (3) [ ] Great extent (4) [ ] Very Great Extent (5)	[ ] Does not apply [ ] Don’t know [ ] Prefer not to answer
Cognitive interview questions for respondents	Why did you answer that way? Walk me through what you were thinking when you chose that response. Probes if needed: How did you arrive at your answer? Why do you believe that?
	How sure are you of your answer?
	How easy or hard was it to answer this question?
	What, to you, does the phrase “improving palliative care quality” mean? Probe if needed: Can you rephrase the question I asked you in your own words? (read question back one more time)

^*Final item changed word “organization” to “program”, defined as “palliative care program”

We then entered interview responses into Q-Notes data entry and analysis software for data analysis.³⁶ Q-Notes is an application used to ensure systematic and transparent analyses across cognitive interviews. The analyses followed a three-stage structured method of data reduction and item revision based on the cognitive interviews. For the first stage, to analyze the findings of the cognitive interviews and accompanying interviewer notes, we conducted a question-by-question review of each of the 18 adapted items using Q-Notes. For the second stage, we used a constant comparative process to assess patterns relevant to the survey issues and to identify potential causes of these patterns. We also evaluated responses for patterns of item interpretation, to assess if respondents answered items in the manner originally intended and how responses deviated from the item intent. Third, we then collated results by item, generated a set of recommended changes to the overall protocol that reflected the results of the pilot interviews, and used these recommendations to revise the survey. As a quality check, we also reviewed the revised survey with several of the experts.

Results

Final survey content

The full adapted survey includes 5 modules: (A) Quality Improvement in Palliative Care – improving the quality of care provided by the palliative care program, (B) Implementing Quality Improvement in the Palliative Care Program, (C) Teamwork & Communication, (D) Measuring the Quality of Palliative Care, and (E) Palliative Care Quality in Your Program (Table 2; survey available upon request). If the survey is used only for evaluating either quality improvement or measurement, the other module(s) can be removed. From the adapted CFIR, for the organizational characteristics domain related to quality, the survey addresses communication; overall implementation climate and specific elements of climate, including tension for change and relative priority; and readiness for implementation, including leadership engagement. For the domain of characteristics and roles of individuals and teams, the survey addresses self-efficacy; perceptions of teamwork; and collective and personal accountability. For the CFIR domain of intervention characteristics, the survey addresses the description of the intervention, evidence strength and quality, patient-centeredness, relative advantage, complexity, and usefulness. Engaging, executing, reflecting and evaluating are included for the CFIR domain of process of implementation. Module E addresses respondents’ perceptions of how well the program is collecting quality data on key quality indicators (and can be easily adapted for other types of indicators), and the final section addresses respondent characteristics. Survey modules and how they correspond to CFIR domains are shown in Table 2.

Summary of cognitive interviews and adaptation for the palliative care context

Input from all palliative care quality experts and frontline team members contributed significantly to the interpretation of survey items and was generally consistent. The cognitive interviews highlighted three main potential problems with the survey, which we revised accordingly: a need to better define respondents, setting and leadership, lack of knowledge about all quality initiatives, and questions about how to describe quality measurement and improvement. First, defining the respondents and setting were key, as well as rewording in clear palliative care terms. The original instruments divided respondents into “physicians” or “providers” and “staff” and worded items differently for physicians, and non-physicians were not sure where to include themselves. From the palliative care perspective, we replaced the terms “providers”, “physicians” or “staff” with “palliative care team members”, to be inclusive and reflect the diverse personnel in palliative care programs.

Regarding the setting, interviewees variably interpreted the word “organization” as department/unit, hospital, health care system, or even at a national level, and interviewees did not use the term “facility”. Given the focus of the survey on quality within palliative care programs, we revised wording throughout the survey to use the term “palliative care program” to reflect the organizational level. The term “leadership” (including organizational or executive) was also challenging in settings with many different types of leaders and was interpreted in different ways. Based on input from the interviews, we reworded items to focus at the program level and at leadership most relevant to quality within the program.

Secondly, interviewees often were unsure about what quality initiatives were ongoing in their program, due to different parts of the program (e.g. hospital consult service vs. outpatient clinic vs. hospice) conducting separate projects or large programs where not everyone was aware of all quality activities. This was a particular issue for frontline team members, who felt they were often not in the communication loop. Interviewees discussed different ways to address this, including a “don’t know” option. Lack of awareness of program measurement or improvement activities could be important to assess, reflecting inadequate communication or culture of quality within the program. Since involvement of all palliative care team members is valuable in quality initiatives, understanding heterogeneity in knowledge of activities would also be important. We therefore reworded selected survey items to reflect uncertainty about ongoing initiatives, such as the item, “Has your program conducted any quality improvement projects for the quality of care provided by your program in the past year?” which includes an option of “Not sure”.

Finally, interviewees identified two key areas of unclear wording about quality. When discussing palliative care quality indicators, the terms “use” and “implementation” were unclear and interviewees often did not distinguish between them. To improve clarity, we removed these terms, rephrasing these as “collection” of quality indicator information throughout for clarity. The focus for quality indicators and improvement was also often unclear to respondents, so we reworded throughout the survey to describe the focus as “quality of care provided by the palliative care program.”

Discussion

We compiled, adapted and evaluated the content validity of measurement instruments to develop a survey for evaluating facilitators and barriers of palliative care program quality measurement and improvement, based on an adaptation of the Consolidated Framework for Implementation Research. The survey addresses key domains of organizational characteristics, characteristics and roles of individual and teams, intervention characteristics and the process of implementation. The survey is adaptable to include either or both quality measurement and improvement and to evaluate perceptions of current quality measurement efforts, and can be used with all palliative care team members as well as palliative care quality experts and leadership.

Evaluating content validity through pilot testing and cognitive interviews led to significant revisions of the survey, particularly in how to define key terms and describe the survey respondents, leadership and organization. These findings emphasize the importance of adapting and evaluating the validity of measurement instruments when applying instruments from other areas to palliative care. The next steps of evaluating reliability and criterion and construct validity, including discriminant and convergent validity, will be important in continued evaluation of this survey. This will include evaluating how survey constructs are associated with collection of quality indicator data, as well as the quality of care that patients receive. Ideally, further evaluation will also include assessment of responsiveness to change for programs that implement quality-related training, improvement programs or other initiatives. Results of these evaluations could also help to guide use of the survey in tailoring initiatives to specific sites, palliative care team members, populations and quality indicators, and inform whether and how quality initiatives’ success in specific sites could be translated to other settings.^{12, 37}

Several recent non-US qualitative studies have evaluated perceptions of barriers and facilitators of quality measurement and improvement, many of which correspond well with the content of this survey and support its content validity. A European study of quality measurement found that characteristics of individuals (attitudes and expertise) and the organization (leadership and resources, including time) were key perceived facilitators.³⁸ Another study of quality improvement across five European countries found that characteristics of individual professionals, such as expertise and motivation, and organizational context, such as leadership, culture of change, team climate, resources (including time) and facilities were perceived facilitators. Key barriers included changes inconsistent with local care philosophy and lack of expertise.^{39, 40}

We developed the survey based on validated tools and rigorous methods, but the process had limitations and the survey requires further development. The CFIR may not include all relevant constructs, and our searches for existing surveys using implementation science databases likely missed some relevant instruments. Our cognitive interviews with six palliative care team members might not be representative of palliative care team members broadly.

Additional conceptual work will continue to evaluate how best to address these issues in palliative care quality initiatives, including evaluating different sub-constructs to include and how best to frame items for this context. Instruments not included in this evaluation and newly developed instruments should also be evaluated for applicability to palliative care quality initiatives. Many key CFIR constructs do not have measurement instruments or cannot be effectively evaluated using surveys, and need to be evaluated through other techniques such as interviews, data collection or observation. Finally, palliative care team survey results may help identify potential issues, but understanding how best to address them also requires additional evaluation, including exploring how palliative care team members responded to the items within a specific setting and integrating other sources of information.

Conclusions

Exploring barriers and facilitators to quality measurement is critical for understanding how best to effectively implement and sustain specialty palliative care quality initiatives.⁴¹ Using a rigorously-developed survey to measure palliative care team members’ perceptions may be useful for framing how best to implement quality measurement initiatives, interpret results to prioritize quality improvement strategies, and enhance ongoing quality improvement. Further research will evaluate the construct and criterion validity of this survey for this purpose and refine how it should best be implemented. Since many barriers and facilitators cannot be measured using surveys, other methods of evaluating quality initiatives will also be needed. Importantly, the patient and family perspective on burdens of data collection, how quality measurement affects their experience of care and ensuring that quality issues that matter most to them are addressed must always be considered in quality measurement and improvement.