Prioritization, Development, and Validation of American Association of Cardiovascular and Pulmonary Rehabilitation Performance Measures

STRUCTURED ABSTRACT

Background

In 2014, the American Association of Cardiovascular and Pulmonary Rehabilitation (AACVPR) Quality of Care Committee was asked to develop performance measures (PMs) to assess program quality and aid in program improvement and certification.

Methods

A 3-step process was used to prioritize, develop, and then validate new PMs for both cardiac and pulmonary rehabilitation programs. First, we surveyed national AACVPR leadership, medical directors, and program directors to identify and rank various potential PM topics. Then, the face validity of the drafted PMs was assessed in a second national survey. Finally, we assessed the inter- and intra-rater reliability and feasibility of each PM by abstracting patient charts at programs throughout the United States. At each step, modifications were made to refine and improve the measures for clarity, reliability, and consistency.

Results

Through survey answers received by 302 people (19% response rate), we identified 5 categories for PM development: optimal blood pressure control, tobacco use cessation, and improvement in functional capacity, depression, and sensation of dyspnea. After drafting the performance measures, a second survey with 82 respondents (57% response rate), found that the proposed PMs had good face validity. Finally, we found excellent inter- and intra-rater reliability for the blood pressure, functional capacity, depression, and dyspnea measures (kappa generally >0.80.) However, validity concerns were raised about the tobacco intervention PM as written, and it continues to undergo further refinement and testing.

Conclusions

We developed and validated five new PMs for use in cardiac and pulmonary rehabilitation program quality assessment, improvement, and certification.

“Quality is never an accident, it is always the result of intelligent effort.”

- John Ruskin, 1819–1900¹

In 2000, the Institute of Medicine published a landmark document, Crossing the Quality Chasm, which launched a national focus on improving quality and safety for patients.² In 2010, the United States (US) Congress passed the Affordable Care Act, which mandated that Medicare begin to incorporate quality scores into payments.³ Accordingly, Medicare currently incorporates factors such as hospital readmission rates, preventable hospital acquired condition rates, and overall patient experience measures in public reports about quality of care and in payment adjustments to hospitals.^4,5 It is now expected that by 2018, more than 50% of Medicare payments will be disbursed through alternative payment models that incorporate measures of quality.⁶ In addition, performance measures (PMs) are increasingly being incorporated into qualified clinical data registries to help programs, hospitals, and hospital systems measure, track, benchmark, and improve processes in order to improve meaningful patient outcomes and to report quality data to Medicare. For example, the American Heart Association and American College of Cardiology registries include performance measures related to referral to cardiac rehabilitation,^7,8 as well as other process and outcomes measures linked to Class IA and IB recommendations in clinical practice guidelines.

In response to these trends, in fall of 2014, the American Association of Cardiovascular and Pulmonary Rehabilitation (AACVPR) Board of Directors charged the AACVPR’s Quality of Care Committee (QCC) to create uniform standards to evaluate program quality. This committee is composed of a diverse group of AACVPR members that includes physicians, nurses, exercise physiologists, psychologists, dietitians, respiratory therapists and physical therapists. Together, the QCC worked on a multiyear initiative to develop and validate new PMs for use in program certification, with a focus on developing outcomes measures that are linked to meaningful patient outcomes which can be influenced by cardiac or pulmonary rehabilitation. The ultimate goal of this project was to provide the tools and standards that programs would need to assure high quality clinical care and ultimately improve patient outcomes. As pulmonary rehabilitation (PR) already had 2 validated PMs that were endorsed by the National Quality Forum (NQF),^9,10 this effort primarily focused on cardiac rehabilitation (CR) PMs but also included developing 1 additional PM for PR programs.

METHODS

In the fall of 2014, the QCC began a deliberate and systematic approach to choosing, writing, and validating new PMs. The QCC followed a distinct 3-step process that included a modified Delphi approach which included structured, purposeful, and repeated cycles of QCC and AACVPR membership involvement to ensure stake holder engagement.¹¹ First, we surveyed national AACVPR leadership and AACVPR members working as CR medical directors and program directors to identify and rank various potential PM topics. Next, we assessed the face validity of the drafted PMs in a second national survey. Finally, the inter- and intra-rater reliability of each PM was assessed by abstracting patient charts at programs throughout the US. As a quality improvement effort, institutional review board (IRB) approval was not required for the first 2 steps because individual patient data were not included, and all surveys were anonymous. For the third step, IRB permission at both Helen Hayes Hospital in Haverstraw, NY (project planning and implementation) and at Baystate Medical Center in Springfield, MA (data collection and analysis) were obtained. In both cases the project was determined to be exempt because it did not meet the federal definition of human subjects’ research.

In an effort to identify key areas for PM development, the QCC utilized the core components of both cardiac and pulmonary rehabilitation^12,13 in addition to several other potential topic areas. A survey regarding these topic areas was then sent to all current program directors and medical directors, as well as current and former AACVPR leaders, fellows, board members, and state affiliate board members. We surveyed professionals in both cardiac and pulmonary rehabilitation, but they only responded to questions directly related to their area(s) of focus and specialization. The survey asked respondents to rate each of the potential topics on 8 characteristics (Table 1), utilizing a Likert scale ranked from 1 to 4, using the following response scale: strongly disagree, disagree, agree, and strongly agree. A free response comment was also included for each topic area. The survey was distributed in December 2014 and used www.surveymonkey.com as the electronic platform. Participation was voluntary, no incentive was provided, and only 1 survey request was sent to each potential respondent. Participants also answered several basic questions about themselves and their roles, experience, and responsibilities. To avoid mixing responses and increase transparency, the members of the QCC also completed the survey and these results were recorded separately.

Table 1.

Characteristics of an Ideal Performance Measure

Scientific basis is strong and well established Results are clinically meaningful Results are easily interpretable by clinicians Results can be influenced by the clinical team Results are can be obtained with reasonable effort and cost Results accurately reflect changes in a patient’s status Results have a low likelihood of producing an unintended negative consequence Results will provide an accurate reflection of the quality of a program and can be used to distinguish good and poor quality

After selection of the highest priority PM topics, 1 member of the QCC drafted each PM with input from other members of the QCC and an ACCVPR experts group, as needed. Initial drafts were reviewed by the QCC and were modified for clarity and content. To assess face validity of the proposed measures, a second survey was then sent to all current AACVPR board members, AACVPR committee members, and state affiliate presidents to establish face validity of the PMs using similar methods as the first survey, except that Likert scale questions were now ranked from 1 to 5 with an additional middle ranking of 3 (neither agree nor disagree). Participants were asked to rate each PM according to both the 8 ideal characteristics of a PM (Table 1), as well as clarity of the PM language and definitions. Each PM had a free response section for comments. Results of this survey were discussed by the QCC and needed adaptations in language and definitions were made by the PM authors.

In the third step of this project, we assessed the feasibility and reliability of the PMs using structured chart abstraction, as described previously.¹⁴ We first solicited volunteer programs from the AACVPR program directory, each of which participated in the project to the extent feasible. However, because of different populations and resources available, only a few programs were able to abstract every PM. Each participating program then identified 3 staff members to participate in the abstraction process for the PMs they volunteered for. One staff member (site coordinator) identified 35 patient charts through purposeful sampling where about 30 of the charts qualified for the measure denominator and 5 of the charts did not, which were then placed in random order.

Next, the site coordinator then provided this list of 35 patients to the 2 other staff members, who reviewed each patient chart to see if the patient met the PM. First, they reviewed the denominator (does the patient have the condition?), then the exclusion (is there an exclusion present?), and finally if the patient met the numerator (was the performance measure met?). They recorded the time needed to complete the chart abstraction and any feasibility issues encountered. All answers were entered into REDCap, a secure on-line research application that allows multiple users to enter data simultaneously from different locations.¹⁵ At no point were names or other identifying information entered into REDCap. Then, 1–2 weeks later, the 2 abstractors again reviewed the same 35 patients following the same procedures as previously noted, so that each individual chart was reviewed 4 times for each PM.

Statistical Analysis

For the prioritization survey and the face validity survey, we averaged the 8 Likert-scale scores of each domain (Table 1) into a single score for each PM. As an alternate measure of priority, we then ranked each PM within each of the 8 domains from highest score to lowest score. PMs were given 1 point for each individual domain score within the top 5. These points were then summed across domains with a summary score range between 0 and 8. In both surveys, comments were summarized, discussed, and needed changes were made to the PMs.

For the reliability assessment phase, all 4 chart abstractions were assembled on the same patient chart for each PM to facilitate calculation of inter and intra-rater reliability. Time required for chart abstraction was averaged across each PM for both the first and second chart abstraction. We used Cohen’s kappa statistic to assess agreement at each step of the PM, including agreement at the denominator, exclusions, and numerator levels. We calculated agreement as a combined group. We considered a kappa ≥0.8 to be excellent agreement, 0.61–0.79 to be good agreement, 0.41–0.60 to be fair agreement, and ≤0.4 to be poor/unacceptable agreement. All calculations were performed on JMP version 12.0.1 (SAS Institute) and Microsoft Excel (2010).

RESULTS

The prioritization survey for both CR and PR were sent to 1562 professionals, of which 302 (19.3%) completed the survey. Of these 302 completers, 214 (70.9%) and 88 (29.1%) were CR and PR professionals, respectively. The majority of participants were program directors with Master’s degrees in either nursing or exercise physiology, who had more than 15 years of experience (Table 2.)

Table 2.

Survey Participants^a

	Prioritization Survey		Face Validity Survey
	Cardiac Rehabilitation n = 214	Pulmonary Rehabilitation n = 88	Both CR/PR n = 82
Professional Field
• CR	214 (100)	---	30 (37)
• PR	---	88 (100)	10 (12)
• Both	---	---	42 (51)
Role in CR/PR
• Program staff	64 (30)	9 (10)	26 (32)
• Program director	135 (63)	61 (69)	47 (57)
• Medical director	12 (6)	13 (15)	1 (1)
• Academic	1 (0)	3 (3)	4 (5)
• Not involved	2 (1)	2 (2)	4 (5)
Experience with Quality Improvement
• Role at hospital	90 (42)	34 (39)	---
• Role in program	175 (82)	76 (86)	---
• Within AACVPR	42 (20)	11 (13)	---
• Within affiliate	64 (30)	17 (19)	---
• Other organizations	18 (8)	11 (13)	---
Highest Academic Degree
• Associate	28 (13)	21 (24)	4 (5)
• Bachelor	68 (32)	26 (30)	22 (27)
• Masters	94 (44)	18 (20)	45 (55)
• Doctorate	24 (11)	23 (26)	6 (7)
• Other	-	-	5 (6)
Profession
• Nurse	111 (52)	23 (26)	23 (28)
• Exercise physiologist	79 (37)	13 (15)	42 (51)
• Physician	18 (8)	14 (16)	5 (6)
• Advanced practitioner	6 (23)	3 (3)	0 (0)
• Respiratory therapist	3 (1)	36 (41)	9 (11)
• Dietitian	2 (1)	0 (0)	0 (0)
• Physical therapist	1 (0)	5 (6)	1 (1)
• Occupational therapist	1 (0)	0 (0)	1 (1)
• Behavioral health/Other	1 (0)	0 (0)	1 (1)
Years Working in Rehabilitation
• <2	4 (2)	1 (1)	1 (1)
• 2–4	18 (8)	7 (8)	3 (4)
• 5–9	18 (8)	16 (18)	13 (16)
• 10–14	32 (15)	14 (16)	12 (14)
• 15–19	39 (18)	16 (18)	13 (16)
• 20–24	41 (19)	14 (16)	17 (21)
• 25 or more	62 (29)	20 (23)	23 (28%)

Abbreviations: AACVPR, American Association for Cardiovascular and Pulmonary Rehabilitation; CR, cardiac rehabilitation; PR, pulmonary rehabilitation.

^aData reported as number (percent).

For CR, of the 14 potential PMs, blood pressure control, improvement in functional capacity, diabetes self-management, psychosocial risk, and abstinence from tobacco use had the highest number of domains ranked in the top 5 and generally had some of the highest overall summary scores (Appendix Table 1). Notably, question number 8, which assessed the ability of the PM to distinguish between a high and low-quality program, consistently scored the lowest of all domains across all PMs and was therefore reported separately. (Appendix Table 1 and 2). QCC scores generally agreed with the primary survey, differing in ranking appropriate utilization of cardiac rehab as a top priority. After discussion and incorporation of survey comments, the QCC decided not to pursue diabetes self-management or appropriate utilization of CR because both were felt to be out of the locus of control of an individual CR program and would require significant physician and hospital coordination. It was decided to make tobacco cessation a process measure rather than an outcome measure, because of the consistently reported feeling that tobacco cessation was difficult to accurately measure and modify in CR. Among the potential topics in the psychosocial risk category, the QCC decided to pursue depression as the first PM given that it is an important comorbidity^16–18, is treatable in CR^19–22, and is strongly proven to be associated with mortality risks.^23–25

Of the 12 topics in the PR survey, improvement in sensation of dyspnea, improvement in functional capacity, improvement in health-related quality of life, and appropriate oxygen utilization had the highest mean scores and number of domains ranking in the top 5. (Appendix Table 2.) These rankings validated the importance of the 2 previously developed PMs (improvement in health-related quality of life and improvement in functional capacity). After incorporation of comments, the QCC decided to further develop the improvement in sensation of dyspnea PM, because improvement in dyspnea is the primary patient symptom addressed by PR. Although appropriate oxygen utilization scored strongly as a topic for measure consideration, the QCC felt that there was not yet sufficient peer-reviewed evidence to require uniform testing, prescription and evaluation of oxygen adherence across all PR programs, so further development on this topic was suspended.

After drafting the initial PMs, we then sent the face validity survey to 144 CR and PR leadership individuals, with 82 (56.9%) completing the survey. Participants’ characteristics were generally similar to the prioritization survey, but this time represented a mixture of both CR and PR professionals (Table 2). Survey results are found in Table 3 and show that, nearly uniformly, the PMs scored between 4 and 5, indicating that participants agreed or strongly agreed on nearly all 8 domains as well as across PM language and definitions. Based upon comments submitted, additional minor modifications in language and definitions were made to each PM.

Table 3.

Mean Performance Summary Scores from Face Validity Survey^a

	Blood Pressure	Functional Capacity	Depression	Tobacco Cessation	Dyspnea
Performance Measure Characteristic
Scientific basis for the measure is well established	4.48	4.6	4.53	4.6	4.47
Measure is reproducible across rehabilitation programs	4.38	4.36	4.32	4.32	4.32
Measure captures meaningful aspects of care and leads to better outcomes for patients	4.37	4.55	4.49	4.43	4.45
Data required for the measure are easily obtained with reasonable effort and cost	4.64	4.43	4.39	4.23	4.55
Likelihood of negative unintended consequences with the measure is low	4.36	4.40	4.42	4.38	4.39
A provider/program can improve this performance measure	4.41	4.75	4.39	4.30	4.50
Scores obtained from the measure, as specified, will provide an accurate reflection of quality and can be used to distinguish good and poor quality	3.83	4.2	3.85	3.98	3.92
Scores obtained from this measure can be used for program improvement	4.38	4.51	4.36	4.34	4.34
Performance Measures Language and Definitions
Patient group that achieves the desired outcome is clearly defined and clinically meaningful (numerator definition)	4.31	4.45	4.37	4.4	4.37
Intended patient group to whom the measure applies is clearly defined and clinically meaningful (denominator definition)	4.40	4.51	4.41	4.36	4.37
Exceptions/exclusions are appropriate for the measure and the patient group	4.46	4.55	4.31	4.43	4.45
Inclusion/exclusion criteria can be easily understood by personnel in programs	4.42	4.52	4.41	4.40	4.47
Inclusion/exclusion criteria can be easily found in program records	4.38	4.50	4.40	4.37	4.42

^aPerformance measures were ranked on a scale of 1 to 5 with 1= strongly disagree, 5 = strongly agree and then averaged.

For feasibility and reliability testing, we recruited 16 sites (both CR and PR) across the US to participate in abstracting up to 4 PMs at each site, with 12 sites completing at least 1 PM abstraction. The number of sites for each PM is shown in Table 4, as well as the number of charts reviewed for each PM at each step of validation. The number of patients who met both the denominator and exclusion definitions for the tobacco cessation measure was small (52 patients) whereas 286 and 253 patients were evaluated for the blood pressure and functional capacity PMs numerator definitions, respectively. In general, both intra-rater and inter-rater agreement was excellent (kappa ≥0.8) or good (kappa 0.61–0.79) for most of the PMs. Abstractors struggled with operationalizing the exclusion definitions for both the depression and dyspnea PMs, but otherwise had excellent intra-rater and inter-rater reliability results. However, the tobacco cessation PM numerator showed low inter- and intra-rater reliability and had the slowest abstractions times. After discussion with the QCC and chart abstractors, the QCC decided that the depression and dyspnea measure inconsistencies could be addressed with better PM definitions, but that the tobacco cessation PM should be simplified and re-tested with a larger eligible population in the future. Current PMs (including the 2 previously validated pulmonary PMs) with their data definitions, algorithm, and frequently asked questions, can be found in the online Appendix or on the AACVPR website.²⁶

Table 4.

Validation Results for the 5 Performance Measures^a

	nb	Blood Pressure	nb	Functional Capacity	nb	Depression	nb	Tobacco	nb	Dyspnea
Participating sites, n		10		9		8		4		5
Time for first chart abstraction, min		1.8 ± 1.2		1.8 ± 1.1		1.9 ± 1.2		2.2 ± 1.6		1.5 ± 1.0
Time for second chart abstraction, min		1.4 ± 0.7		1.4 ± 0.7		1.4 ± 0.7		2.1 ± 1.3		1.4 ± 0.8
Intra-rater reliability Kappa, 95% CI	350		315		245		140		175
• Denominator	295	0.92 (0.87, 0.96)	262	0.96 (0.81, 0.91)	199	0.86 (0.80,0.91)	56	0.94 (0.90, 0.98)	104	0.94 (0.91, 0.97)
• Denominator after exclusions	286	0.91 (0.80, 1.00)	253	0.81 (0.68, 0.95)	173	0.92 (0.86, 0.98)	52	0.94 (0.82, 1.00)	97	0.59 (0.36, 0.82)
• Numerator	239	0.90 (0.84, 0.94)	166	0.86 (0.81, 0.91)	125	0.92 (0.88, 0.97)	28	0.70 (0.58, 0.82)	57	0.92 (0.87, 0.98)
Inter-rater reliability Kappa, 95% CI	315		315		245		140		175
• Denominator	265	0.88 (0.82, 0.96)	262	0.84 (0.76, 0.92)	199	0.73 (0.62, 0.84)	56	0.88 (0.80, 0.96)	104	0.87 (0.80, 0.94)
• Denominator after exclusions	258	0.93 (0.80, 1.00)	253	0.79 (0.59, 0.99)	173	0.40 (0.20, 0.60)	52	0.73 (0.38, 1.00)	97	0.58 (0.21, 0.95)
• Numerator	209	0.84 (0.75, 0.92)	166	0.85 (0.78, 0.92)	125	0.87 (0.78, 0.96)	28	0.21 (0.00, 0.43)	57	0.93 (0.85, 1.00)
Met performance measure using inter-rater data, %c		81.0		65.6		72.2		53.8		58.7

^aData reported as mean ± standard deviation or kappa (95% confidence interval unless otherwise noted.

^bNumber of patients who met each criterion at each step of the analysis combined across all sites. For example, the blood pressure measure started with 350 charts at 10 sites. Of these, 295 met criteria for the denominator and 9 charts had exclusions. In total, 239 of the eligible 286 (83.6%) patients met criteria for the measure. There was some disagreement (kappa <1.0) between raters at each step, so the data presented here is the average number at each validation step, recognizing that raters sometimes disagreed about which patients met the measure definitions.

^cClaculated by dividing the numerator by the denominator after exclusions. SD = standard deviation; CI = confidence interval

DISCUSSION

Using a methodical approach for identification and validation of new PMs, we introduced 5 new performance measures for use among both CR and PR programs across the US. These were identified through significant engagement with stakeholders including AACVPR membership, AACVPR national and regional leaders, and the QCC. Development was an iterative process with modifications made at each step to improve and clarify each PM based upon the feedback received. Overall, results suggest that these PMs should have strong AACVPR member buy-in, are feasible to implement, and, when applied to programs across the US, should perform adequately to distinguish high and low quality programs. Such information will be valuable to payers, referring clinicians, and to the programs themselves as they will give them objective feedback on their performance and should stimulate changes in clinical practice and improvements in quality of care.

The first PM within cardiopulmonary rehabilitation was originally developed in 2007 and focused on CR referral.²⁷ This PM was validated and endorsed by the National Quality Forum (NQF), updated in 2010, and is currently undergoing an additional update.^14,28 Similarly, the 2 PR measures were initially endorsed by the NQF in January 2011.^9,10 Now, with the addition of the PM for dyspnea in PR as described in this paper, PMs for the top 3 major outcomes of PR that have strong scientific evidence are developed and validated. Additionally, although structure and process measures for CR programs were initially described in 2007,²⁷ this is the first time any PM for CR programs has focused primarily on outcomes and been tested and validated. In the future, we anticipate that additional PMs will be developed and existing PMs will be updated and modified as new information becomes available. We also anticipate assessing how effective these PMs are in improving the actual quality of CR and PR services.

One of the major impacts of these PMs will be their use in AACVPR program certification and data registries^29,30 to assess CR and PR program quality. Beginning with the 2018 program certification cycle, programs will be required to submit data on these new PMs as part of the application for AACVPR certification. However, currently, no standards have been set at which a program would be certified or otherwise denied certification, although some data regarding expected mean exercise workloads in CR has recently been published.³¹ Consequently, 2 of the major goals for this first round of program certification are to establish baseline PM data and to assure that these PMs are incorporated into the individual treatment plans of programs across the US. Then, when such data are created, collected, and submitted, clear standards for program quality can be set and justified scientifically. To this end, both the cardiac and pulmonary registries are in the process of being modified to capture all the necessary elements to evaluated program performance on these PMs.

Ultimately, the goal of these PMs is to assure that all CR and PR programs in the US are meeting or exceeding high quality standards for patient care and that certification through the AACVPR is a marker of true program quality. Over time, we anticipate these PMs will be used in quality improvement projects and will help standardize and improve care.

These data should be viewed in light of several limitations. First, there was a low response rate to the first prioritization survey, which suggests these results may not be fully generalizable because they represent a select group of respondents. However, we still had 302 participants and survey responses were received from clinicians all over the US. Second, during the validation, only a limited number of programs participated and chart abstractions varied substantially between PMs. This heterogeneity could mean that PMs might perform differently when used in settings other than the ones in which we tested them and have limited generalizability. This limitation may be overcome by future validation studies or as data about these PMs become available in AACVPR registries. Third, although the tobacco cessation PM had low reliability, this measure has now been simplified and is currently undergoing additional testing. This initial low performance is not a critical problem because, just like every other PM introduced here, it will not be immediately used to guide certification or reimbursement. This will allow the tobacco cessation measure to be retested before it is tied to any certification or financial outcomes.

Conclusions

We identified, drafted, modified, and validated 5 new PMs for use in cardiac and pulmonary rehabilitation programs across the US. We anticipate these PMs will be the beginning of a new focus on improving quality and consistency in cardiopulmonary rehabilitation care. Modifications will continue and new PMs will be validated. Ultimately, we believe these PM will help programs provide high quality care and ultimately may be useful in future value-based purchasing agreements in the ever changing landscape of US healthcare.

Supplementary Material

SDC Appendix

Appendix Table 1. Summary Survey Results for Prioritization Survey for Cardiac Rehabilitation Outcomes

Appendix Table 2. Summary Survey Results for Prioritization Survey for Pulmonary Rehabilitation Outcomes