Abstract
Despite their widespread use, audit and feedback (A&F) interventions show variable effectiveness on improving professional performance. Based on known facilitators of successful A&F interventions, we developed a web-based A&F intervention with indicator-based performance feedback, benchmark information, action planning and outreach visits. The goal of the intervention was to engage with multidisciplinary teams to overcome barriers to guideline concordance and to improve overall team performance in the field of cardiac rehabilitation (CR). To assess its effectiveness we conducted a cluster-randomized trial in 18 CR clinics (14,847 patients) already working with computerized decision support (CDS). Our preliminary results showed no increase in concordance with guideline recommendations regarding prescription of CR therapies. Future analyses will investigate whether our intervention did improve team performance on other quality indicators.
Introduction
The widespread uptake of electronic patient records (EPRs) provides unprecedented opportunities to monitor clinical performance and improve care quality. As such, audit and feedback (A&F) interventions based on interrogating EPR databases are increasingly used to aid health care professionals in improving their performance. A&F provide care professionals with an objective summary of their clinical performance over a specified period of time [1]. Yet despite their expanded use, A&F interventions show variable effectiveness on improving quality of care. A recent Cochrane review of 140 randomized trials of A&F interventions reported a median 4.3% absolute improvement (interquartile range 0.5% to 16%) in quality of care, with a quarter of the studies showing a strong positive effect, but with another quarter showing a negative or null effect [1].
Previous studies have attributed much of the observed variability in effect of A&F interventions to feedback design characteristics and contextual factors. They suggested A&F to be most effective if baseline performance is low, when feedback is provided by a supervisor or colleague, more than once, both verbally and in writing, and when it includes explicit targets and an action plan [1–4]. Furthermore, the effect of indicator-based performance feedback is likely to be stronger when it is combined with educational meetings [1]. Other suggested effect modifiers are the perceived quality of the data underlying the feedback, motivation and interest of the recipient, organizational support for quality improvement (QI), and how performance targets or benchmarks are derived [5]. Besides these literature results we had a conjunctional expectancy based on the actuality that modern medicine, including the care for chronically ill patients, is not just a matter of individual professionals but largely the responsibility of multidisciplinary teams embedded in complex organizations. Therefore we expected that, specifically in chronic disease management, engaging the entire multidisciplinary teams and their managers in the QI process is an important success factor of A&F interventions [6]. We developed a multifaceted A&F intervention that both incorporates successful characteristics described in the literature [1–5] and that is specifically directed at multidisciplinary teams [7]. It comprises the use of a web-based system that provides periodic performance feedback with benchmark comparisons and support for concrete QI action planning. In combination with educational outreach visits the systems facilitates active team engagement in improving their performance [8].
We implemented our intervention in the field of cardiac rehabilitation (CR) in the Netherlands. Within this field an EPR system with computerized decision support (CDS) functionalities was previously developed to stimulate concordance with guideline recommendations for the patient tailored CR program [9]. Although the CDS has proven to be effective [10], there remained considerable non-concordance due to a lack of resources and other organizational constraints [11]. Further improvement of guideline concordance required organizational and procedural changes that individual users considered to be beyond their own tasks, influence and responsibilities. Use of the CDS system alone was insufficient for inciting users to involve decision makers at team and organizational level to realize those changes [11]. This finding stressed the need for an intervention specifically directed at the decision-making processes at these levels to create the necessary conditions and resources for further improving guideline concordance. Hence, guideline concordance was one of the targeted behaviors of our A&F intervention.
We performed a multicenter cluster randomized trial to assess the effect of a multifaceted A&F intervention on clinical performance of multidisciplinary teams in the field of CR. In this paper we present preliminary results regarding the intervention’s effect on concordance of CR therapies with guideline recommendations.
Methods
Setting: cardiac rehabilitation and computerized decision support
CR is a therapy provided by multidisciplinary care teams to support cardiac patients recovering from a cardiac incident or intervention on both the physical and psychosocial domain [12, 13]. CR is recommended for all patients who have been hospitalized for an acute coronary syndrome (ACS) and for those who have undergone a cardiac intervention [12, 14]. A meta-analysis shows consistent evidence of the effectiveness of exercise-based and multimodal (e.g., psychosocial and stress management) CR interventions with regard to mortality and prevention of future cardiac events (relative-risk reduction 21–47%) [15]. CR teams usually include cardiologists, physical therapists, nurses, psychologists, dieticians, social workers, and rehabilitation physicians. However, in many Western countries, CR services are under-utilized and poorly standardized and do not follow the available scientific evidence [16]. A recent study in the Netherlands shows that only a minority of patients eligible for CR actually receive it [17]. The CR uptake rate was 28.5% among patients with an ACS and/or intervention.
Consistent with international standards, the Dutch Guidelines for CR [18] state that professionals should conduct an extensive needs assessment procedure (NAP) where 80 to 130 data items concerning the patient’s medical, physical, psychological, and social condition and lifestyle are gathered. Based on this procedure, a patient-tailored rehabilitation program should be prescribed which can contain up to four group-based therapies: two psychosocial therapies (disease-specific education; lifestyle modification) and two physical therapies (exercise training; relaxation and stress management training), and can be supplemented by individual counselling (e.g., by a psychologist, dietician or social worker) when needed. In the Netherlands, there are two commercial vendors of EPR systems with CDS for CR (referred to as EPR1 and EPR2) that can be used for data collection. Both systems are based on the Dutch guidelines for CR [18] and follow the same data model. They guide their users through the NAP and provide advice for the decision about each out of the four considered CR therapies for the prescribed CR program [19]. However, one of the EPRs is a stand-alone product in which, based on results of usability evaluation of a beta version of the system [20], the data entry navigational structure is organized flexible around an overview screen. Complete data collection is stimulated by showing users which steps of the NAP they already have finished and which steps they still need to complete. The other system is integrated into the hospital EPR from one vendor and offers a more straightforward data entry structure. This system does not provide feedback on finished NAP steps. After data collection in both systems, the patient specific CDS advice is discussed with the patient. Thereafter the prescribed CR program, including the decisions for each out of the four CR therapies (which can deviate from the CDS advice), are recorded in the EPR. While we focus on prescribed therapies in this study, we note that there are sometimes discrepancies between prescriptions and therapies that are actually received by patients. After participation in the program (which typically lasts for 6–12 weeks), patient are reassessed to determine results.
Study design
The effect of the intervention was evaluated in a multicenter cluster-randomized study in which each CR clinic received the A&F intervention, but its contents were randomly limited to one of two complementary domains that jointly constitute CR: the psychosocial domain (disease-specific education; lifestyle modification) or the physical domain (exercise training; relaxation and stress management training). In this way, both study arms served as each other’s control, and we minimized the risk of clinics dropping out of the study because they did not receive any intervention. Cluster-randomization was chosen to avoid contamination among professionals within the same clinic. We refer to the study protocol for further details of the experimental design [7].
Eligible CR clinics and patients
All CR clinics that used an EPR with CDS during the NAP and that were willing to share their data for research and to set up a local QI team were eligible to participate in the study. There were 91 CR clinics in the Netherlands, the majority affiliated with hospitals [21]. Twelve clinics were located in specialized rehabilitation clinics [21], which have regional functions and can treat both simple and more complex referred patients. All types of clinics were eligible to participate in the study, provided that they worked with either one of two commercial EPR systems for CR that could be used for data collection for our study. During the inclusion period of the trial from July 2012 until December 2013, this was the case for 22 clinics. The study dataset consisted of (i) patient identification data (31 items), (ii) CR needs assessment data (80–130 items), (iii) data on selected rehabilitation goals and therapies (79 items), and (iv) CR evaluation data (105 items). All consecutive CR patients that underwent the NAP in one of the participating clinics during the study period were eligible for enrollment in the study. Clinics that participated agreed to enter all data of these patients in their EPR.
Intervention
Our multifaceted A&F intervention was provided through a web-based system, called CARDSS Online [8]. When designing the intervention we followed the A&F literature which underlines the importance of combining periodic performance feedback with benchmark comparisons, action planning with concrete, self-formulated goals, and educational outreach visits to actively involve care professionals in a continuous QI process. To this end CARDSS Online supports four tasks: (i) monitoring of indicator-based performance by means of quarterly feedback reports including benchmark information, (ii) selecting indicators for QI that are locally perceived as important and upon which improvement is deemed feasible, (iii) developing a QI plan consisting of QI goals and concrete actions to accomplish these goals, and (iv) during follow-up iterations updating the QI plan based on new performance measurements and experiences with executing the QI actions in practice. Benchmark comparisons were summarized by a colored icon next to each indicator score which depicted whether the performance was acceptable (green checkmark), borderline (yellow checkmark), or poor (red exclamation mark). The benchmark comparisons were based on the clinic’s performance score and the average score across all clinics (details available in [7]).
Educational outreach visits were held following each feedback report. During these visits CARDSS Online was used to guide the clinics’ local QI teams through the process of systematically defining, implementing and monitoring QI actions. The QI teams consisted of at least one local CR coordinator (usually a specialized nurse), one professional from another discipline (e.g. a physical therapist), their manager and the responsible cardiologist. The visits were chaired by an investigator (MvEV) who supported the QI team with interpretation of the feedback and drafting (or during follow-up visits monitoring and updating) a concrete QI plan. Indicators were included in the plan based on the benchmark information and discussion on importance, feasibility, and expected time needed to improve. For each quality indicator, the QI team could specify the problem, presumed causes, improvement goal, and concrete actions on how to reach that goal. If clinics agreed upon extended participation after the minimum study period of one year (comprising of four A&F iterations), they received up to two more quarterly feedback reports in combination with telephone support rather than a face-to-face visit.
We previously developed a set of eighteen primary quality indicators to provide performance feedback in our system [22]. This was done in close collaboration with an expert panel (representatives from all disciplines involved in CR) and patient panel, using a modified RAND method [22]. Results from both panels were combined with results from a literature search and guideline review in an extensive rating and consensus procedure. The expert panel did not select concordance of prescribed therapies with the guidelines as one of the eighteen primary quality indicators. However, the tailoring of CR programs to individual needs of patients is an important quality theme in CR, and indirectly reflected by many indicators that were chosen. Furthermore, we did include concordance of prescribed therapies with the guidelines in the feedback reports, which enabled QI teams to include improvement actions aimed at guideline concordance in their QI plans. Besides results on indicators and concordance, the feedback also included patient characteristics (e.g. age and diagnosis), information referring to general processes (e.g. time between discharge and NAP) and structures (e.g. presence of patient satisfactory research) to reduce the risk of attrition.
Outcome measurement
Our intervention was targeted at health care professionals and was therefore expected to have a direct effect on process outcomes but only an indirect, long-term effect on patient outcomes. The outcome measure was therefore concordance to national CR guidelines regarding the CR program that was prescribed during the NAP. We defined concordance at the level of the patient; it implied prescribing therapy to patients who should be treated and not prescribing therapy to patients who should be untreated, according to the guidelines. This was determined for each of the four group-based therapies separately.
Cluster randomization and allocation
Randomization of CR clinics was stratified by size (more/less than 30 patients starting CR per month). Per stratum, we generated a randomization scheme with randomly assigned block sizes of either two or four CR clinics using dedicated software. This scheme was concealed to those enrolling and assigning CR clinics [7]. Due to the character of the intervention, it was not possible to blind participants or those involved in providing the intervention.
Statistical analysis
For each of the four CR therapies (education, lifestyle modification, exercise training, and relaxation training) we performed a separate mixed-effect logistic regression analysis [10, 23] to assess the effect of the intervention on concordance with guideline recommendations. To this end we included covariates study arm, time, and the interaction between study arm × time. We focused on the interaction term to assess the difference in change over the study period between the two arms—that is, the effect of the intervention—because we expected concordance to improve gradually. We used random effects to model the variation in baseline concordance between clinics (random intercept for each clinic) and the variation in change in concordance over time (random slope for time). To adjust for differences in case mix between the study arms, we included in our analysis three patient level variables (age, sex, and indication for CR) and two clinic level variables (weekly volume of new patients, and whether the clinic is a specialized rehabilitation center or part of a university or teaching hospital) as covariates.
Patients who were seen in the last month of a clinic’s study period were excluded from the analysis because their prescription data was often not yet complete. We also excluded patients for whom the indication for CR was missing. Furthermore, for each of the four analyses of guideline concordance on specific CR therapies we excluded patients for whom it could not be determined whether the prescription of that therapy concordant with the guideline (either because the guideline’s recommendation or the actual prescription could not be determined).
Results
Participants
Of the 22 eligible CR clinics 18 clinics accepted our invitation to participate in the trial, of which twelve clinics were assigned to intervention arm A (receiving multifaceted A&F intervention on psychosocial therapies), and six to arm B (receiving intervention on physical therapies) (see Figure 1). CR clinics were enrolled in the study between July 2012 and December 2013. On average, the time between randomization and the first educational visit was 3.5 months (standard deviation [SD] 0.7). The average time between subsequent visits was 4.0 months (SD 1.4). Table 1 shows the baseline characteristics of clinics and patients. During the study period a total of 14,847 patients were seen for a NAP in the participating CR clinics. After exclusions in the overall database 11,932 patients were included for the analyses on concordance per CR therapy. The analyses were performed on data from 10,730 (education), 10,774 (lifestyle modification), 10,953 (exercise training), and 8,804 (relaxation training) patients.
Figure 1.
Flow diagram of CR clinics through the trial.
Table 1.
Baseline characteristics of clinics (N=18) and patients (N=11,932) per study arm; values are numbers (%), unless indicated otherwise.
| Characteristics | Arm A (A&F on psychosocial domain) | Arm B (A&F on physical domain) |
|---|---|---|
|
| ||
| Clinics | ||
|
| ||
| Number participating | 12 (66.6) | 6 (33.3) |
|
|
|
|
| Median (min-max) number of patients per year | 431 (183–1,156) | 370 (256–988) |
|
|
|
|
| Stratum ‘large’ (>30 patients monthly starting CR) | 6 (50.0) | 3 (50.0) |
|
|
|
|
| Use of EPR1 | ||
|
|
|
|
| CR outpatient clinic type: | 9 (75.0) | 1 (16.7) |
| Non-teaching hospital | 7 (58.3) | 3 (50.0) |
| Teaching hospital | 2 (16.7) | 3 (50.0) |
| University hospital or specialized rehabilitation | 3 (25.0) | 0 (0.0) |
|
| ||
| Patients | ||
|
| ||
| Number included in analyses | 7,692 | 4,240 |
|
|
|
|
| Mean (SD) age in years | 64.9 (11.4) | 65.8 (11.8) |
|
|
|
|
| Male gender | 5,533 (71.9) | 3,027 (71.4) |
|
|
|
|
| Indications for CR | ||
| ACS with revascularization | 4,446 (57.8) | 2,489 (58.7) |
| ACS without revascularization | 440 (5.7) | 386 (9.1) |
| Elective CABG or valvular surgery | 1,262 (16.4) | 598 (14.1) |
| Elective PCI | 517 (6.7) | 321 (7.6) |
| Other elective interventions | 341 (4.4) | 113 (2.7) |
| CHF or stable AP, no intervention | 252 (3.3) | 179 (4.2) |
| Other diagnosis, no intervention | 434 (5.6) | 154 (3.6) |
Abbreviations: A&F= audit and feedback, ACS= acute coronary syndrome, AP= angina pectoris, CABG= coronary artery bypass graft surgery, CHF= chronic heart failure, CR= cardiac rehabilitation, EPR= electronic patient record, PCI= percutaneous coronary intervention, QI= quality improvement, SD= standard deviation.
Implementation of the intervention
Table 2 shows detailed information on how, and to what extent, the main components of the A&F intervention were implemented in the participating clinics. Due to limited availability of the QI team, one clinic in arm A completed only three A&F iterations during the study period instead of four. There were no differences in QI team size, number of indicators selected as QI goal and number of actions per goal in the QI plan, attendance to the visits, and mean study period between the two study groups. Attendance to the visits remained the same during the study period. The mean number of selected QI goals in each QI plan decreased from 8.0 (SD 2.4) during the initial A&F iteration to 5.0 (SD 3.2) in the final iteration. QI teams in both groups reportedly resolved 1.8 of these QI goals per A&F iteration, on average.
Table 2:
Implementation of the multifaceted A&F intervention in daily practice per study arm; values are mean (SD) unless indicated otherwise.
| Implementation of the A&F intervention | Arm A (A&F on psychosocial domain) |
Arm B (A&F on physical domain) |
||
|---|---|---|---|---|
| Range | Range | |||
| QI teams | ||||
| Length of study period per clinic in months | 19.8 (6.0) | 12–30 | 22.5 (4.1) | 14–27 |
| Number of A&F iterations | 4.6 (1.0) | 3–6 | 5.7 (0.7) | 4–6 |
| Size of local multidisciplinary QI team | 7.5 (2.8) | 3–13 | 6.3 (1.3) | 4–8 |
| Number of QI team members attending outreach visits | 5.4 (1.9) | 1–11 | 4.7 (1.8) | 2–8 |
| Number (%) of QI teams receiving first telephone follow-up | 5 (41.7) | n.a. | 5 (83.3) | n.a. |
| Number (%) of QI teams receiving second telephone follow-up | 3 (25.0) | n.a. | 5 (83.3) | n.a. |
| QI action planning | ||||
| Number of indicators selected as QI goal in QI plan | 6.9 (3.1) | 1–14 | 6.3 (2.5) | 0–10 |
| Mean number of actions per QI goal in QI plan | 1.9 (0.5) | 1.0–3.3 | 1.6 (0.4) | 1.0–2.6 |
| Number of achieved QI goals per follow-up A&F iteration | 1.7 (1.5) | 0–5 | 1.9 (1.5) | 0–6 |
| Number of unachieved QI goals in final A&F iteration | 5.9 (3.5) | 1–13 | 3.5 (2.2) | 0–7 |
Abbreviations: A&F= audit and feedback, CDS= computerized decision support, CR= cardiac rehabilitation, n.a.= not applicable, QI= quality improvement, SD= standard deviation.
Effect of the intervention
Table 3 compares crude concordance rates between baseline (first three months) and follow-up (remaining time in the study period) for each of the four therapies. Despite random allocation of the participating clinics into two study arms, Chi-squared testing showed significant differences in baseline concordance for lifestyle modification (p<0.001), exercise (p=0.004), and relaxation training (p<0.001) between the two study groups. Table 3 also presents the results of the mixed-effect logistic regression analyses, which compare the trend in concordance over time between intervention and control groups while adjusting for patient age, sex, and indication for CR and adjusting for clinic type and weekly patient volume. No significant differences were found for any of the four therapies. For three of the four therapies (education, lifestyle modification, and exercise training) there were few missing data (around 10%) with respect to recommended and prescribed care, but for relaxation training we found missing data in 26.2% of cases. This was due to six clinics having substantially lower data quality for the relaxation therapy. A sensitivity analysis in which we excluded these clinics from our dataset did not yield different results.
Table 3.
Concordance rates and difference in concordance between study arms for the four prescribed CR therapies (N=12,111).
| CR therapies | Crude concordance at baselinea) | Crude concordance at follow-upb) | Adjusted odds ratio (95% CI)c) | N (Clinics) | Missing valuesd) | ||
|---|---|---|---|---|---|---|---|
| Intervention | Control | Intervention | Control | ||||
| Psychosocial domain (A&F intervention for arm A) | |||||||
| Education | 87.5% (1,411/1,612) | 81.2% (375/462) | 90.4% (5,045/5,584) | 71.3% (2,191/3,072) | 1.28 (0.65 to 2.54) | 10,730 (18) | 1,202 (10.1%) |
| Lifestyle | 63.0% (1,023/1,624) | 34.3% (155/452) | 63.2% (3,548/5,610) | 25.9% (800/3,088) | 0.75 (0.14 to 4.03) | 10,774 (18) | 1,158 (9.7%) |
| Physical domain (A&F intervention for arm B) | |||||||
| Exercise | 82.6% (399/483) | 89.9% (1,460/1,625) | 83.6% (2,667/3,191) | 95.1% (5,378/5,654) | 0.58 (0.24 to 1.38) | 10,953 (18) | 979 (8.2%) |
| Relaxation | 38.8% (124/320) | 72.6% (976/1,345) | 51.6% (1,211/2,348) | 82.5% (3,952/4,791) | 0.4 (0.06 to 2.79) | 8,804 (18) | 3,128 (26.2%) |
Abbreviations: A&F= audit and feedback, CDS= computerized decision support, CI= confidence interval, CR= cardiac rehabilitation.
Observed concordance during first 3 months of study period.
Observed concordance after baseline period until end of study.
Odds ratio of improvement in guideline concordance after receiving the A&F intervention for 1 year versus no intervention; adjusted for patients’ age, gender, indication for CR, and clinics’ type and size.
Patients for whom the CDS could not provide advice caused by missing data and/or it was not recorded whether the therapy was included in the patients’ CR program.
Discussion
Summary of findings
Our multifaceted A&F intervention did not increase concordance of prescribed CR therapies with guideline recommendations. There appeared to be a high variation in baseline performance and data quality between participation CR clinics. Especially for the relaxation training we had a high percentage of missing data on guideline concordance. Although our intervention facilitated active engagement of local multidisciplinary QI teams in setting their own performance improvement goals, the teams often did not succeed in completing the actions that were needed to achieve those goals.
Strengths and weaknesses of this study
The main strength of our study is that we designed our multifaceted A&F intervention based on both existing knowledge from the literature on effective characteristics of A&F interventions [1–5] and an extensive analysis of potential barriers to further increase guideline concordance in the field of CR [11]. The use of CARDSS Online combined with outreach visits actively involved local QI teams, including managers and cardiologists, in the improvement process. By the development and regularly update of a QI plan with concrete, self-formulated goals, we focused on the decision-making processes at the organizational level to create the necessary conditions for improving guideline concordance. Also, as participating clinics were already working with an EPR with CDS functionality, they did not need to change their workflow to participate in the study and collect data. This pragmatic aspect of our study design may have optimized CR clinics’ willingness to participate and minimized the loss to follow-up. Although this resulted in a relatively large sample of CR clinics in which all different CR clinic types were represented, there were large differences in baseline performance between participants.
A limitation of our study is that only CR clinics that used an EPR with CDS that facilitates registration of our study dataset were eligible to participate. These clinics needed to be willing to share their data for research and to allocate resources to establish a QI team. This potentially resulted in a volunteer bias, as eligible CR clinics were less likely to be understaffed and more likely to have information technology to facilitate routine collection of CR data. The generalizability of our results may thus be limited to clinics that are motivated and equipped to systematically monitor and improve the quality of care they deliver. Second, the intervention allowed QI teams to formulate any improvement actions, even if those were not specifically targeted at improving concordance to a specific guideline recommendation. Although this may have optimized the engagement of the team in the improvement process and the commitment to goal attainment, it undermined the connection between the intervention and our primary outcome measure. This link was further diluted because the set of quality indicators chosen by the expert panel did not include guideline concordance for prescribed CR therapies as a separate indicator. However, we did include concordance statistics on each of the four therapies in our feedback reports. In addition, clinics might have started to improve both CR domains and not just the domain covered in their study arm because the intervention has raised their overall awareness for QI. Last, sometimes there are discrepancies between prescriptions and therapies that are actually received by patients caused by e.g. quality of the content of therapies or patient motivation. The effect of our intervention on received rather than prescribed CR therapies might be different.
Strengths and weaknesses in relation to other studies
Our multifaceted A&F intervention included the development and revision (up to five times) of a QI plan based on indicator-based performance in quarterly feedback reports. Such iterative cycles and repeated use of data over time are generally considered key features to improve health care processes. However a recent systematic review of studies employing the plan-do-study-act method, showed that less than 20% of such studies use iterative cycles of change, and only 14% of them repeatedly use data over time [24].
Furthermore, the combination of A&F with both web-based guidance through the process of systematically developing a QI plan and outreach visits to encourage the local QI team to regularly monitor the feedback and update their plan, stimulated engagement of the QI team. Although this is a known characteristic of effective A&F interventions [1–5], other studies struggle with active engagement of health care professionals in goal setting and action planning to improve their performance [25–27]. Ivers et al [27] performed a qualitative study to understand the usefulness of A&F among family physicians and examined barriers to using it to improve quality of care. Their main findings address some general concerns during implementation of A&F interventions to improve professional performance. Participants reported that the feedback increased their awareness of gaps between ideal and actual performance. This resulted mainly in efforts to “try harder” patient by patient. Key barriers to acting upon feedback in a systematic manner included a perceived discordance between population-level quality targets and patient-centered care (“It [A&F] talks about whole populations as opposed to the one individual and I think my approach to this job is the one person at a time”), as well as competing priorities at both the patient and organizational levels (“How much time do you want your doctor devoting to that [A&F], because the more … the less time I am [devoting] to the patient”). A qualitative analysis which is currently underway should point out if similar barriers were present during the implementation of our multifaceted A&F intervention.
Meaning and implications of findings
Further analyses should point out whether participating clinics were able to improve their performance on individual quality indicators, and whether this was related to the selection of these quality indicators in QI plans and to achieving self-formulated improvement goals. If there was indeed improvement on individual indicators, the failure to achieve progress in concordance of prescribed CR therapies with guideline recommendations is probably due to a poor link between these indicators and guideline concordance of therapeutic prescriptions. If there is no improvement on individual indicators, then our A&F intervention has simply failed to stimulate clinicians to work on QI actions outside their daily routine. The large number of unattained QI goals (Table 2) points in this direction.
According to Ancker et al [28] the evaluation of health information systems, like our web-based A&F intervention, often show mixed results. This may be in part attributable to the evaluation frameworks used. They developed a model for evaluation, named the Triangle Model, in which they emphasize the sociotechnical view that organization, technology, and users influence and change each other during implementation processes. The lack of success of our web-based A&F intervention might not have only depended on the technology used but also on the organizations and professionals involved. Similarly, the Team Strategies and Tools to Enhance Performance and Patient Safety (TeamSTEPPS) framework [29] emphasizes that a thorough needs analysis should be performed to determine organizational readiness before initiating change. This might uncover underlying issues within the institution (e.g. equipment problems or staffing shortages) which first should be resolved to make the QI effort succeed [29]. Also the Systems Engineering Initiative for Patient Safety (SEIPS) model [30] presents a broad approach with a focus on system design and its impact on processes and outcomes. This model describes the structure of a health care organization as a work system with five components (person, tasks, tools and technologies, physical environment, organizational conditions) who interact with each other and affect both work (e.g. maintenance and supply chain management) and clinical care processes. Both processes in turn influence the patient, employee, and organizational outcomes of care [30]. Capturing more detailed predictor variables about the technology, users, and the surrounding context might have increased the ability to interpret our findings of process variables (e.g. organization – professional processes such as culture and workflow [28]) during the evaluation of our intervention.
Future work
Although A&F interventions are increasingly used to aid health care professionals in improving their performance, they might not qualify as the best basis for improving concordance of prescribed therapies with guideline recommendations. Our future work will include results on the intervention’s effect on concordance of the received CR therapies with guideline recommendations, as well as results on team performance (the intervention’s effect on all quality indicators); and results of a qualitative process evaluation. During this evaluation we use the concept mapping methodology (including focus group sessions) to explore experiences from participating CR clinics with the intervention to gain insight into barriers and facilitators of the implementation.
Conclusion
A web-based A&F intervention with outreach visits did not increase concordance of prescribed CR therapies with guideline recommendations in a pragmatic evaluation using EPRs for data collection. There appeared to be a high variation in baseline performance and in data quality among participating CR clinics. Although QI teams in the clinics formulated QI goals and associated actions at the start of each quarterly A&F iteration, most goals were not attained. We recommend to align data registration in participating clinics before starting an A&F intervention that uses EPRs for data collection. Future analyses should show whether our intervention did improve the overall CR team performance measured by change in quality indicators results, complemented with qualitative information on factors which influenced the implementation of the A&F intervention.
References
- 1.Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;6:CD000259. doi: 10.1002/14651858.CD000259.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Hysong SJ. Meta-analysis: audit and feedback features impact effectiveness on care quality. Medical care. 2009;47(3):356–63. doi: 10.1097/MLR.0b013e3181893f6b. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Gardner B, Whittington C, McAteer J, Eccles MP, Michie S. Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. Social science & medicine (1982) 2010;70(10):1618–25. doi: 10.1016/j.socscimed.2010.01.039. [DOI] [PubMed] [Google Scholar]
- 4.Foy R, MacLennan G, Grimshaw J, Penney G, Campbell M, Grol R. Attributes of clinical recommendations that influence change in practice following audit and feedback. J Clin Epidemiol. 2002;55(7):717–22. doi: 10.1016/s0895-4356(02)00403-1. [DOI] [PubMed] [Google Scholar]
- 5.van der Veer SN, de Keizer NF, Ravelli AC, Tenkink S, Jager KJ. Improving quality of care. A systematic review on how medical registries provide information feedback to health care providers. Int J Med Inform. 2010;79(5):305–23. doi: 10.1016/j.ijmedinf.2010.01.011. [DOI] [PubMed] [Google Scholar]
- 6.Kiefe CI, Allison JJ, Williams OD, Person SD, Weaver MT, Weissman NW. Improving quality improvement using achievable benchmarks for physician feedback: a randomized controlled trial. JAMA. 2001;285(22):2871–9. doi: 10.1001/jama.285.22.2871. [DOI] [PubMed] [Google Scholar]
- 7.van Engen-Verheul MM, de Keizer NF, van der Veer SN, Kemps H, Scholte op Reimer W, Jaspers M, et al. Evaluating the effect of a web-based quality improvement system with feedback and outreach visits on guideline concordance in the field of cardiac rehabilitation: rationale and study protocol. Implementation science: IS. 2014;9(1):780. doi: 10.1186/s13012-014-0131-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.van Engen-Verheul MM, van der Veer SN, de Keizer NF, Tjon Sjoe Sjoe W, van der Zwan EP, Peek N. A Web-based System to Facilitate Local, Systematic Quality Improvement by Multidisciplinary Care Teams: Development and First Experiences of CARDSS Online. Stud Health Technol Inform. 2013;192:248–52. 248–52. [PubMed] [Google Scholar]
- 9.Goud R, Hasman A, Peek N. Development of a guideline-based decision support system with explanation facilities for outpatient therapy. Comput Methods Programs Biomed. 2008;91(2):145–53. doi: 10.1016/j.cmpb.2008.03.006. [DOI] [PubMed] [Google Scholar]
- 10.Goud R, de Keizer NF, ter Riet G, Wyatt JC, Hasman A, Hellemans IM, et al. Effect of guideline based computerised decision support on decision making of multidisciplinary teams: cluster randomised trial in cardiac rehabilitation. BMJ. 2009;338:b1440. doi: 10.1136/bmj.b1440. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Goud R, van Engen-Verheul MM, de Keizer NF, Bal R, Hasman A, Hellemans IM, et al. The effect of computerized decision support on barriers to guideline implementation: a qualitative study in outpatient cardiac rehabilitation. Int J Med Inform. 2010;79(6):430–7. doi: 10.1016/j.ijmedinf.2010.03.001. [DOI] [PubMed] [Google Scholar]
- 12.Piepoli MF, Corra U, Adamopoulos S, Benzer W, Bjarnason-Wehrens B, Cupples M, et al. Secondary prevention in the clinical management of patients with cardiovascular diseases. Core components, standards and outcome measures for referral and delivery: A Policy Statement from the Cardiac Rehabilitation Section of the European Association for Cardiovascular Prevention & Rehabilitation. Endorsed by the Committee for Practice Guidelines of the European Society of Cardiology. European journal of preventive cardiology. 2012;21(6):664–81. doi: 10.1177/2047487312449597. [DOI] [PubMed] [Google Scholar]
- 13.Thomas RJ, King M, Lui K, Oldridge N, Pina IL, Spertus J, et al. AACVPR/ACC/AHA 2007 performance measures on cardiac rehabilitation for referral to and delivery of cardiac rehabilitation/secondary prevention services endorsed by the American College of Chest Physicians, American College of Sports Medicine, American Physical Therapy Association, Canadian Association of Cardiac Rehabilitation, European Association for Cardiovascular Prevention and Rehabilitation, Inter-American Heart Foundation, National Association of Clinical Nurse Specialists, Preventive Cardiovascular Nurses Association, and the Society of Thoracic Surgeons. J Am Coll Cardiol. 2007;50(14):1400–33. doi: 10.1016/j.jacc.2007.04.033. [DOI] [PubMed] [Google Scholar]
- 14.Leon AS, Franklin BA, Costa F, Balady GJ, Berra KA, Stewart KJ, et al. Cardiac Rehabilitation and Secondary Prevention of Coronary Heart Disease: An American Heart Association Scientific Statement From the Council on Clinical Cardiology (Subcommittee on Exercise, Cardiac Rehabilitation, and Prevention) and the Council on Nutrition, Physical Activity, and Metabolism (Subcommittee on Physical Activity), in Collaboration With the American Association of Cardiovascular and Pulmonary Rehabilitation. Circulation. 2005;111(3):369–76. doi: 10.1161/01.CIR.0000151788.08740.5C. [DOI] [PubMed] [Google Scholar]
- 15.Muller-Riemenschneider F, Meinhard C, Damm K, Vauth C, Bockelbrink A, Greiner W, et al. Effectiveness of nonpharmacological secondary prevention of coronary heart disease. Eur J Cardiovasc Prev Rehabil. 2010;17(6):688–700. doi: 10.1097/HJR.0b013e32833a1c95. [DOI] [PubMed] [Google Scholar]
- 16.Bjarnason-Wehrens B, McGee H, Zwisler AD, Piepoli MF, Benzer W, Schmid JP, et al. Cardiac rehabilitation in Europe: results from the European Cardiac Rehabilitation Inventory Survey. Eur J Cardiovasc Prev Rehabil. 2010 doi: 10.1097/HJR.0b013e328334f42d. [DOI] [PubMed] [Google Scholar]
- 17.van Engen-Verheul MM, de Vries H, Kemps H, Kraaijenhagen R, de Keizer N, Peek N. Cardiac rehabilitation uptake and its determinants in the Netherlands. European journal of preventive cardiology. 2013;20(2):349–56. doi: 10.1177/2047487312439497. [DOI] [PubMed] [Google Scholar]
- 18.Rehabilitation Committee: Netherlands Society for Cardiology (NVVC) and Netherlands Heart Foundation (NHS) (both Guidelines on Cardiac Rehabilitation 2004) and Working Group PAAHR (partial revision 2011) Multidisciplinary Guidelines for Cardiac Rehabilitation (in Dutch) Utrecht: Netherlands Society for Cardiology (NVVC); 2011. Available at http://www.nvvc.nl/hr. Last accessed June 2015. [Google Scholar]
- 19.van Engen-Verheul MM, de Rijk AE, Peek N. Dutch Clinical algorithm for assessment of patient needs in cardiac rehabilitation and secondary prevention (in Dutch) Utrecht: Netherlands Society for Cardiology (NVVC); 2012. May 22, 2012. [Google Scholar]
- 20.van Engen-Verheul MM, Peute LWP, Kilsdonk E, Peek N, Jaspers MWM. Usability evaluation of a guideline implementation system for cardiac rehabilitation: think aloud study. Stud Health Technol Inform. 2012;180:403–7. [PubMed] [Google Scholar]
- 21.Reulings PG, van der Lans S. Cardiac rehabilitation with lifestyle counselling after myocardial infarction: it helps, but not everyone undergoes it. Ned Tijdschr Geneeskd. 2012;156(49):A5758. Article in Dutch. [PubMed] [Google Scholar]
- 22.van Engen-Verheul MM, Kemps H, Kraaijenhagen R, de Keizer N, Peek N. Modified Rand method to derive quality indicators: a case study in cardiac rehabilitation. Stud Health Technol Inform. 2011;169:88–92. [PubMed] [Google Scholar]
- 23.Pinheiro JC, Bates DM. Mixed Effects Models in S and S-plus. New-York: Springer; 2000. p. 2000. [Google Scholar]
- 24.Taylor MJ, McNicholas C, Nicolay C, Darzi A, Bell D, Reed JE. Systematic review of the application of the plan-do-study-act method to improve quality in healthcare. BMJ quality & safety. 2014;23(4):290–8. doi: 10.1136/bmjqs-2013-001862. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Ivers NM, Tu K, Young J, Francis JJ, Barnsley J, Shah BR, et al. Feedback GAP: pragmatic, cluster-randomized trial of goal setting and action plans to increase the effectiveness of audit and feedback interventions in primary care. Implementation science: IS. 2013;8:142. doi: 10.1186/1748-5908-8-142. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.van der Veer SN, de Vos ML, van der Voort PH, Peek N, Abu-Hanna A, Westert GP, et al. Effect of a multifaceted performance feedback strategy on length of stay compared with benchmark reports alone: a cluster randomized trial in intensive care*. Critical care medicine. 2013;41(8):1893–904. doi: 10.1097/CCM.0b013e31828a31ee. [DOI] [PubMed] [Google Scholar]
- 27.Ivers N, Barnsley J, Upshur R, Tu K, Shah B, Grimshaw J, et al. “My approach to this job is…one person at a time”: Perceived discordance between population-level quality targets and patient-centred care. Canadian family physician Medecin de famille canadien. 2014;60(3):258–66. [PMC free article] [PubMed] [Google Scholar]
- 28.Ancker JS, Kern LM, Abramson E, Kaushal R. The Triangle Model for evaluating the effect of health information technology on healthcare quality and safety. Journal of the American Medical Informatics Association. JAMIA. 2012;19(1):61–5. doi: 10.1136/amiajnl-2011-000385. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.King HB, Battles J, Baker DP, Alonso A, Salas E, Webster J, et al. Advances in Patient Safety TeamSTEPPS: Team Strategies and Tools to Enhance Performance and Patient Safety. In: Henriksen K, Battles JB, Keyes MA, Grady ML, editors. Advances in Patient Safety: New Directions and Alternative Approaches. Vol. 3. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008. (Performance and Tools). [PubMed] [Google Scholar]
- 30.Carayon P, Schoofs Hundt A, Karsh BT, Gurses AP, Alvarado CJ, Smith M, et al. Work system design for patient safety: the SEIPS model. Quality & safety in health care. 2006;15(Suppl 1):i50–8. doi: 10.1136/qshc.2005.015842. [DOI] [PMC free article] [PubMed] [Google Scholar]