Abstract
Background:
In Ontario, a province-wide quality-improvement program (Quality Improvement and Innovation Partnership [QIIP]) was implemented between 2008 and 2010 to support improved outcomes in Family Health Teams, a care model that includes many features of the patient-centred medical home. We assessed the impact of this program on diabetes management, colorectal and cervical cancer screening and access to health care.
Methods:
We used comprehensive linked administrative data sets to conduct a population-based controlled before-and-after study. Outcome measures included diabetes process-of-care measures (test ordering, retinal examination, medication prescribing and completion of billing items specific to diabetes management), colorectal and cervical cancer screening measures and use of health care services (emergency department visits, hospital admission for ambulatory-care-sensitive conditions and rates of readmission to hospital). The control group consisted of Family Health Team physicians with at least 100 assigned patients during the study follow-up period (November 2009-February 2013).
Results:
There were 53 physicians in the intervention group and 1178 physicians in the control group. Diabetes process-of-care measures improved more in the intervention group than in the control group: hemoglobin A1c testing 4.3% (95% confidence interval [CI] 1.2-7.5) more, retinal examination 2.5% (95% CI 0.8-4.4) more and preventive care visits 8.9% (95% CI 2.9-14.9) more. Medication prescribing also improved for use of statins (3.4% [95% CI 0.8-6.0] more) and angiotensin-converting-enzyme inhibitors or angiotensin receptor blockers (4.1% [95% CI 1.8-6.4] more). Colorectal cancer screening improved 5.4% (95% CI 3.1-7.8) more in the intervention group than in the control group, and cervical cancer screening improved 2.7% (95% CI 0.9-4.6) more. There were no significant differences in any of the measures of use of health care services.
Interpretation:
This large controlled evaluation of a broadly implemented quality-improvement initiative showed improvement for diabetes process of care and cancer screening outcomes, but not for proxy measures of access related to use of health care services.
Primary health care plays a key role in health care systems in Canada and around the world.1,2 Studies consistently show that the vast majority of care is delivered in primary care settings3-7 and that strong primary care systems are associated with improved outcomes and decreased health care costs.5,8,9 In Ontario, primary care is the backbone of the publicly funded health care system, accounting for about 80% of all visits annually.7,10 Over the past decade, improving and strengthening primary health care have been key priorities of successive governments in Ontario, which have implemented a series of reforms and initiatives in this key sector.10 These include changes to payment models for physicians, formal enrolment of patients with primary care providers, support for multidisciplinary teams, support for the adoption of electronic health records and province-wide quality-improvement initiatives.8,11,12 Family Health Teams represent the most highly reformed model of primary health care in Ontario and include all of the elements described above.13 There is also consistent evidence that there is room for improvement in the quality of delivery of primary health care in Canada and in Ontario. The much-cited 2010 Commonwealth Fund International Health Policy Survey shows that Canada's performance, last or next to last on many measures, leaves much to be desired.14 Recent studies comparing aspects of the new payment and organizational models in Ontario have shown modest improvements or no differences between models but have shown improvement in quality of care over time.15-20
To help maximize the impact of the new models, a provincial quality-improvement initiative, the Quality Improvement and Innovation Partnership (QIIP) (subsequently incorporated into Health Quality Ontario), was created to assist Family Health Teams in 3 targeted areas - diabetes management, colorectal cancer screening and access to care - through a learning collaborative program based on the Institute for Healthcare Improvement model.21-23. Quality-improvement teams in participating practices received training on quality-improvement methods and the chronic care model as described by Wagner and colleagues24 and others10 as well as professional practice facilitation and participated in learning sessions with other teams. Tricco and colleagues25 conducted a systematic review and meta-analysis of any approach to quality improvement for diabetes management. Within their selected studies, only 4 cluster randomized trials were classified as "continuous quality improvement," which was the closest match for the learning collaborative model in their taxonomy of interventions. They did not find any significant improvement in any process of care measures for this approach. Schouten and colleagues26 conducted a systematic review of quality-improvement collaboratives and concluded that, although there is evidence that this approach can be effective, further evaluation, particularly with controlled study designs, is warranted, as outcome improvements in prior studies have been modest and not always consistent or predictable. Of the controlled studies included in those authors' review, only the single-clinic study on diabetes by Benedetti and colleagues27 reported on any of the outcomes targeted by the QIIP program. The QIIP program and the overall results of a comprehensive mixed-methods evaluation are described in detail elsewhere.23,28 The mixed-methods evaluation did not show statistically significant differences in the primary outcomes, but it was not powered to detect modest effect sizes. We conducted a supplementary population-level controlled before-and-after evaluation to assess the impact of the QIIP learning collaborative program on outcomes that could be assessed with the use of administrative data.
Methods
Participants and setting
We used a controlled before-and-after-study design. All patients of physicians included in the study during the study period were included in the study population. Patients were assigned to physicians if they had formally enrolled as a patient with the physician or if they could be virtually assigned by means of an established method of allocating patients to the physician who delivered the majority of a basket of primary care services.19,20,29 All Family Health Team physicians in Ontario with at least 100 assigned patients during the study follow-up period (November 2009-February 2013) were included in the study. We asked physicians who had participated in the QIIP learning collaborative program to provide their registration number and to consent to the use of this number to identify their data within the health administrative data. Physicians who agreed and who met the inclusion criteria constituted the intervention group. Privacy limitations meant that we were unable to determine which physicians who had participated in the QIIP program did not consent, so all other eligible physicians made up the control group. Therefore, the control group included physicians who had participated in the QIIP program but did not consent to take part in the current evaluation.
QIIP learning collaborative program (intervention)
The QIIP learning collaborative program was available to all Family Health Teams in Ontario on a voluntary basis. The learning collaboratives used the Institute for Healthcare Improvement's Breakthrough Series learning model21 and included three 2-day learning sessions, action periods and a summative congress. The learning sessions enabled teams to learn quality-improvement methods and concepts from experts, and also provided a rich opportunity to network and learn from each other. Teams were educated in the use of the Ontario Chronic Disease Prevention and Management Framework's plan-do-study-act methodology and clinical practice guidelines.10 The program provided each team the support of a dedicated quality-improvement coach, whose role was to facilitate and mentor participants throughout the program. Participating practices were required to form a core interdisciplinary quality-improvement team that would be representative of the interdisciplinary composition of their organization. Participating teams included family physicians and allied health care clinicians (e.g., pharmacists, social workers, nurses) as well as senior administrators and front-line office staff. Practices could focus on any or all of the 3 priority areas of diabetes management, colorectal cancer screening and access to care. The program was offered in 3 waves between April 2008 and October 2010, with each wave occurring over about 15 months.
Design and data sources
Administrative data sets were linked with the use of unique, encoded identifiers and were analyzed at the Institute for Clinical Evaluative Sciences to measure the impact of the quality-improvement program on diabetes management, screening for colorectal cancer and cervical cancer, and access to care (emergency department visits, hospital admission for ambulatory-care-sensitive conditions and rates of readmission to hospital). We included cervical cancer screening to see whether participation in the program resulted in process changes that affected cancer screening in general, not just the targeted condition (spread); we considered both 2-year and 3-year screening intervals on account of changes in practice guidelines over the study period. We also assessed continuity of care with the primary care physician. The data sets included payments to physicians from the Ontario Health Insurance Plan, the hospital Discharge Abstracts Database, the National Ambulatory Care Reporting System (for emergency department visits), laboratory and diagnostic imaging ordering data, the Ontario Cancer Registry, the Ontario Drug Benefit database, physician workforce data, data on patient enrolment in primary care models, the Ontario Registered Persons Database, the census and vital statistics. We randomly assigned control physicians index dates corresponding to the 3 waves between 2008 and 2010. We assessed physician and practice demographic characteristics including case-mix, health status of the practice population, patterns of use of health care services, and measures for chronic disease prevention and management. Data definitions and sources for each of the measures used in this paper are in alignment with the Primary Care in Ontario: ICES Atlas.29
Statistical analysis
We compared results 12-24 months before the program to results 12-24 months after the program using generalized linear regression with adjustment for baseline value, patient demographic characteristics (age, sex, rurality) and case-mix (using the Johns Hopkins Adjusted Diagnosis Groups30). All analysis was conducted with the use of SAS version 13.1. We conducted the analysis at the level of the physician practice as that is where the intervention was targeted. A p value less than 0.05 was considered statistically significant.
Ethics approval
This study was approved by the research ethics boards at Sunnybrook Hospital (Institute for Clinical Evaluative Sciences Central), Western University and Queen's University.
Results
We approached 118 physicians who had participated in the QIIP program. Of the 118, 45 did not respond and 10 refused, for a recruitment rate of 53.4% (63/118). Ten physicians were subsequently found to be ineligible (4 had a panel size less than 100 patients, 4 did not have rostered patients, and 2 did not have a Family Health Team designation) and were removed from the study.
A total of 1178 control physicians with at least 100 patients were identified and randomly assigned to one of the 3 index dates. Demographic and practice characteristics of the QIIP and control physicians are presented in Table 1. QIIP physicians were slightly more likely to be male, Canadian trained and from rural areas than the control physicians, but only the mean Rurality Index of Ontario score was significantly different between the 2 groups (p = 0.006).
Table 1: Demographic and practice characteristics of physicians in the QIIP and control groups.
| Characteristic | QIIP group n = 53 |
Control group n = 1178 |
p value |
|---|---|---|---|
| Male sex, no. (%) | 34 (64.2) | 699 (59.3) | 0.5 |
| Canadian graduate, no. (%) | 49 (92.4) | 1035 (87.9) | 0.3 |
| Rurality Index of Ontario category, no. (%) | 0.1 | ||
| Major urban (1-9) | 26 (49.0) | 644 (54.7) | |
| Suburban (10-39) | 12 (22.6) | 330 (28.0) | |
| Rural (≥ 40) | 15 (28.3) | 190 (16.1) | |
| Missing | 0 (0.0) | 14 (1.2) | |
| Rurality Index of Ontario score, mean ± SD | 27.00 ± 30.86 | 17.95 ± 23.12 | 0.006 |
| Age, mean ± SD | 49.85 ± 8.40 | 48.27 ± 9.47 | 0.2 |
| Years since graduation, mean ± SD | 24.9 ± 9.12 | 22.5 ± 9.87 | 0.2 |
| No. of patients, mean ± SD | 1475 ± 620 | 1410 ± 694 | 0.5 |
Note: QIIP = Quality Improvement and Innovation Partnership, SD = standard deviation..
There were no clinically relevant differences in demographic and clinical characteristics between the patients of the 2 groups of physicians other than a higher proportion residing in a rural location in the QIIP group than in the control group (18.1% v. 15.1%) (Table 2). No statistical analysis is presented because, owing to the large sample, all reported differences were statistically significant.
Table 2: Demographic and clinical characteristics of patients of physicians in the QIIP and control groups.
| Characteristic | No. (%) of patients* | |
|---|---|---|
| QIIP group n = 78 192 |
Control group n =1 661 152 |
|
| Male sex | 37 061 (47.4) | 773 869 (46.6) |
| Health card registration within 10 yr of baseline | 2431 (3.1) | 70 507 (4.2) |
| Age, yr | ||
| ≤ 4 | 3797 (4.8) | 87 171 (5.2) |
| 5-9 | 4242 (5.4) | 90 263 (5.4) |
| 10-18 | 9386 (12.0) | 194 222 (11.7) |
| 19-34 | 13 713 (17.5) | 314 618 (18.9) |
| 35-49 | 17 421 (22.3) | 374 856 (22.6) |
| 50-64 | 17 062 (21.8) | 336 481 (20.2) |
| 65-74 | 6653 (8.5) | 136 228 (8.2) |
| 75-84 | 4421 (5.6) | 93 977 (5.6) |
| ≥ 85 | 1497 (1.9) | 33 336 (2.0) |
| Rurality Index of Ontario category | ||
| Major urban (1-9) | 38 688 (49.5) | 883 085 (53.2) |
| Suburban (10-39) | 22 644 (29.0) | 502 829 (30.3) |
| Rural (≥ 40) | 14 148 (18.1) | 250 538 (15.1) |
| Missing | 2712 ( .5) | 24 700 (1.5) |
| Income quintile | ||
| 1 (low) | 13 171 (16.8) | 288 115 (17.3) |
| 2 | 14 218 (18.2) | 318 724 (19.2) |
| 3 | 14 925 (19.1) | 330 089 (19.9) |
| 4 | 17 296 (22.1) | 358 818 (21.6) |
| 5 (high) | 18 062 (23.1) | 351 419 (21.2) |
| Missing | 520 (0.7) | 13 987 (0.8) |
| Diagnosis | ||
| Diabetes | 6225 (8.0) | 129 523 (7.8) |
| Previous acute myocardial infarction | 1090 (1.4) | 23 082 (1.4) |
| Asthma | 10 539 (13.5) | 223 053 (13.4) |
| Chronic heart failure | 1602 (2.0) | 32 455 (2.0) |
| Chronic obstructive pulmonary disorder | 5462 (7.0) | 105 023 (6.3) |
| Hypertension | 16 859 (21.6) | 352 751 (21.2) |
| Mental health disorder | 15 295 (19.6) | 316 221 (19.0) |
| Adjusted Diagnosis Group | ||
| 0 | 4538 (5.8) | 94 961 (5.7) |
| 1-5 | 41 814 (53.5) | 871 048 (52.4) |
| 6-9 | 26 956 (34.5) | 587 558 (35.4) |
| ≥ 10 | 4884 (6.2) | 107 585 (6.5) |
| Resource Utilization Band, mean ± SD | 2.62 ± 1.10 | 2.63 ± 1.10 |
Note: QIIP = Quality Improvement and Innovation Partnership, SD = standard deviation.
*Except where noted otherwise.
Table 3 presents the results for diabetes process-of-care measures including test ordering, completion of billing items specific to diabetes management and medications prescribed for eligible patients. Process-of-care measures improved more in the intervention group than the control group: hemoglobin A1c testing improved 4.3% (95% confidence interval [CI] 1.2-7.5) more, and retinal examination improved 2.5% (95% CI 0.8-4.4) more. The differential increase in the completion of lipid testing, 1.3%, was not significant. Billing items specific to diabetes management also improved more in the intervention group than in the control group: flow sheet completion improved 8.8% (95% CI 4.1-13.5) more, and preventive care visits improved 8.9% (95% CI 2.9-14.9) more. Differential increases in medication prescribing were noted for statins (3.4% [95% CI 0.8-6.0]) and angiotensin-converting-enzyme inhibitors/angiotensin receptor blockers (4.1% [95% CI 1.8-6.4]). There were no significant differences in the use of orally administered hypoglycemic agents or insulin.
Table 3: Rates of completion of diabetes management measures by physicians in the QIIP and control groups.
| Measure | Time; % of physicians | Adjusted % change, QIIP group v. control group (95% CI)* | ||
|---|---|---|---|---|
| 12 mo before intervention or index date | 12 mo after intervention or index date | |||
| Hemoglobin A1c test: ≥ 2 in previous 12 mo | QIIP group | 41.1 | 51.4 | 4.3 (1.2 to 7.5) |
| Control group | 42.7 | 48.0 | ||
| Retinal examination: ≥ 1 in previous 24 mo | QIIP group | 72.5 | 76.6 | 2.5 (0.8 to 4.4) |
| Control group | 71.6 | 73.3 | ||
| Low-density lipoprotein cholesterol test: ≥ 1 in previous 12 mo | QIIP group | 56.9 | 64.0 | 1.3 (-2.1 to 4.6) |
| Control group | 59.5 | 64.7 | ||
| Billing for diabetes flow sheet (K030): ≥ 1 in previous yr | QIIP group | 27.6 | 42.8 | 8.8 (4.1 to 13.5) |
| Control group | 34.4 | 39.0 | ||
| Billing for preventive care of diabetes (Q040) | QIIP group | 21.7 | 39.1 | 8.9 (2.9 to 14.9) |
| Control group | 28.8 | 35.0 | ||
| Diabetes medication management for patients with type 2 diabetes > 65 yr | ||||
| Prescribed statin | QIIP group | 66.5 | 74.5 | 3.4 (0.8 to 6.0) |
| Control group | 67.6 | 71.9 | ||
| Prescribed angiotensin-converting-enzyme inhibitor or angiotensin receptor blocker | QIIP group | 74.1 | 78.4 | 4.1 (1.8 to 6.4) |
| Control group | 76.0 | 75.0 | ||
| Prescribed orally administered hypoglycemic agent | QIIP group | 59.6 | 59.2 | 0.8 (-1.8 to 3.3) |
| Control group | 58.7 | 57.6 | ||
| Prescribed insulin | QIIP group | 17.1 | 18.5 | -0.3 (-2.3 to 1.7) |
| Control group | 15.9 | 17.9 | ||
Note: CI = confidence interval, QIIP = Quality Improvement and Innovation Partnership.
*Generalized linear regression adjusted for baseline value, sex, rurality, age and comorbidity (Johns Hopkins Adjusted Diagnosis Groups30).
Screening measures for colorectal cancer increased more in the intervention group than in the control group: fecal occult blood testing improved 8.5% (95% CI 5.1-12.0) more, and any screening improved 5.4% (95% CI 3.1-7.8) more (Table 4). There was a small but statistically significant increase in cervical cancer screening in the intervention group, for both 2- and 3-year intervals (percent adjusted change 2.3% [95% CI 0.5-4.1] and 2.7% [95% CI 0.9-4.6), respectively).
Table 4: Rates of completion of screening measures for colorectal cancer and cervical cancer by physicians in the QIIP and control groups.
| Measure | Time; % of physicians | Adjusted change, QIIP group v. control group (95% CI)* | ||
|---|---|---|---|---|
| 12 mo before intervention or index date | 12 mo after intervention or index date | |||
| Colorectal cancer | ||||
| Fecal occult blood testing within previous 2 yr | QIIP group | 41.1 | 52.2 | 8.5 (5.1 to 12.0) |
| Control group | 39.5 | 42.8 | ||
| Colonoscopy within previous 5 yr | QIIP group | 24.5 | 29.7 | 0.02 (-1.7 to 1.8) |
| Control group | 26.6 | 31.7 | ||
| Flexible sigmoidoscopy/barium enema | QIIP group | 5.8 | 3.8 | -0.03 (-0.1 to 0.3) |
| Control group | 6.3 | 4.4 | ||
| Any screening | QIIP group | 57.2 | 67.1 | 5.4 (3.1 to 7.8) |
| Control group | 57.6 | 62.4 | ||
| Cervical cancer | ||||
| Papanicolaou test within previous 2 yr | QIIP group | 61.7 | 63.5 | 2.3 (0.5 to 4.1) |
| Control group | 62.3 | 61.4 | ||
| Papanicolaou test within previous 3 yr | QIIP group | 72.2 | 75.1 | 2.7 (0.9 to 4.6) |
| Control group | 72.9 | 72.7 | ||
Note: CI = confidence interval, QIIP = Quality Improvement and Innovation Partnership.
*General linear regression for baseline value, sex, rurality, age and comorbidity (Johns Hopkins Adjusted Diagnosis Groups30).
Table 5 presents the results for access to care measures. There were no significant changes in emergency department visits, hospital admission for ambulatory-care-sensitive conditions or hospital readmission, with the exception of admission for chronic obstructive pulmonary disease, for which there was a small but statistically significant increase in the intervention group (percent adjusted change 6.0% [95% CI 1.0-11.0]). Continuity of care with the usual provider was similar at baseline for the intervention and control groups (72.2% and 72.0%, respectively), and there were no significant changes over the study period (increase of 0.7% v. decrease of 0.3%) (p = 0.5) (data not shown).
Table 5: Rates of access to care (emergency department visits, hospital admission and hospital readmission) for physicians in the QIIP and control groups.
| Measure | Time; rate | Adjusted change, QIIP group v. control group (95% CI) | ||
|---|---|---|---|---|
| 12 mo before intervention or index date | 12 mo after intervention or index date | |||
| Emergency department visits per 100 patients per yr | ||||
| Canadian Triage and Acuity Scale score 1-3 | QIIP group | 20.8 | 24.7 | 0.1 (-1.3 to 1.5) |
| Control group | 22.0 | 24.9 | ||
| Canadian Triage and Acuity Scale score 4-5 (low acuity) | QIIP group | 36.7 | 29.3 | -1.0 (-3.3 to 1.2) |
| Control group | 28.8 | 24.6 | ||
| Hospital admission for ambulatory-care-sensitive condition per 10 000 patients per yr | ||||
| Overall | QIIP group | 52.8 | 51.5 | 3.0 (-5.0 to 11.0) |
| Control group | 42.7 | 42.1 | ||
| Diabetes | QIIP group | 8.0 | 8.4 | 0 (-3.0 to 3.0) |
| Control group | 7.5 | 7.2 | ||
| Asthma | QIIP group | 5.0 | 2.9 | -1.0 (-2.0 to 1.0) |
| Control group | 4.1 | 3.3 | ||
| Chronic obstructive pulmonary disease | QIIP group | 25.9 | 28.1 | 6.0 (1.0 to 11.0) |
| Control group | 18.5 | 18.3 | ||
| Cardiac heart failure | QIIP group | 13.9 | 12.1 | -3.0 (-6.0 to 1.0) |
| Control group | 12.7 | 13.2 | ||
| Hospital readmission (% of those admitted to hospital) | ||||
| Within 30 d | QIIP group | 5.5 | 5.5 | -0.03 (-0.08 to 0.08) |
| Control group | 5.1 | 5.3 | ||
| Within 1 yr | QIIP group | 17.0 | 17.4 | 0.2 (-1.1 to 1.6) |
| Control group | 15.8 | 15.9 | ||
Note: CI = confidence interval, QIIP = Quality Improvement and Innovation Partnership.
*Generalized linear regression adjusted for baseline value, sex, rurality, age and comorbidity (Johns Hopkins Adjusted Diagnosis Groups30).
Interpretation
For most of the measures that were targeted by the QIIP program, outcomes were improved compared with the control group, with rates of change about double those in the control group. These differences were statistically significant owing to the relatively large samples that are possible with population-based analyses but were in general modest in magnitude (absolute differences 2%-11%). However, even small changes can be important if they are broadly applied to the population, so findings such as ours have the potential to provide system- or population-level benefits despite their modest effect size.
The observed changes in diabetes process-of-care measures are consistent with those noted in other studies. Benedetti and colleagues27 found improvements in both process measures (test ordering, documentation of self-management plans) in diabetes-focused learning collaboratives in Washington State. Valk and colleagues31 compared programs to improve the quality of diabetes care in the Netherlands and the United States and also found both process and outcome (hemoglobin A1c testing) improvements in both groups. The magnitude of changes (crude rate) observed in our intervention group after participation in the QIIP program is similar to that noted in an uncontrolled before-and-after evaluation of a quality-improvement initiative (Partnerships for Health) that also used the Institute for Healthcare Improvement model (9% increase in annual hemoglobin A1c testing and 9% increase in testing for low-density lipoprotein cholesterol).32 Our adjusted analysis, which took into account temporal changes in the control group, showed more modest increases, which highlights the importance of including a control group.
In contrast to the Supporting Colorectal Cancer Outcomes through Participatory Enhancements trial,33 which showed no significant improvement in colorectal cancer screening after an intervention that combined learning collaboratives and on-site facilitation, in this population-based study we found an improvement in screening comparable to that with other interventions, such as audit and feedback, noted in the Colorectal Cancer Screening in Primary Care Practice study34 and Building on Existing Tools to Improve Chronic Disease Prevention and Screening in Family Practice study.35 Our results also contrast with those of our chart audit, which failed to show a significant difference in screening rates in a small subsample of patients.28 This highlights one of the advantages of using population-level data to achieve the power required to detect changes in situations in which event frequency or effect sizes mean that resource-intensive audits of patient records are not feasible. Analysis of rates of cervical cancer screening showed a statistically significant increase in the intervention group, but the magnitude of the difference between the intervention and control groups was smaller than that for colorectal cancer screening. This could be an indication of spread if, for example, recall processes for cancer screening were introduced that covered more than just colorectal cancer.
We found no clinically relevant changes in measures of use of health care services such as low-acuity emergency department visits and hospital admission for ambulatory-care-sensitive conditions that are commonly used as outcome measures or proxy measures for access to primary health care. This is consistent with the limited number of other studies that have explored the relation between advanced access scheduling (the focus of the QIIP program) and use of health care services. In a 2011 systematic review of advanced access scheduling, Rose and colleagues36 found that, although most studies showed improvements in time to third-next available appointment and reduced no-show rates, the effects on patient satisfaction were mixed, and the data on use were limited to very few studies. Those authors identified only 2 studies that reported on use of health care services, and only 1 of these was controlled. Solberg and colleagues37 reported on an uncontrolled before-and-after study of advanced access for patients with chronic conditions and found that, despite significant improvements in access and improved continuity of care, there was very limited change in use of health care services, including emergency department visits and hospital admissions, or overall costs. Finally, in a controlled study of primary care clinics in Indiana that transitioned or did not transition to advanced access, no significant differences were found in emergency department and urgent care visits, hospital admissions or total outpatient visits.38
Limitations
First, by using administrative data, we were limited to collected process measures and measures of use of health care services. Second, as we were not permitted access to the list of all participating physicians because of privacy restrictions, physicians who participated in the QIIP program but who did not complete consent to take part in the current evaluation were included in the control group. Although this may have introduced a bias toward a null result, nonparticipants represented only 5% of the control group, which makes bias toward a null result unlikely. Third, this was a program implementation rather than a trial of an intervention, so the degree to which recommended processes were implemented and the way in which they were implemented likely varied significantly across the participating teams. This reduced implementation fidelity, would generally result in a bias toward a null result. In addition, the implementation took place during a time of reform, with other changes and initiatives being implemented concurrently. In particular, province-wide programs to improve the quality of diabetes care and cancer screening were being conducted that targeted all primary care providers, including physicians in Family Health Teams. To mitigate against the risk of bias toward a null result we applied a controlled before-and-after design and also limited patients in the control group to Family Health Team patients, as the concurrent changes would likely have been similar in the control and intervention practices. Fourth, as this was a voluntary program, it is possible that participating physicians were more motivated to improve their practices than nonparticipating physicians. A final caveat is that this evaluation was based on a relatively limited follow-up period. Repeating this analysis later in time to assess for the sustainability of the changes would provide important additional information about the role of the learning collaborative approach in improving quality of primary care.
Conclusion
Our overall results were positive for diabetes management and for colorectal and cervical cancer screening, with both clinically and statistically significant improvements in multiple outcomes noted. These findings support the use of the Institute for Healthcare Improvement's learning collaborative strategy for quality improvement in multidisciplinary team practices such as Family Health Teams. There were no improvements in measures of use of health care services that were used as a proxy for improved primary care access. This is likely because there are only weak links between actual improved access and these outcomes. Direct assessment of access to primary health care as well as patient experience with access would be better measures of success in this domain. Despite the relatively modest absolute levels of improvement noted, the success in achieving improvements of this scale for large populations is important. In our setting, the learning collaborative approach seemed to be beneficial for outcomes that were more directly under the control of health care providers when applied at large scale to a broad range of primary care practices. It seemed less effective in changing outcomes such as medication use or use of health care services that are dependent on patient or system factors outside the control of the practice.
Supplemental information
For reviewer comments and the original submission of this manuscript, please see www.cmajopen.ca/content/5/2/E281/suppl/DC1
Supplementary Material
Acknowledgements
This study was supported by the Quality Improvement and Innovation Partnership (QIIP), Health Quality Ontario and a Canadian Institutes of Health Research Knowledge Dissemination grant. Michael Green is supported by the Clinical Teachers' Association of Queen's University Chair in Applied Health Policy/Health Economics. Stewart Harris is supported by the Canadian Diabetes Association Chair in Diabetes Management and the Ian McWhinney Chair of Family Medicine Studies, Western University. Richard Glazier is supported as a Clinician Scientist in the Department of Family and Community Medicine at the University of Toronto and at St. Michael's Hospital. No endorsement by Health Quality Ontario is intended or should be inferred. Parts of this material are based on based on data and information compiled and provided by the Canadian Institutes of Health Information (CIHI) or Cancer Care Ontario (CCO). The analysis, conclusions, opinions and statements reported in this paper are those of the authors and do not necessarily reflect those of the CIHI or CCO. No endorsement by CCO or the CIHI is intended or inferred.
Footnotes
Competing interests: This work was supported by an award to Stewart Harris, Richard Birtwhistle, Michael Green and Richard Glazier from Health Quality Ontario to conduct the evaluation of the Quality Improvement and Innovation Partnership program. Stewart Harris reports grants and personal fees from Abbott, AstraZeneca, Boehringer Ingelheim/Lilly, Janssen and Novo Nordisk, and personal fees from Merck outside the submitted work. Richard Birtwhistle reports a contract with the Ontario Ministry of Health and Long-Term Care and a grant from the Canadian Institutes of Health Research outside the submitted work. No other competing interests were declared.
Funding: Health Quality Ontario, Canadian Institutes of Health Research.
Prior presentations: Health Services Research Association of Australia and New Zealand biennial conference, Melbourne, Australia, Dec. 7, 2015; Canadian Association for Health Services and Policy Research meeting, Montréal, May 2016.
ICES disclaimer: This study was supported by the Institute for Clinical Evaluative Sciences (ICES), which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). The opinions, results and conclusions reported in this article are those of the authors and are independent from the funding sources. No endorsement by the ICES or the Ontario MOHLTC is intended or should be inferred.
References
- 1.The World health report 2008 - primary health care (now more than ever). Geneva: World Health Organization. 2008. [Google Scholar]
- 2.Schoen C, Osborn R, Huynh PT, et al. On the front lines of care: primary care doctors' office systems, experiences, and views in seven countries. Health Aff (Millwood) 2006;25:w555–71. doi: 10.1377/hlthaff.25.w555. [DOI] [PubMed] [Google Scholar]
- 3.White KL, Williams TF, Greenberg BG. The ecology of medical care. N Engl J Med. 1961;265:885–92. doi: 10.1056/NEJM196111022651805. [DOI] [PubMed] [Google Scholar]
- 4.Green LA, Cifuentes M, Glasgow RE, et al. Redesigning primary care practice to incorporate health behavior change: prescription for health round-2 results. Am J Prev Med. 2008;35(Suppl):S347–9. doi: 10.1016/j.amepre.2008.08.013. [DOI] [PubMed] [Google Scholar]
- 5.Starfield B, Shi L, Macinko J. Contribution of primary care to health systems and health. Milbank Q. 2005;83:457–502. doi: 10.1111/j.1468-0009.2005.00409.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Boucai L, Zonszein J. Effects of quality improvement strategies for type 2 diabetes in Bronx, N.Y. Clin Diabetes. 2007;25:155–9. [Google Scholar]
- 7.McPherson C, Kothari A, Sibbald S.Quality improvement in primary health care in Ontario: an environmental scan and capacity map. Final report to the Quality Improvement in Primary Healthcare Project Planning Group. Toronto: Cancer Quality Council of Ontario 2010Availablewww.cqco.ca/common/pages/UserFile.aspx?fileId=250528accessed 2015
- 8.O'Brien BD, Brown MG, Kephart G. Estimation of hospital costs for colorectal cancer care in Nova Scotia. Can J Gastroenterol. 2001;15:43–7. doi: 10.1155/2001/975208. [DOI] [PubMed] [Google Scholar]
- 9.Friedberg MW, Hussey PS, Schneider EC. Primary care: a critical review of the evidence on quality and costs of health care. Health Aff (Millwood) 2010;29:766–72. doi: 10.1377/hlthaff.2010.0025. [DOI] [PubMed] [Google Scholar]
- 10.Preventing and managing chronic disease: Ontario's framework. Toronto: Ministry of Health and Long-Term Care. 2007. [accessed 2014 Nov. 2]. Available www.health.gov.on.ca/en/pro/programs/cdpm/pdf/framework_full.pdf.
- 11.Hutchison B, Levesque JF, Strumpf E, et al. Primary health care in Canada: systems in motion. Milbank Q. 2011;89:256–88. doi: 10.1111/j.1468-0009.2011.00628.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Kates N, Hutchison B, O'Brien P, et al. Framework for advancing improvement in primary care. Healthc Pap. 2012;12:8–21. doi: 10.12927/hcpap.2012.22978. [DOI] [PubMed] [Google Scholar]
- 13.Rosser WW, Colwill JM, Kasperski J, et al. Progress of Ontario's family health team model: a patient-centred medical home. Ann Fam Med. 2011;9:165–71. doi: 10.1370/afm.1228. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.How engaged are Canadians in their primary care? Results from the 2010 Commonwealth Fund International Health Policy Survey. Canadian Health Care Matters Bulletin 5. Toronto: Health Council of Canada; 2011. [accessed 2017 Mar. 26]. Available www.healthcouncilcanada.ca/tree/2.30-Commonwealth5_EN_Final.pdf.
- 15.Dahrouge S, Hogg W, Russell G, et al. The comparison of models of primary care in Ontario (COMP-PC) study: methodology of a multifaceted cross-sectional practice-based study. Open Med. 2009;3:e149–64. [PMC free article] [PubMed] [Google Scholar]
- 16.Collier R. Verdict still out on family health teams. CMAJ. 2011;183:1131–2. doi: 10.1503/cmaj.109-3864. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Jaakkimainen L, Glazier R, Barnsley J, et al. Waiting to see the specialist: patient and provider characteristics of wait times from primary to specialty care. BMC Fam Pract. 2014;15:16. doi: 10.1186/1471-2296-15-16. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Russell GM, Hogg W, Lemelin J. Integrated primary care organizations: the next step for primary care reform. Can Fam Physician. 2010;56:216–8. e87–9. [PMC free article] [PubMed] [Google Scholar]
- 19.Final report: an external evaluation of the Family Health Team (FHT) initiative. Ottawa: Conference Board of Canada 2014Availablewww.conferenceboard.ca/e-library/abstract.aspx?did=6711accessed 2015
- 20.Kiran T, Kopp A, Moineddin R, et al. Longitudinal evaluation of physician payment reform and team-based care for chronic disease management and prevention. CMAJ. 2015;187:E494–502. doi: 10.1503/cmaj.150579. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.The Breakthrough Series: IHI's collaborative model for achieving breakthrough improvement. Boston: Institute for Healthcare Improvement. 2003. [Google Scholar]
- 22.Jones K, Piterman L. The effectiveness of the breakthrough series methodology. Aust J Prim Health. 2008;14:59–65. [Google Scholar]
- 23.Kotecha J, Brown JB, Han H, et al. Influence of a quality improvement learning collaborative program on team functioning in primary healthcare. Fam Syst Health. 2015;33:222–30. doi: 10.1037/fsh0000107. [DOI] [PubMed] [Google Scholar]
- 24.Wagner EH, Austin BT, Davis C, et al. Improving chronic illness care: translating evidence into action. Health Aff (Millwood) 2001;20:64–78. doi: 10.1377/hlthaff.20.6.64. [DOI] [PubMed] [Google Scholar]
- 25.Tricco AC, Ivers NM, Grimshaw JM, et al. Effectiveness of quality improvement strategies on the management of diabetes: a systematic review and meta-analysis. Lancet. 2012;379:2252–61. doi: 10.1016/S0140-6736(12)60480-2. [DOI] [PubMed] [Google Scholar]
- 26.Schouten LMT, Hulscher MEJL, van Everdingen JE, et al. Evidence for the impact of quality improvement collaborative: systematic review. BMJ. 2008;336:1491. doi: 10.1136/bmj.39570.749884.BE. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Benedetti R, Flock B, Pedersen S, et al. Improved clinical outcomes for fee-for-service physician practices participating in a diabetes care collaborative. Jt Comm J Qual Saf. 2004;30:187–94. doi: 10.1016/s1549-3741(04)30020-1. [DOI] [PubMed] [Google Scholar]
- 28.Harris SB, Green ME, Brown JB, et al. Impact of a quality improvement program on primary healthcare in Canada: a mixed-method evaluation. Health Policy. 2015;119:405–16. doi: 10.1016/j.healthpol.2014.10.019. [DOI] [PubMed] [Google Scholar]
- 29.Jaakkimainen L, Upshur REG, Klein-Geltink JE, et al. editors. Primary care in Ontario: ICES atlas. Toronto: Institute for Clinical Evaluative Sciences 2006Availablewww.ices.on.ca/flip-publication/primary-care-2006/index.htmlaccessed 2015
- 30.The Johns Hopkins ACG SystemAvailable: https://www.hopkinsacg.org/ accessed 2017 Mar. 29
- 31.Valk GD, Renders CM, Kriegsman DMW, et al. Quality of care for patients with type 2 diabetes mellitus in the Netherlands and the United States: a comparison of two quality improvement programs. Health Serv Res. 2004;39:709–25. doi: 10.1111/j.1475-6773.2004.00254.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Harris S, Paquette-Warren J, Roberts S, et al. Results of a mixed-methods evaluation of partnerships for health: a quality improvement initiative for diabetes care. J Am Board Fam Med. 2013;26:711–9. doi: 10.3122/jabfm.2013.06.120211. [DOI] [PubMed] [Google Scholar]
- 33.Shaw EK, Obman-Strickland PA, Piasecki A, et al. Effects of facilitated teammeetings and learning collaboratives on colorectal cancer screening rates in primary care practices: a cluster randomized trial. Ann Fam Med. 2013;11:220–8. S1–8. doi: 10.1370/afm.1505. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Ornstein S, Nemeth LS, Jenkins RG, et al. Colorectal cancer screening in primary care: translating research into practice. Med Care. 2010;48:900–6. doi: 10.1097/MLR.0b013e3181ec5591. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Grunfeld E, Manca D, Moineddin R, et al. BETTER Trial Investigators. Improving chronic disease prevention and screening in primary care: results of the BETTER pragmatic cluster randomized controlled trial. BMC Fam Pract. 2013;14:175. doi: 10.1186/1471-2296-14-175. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Rose KD, Ross JS, Horwitz LI. Advanced access scheduling outcomes: a systematic review. Arch Intern Med. 2011;171:1150–9. doi: 10.1001/archinternmed.2011.168. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Solberg LI, Maciosek MV, Sperl-Hillen JM, et al. Does improved access to care affect utilization and costs for patients with chronic conditions? Am J Manag Care. 2004;10:717–22. [PubMed] [Google Scholar]
- 38.Subramanian U, Ackermann RT, Brizendine EJ, et al. Effect of advanced access scheduling on processes and intermediate outcomes of diabetes care and utilization. J Gen Intern Med. 2009;24:327–33. doi: 10.1007/s11606-008-0888-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.