Patient satisfaction and quality of care in walk-in clinics, family practices and emergency departments: the Ontario Walk-In Clinic Study

Abstract

Background

Although walk-in clinics are an increasingly common feature of Ontario's health care system, the quality of care they provide is the subject of continuing debate. In this study we examined differences in patient satisfaction and quality of care for common acute conditions in walk-in clinics, family practices and emergency departments.

Methods

For this prospective cohort study, we recruited 12 walk-in clinics, 16 family practices and 13 emergency departments from 11 geographic areas in greater Toronto, Hamilton-Burlington and London, Ont. An expert review panel selected and established quality-of-care criteria for 8 common acute conditions. Patients who sought initial care for 1 of the 8 conditions were recruited by an on-site data collector. We used a questionnaire to assess the satisfaction of 433 patients with patient-centred communication, the physician's attitude and any delay in the waiting room during the study visit. Abstractors reviewed 600 charts for the study patients to assess whether the quality-of-care criteria had been met. A quality score for each case was computed as the percentage of applicable criteria that were met. Mean quality scores for the 3 settings were computed, with adjustment for potentially confounding variables (sex, age, city and diagnosis).

Results

After adjustment for 12 patient characteristics, walk-in clinic patients were significantly more satisfied than emergency department patients on all 3 satisfaction scales. Family practice patients were more satisfied than walk-in clinic patients on all 3 satisfaction scales, but the difference was statistically significant only for satisfaction with waiting time. Adjusted mean quality-of-care scores were 73.1% for emergency departments, 69.9% for walk-in clinics and 64.1% for family practices. The scores for walk-in clinics and emergency departments were significantly higher than that for family practices.

Interpretation

Satisfaction with waiting time was highest among family practice patients. Both family practices and walk-in clinics were perceived more positively than emergency departments on all 3 dimensions of satisfaction. Overall quality-of-care scores were higher in walk-in clinics and emergency departments than in family practices.

Over the past decade, fiscal restraint and changing public perceptions about health care have resulted in controversies about the best ways to organize and deliver primary care. Since 1984, when they first emerged in Ontario, walk-in clinics have proliferated. From the patient's perspective, a potential appeal of walk-in clinics is that they provide ready access to immediate care, especially outside regular office hours. It is widely believed that the volume of services these clinics provide increased significantly in the 1990s.

Walk-in clinics have been met with a lack of enthusiasm from both family and emergency physicians.^1,2 Concerns about walk-in clinics include quality^2,3,4,5 as well as continuity^4,5,6,7 and costs^2,3,4,5 of care. However, perceptions of low quality of care in walk-in clinics are not supported by empirical evidence. In our review of the literature we identified only one study comparing quality of care in walk-in clinics and alternative sources of primary care such as family practices and emergency departments. Using standardized patients and consensus-based process-of-care criteria for 5 clinical scenarios, Grant and colleagues⁸ found that walk-in centres achieved a significantly higher mean score for overall quality of care than general practices. Parks and associates⁹ assessed the quality of processes of care in a chain of US walk-in centres for 5 acute diagnoses using explicit criteria developed by an independent panel of physicians. However, their study did not include a direct comparison with quality of care in other primary care settings.

Although some physicians are concerned about the continuity and quality of care given in walk-in clinics, patients may be satisfied because their demands for convenient care are being met. Patient satisfaction can be considered one measure of quality of medical care.

We report here on a prospective cohort study to assess and compare patient satisfaction (as determined by questionnaire) and quality of care (as determined by chart abstraction) for common, minor, acute conditions in 3 primary care settings in Ontario: walk-in clinics, family physicians' offices and emergency departments.

Methods

This study was part of a larger multicentre study, “The Role and Impact of Walk-In Clinics in Ontario's Health Care System,” which compared utilization, cost and quality of services in walk-in clinics with those provided in family physicians' offices and emergency departments.

We conducted our study in greater Toronto, Hamilton-Burlington and London, Ont. We approached 20 walk-in clinics (chosen by random selection), 35 family practices (32 randomly selected and 3 identified by targeted recruitment) and 13 emergency departments (randomly selected in Toronto and all departments in Hamilton-Burlington and London) from 11 geographic areas in the 3 cities. These 3 metropolitan areas were chosen because they had a relatively high concentration of walk-in clinics and were readily accessible to the research team. At least one walk-in clinic, one family practice and one emergency department were recruited from each of the 11 geographic areas to ensure that the 3 care settings drew patients from the same population pool.

Family practices were defined as settings where more than 50% of patient visits were by “regular patients,” people who used the clinic or saw a physician in the clinic for ongoing care of common medical problems and for preventive care. To be included in the study, family practices had to consist of 2 or more physicians (to increase the efficiency of patient recruitment) and remuneration had to be on a fee-for-service basis. Walk-in clinics were defined as clinics where less than 50% of visits were by “regular patients.” All walk-in clinics included in the study received fee-for-service payment.

An expert review panel was convened to select tracer conditions and to establish quality-of-care criteria. The panel consisted of 2 family physicians, 1 family/walk-in clinic physician, 1 family/emergency physician and 1 emergency/walk-in clinic pediatrician. On the basis of available data regarding frequency of diagnoses in the 3 settings¹⁰ (J. Ivan Williams, Senior Scientist, Institute for Clinical Evaluative Sciences, Toronto, personal communication 1997), 8 acute conditions commonly seen in all 3 settings were selected for study: pharyngitis, gastroenteritis, serous otitis media, acute otitis media, upper respiratory infection, acute bronchitis, urinary tract infection and low back pain. Quality-of-care criteria previously developed by the College of Family Physicians of Canada^11,12 were reviewed by the panel and modified by consensus. The criteria dealt with processes of care and included performance of appropriate clinical actions and avoidance of inappropriate clinical actions. The criteria covered the domains of history-taking, physical examination, diagnostic procedures, imaging, verification of diagnosis, drugs and physical therapy, education and preventive care, and follow-up. Copies of the final criteria are available from the authors on request. The number of criteria ranged from 4 for serous otitis media to 23 for gastroenteritis.

On the basis of previous studies of patient satisfaction¹³ and quality of care¹¹ and using standard methods,¹⁴ we estimated that a sample size of 150 patients per setting (total 450 patients) would be adequate to detect clinically meaningful differences in outcomes between settings. To ensure a similar patient mix in each setting, only patients being seen for an initial assessment for 1 of the 8 tracer conditions were invited to participate. Patients were approached by data collectors while they were waiting to see the primary care physician. Information from patients under 16 years of age and those not competent to respond was collected through a proxy adult respondent using questionnaires with slightly modified wording. After eligibility questions were asked and informed consent was obtained, each English-proficient patient or proxy was interviewed to ascertain demographic characteristics and whether the patient had a regular family physician. Only patients with 1 of the 8 conditions confirmed by the physician (as determined by chart audit) were eligible for the study. As well, eligibility was limited to regular patients in the family practices and “nonregular” patients in the walk-in clinics. Data were collected between Feb. 15 and Dec. 21, 1998. Data collectors recruited 600 eligible patients with a total of 625 tracer conditions (some of the 600 patients had more than one tracer condition). All 600 patients were included in the quality-of-care analyses. Of these 600 patients, 486 completed all data collection instruments, and 433 of these (174 from walk-in clinics, 122 from family practices and 137 from emergency departments), for whom we had complete information on all relevant variables, were included in the patient satisfaction analyses. For the analyses related to satisfaction with care, the diagnoses upper respiratory infection, pharyngitis and acute bronchitis were collapsed into a single category, upper respiratory illness, and the diagnoses acute otitis media and serous otitis media were collapsed into a single category, otitis media.

A tridimensional model of patients' satisfaction with the study visit was used. After seeing the physician, each patient completed a self-administered questionnaire which included 3 instruments, each measuring a separate dimension of satisfaction previously identified as important:^15,16,17,18 perceptions of patient-centred communication^19,20 (8 items with 4 response choices per item, averaged to a total score ranging from 1 to 4), perceptions of the physician's attitude²¹ (5 items with 5 response choices per item, converted to a total score ranging from 20 to 100) and delay in the waiting room²¹ (3 items with 5 response choices per item, converted to a total score ranging from 20 to 100). The items included in each scale are listed in Appendix 1 (see www.cmaj.ca). The internal reliability of the 3 satisfaction scales was tested with Cronbach's α.

Although patients were clustered by provider and providers by practice, the fact that some patients were seen by more than one provider made analyses allowing for such clustering infeasible, and the subjects within each setting were considered as independent observations. Univariable comparisons of satisfaction among the 3 settings were performed by analysis of variance (ANOVA). In multivariable analyses, the different satisfaction outcomes were modelled separately with linear multiple regression analyses (backward stepwise elimination method with significance level for exclusion of p = 0.15). The independent variables considered for inclusion were setting (walk-in clinic, family practice, emergency department), sex, age, education, main activity (employment or other), income, language spoken at home, whether the patient was living with a partner, whether the patient had children, self-reported health status, perceived seriousness of condition, diagnosis (conditions of interest) and whether the patient had a regular family physician. Statistical significance was set at p < 0.05 for both univariable and multivariable analyses.

For the quality-of-care analysis, chart abstraction was performed by 12 experienced abstractors between Feb. 15 and Dec. 21, 1998. Only the record of the initial visit for the tracer condition was reviewed. The abstractors were provided with a detailed abstraction manual and participated in a 3-day training session. Intra- and inter-rater reliability and criterion validity were assessed on the basis of duplicate abstraction of 10 charts covering 7 of the tracer conditions, for which kappa statistics²² and reliability coefficients²³ were generated.

A quality-of-care score for each case was computed as the percentage of applicable criteria met. Therefore, each case received the same weight regardless of the number of applicable criteria. Scores for each of the 3 settings were computed as the mean of scores for all cases managed in that setting.

To adjust for potentially confounding factors, we computed mean scores for each setting, controlling for sex of the patient, age of the patient (grouped as less than 12 years and 12 years or older), city and diagnosis. In addition, preliminary investigation revealed the need to control for a significant interaction between city and setting.

We explored the potentially confounding effects of patients' health status (“In general, would you say your health is excellent, very good, good, fair or poor?”) and perceived condition severity (“On a scale of 1 to 5, with 1 being not serious at all and 5 being very serious, how would you rate the seriousness of your condition?”). Neither perceived health status nor perceived condition severity was significantly related to quality-of-care scores. Accordingly, neither of these variables was included in our analytic model.

We developed an ANOVA model from which the population marginal means of the scores for each setting were estimated. Estimated marginal means, also known as least squares means, are a function of the model parameters, without regard for the distribution of observations through the model factors.²⁴ These means are adjusted for other factors in the model, and thus we refer to them as adjusted scores. The proportion of variability explained is given by the ratio of the between-groups sum of squares to the total sum of squares. After including setting, sex, age, city, diagnosis, and the interaction of setting and city, we checked for significant interactions between sex and age and other model factors. The only other significant interaction was between sex and diagnosis.

The study was approved by the Research Ethics Boards at The University of Western Ontario, The University of Toronto, McMaster University and participating hospitals.

Results

Of the 20 randomly selected walk-in clinics, 12 (60%) agreed to participate. All 13 emergency departments that we approached participated in the study. Of 35 family practices approached, 17 (49%) agreed to participate. Of these 17, 3 did so after targeted recruitment, which was undertaken because as the deadline for completion of data collection neared, no family practices had been recruited in 3 of the 6 geographic areas in Toronto. The final 3 family practices were recruited from the sampling frame on the basis of a personal relationship with a member of the research team. One of the 17 family practices that agreed to participate contributed no patients to the study, which left 16 family practices for inclusion in the analysis. Most of the participating family practices provided care exclusively to regular patients. No participating practice reported walk-in patients as more than one-third of total patient volume.

The reference standard for the assessment of criterion validity of quality-of-care chart abstraction was based on chart abstraction by a physician member of the research team (E.V.). The kappa statistic, a measure of chance- corrected agreement, was 0.89 for intra-rater reliability. The reliability coefficient for inter-rater reliability and the overall reliability coefficient (which reflects both inter- and intra-rater reliability) were both 0.765. The kappa statistic for chance-corrected agreement with the reference standard chart abstraction (criterion validity) was 0.84.

The distribution of tracer conditions varied markedly across settings (Table 1). For example, a large majority of the cases of gastroenteritis were seen in emergency departments, whereas acute bronchitis and pharyngitis were more common in walk-in clinics and family practices. The variable distribution of cases among the 3 settings illustrates the need for case-mix adjustment in comparing performance across settings. The characteristics of the 433 patients included in the satisfaction analysis are summarized in Table 2.

Table 1

Table 2

The internal reliability of the 3 satisfaction scales (Cronbach's α) was 0.91 for patient-centred communication, 0.87 for doctor's attitude and 0.77 for delay in the waiting room. The distribution of the satisfaction measures by setting is shown in Fig. 1. For each individual item in all 3 satisfaction scales, the mean values were ordered consistently, with values for family practice patients higher than those for walk-in clinic patients, and values for walk-in clinic patients higher than those for emergency department patients (results not shown), although the differences were not always statistically significant.

Fig. 1: Frequency distributions of the 3 satisfaction scales by setting. Black bars = family practice patients, grey bars = walk-in clinic patients, open bars = emergency department patients. For all 3 satisfaction scales, p < 0.001 for differences in distribution across settings.

In the final multivariable model regarding patient-centred communication (Table 3), satisfaction among walk-in clinic patients was significantly higher than among emergency department patients but not significantly lower than among family practice patients. Having children was associated with higher levels of satisfaction.

Table 3

Patients' satisfaction with the doctor's attitude was related in multivariable analysis to setting (with walk-in clinic patients being more satisfied than emergency department patients), language (with those who usually spoke a language other than English at home being less satisfied), having children, main activity and income level (Table 3).

In the final multivariable model of satisfaction with delay in the waiting room, only setting was significant, with patients in walk-in clinics reporting levels of satisfaction significantly lower than those in family practices but higher than those in emergency departments (Table 3).

Interactions among variables that were significant in the multivariable models were also investigated, but none of the interaction terms were significant.

The final ANOVA model explained 28% of the variance in quality-of-care scores. Of the variables included in the model, diagnosis made the largest contribution to variation in quality-of-care scores, followed, in descending order of contribution, by city (greater Toronto, Hamilton-Burlington or London), interaction of city and setting (walk-in clinic, family practice or emergency department), setting, interaction of diagnosis and sex, and sex.

Overall unadjusted and adjusted mean quality-of-care scores for each of the 3 settings are shown in Table 4. The adjusted mean quality-of-care scores were significantly higher for patients in walk-in clinics and emergency departments than for patients in family practices.

Table 4

Interpretation

This study examined 2 aspects of care in different settings: patient satisfaction and quality of care received for specific tracer conditions.

Patient satisfaction is an important dimension of good medical care. In addition to being a desirable outcome in its own right,²⁵ patient satisfaction is also associated with health status^15,26 and with health-related behaviours such as compliance with medical regimens and appointment-keeping.^27,28,29 In our study walk-in clinic patients reported lower satisfaction than family practice patients (statistically significant for delay in the waiting room) and higher satisfaction than emergency department patients (statistically significant for all 3 dimensions of satisfaction).

One possible explanation for the pattern of satisfaction results is selection bias. Most of those who chose a walk-in clinic or emergency department had regular family physicians; however, one-third reported that their family physician was not available (or at least was perceived to be unavailable), which led to their attendance at a clinic or emergency department. These patients' frustration with not being able to see their family physician might have resulted in lower levels of satisfaction with care received. Since the analyses controlled for a series of patient characteristics, including diagnosis and perceived seriousness of the presenting condition, it is unlikely that these results can be attributed to different types of patients seeking care in the various settings. However, it is possible that some walk-in clinic patients are hard to please and were “doctor-shopping.” The scores would then reflect, in part, this element of their personalities. Physician selection bias is also possible, given the differential participation rates across settings.

The difference in patient satisfaction between walk-in clinic and family practice patients on the one hand and emergency department patients on the other may be due in part to differential training of physicians working in these settings. Since 1993, it has been a requirement in Ontario that all walk-in clinic and family physicians complete 2 years of family medicine training. In contrast, emergency physicians may or may not have family medicine training. Those accredited by the College of Family Physicians of Canada must complete 2 years of such training followed by 1 year of emergency medicine training, but the 5-year training program for accreditation by the Royal College of Physicians and Surgeons of Canada has no family medicine component. Another possible reason for the differences might be the personality types of the physicians who choose the different work settings. Emergency physicians may be interested in a fast pace and episodic care, which could imply less interest in the interpersonal aspects of the patient encounter. The differences found may also be due to the context of the visits. The family practice office may offer the comfort of familiarity, short waiting times and a noninstitutional atmosphere. The walk-in clinic may offer longer waiting times and less familiarity but still a noninstitutional atmosphere. Visiting the emergency department may entail even longer waiting times and less familiarity, as well as an institutional atmosphere. Finally, one cannot rule out the possibility that continuity of care, which by definition characterizes visits to family practices, was the reason for higher satisfaction in that setting. Continuity of care and length of relationship with primary care physicians have, in many other studies, been positively correlated with satisfaction.^30,31,32,33

The overall high level of satisfaction found and the relation between satisfaction and patient characteristics (sex, age, family size and socioeconomic status) was similar to findings in some other studies in primary care settings, although there is a lack of consistency in this literature.^{15,16,18,34,35,36,37,38,39}

High scores and little variability in scores are common problems in studies of satisfaction.¹⁸ In spite of these potential limitations of measurement, the current study still showed a range of scores for all 3 scales. Furthermore, the study sample was large enough to detect statistically significant differences.

The second aspect of the study examined quality of care received. Although overall quality-of-care scores were significantly higher among walk-in clinic and emergency department patients than among family practice patients, the absolute differences in adjusted scores were not large: 5.8 percentage points between walk-in clinics and family practices and 9.0 percentage points between emergency departments and family practices. Differences of this magnitude may or may not be reflected in differences in patient outcomes for 2 reasons. First, quality-of-care criteria were based on the consensus of a panel of expert clinicians informed by, but not exclusively based on, high-quality evidence. For many processes of care such evidence is not available. It is therefore possible that conformity with some of these criteria is unrelated to outcome. Second, it is conceivable that family physicians' continuing relationship with their patients allows more limited assessment and management or less complete documentation without compromising patient outcomes.

Lower quality-of-care scores in family practices could be related to the broader mandate of this setting compared with walk-in clinics and emergency departments. If family physicians attend to preventive care, chronic disease management and psychosocial issues in the course of some visits for acute minor conditions, they may be less rigorous in their assessment and management or documentation of the acute illness.

Because of the variable participation rates across the 3 settings — from 49% of family practices to 100% of emergency departments — our results may be affected by selection bias. If higher-quality (better-performing) walk-in clinics and family practices were more likely to agree to participate in the study, the differences between emergency departments on the one hand and walk-in clinics and family practices on the other might have been underestimated by our results. Given the higher participation rate of walk-in clinics than of family practices, we feel that our results provide strong reassurance that the quality of the processes of care in walk-in clinics for the conditions studied was at least as good as in family practices.

Similar quality of care in walk-in clinics and family practices should not be surprising, because most physicians in both settings have been trained in family medicine residency programs. Furthermore, many walk-in clinic physicians divide their time between walk-in clinic work and family practice.

Our study had several limitations. In the course of pilot work for this study we found that some walk-in clinics provided continuing care to regular patients in addition to walk-in services and some family practices provided walk-in services in addition to continuing care for regular patients. Limiting our study to “pure” walk-in clinics and family practices would have excluded a significant proportion of walk-in clinics currently in operation. However, including these mixed practices might have made walk-in clinics and family practices appear more similar than would otherwise have been the case, even though only regular patients of family practices and walk-in patients of walk-in clinics were included in the study. In spite of this potential, we did observe differences between these 2 settings in terms of both quality and satisfaction.

Our inability to adjust our statistical analysis for the clustering of patients at both the practice and physician levels would tend to inflate the statistical significance of differences between settings. However, with one exception (patient-centred communication in emergency departments compared with walk-in clinics and family practices) the p values for the differences between settings that we report ranged from 0.006 to less than 0.001. Even if scores for quality or satisfaction were highly correlated within practices or among patients seen by the same physician, these differences would almost certainly remain statistically significant in an analysis that adjusted for cluster effects.

The contrasting results related to quality and patient satisfaction are notable. Patient satisfaction scores were highest in family practices and lowest in emergency departments, whereas for quality-of-care scores the order was reversed. This suggests the possibility of a trade-off in the treatment of acute minor conditions between patient satisfaction and adherence to “best practice” standards for the provision and documentation of care.