Abstract
Background
Strategies to improve preventive services delivery (PSD) have yielded modest effects. A multidimensional approach that examines distinctive configurations of physician attributes, practice processes, and contextual factors may be informative in understanding delivery of this important form of care.
Objective
We identified naturally occurring configurations of physician practice characteristics (PPCs) and assessed their association with PSD, including variation within configurations.
Design
Cross-sectional study.
Participants
One hundred thirty-eight family physicians in 84 community practices and 4,046 outpatient visits.
Measurements
Physician knowledge, attitudes, use of tools and staff, and practice patterns were assessed by ethnographic and survey methods. PSD was assessed using direct observation of the visit and medical record review. Cluster analysis identified unique configurations of PPCs. A priori hypotheses of the configurations likely to perform the best on PSD were tested using a multilevel random effects model.
Results
Six distinct PPC configurations were identified. Although PSD significantly differed across configurations, mean differences between configurations with the lowest and highest PSD were small (i.e., 3.4, 7.7, and 10.8 points for health behavior counseling, screening, and immunizations, respectively, on a 100-point scale). Hypotheses were not confirmed. Considerable variation of PSD rates within configurations was observed.
Conclusions
Similar rates of PSD can be attained through diverse physician practice configurations. Significant within-configuration variation may reflect dynamic interactions between PPCs as well as between these characteristics and the contexts in which physicians function. Striving for a single ideal configuration may be less valuable for improving PSD than understanding and leveraging existing characteristics within primary care practices.
Key words: practice patterns, preventive service delivery, primary care
Introduction
The important thing in science is not so much to obtain new facts as to discover new ways of thinking about them. Sir William Bragg (1862–1942).
Despite ample scientific evidence and the recommendations of numerous professional organizations, preventive services are delivered at low rates.1–3 This “treatment gap” may be related to a variety of factors, including physician disagreement with clinical guidelines,4 the absence of office systems that enable their implementation, or the absence of incentives, financial and otherwise, that reinforce delivery of preventive services. Previous efforts to improve preventive service delivery (PSD) in primary care practice are numerous, but have yielded small to modest changes.5–9 Interventions in many of these studies often target individual barriers such as clinician knowledge, attitudes, or the absence of clinical decision support systems for change efforts.10–12 Previous work suggests that multicomponent strategies that create multiple complementary changes in practice systems13 and those tailored to the way in which a practice is organized and functions lead to increased and sustained improvements in PSD.8,9,14,15 The success of these strategies may be based, in part, in an acknowledgement that physicians’ practices are uniquely configured “systems”16–18 and that an understanding of physicians’ practices is needed before attempting change.19
Efforts to understand physicians’ practices have taken two general approaches, both with limitations. The first assesses characteristics across groups of practices to identify features independently associated with greater delivery of specific forms of care.20–22 Although this approach suggests that specific characteristics are essential to high performance, it generally fails to anticipate and evaluate patterns of interactions across all variables under consideration. A second strategy uses case study–based methods such as positive deviance23 and case comparisons.17,24–26 Although producing rich descriptions of patterns of important characteristics, the sampling method for case study–based approaches does not permit quantitative evaluation of association between identified characteristics and the “success” variable of interest.
In this study, we integrate elements of both approaches. Rather than purposefully selecting high- and low-performing cases, we followed an approach that allowed distinctive configurations of physician practice characteristics (PPCs) to emerge from the data. Our objectives were to examine the relationship between empirically derived practice configurations and the delivery of preventive services, and to explore variation in rates of PSD among those with similar and differing configurations.
Methods
Data were drawn from the Direct Observation of Primary Care (DOPC) study, a cross-sectional multimethod study that examined the content of 4,454 outpatient visits to family physicians in northeast Ohio. Four teams of two research nurses directly observed consecutive patient visits to 138 participating physicians in 84 practice sites between October 1994 and August 1995.27,28 Concurrent with the direct observation, medical record review and surveys of patients and physicians were conducted. After 4 days or more of observation at each site, study nurses generated ethnographic field notes used to discern features of clinicians and their practice relevant to PSD.29 This process, described elsewhere in detail,29 involved summarization of salient themes and key practice features after a repetitive series of intense readings. Preliminary findings were iteratively reviewed, refined, and cross-checked against the original data by two independent analysts. The qualitative and quantitative variables selected for this study are described below. The Institutional Review Board of University Hospitals of Cleveland approved the study protocol.
Measures
Although several general theoretical models for practice assessment exist, we selected the Precede–Proceed model30 because of its extensive application to PSD. This model includes characteristics from three domains, each of which were represented by at least one variable in our analysis. Predisposing factors represent physician attitudes and knowledge affecting the likelihood of PSD. Of the six variables we used to represent this domain, four were assessed from a physician survey and included (1) the rated importance of smoking and diet counseling; (2) self-efficacy in counseling on smoking and diet modification; (3) personal smoking, diet and exercise behaviors; and (4) the degree to which prevention was a focus in their practice. Field notes assessed whether physicians strongly advocated for specific preventive services and observer rating assessed whether physicians used illness visits as opportunities for PSD.
Enabling factors represent skills and resources that facilitate or impede PSD. The physician survey assessed familiarity with the United States Preventive Service Task Force (USPSTF) guidelines and the presence of demands perceived to compete with PSD. Daily patient volume, ability to work in unscheduled patients, degree to which the physician got behind schedule in seeing patients, practice efficiency, use of recommended patient education techniques for health promotion, and involvement of nonphysician staff in PSD were assessed using direct observation of patient care and office operations and the practice environment checklist. Service options included availability of a dietician and training and equipment to conduct sigmoidoscopy. Medical record review assessed the proportion of well-care visits and use of prevention flow sheets. Reinforcing factors, reflecting the presence of practice features that motivate specific actions, were represented by a single variable—use of chart audits to monitor care delivery.
Outcome Measures
Three composite scores reflected receipt of preventive screening procedures, counseling, and immunizations for each eligible patient based on USPSTF recommendations regarding sex, age, and the appropriate time interval for service delivery.3,31 Scores, representing the proportion of services an individual received out of those for which he/she was eligible, ranged from 0 to 100, indicating being up-to-date on none versus all services, respectively.3
Control Variables
Patient age, sex, and health care insurance were assessed from the medical record. Insurance was coded as commercial (fee for service, preferred provider organization), government (Medicare and Medicaid), or none/other.
Analyses
The analyses were conducted in two stages using two different techniques. In the first, we used a hierarchical cluster approach32,33 to group physicians and their practice characteristics (referred to as physician practices) into similar, naturally occurring configurations of PSD-predisposing, PSD-enabling, and PSD-reinforcing characteristics. Here, physician practices were the unit of analysis because many physicians in our sample shared office space, but functioned independently and organized and used their staff differently. Multicolinearity among study variables was assessed and each physician or practice characteristic was then standardized to permit the use of the same metric. The estimated number of physician practice configurations was confirmed using a bootstrap stopping rule procedure.34 During this process, we identified and excluded one physician as an extreme outlier because outliers can greatly influence the results of a cluster analysis. Defining features of each configuration (i.e., variables on which a configuration differed by >0.5 standard deviations from the overall mean score)32,35 were identified and used in generating labels for each. Analysis of variance confirmed that each defining feature differed significantly between at least two of the identified configurations and thus contributed to distinct combinations of PPCs.
In the second analytic stage, we used multilevel random effects models to assess the association between membership in a physician practice configuration and patient-level PSD scores as the dependent or response variables. Patient sex, health care insurance, and age (centered on grand mean) were adjusted for potential confounding. The multilevel analyses were specified by a patient-level equation:
where yij is the PSD score for patient i of physician j. β0j is the intercept for physician j and ɛij is a patient-level random effect that represents the deviation of patient ij’s score from the predicted score.
The physician-level equation specifies the relationship between the physician-level predictors and coefficients in the patient-level equation. The single variable used here was the dummy variable representing each physician practice configuration (PPCC):
where π00 represented the PSD score for the referent on the PPCC variable. π01 to π05 represent the physician-level coefficients and u0j represents physician-level random effects. SPSS 12.1, SAS 9.1, and HLM 6.0 were used to conduct all analyses.
Results
Of the 4,994 patients presenting for care at the study practices, 4,454 (89%) agreed to participate in the DOPC study. The distribution of patient age, sex, and race in this sample was similar to that of patients seeing family physicians and general practitioners nationally as reported in the National Ambulatory Medical Care Survey.28 The average duration of relationship with a physician was 5.4 years. Physicians participating in the DOPC study were similar in age to national samples of family physicians but overrepresented female and residency trained physicians.28 Ten physicians did not complete a physician survey and one physician was excluded as an outlier. Of 4,083 eligible patients seen by the 127 eligible physicians, 4,046 (99.1%) had complete PSD data.
Cluster analysis identified six distinct configurations. Using the magnitude and sign of the standardized score for each defining characteristic, descriptive names were assigned to each (Table 1).
Table 1.
Characteristic | Illness-focused | Prevention team | Average | Well-care focused | Efficiency-minded | Chaotic |
---|---|---|---|---|---|---|
n = 14 | n = 19 | n = 42 | n = 12 | n = 25 | n = 15 | |
Predisposing | ||||||
1. Feels that counseling on behaviors is important | − − | − | + | |||
2. Perceives high efficacy in counseling on behaviors | ||||||
3. Strongly advocates for specific forms of preventive care or health behavior counseling (bee in bonnet) | − | + | − | − | ||
4. Personal health behaviors consistent with behavioral counseling provided patients | − | |||||
5. Reports a high degree of focus on prevention | − − | |||||
6. Frequently uses illness visits as opportunity for prevention and teaching | − − | + | − − | |||
7. Uses multiple patient education techniques when engaging in health promotion or counseling behavior change* | − | + | − − | |||
Enabling | ||||||
8. High degree of familiarity with USPSTF guidelines | − | − | ||||
9. Has high daily patient volume | ++ | − | − | |||
10. Able to work in unscheduled patients | + | − − | + | |||
11. High proportion of well-care visits | − | + | − | |||
12. Gets behind schedule in seeing patients | − | + | − | |||
13. Perceives numerous competing demands | − | |||||
14. Uses prevention flow sheet | − | + | − | |||
15. Uses patient reminder cards | − | + | ||||
16. Has additional service providers available in office to provide counseling | − | + | ||||
17. Nonphysician staff involved in patient education | − | + | − | |||
18. High level of office efficiency | − − | |||||
19. Strong sense of shared vision among workers within practice | + | − | − − | |||
20. High clarity of communication within practice | − − | |||||
Reinforcing | ||||||
21. Periodically audits charts for quality improvement | ++ |
“+” or “−” correspond to a z score >0.5 SD; “++” or “− −” correspond to a z score >1 SD. An empty cell indicates that configuration was average on the characteristic.
USPSTF = United States Preventive Service Task Force.
*Strategies include use of feedback, reinforcement, individualization, goal setting, relevance, and multiple educational channels.
Description of Physician Practice Configurations
Fourteen physician practices (10%) were classified as “illness-focused.” A very high volume of patients and the infrequent use of well-care visits characterized this group. Physicians in this configuration did not identify prevention as a focus of their practice and did not rate health behavior counseling as important. They also scored low on personal health behaviors, did not use illness visits as an opportunity for PSD, did not use recommended patient education techniques to facilitate health behavior advice, and were unfamiliar with USPSTF guidelines. Nonphysician staff was not involved in prevention, service options were not available, and physicians in this configuration did not use prevention flow sheets or reminders to prompt PSD.
In contrast, members of the “prevention team” group (n = 19, 14%) involved other staff in PSD, offered other service options, and made significant use of chart audit to provide performance feedback. They demonstrated a shared vision of purpose that reflected a unified team approach. Patient volume for this group was relatively lower than that for other physician practice configurations. Despite rating health behavior counseling as not being very important, they more commonly used multiple health behavior advice techniques and illness visits as opportunities to provide health behavior advice or other preventive services.
The “well-care-focused” group included 12 physician practices (8.7%), whose primary venue for PSD was scheduled well-care visits. Although these physicians rated health behavior counseling as very important, they did not utilize multiple health behavior advice techniques and did not use illness visits as opportunities for PSD. Low efficiency, the absence of a shared practice vision, an inability to work in unscheduled patients, and a lack of involvement of other staff in providing preventive services also characterized members of this group.
The “chaotic” configuration (n = 15, 11%) had two dominant characteristics: poor clarity of roles and communication among the staff and a lack of a practice vision. Use of prevention flow sheets was also uncommon; practice efficiency was low and well-care visits were generally not scheduled.
The “efficiency-minded” configuration (n = 25, 18%) used reminders systems. Physicians in this group also perceived few competing demands that diverted attention away from direct patient care delivery. While patient volume was average, this group of physicians was able to accommodate unscheduled patients, rarely fell behind schedule, and was described by study personnel as having above average efficiency.
The “average” configuration (n = 42, 31%) had no prominent characteristics to differentiate it from the average of the entire group of physicians. Less prominent but important features included focus on a particular prevention issue, use of prevention flow sheets, and a greater likelihood of being behind schedule.
Overall, patients in the study sample were up-to-date on 11.7% of health behavior counseling services, 59.1% of screening procedures, and 31.3% of immunizations they were eligible to receive. Controlling for selected patient characteristics and the multilevel study design, PSD scores significantly differed among several of the configurations (Fig. 1). Contrary to our hypotheses, however, the physician practice configuration best positioned for PSD (prevention team) did not deliver the highest level of preventive services. Instead, the average physician practice configuration without predisposing, enabling, or reinforcing characteristics differentiating them from the overall group average had scores comparable to or exceeding those of the prevention team configuration for each PSD indicator. Consistent with our hypothesis, however, the illness-focused configuration, characterized by the absence of several theoretical characteristics associated with PSD, performed at the lowest level for each indicator. The intracluster correlation was 0.16, 0.11, and 0.06 for counseling, screening, and immunization PSD scores, respectively.
Although PSD differed significantly across configurations, mean differences between the configurations with the lowest and highest PSD scores were relatively small (i.e., 3.4, 7.7, and 10.8 points for health behavior counseling, screening, and immunizations, respectively, on a 100-point scale). For example, the mean proportion of patients up-to-date on health behavior counseling among physicians in the average configuration was 3.4 points higher than patients seen by physicians in the lowest performing configuration (illness-focused). Within configurations, we observed substantial variation in rates of screening, immunization, and behavioral counseling (see Fig. 1).
Discussion
In a dynamic and complex health care environment, it is not surprising that primary care physicians organize their patient care practices differently. This work identified six distinct physician practice configurations reflecting different approaches to organizing across a range of characteristics with the potential to promote or impede PSD. Although we found statistically significant differences in PSD between the configurations, these differences were not substantive from a clinical perspective. Furthermore, we were unable to consistently predict those that were most likely to perform best on PSD and found large variability in PSD within each configuration. Given a well-recognized gap in implementing recommendations for PSD in community practice,3,21,22,36 further exploration of these patterns of variation is warranted.
The idea that different configurations can achieve equally good performance is not new; it has been called equifinality.37,38 Our finding of comparable PSD rates across several distinct physician practice configurations with differing enabling and predisposing attributes supports its relevance in health care settings. Thus, not all variation in primary care processes is necessarily undesirable.16,21,23
In this study, interactions among the characteristics of physician practices may account for some of the observed variation in PSD rates within seemingly similar configurations. For example, the availability of a prevention flow sheet may have very different effects on PSD depending on the inclination of nonclinician staff to incorporate it into their daily workflow. Likewise, excellent clarity of staff roles may be less influential on PSD if staff are not involved in assisting with or providing preventive services. Thus, although the presence of characteristics may be the same within a configuration, the ways in which they are deployed and their interaction with other practice characteristics may influence actual outcomes.
Acknowledging that system equifinality exists with regard to PSD has implications on the development and implementation of interventions to improve the quality of primary care. Interventions that tailor tools and training to provide missing structural components or processes to enable, predispose, or reinforce delivery of specific forms of care may not result in desirable change. Indeed, reconfiguring practices to conform to a single “ideal” configuration may have the unintended effect of reducing or eliminating variation in work processes that is appropriate to local circumstances.24
Our study has implications for future work. First, these findings support the need for a careful assessment of the context, structures, and processes as well as important interactions between each that might influence the implementation of interventions to promote change in practice outcomes. Examining the dynamic interactions may, for example, be useful in guiding quality improvement efforts.39–41 However, more fully developed conceptual frameworks of practice change16,17,42,43 and measurement strategies are needed to assess the importance of specific practice characteristics.43–45 Furthermore, application of analytic methods that facilitate examination of naturally occurring configurations of practice characteristics, such as the methods used in this study, could aid in identifying patterns across practices. Second, it is also clear that there is much we have to learn from practices that may be useful in guiding future work. As we demonstrated, the presence of variation can lead to unanticipated insights that are informative and therefore highly useful. Being open to discovery as one intervenes with practices increases the opportunities for mutual learning and significant enhancements to proposed interventions.46 Both may contribute, in turn, to the potential for greater intervention effectiveness and opportunities for unanticipated insights into approaches useful in other practice settings.
Several limitations deserve note. First, this was a cross-sectional study in which causal inferences could not be established. Second, we analyzed previously collected data, and our choice of variables was constrained. Although a variety of predisposing and enabling variables were available, additional variables representing features that “reinforced” PSD or demonstrated actual implementation of specific strategies as well would have been desirable. However, the PPCs evaluated in this study are similar to those considered important today although they were collected a decade ago.47 These data are also strong in terms of measurement of all USPSTF-recommended services and utilization of the combination of direct observation of patient visits and medical record review methods. Third, cluster analysis is a technique sensitive to both the inclusion of outliers and to the specific variables included in the analysis.32 Only one physician was identified and excluded as an outlier. However, it is possible that the composition of the configurations would have differed if we had incorporated other predisposing, enabling, or reinforcing variables into our analysis. Fourth, our analyses were limited to evaluating PSD, therefore the findings cannot be generalized to other aspects of primary care such as chronic disease management. Finally, the generalizability of our results may also be limited by the nonrandom and geographically homogeneous sample of participating physicians. However, previous work suggests that both patients and community practicing physicians participating in the DOPC study were comparable to those found in most family practices.28
Conclusions
Different ways of configuring physician practices can result in similar rates of PSD. In addition, physician practices sharing similar configurations appear to vary significantly in their delivery of preventive services. Striving for a single ideal configuration may be less important for improving PSD than understanding and leveraging the dynamic interactions between PPCs. Thus, developing a deeper understanding of physician practices may be valuable in designing more effective interventions to enhance delivery of preventive services and other forms of health care.
Acknowledgements
The authors are indebted to the physicians, office staff members, and patients who participated in this study. This study was supported in part by a grant from the National Cancer Institute (1R01 CA60862) and in part by the Center for Research in Family Practice and Primary Care and the American Academy of Family Practice. Dr. Flocke was supported by a career development award from the National Cancer Institute (K07 CA86046). Dr. Litaker was supported by a career development award from the Veteran’s Administration Health Services Research and Development Service.
Potential Financial Conflicts of Interest None disclosed.
References
- 1.Anderson L, May DS. Has the use of cervical, breast, and colorectal cancer screening increased in the United States? Am J Public Health. 1995;85:840–2. [DOI] [PMC free article] [PubMed]
- 2.Kottke TE, Solberg LI, Brekke ML, Cabrera A, Marquez MA. Delivery rates for preventive services in 44 Midwestern clinics. Mayo Clin Proc. 1997;72(6):515–23. [DOI] [PubMed]
- 3.Stange KC, Flocke SA, Goodwin MA, Kelly RB, Zyzanski SJ. Direct observation of rates of preventive service delivery in community family practice. Prev Med. 2000;31:167–76. [DOI] [PubMed]
- 4.Cabana MD, Rand CS, Powe NR, et al. Why don’t physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999;282(15):1458–65. [DOI] [PubMed]
- 5.Solberg LI, Kottke TE, Brekke ML. Will primary care clinics organize themselves to improve the delivery of preventive services? A randomized controlled trial. Prev Med. 1998;27(4):623–31. [DOI] [PubMed]
- 6.Solberg LI, Kottke TE, Brekke ML, et al. Failure of a continuous quality improvement intervention to increase the delivery of preventive services. A randomized trial. Eff Clin Pract. 2000;3(3):105–15. [PubMed]
- 7.McBride P, Underbakke G, Plane MB, et al. Improving prevention systems in primary care practices: the Health Education and Research Trial (HEART). J Fam Pract. 2000;49(2):126–9. [PubMed]
- 8.Goodwin MA, Zyzanski SJ, Zronek S, et al. A clinical trial of tailored office systems for preventive service delivery: the Study To Enhance Prevention by Understanding Practice (STEP-UP). Am J Prev Med. 2001;21:20–28. [DOI] [PubMed]
- 9.Stange KC, Goodwin MA, Zyzanski SJ, Dietrich AJ. Sustainability of a practice-individualized preventive service delivery intervention. Am J Prev Med. 2003;25(4):296–300. [DOI] [PubMed]
- 10.Davis D, O’Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA. 1999;282(9):867–74. [DOI] [PubMed]
- 11.Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA. 1995;274(9):700–5. [DOI] [PubMed]
- 12.Glasgow RE. Translating research to practice: lessons learned, areas for improvement, and future directions. Diabetes Care. 2003;26(8):2451–6. [DOI] [PubMed]
- 13.Stone EG, Morton SC, Hulscher ME, et al. Interventions that increase use of adult immunization and cancer screening services: a meta-analysis. Ann Intern Med. 2002;136:641–51. [DOI] [PubMed]
- 14.Evashwick C, Ory MG. Organizational characteristics of successful innovative health care programs sustained over time. Fam Commun Health. 2003;26(3):177–93. [DOI] [PubMed]
- 15.Bordley WC, Margolis PA, Stuart J, Lannon C, Keyes L. Improving preventive service delivery through office systems. Pediatrics. 2001;108(3):E41. [DOI] [PubMed]
- 16.Miller WL, McDaniel RR, Crabtree BF, Stange KC. Practice jazz: understanding variation in family practices using complexity science. J Fam Prac. 2001;50(10):872–8. [PubMed]
- 17.Crabtree BF, Miller WL, Stange KC. Understanding practice from the ground up. J Fam Prac. 2001;50(10):881–7. [PubMed]
- 18.Rhydderch M, Elwyn G, Marshall M, Grol R. Organisational change theory and the use of indicators in general practice. Qual Saf Health Care. 2004;13(3):213–7. [DOI] [PMC free article] [PubMed]
- 19.Stange KC. “One size doesn’t fit all.” Multimethod research yields new insights into interventions to improve preventive service delivery in family practice. J Fam Pract. 1996;43(4):358–60. [PubMed]
- 20.Carpiano RM, Flocke SA, Frank SH, Stange KC. Tools, teamwork, and tenacity: an examination of family practice office system influences on preventive service delivery. Prev Med. 2003;36:131–40. [DOI] [PubMed]
- 21.Randolph G, Fried B, Loeding L, Margolis P, Lannon C. Organizational characteristics and preventive service delivery in private practices: a peek inside the “black box” of private practices caring for children. Pediatrics. 2005;115(6):1704–11. [DOI] [PubMed]
- 22.Pham HH, Schrag D, Hargraves JL, Bach PB. Delivery of preventive services to older adults by primary care physicians. JAMA. 2005;294(4):473–81. [DOI] [PubMed]
- 23.Marsh DR, Schroeder DG, Dearden KA, Sternin J, Sternin M. The power of positive deviance. BMJ. 2004;329(7475):1177–9. [DOI] [PMC free article] [PubMed]
- 24.Crabtree BF, Miller WL, Tallia AF, et al. Delivery of clinical preventive services in family medicine offices. Ann Fam Med. 2005;3(5):430–5. [DOI] [PMC free article] [PubMed]
- 25.Stake RE. Case Studies, in Handbook of qualitative research. Denzin NK, Lincoln YS, eds. 2000, Sage Publications, Inc.: Thousand Oaks. pp 435–54.
- 26.Backer EL, Geske JA, McIlvain HE, Dodendorf DM, Minier WC. Improving female preventive health care delivery through practice change: an Every Woman Matters Study. J Am Board Fam Pract. 2005;18(5):401–8. [DOI] [PubMed]
- 27.Stange KC, Zyzanski SJ, Jaen CR, et al. Illuminating the ‘black box’. A description of 4454 patient visits to 138 family physicians. J Fam Pract. 1998;46(5):377–89. [PubMed]
- 28.Stange KC, Zyzanski SJ, Smith TF, et al. How valid are medical records and patient questionnaires for physician profiling and health services research? A comparison with direct observation of patient visits. Med Care. 1998;36:851–67. [DOI] [PubMed]
- 29.Crabtree BF, Miller WL, Aita V, Flocke SA, Stange KC. Primary care practice organization: a qualitative analysis. J Fam Pract. 1998;46(5):403–9. [PubMed]
- 30.Green LA, Niebauer L, Miller RS, Lutz LJ. An analysis of reasons for discontinuing participation in a practice-based research network. Fam Med. 1991;23:447–9. [PubMed]
- 31.U.S. Preventive Services Task Force. Guide to Clinical Preventive Services: An Assessment of the Effectiveness of 169 Interventions. 1989, Baltimore: Williams & Wilkins.
- 32.Aldenderfer MS, Blashfield RK. Cluster Analysis. Quantitative Applications in the Social Sciences. Lewis-Beck MS, ed. 1984, Newbury Park: Sage.
- 33.Hair JF, Anderson RE, Tatham RL, Black WC. Multivariate Data Analysis, 5th edn. 1998, Upper Saddle River, NJ: Prentice Hall.
- 34.Tonidandel S, Overall JE. Determining the number of clusters by sampling with replacement. Psychol Methods. 2004;9(2):238–49. [DOI] [PubMed]
- 35.Norman GR, Sloan JA, Wyrwich KW. Interpretation of changes in health-related quality of life. The remarkable universality of half a standard deviation. Med Care. 2003;41(5):582–92. [DOI] [PubMed]
- 36.U.S. Department of Health and Human Services. Healthy People 2010: understanding and improving health, 2nd edn. 2000, U.S. Department of Health and Human Services: Washington, DC: U.S. Government Printing Office.
- 37.Gresov C, Drazin R. Equifinality: functional equivalence in organization design. Acad Manage Rev. 1997;22:403–8. [DOI]
- 38.von Bertalanffy L. General Systems Theory. 1968, New York, NY: George Braziller, Inc.
- 39.Engels Y, Dautzenberg M, Campbell S, et al. Testing a European set of indicators for the evaluation of the management of primary care practices. Fam Pract. 2006;23(1):137–47. [DOI] [PubMed]
- 40.Rhydderch M, Edwards A, Elwyn G, et al. Organizational assessment in general practice: a systematic review and implications for quality improvement. J Eval Clin Pract. 2005;11(4):366–78. [DOI] [PubMed]
- 41.Engels Y, Campbell S, Dautzenberg M, et al. Developing a framework of, and quality indicators for, general practice management in Europe. Fam Pract. 2005;22(2):215–22. [DOI] [PubMed]
- 42.Stroebel CK, McDaniel RR, Crabtree BF, Miller WL, Nutting PA, Stange KC. How complexity science can inform a reflective process for improvement in primary care practices. Jt Comm J Qual Saf. 2005;31(8):438–46. [DOI] [PubMed]
- 43.Kairys JA, Orzano J, Gregory P, et al. Assessing diversity and quality in primary care through the multimethod assessment process (MAP). Qual Manag Health Care. 2002;10(4):1–4. [DOI] [PubMed]
- 44.Creswell JW, Fetters MD, Ivankova NV. Designing a mixed methods study in primary care. Ann Fam Med. 2004;2(1):7–12. [DOI] [PMC free article] [PubMed]
- 45.Cohen D, McDaniel R, Crabtree BF, et al. A practice change model for quality improvement in primary care practice. J Healthc Manag. 2004;49(3):155–70. [PubMed]
- 46.Gordon LE, Flocke SA. Mutual learning and the transformation of study intervention tools. Ann Fam Med. 2005;3:S54–56. [DOI] [PMC free article] [PubMed]
- 47.Hung DY, Rundall TG, Crabtree BF, Tallia AF, Cohen DJ, Halpin HA. Influence of primary care practice and provider attributes on preventive service delivery. Am J Prev Med. 2006;30(5):413–22. [DOI] [PubMed]