Abstract
OBJECTIVE
To develop a sound method to identify patient and physician characteristics that influence specialty referrals.
DESIGN
A retrospective cohort analysis of medical claims data from 1996 supplemented with surveys of primary care physicians.
SETTING
A 600-member independent practice association in southeastern Michigan that provided care for 90,000 members of an HMO.
PATIENTS
Five cohorts, each of 2,000 to 6,000 patients with diagnoses that could be referred to cardiologists, ophthalmologists, pulmonologists, orthopedists, or general surgeons.
MAIN RESULTS
The referral rates for the different cohorts ranged from 1% to 7%. The discriminatory ability of the multivariate logistic models (c-statistic) ranged from 0.66 to 0.79. The likelihood of referral was associated with the patient's diagnoses and medications and with the referring physician's age, years out of medical school, satisfaction with the specialty being referred to, and the importance of making or confirming a diagnosis.
CONCLUSIONS
Because these methods were not difficult to implement and the results were credible, we believe that other organizations should be able to use them.
Keywords: physician resource utilization, managed care, case-mix adjustment
Awidely used mechanism for controlling costs in managed care is to require authorization from a primary care physician before a patient can see a specialist.1 Concern has been raised that the gatekeeper function could restrict patient access to necessary services.2 The challenge for a physicians' organization is to assess whether its frequency of specialty referrals is appropriate, given the variability in physician referral practices.3 Explicit criteria have been developed to determine if a referral is medically indicated,4 such as the rate of false-positive referrals for strabismus,5 or the percentage of “appropriate” surgical referrals for breast symptoms.6 Unfortunately, criteria may not be developed for the conditions of greatest financial or clinical importance, or may require data sources beyond those generally available.4 In the absence of criteria, over-utilization and under-utilization can be suspected if an organization's rate of referrals deviates greatly from that of similar entities.4 However, the optimal referral rate remains unknown.7
Previous work has shown that the likelihood of specialty referrals varies in part owing to the illness burden of the patient population,8,9 and in part owing to physician practice style.7 Although it is intuitive that patients with more complex or graver illness might need to be referred more often to specialists, it is not known how best to adjust the referral rate for patient case mix. One frequently cited article9 used the Ambulatory Diagnosis Groups to adjust the referral rate for patient case mix.10 However, this study has been criticized,11 because physician practice style (in this case, a low threshold for referring) could confound the association between the outcome of interest (referral rates) with the explanatory variables (complexity and severity of illness, which are also determined by the physician).11,12 Physicians predisposed to assuming the worst-case scenario, for example, might be expected to assign more severe diagnostic codes to their patients and refer more frequently than peers. The confounding is compounded if diagnoses made subsequent to the specialty referral are used to determine the complexity level of the case.11
Physician-dependent factors have also been offered as possible explanations for the variability, including specialist availability,13,14 pressure from consumers,15,16 and diagnostic certainty.17–19 Despite the numerous investigations reported in the literature, much of the variability has been left unexplained.7
A physician organization interested in examining specialty referrals therefore faces the dual challenge of adjusting for patient complexity in a methodologically sound manner and identifying important physician-level variables. A further challenge is to assess and measure these without burdening busy practitioners. We report one approach.
METHODS
This study examined referrals to five specialties within an independent practice association (IPA) of approximately 200 primary care and 400 specialty physicians in southeastern Michigan during 1996. At this time, the IPA had imposed a partial moratorium on adding more specialists. The IPA had a full-risk contract with a local HMO to provide care for approximately 90,000 people. Thus, primary care physicians and specialists agreed to share financial surpluses or deficits if their patients' expenditures exceeded targets.1 Patients could not see a specialist without the authorization of their primary care physician. No formal guidelines for referral were in place.
Because the focus of this article is on the methods that can be used to study referral patterns, we will present our approach in some detail, beginning with an overview. The conceptual model was that patients were at risk of being referred to a specialist each time they had an encounter with a primary care physician and they had a condition that was managed frequently by a specialist. The encounter with a primary care physician was therefore the unit of analysis. Whether or not a given encounter was followed by a specialty referral was hypothesized to be a function of patient characteristics, including diagnoses, and characteristics of the primary care physician. We thought that some of these physician characteristics could be measured by asking primary care physicians about themselves in a survey (Figure 1). To create the survey instrument, we started by identifying items or concepts from the literature.7,16,20–22 We showed sample questions to several primary care physicians and specialists and asked for their feedback. In March 1997, the survey forms were distributed to all 144 primary care physicians caring for more than 100 patients in the IPA. Primary care physicians were told their responses would be confidential. Responses from individual physicians were linked to demographic information in the IPA's physician membership file.
FIGURE 1.
Survey questions for primary care physicians.
Please think back over the past year and try to remember patients you saw with a problem in this specialty.* Please consider the specialists within the HMO.
1. How long does it take on average to have an outpatient seen (for a non-emergent condition)?
<1 week … 1–2 weeks … >2 weeks
2. How much do you agree with the following?
Scale: 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree
a. The specialists in this specialty are equally trying to keep costs down.
b. The quality of service provided is similar among these specialists.
c. Your referral patterns are the same for both HMO patients and other patients.
d. Your preferred specialist is the one who actually sees the patient.
3. Based on your experience with your patients, how satisfied were you with the following:
Scale: 1 = very dissatisfied, 2 = mostly dissatisfied, 3 = indifferent, 4 = mostly satisfied, 5 = very satisfied
a. the consultant's attitude?
b. the extent to which your questions were addressed?
c. the patient benefits of the tests/procedures performed by the specialist in light of the costs?
d. the patient benefits of the tests/procedures recommended by the specialist in light of the costs?
e. your patients' satisfaction with the consultant?
f. the extent to which the consultation enhanced your relationship with the patient?
g. the extent to which the consultant involved you in the decision making?
h. the timeliness of the written consultation?
i. the overall service provided?
4. Consider all the patients you referred to this specialty over the past year. How often did the following affect your decision to refer?
Scale: 1 = none of the time, 2 = little of the time, 3 = some of the time, 4 = most of the time, 5 = all of the time
a. to learn more about a disease or condition
b. to establish a diagnosis
c. to confirm your diagnosis
d. to establish a treatment plan
e. to confirm your treatment plan
f. to have a specific procedure performed that you do not perform yourself
g. to comply with patient or family request
h. to obtain assistance due to the severity of the patient's illness
Survey items were transformed to a scale ranging from 0–100.
Summary SATISFACTION score 5 mean score for items (2b + 2d +3a + 3b + 3e +3f + 3g + 3h + 3i).
Summary COST-EFFECTIVENESS score 5 mean score for items (2a + 3c + 3d).
* Primary care physicians completed a separate survey form for each of the five specialties.
To identify the conditions that were managed frequently by a specialist, we started with the 260 categories of the Clinical Classification for Health Policy Research,23–25 which themselves are groupings of a much larger number of diagnoses from the International Classification of Diseases, Ninth Revision. In earlier work, we used multivariate linear regression models to identify the 57 mutually exclusive groupings of these 260 categories that best explained overall medical conditions. (The adjusted R2values ranged from .42 to .48 in development and validation samples from different places and different years.) For each specialty, we ranked these 57 groups according to the number of patients seen by physicians starting at the top of the rank list with the group representing the most patients seen in that specialty. We then added the next largest group until we had accounted for approximately 90% of the patients in that specialty (86%–94%, depending on the specialty). Table 1 describes the final list of what we will refer to as index conditions.
Table 1.
Conditions Managed Frequently by a Specialist
| Specialty | Index Condition* | Categories from theClinical Classification for HealthPolicy Research |
|---|---|---|
| Cardiology–related | Coronary artery disease | 101 |
| Congestive heart failure | 96, 97, 108 | |
| Acute myocardial infarction | 100 | |
| Arrhythmia | 105–107, 245, 249 | |
| Congenital anomaly | 213–217 | |
| Thromboembolism and aneurysms | 114–118, 248 | |
| Nonspecific chest pain | 102, 104 | |
| Opthalmology–related | Diabetes | 49, 50 |
| Headache | 84 | |
| HIV | 5 | |
| Congenital anomaly | 213–217 | |
| Collagen vascular disease | 201, 202, 210 | |
| Cataract, glaucoma and other eye problems | 86–91 | |
| Pulmonary disease–related | Lung cancer | 19 |
| COPD or asthma | 127, 128 | |
| Pneumonia | 122, 129–133 | |
| Other conditions | 254, 257, 259 | |
| Orthopedics–related | Other injuries | 237, 238, 241–244 |
| Congenital | 213–217 | |
| Nontraumatic musculoskeletal | 203–206, 208, 209, 211, 212 | |
| Major joint trauma or fractures | 207, 225, 226 | |
| Skull or spinal cord injuries | 227, 228, 233, 234 | |
| Soft tissue injuries | 232, 235, 236, 239, 240 | |
| Collagen vascular disease | 201, 202, 210 | |
| Fractures | 229–231 | |
| Skin ulcers and other skin problems | 198–200 | |
| General surgery–related | Colon cancer | 14 |
| Breast cancer | 24 | |
| Other cancer | 11–13, 15–18, 20–23, 25, 27, 28, 30–47 | |
| Kidney disease | 156–158, 161 | |
| Thromboembolism and aneurysms | 114–118, 248 | |
| Serious GI condition | 144–146, 153 | |
| Other conditions | 254, 257, 259 | |
| Venous conditions | 103, 119–121 | |
| Biliary or pancreatic disorders | 149, 152 | |
| Nonmalignant breast | 167 | |
| Abdominal pain symptoms | 138–141, 147, 151, 154, 155, 250, 251 | |
| Appendicitis | 142, 148 | |
| Skin ulcers and other skin problems | 198–200 | |
| Hernia | 143 |
COPD indicates chronic obstructive pulmonary disease; GI, gastrointestinal.
Once the index conditions had been identified, we constructed a separate data set for each specialty and treated each specialty as a separate study, so the same patient could be considered at risk of one referral in one data set, and at risk of another referral in a different data set. Inclusion criteria limited each data set to patients with new or recent onset of the index conditions to minimize differences in the duration of the conditions. Patients were excluded if they either had medical claims for any of the index conditions in that specialty or had been referred to that specialty in the previous year. Patients also were required to be at least 18 years old and members of the HMO in both the year of the study and the previous year. The patient's encounter with the primary care physician could occur in the office, hospital, or emergency department. The outcome of interest was referral to a specialty physician within 90 days of the primary care physician encounter, which was defined as a face-to-face encounter with a specialist that was billed as a consultation, referral, hospital visit, critical care visit, or emergency department visit.
We included as binary covariates diagnoses assigned to the patient before and during the encounter with the primary care physician but not those assigned subsequently. We searched medical and pharmacy claims databases for potential covariates. For the cardiology cohort, variables indicated the presence or absence of hypertension, hyperlipidemia, and diabetes. To use the information in the pharmacy claims, we grouped into 21 categories the American Hospital Formulary Service (AHFS) pharmacologic-therapeutic classifications.26 We derived these categories during a previous study of pharmacy expenditures, and our analyses indicated which categories were likely to be prescribed for patients with specific index conditions. For all cohorts, we identified which patients were taking drugs in each category at the index visit to the primary care physician. In our final models, we included as binary covariates only those drugs in the following categories: cardiovascular drugs (AHFS 24:00–24:04, 24:08–24:16), hormones (AHFS 68:00–68:12, 68:18–68:28, 68:34–68:36.08), and analgesics (AHFS 28:04–28:10).26
Statistical Analysis
All analyses were performed using SAS statistical software, version 6.12 (SAS Institute Inc., Cary, NC). Questionnaire reliability was tested with Cronbach's α. Principal components analyses were used to identify survey items that could be combined into summary scores. These items were subsequently transformed to a 0-to-100 scale and combined into summary satisfaction and cost-effectiveness scores. Survey items that were not combined into summary scores were collapsed into binary variables (none–little of the time vs some–all of the time) and were tested for association with referral (yes/no) by means of χ2testing for each cohort. Items achieving a significance level of .20 were considered for multivariate logistic regression models.
Multivariate models were constructed to identify factors that had a significant independent influence on the likelihood of referral. Three steps were used. First, multivariate logistic regression models were constructed using a stepwise procedure (significance levels for entry/retention, .15/.20). For each cohort, candidate variables included the index conditions pertaining to that cohort; medication categories; the gender, age, and years since graduation from medical school for the primary care physician; a summary cost-effectiveness score; a summary satisfaction score; and the survey items. Second, these models were examined using a backward elimination procedure (significant level for retention, .05). The discriminatory ability of these models was measured by the c-statistic.27 The effect of nonresponse bias was assessed by including a dummy variable for missing survey data in models containing patient characteristics. Variables retained in the multiple logistic regression models were entered into generalized estimating equations (GEE) using proc genmod. This additional step allowed for the possibility that patients with the same primary care physician might have referral patterns that were more similar to each other than patients with different primary care physicians. The GEE models assumed a common outcome correlation coefficient for any two patients with the same physician.
RESULTS
One hundred fourteen physicians (79%) returned 570 surveys, applicable to 86% of the encounters. On average, physicians were 46 years old (SD = 11 years), and had graduated 19 years earlier from medical school (SD = 11 years); 28% of them were women. Patient cohorts ranged in size from 2,204 to 5,640 patients (Table 2), representing 10% to 20 % of the IPA's managed care patients with the index conditions.
Table 2.
Characteristics of the Study Population
| Specialty | Patients inCohort, n | HMO Patients with TheseConditions, % | Female,% | Mean Patient Age,Years (SD) | PercentageReferred |
|---|---|---|---|---|---|
| Cardiology | 2,319 | 18.6 | 53.3 | 46 (12) | 6.7 |
| Ophthalmology | 2,204 | 13.7 | 61.5 | 42 (12) | 7.0 |
| Pulmonary disease | 2,761 | 16.8 | 55.9 | 43 (13) | 1.6 |
| Orthopedics | 5,640 | 15.1 | 54.0 | 42 (12) | 6.0 |
| General surgery | 3,776 | 11.1 | 49.7 | 40 (12) | 4.0 |
Principal components analyses identified two factors in the surveys. Factor 1 loaded heavily on variables considered to be measures of physician satisfaction with the specialty, and factor 2 loaded heavily on variables that addressed the perceived cost-effectiveness. Together, these factors explained 64% of the variability. Cronbach's α was .91 for the items in factor 1 and .85 for the items in factor 2.
Approximately one fourth of encounters were with a primary care physician other than the patient's assigned physician; however, less than 1% of cohort patients changed primary care physician assignment during this time. The likelihood of referral after an encounter was not different when information was missing on the physician survey, except for pulmonary disease referrals, for which encounters with missing items were more likely to be followed by a referral (odds ratio 1.8, p = .048). The multivariate logistic models had the following c statistics: cardiology, .74; ophthalmology, .79; pulmonary, .73; orthopedics, .66; and general surgery, .77.
The referral rates to each specialty and results of the multivariate GEE models are shown in Table 3. Although each parameter in these models initially had a p value less than .05 in the multivariate logistic models, several items lost statistical significance in the GEE models, indicated by confidence intervals that include 1.0.
Table 3.
Odds Ratios and 95% Confidence Intervals from the Multivariate General Estimating Equation Models
| Specialty | Independent Variable | Odds Ratio | 95% CI |
|---|---|---|---|
| Cardiology | Acute myocardial infarction | 11.71 | 4.63, 29.60 |
| Coronary artery disease | 3.19 | 1.93, 5.28 | |
| Increasing patient age | 1.03 | 1.01, 1.04 | |
| Increasing physician satisfaction | 1.01 | 1.00, 1.03 | |
| Increasing age of physician | 0.98 | 0.96, 0.99 | |
| Patient female gender | 0.58 | 0.39, 0.86 | |
| Thromboembolism and aneurysms | 0.19 | 0.08, 0.44 | |
| Opthalmology | Visit for cataract, glaucoma, or other eye problems | 9.79 | 6.39, 14.99 |
| Increasing patient age | 1.03 | 1.02, 1.04 | |
| Increasing physician satisfaction | 1.02 | 1.01, 1.04 | |
| Physician years out of medical school | 0.98 | 0.97, 0.99 | |
| Physician often referred to confirm diagnosis | 0.59 | 0.42, 0.85 | |
| Pulmonary | Patient on cardiovascular medicine | 2.34 | 1.11, 4.94 |
| Increasing perceived cost-effectiveness | 1.02 | 1.00, 1.05 | |
| Increasing physician satisfaction | 0.97 | 0.95, 1.00 | |
| COPD or asthma | 0.22 | 0.10, 0.47 | |
| Orthopedic | Major joint trauma or fracture | 7.76 | 4.58, 13.15 |
| Fractures | 3.80 | 2.27, 6.38 | |
| Nontraumatic musculoskeletal condition | 2.15 | 1.52, 3.03 | |
| Physician often referred to establish a diagnosis | 1.65 | 1.04, 2.61 | |
| Patient on a hormonal agent | 1.57 | 1.07, 2.29 | |
| Patient on an analgesic agent | 0.63 | 0.41, 0.98 | |
| Skin ulcers and other skin problems | 0.49 | 0.31, 0.78 | |
| General surgery | Breast cancer | 54.36 | 2.26, 1,307.80 |
| Hernia | 16.82 | 8.54, 33.12 | |
| Biliary or pancreatic disorders | 4.56 | 1.47, 14.18 | |
| Nonmalignant breast | 3.02 | 1.86, 4.91 | |
| Venous conditions | 2.34 | 1.43, 3.82 | |
| Abdominal pain symptoms | 0.42 | 0.26, 0.68 | |
| Skin ulcers and other skin problems | 0.34 | 0.21, 0.56 | |
| Physician often referred to have specific procedure performed | 0.20 | 0.15, 0.26 | |
| Other conditions | 0.15 | 0.06, 0.40 | |
| Physician female gender | 0.65 | 0.42, 1.01 |
When we compared one specialty with another, there were large differences in referral rates for patients with index conditions in those specialties, from 1.6% for patients with pulmonary conditions to 7.0% for those with ophthalmologic conditions (Table 3). When we looked at the effects of the two types of characteristics, we found that both characteristics of the patient and characteristics of the primary care physician were associated with the referral rate. We observed some interesting patterns in these associations. The relative effects of the two types of characteristics were different for different specialties. For example, the ratio of patient characteristics to physician characteristics was high for patients with orthopedic and general surgical conditions, low for patients with ophthalmologic and pulmonary conditions, and intermediate for patients with cardiology conditions. Also, at least one patient characteristic had a stronger effect than any physician characteristic, regardless of specialty. Finally, different physician characteristics were associated with referral rates in different specialties, except for increasing primary care physician satisfaction, which had a small positive effect in two specialties and a small negative effect in one specialty.
DISCUSSION
Given the known variability in referral practices, our particular findings may not apply to other settings. Therefore, we will focus on the inherent strengths and limitations of the methods.
The first consideration is utility. This approach was relatively inexpensive and was accepted by a large majority of the physicians. The high response rate to the survey suggests the study could be replicated without much difficulty, assuming an honest and collegial physician environment. We believe the type of information displayed in Table 3 is useful to a physician organization. The diagnoses identified are potential candidates for referral guidelines, or more focused analyses of over-utilization or under-utilization. The physician characteristics, such as the level of diagnostic certainty in the orthopedics model and the primary care physician satisfaction in the cardiology and ophthalmology models, could inform planning for continuing education or marketing programs.
The extent to which the characteristics we identified explained the variability in the referral patterns is reflected by the favorable c statistics of the multivariate logistic models. Values of this measure can range from 0.5 (no explanatory ability over chance alone) to 1.0 (complete explanation).28 The values of 0.66 to 0.79 that we found in this study indicate that given two randomly chosen primary care physician encounters in the data set, the one more likely to be followed by a referral could be identified 66% to 79% of the time.27
The next consideration concerns the validity of the methods. A unique aspect of this study was the cohort approach to inpatient and outpatient medical claims, incorporating information from a one-time survey of primary care physicians. The data were arranged chronologically, so that referrals which followed a particular encounter were influenced only by diagnoses assigned up to and including that encounter. This approach addressed a common flaw in using retrospective risk-adjustment methods. Another strength was the use of models based on GEE to adjust for the influence of individual physicians on patient outcomes. This approach was relevant because some variables that appeared to be statistically significant in the logistic regression models became less significant when using the GEE models, indicating that some erroneous conclusions could occur if only models based on logistic regression were used.
There are, however, limitations to our approach. First, claims data can inaccurately reflect clinical reality.29 Further, the level of explanation achieved is certainly due in part to the inclusion and exclusion criteria. Study patients had the same HMO insurance for at least 2 years, were cared for by the same IPA, and had symptoms or diagnoses that had not been present in the previous year. These criteria permitted a more focused evaluation of referral patterns for patients with recent onset of symptoms, but meant that only 10% to 20% of patients with these diagnoses were studied. Whether or not these findings would apply to patients whose problems are more chronic is not known. Furthermore, the relatively small number of referrals raises the possibility that the models were overfit.
Even with strict inclusion criteria, about one fourth of the encounters were with a primary care physician other than the patient's assigned physician. This introduced another source of variability in the data. Although holding the assigned physician responsible for the referral is consistent with common administrative practices, the activity of other physicians should not be ignored when profiling individual physicians.
The diagnostic categories were broad, particularly in the case of the orthopedic model. These categories were originally developed to explain overall medical expenditures, not referral patterns specifically. Broad grouping could result in the loss of discriminatory information. Even then, the potential for confounding referral practices with the assigned diagnoses was not eliminated. For example, a physician with a low threshold for diagnosing coronary artery disease in patients with chest pain might be expected to refer to cardiologists more frequently than a physician who would code similar presentations as nonspecific chest pain. At present, there does not appear to be an easy solution to this problem. A conservative interpretation of the diagnostic categories is that they reflect physicians' undererlying uncertainty and propensity for referral, as well as the condition of a patient. Future studies will be required to clarify the optimal way to define diagnostic groups and to separate variability due to physicians from variability due to patients.
The physician survey yielded a counterintuitive finding that increasingly satisfied primary care physicians were less likely to refer patients to pulmonologists. This finding could have been due to nonresponse bias, because encounters with primary care physicians who did not complete survey items regarding this specialty were also less likely to result in referrals. This finding could also have resulted from a survey process that was not anonymous, from changing levels of physician satisfaction over time, or the possibility that pulmonary disease specialists had educated some primary care physicians to handle certain pulmonary problems without referral (which could decrease referrals and increase satisfaction). It is not known if important physician-level factors were missed or imperfectly measured by the survey.
No one-time assessment of a complex subject such as physician referral practices will identify all important factors. Also, further studies will be needed to address methodologic concerns. The challenge will be to perform these investigations in a way practicing physicians can tolerate. From our experience, we believe the basic strategy of combining a physician survey with a careful analysis of claims data holds promise for an iterative approach to understanding referral patterns.
Acknowledgments
The authors gratefully acknowledge the assistance of E. Francis Cook, ScD, and Gregory Levine; and the helpful comments of Paul N. Valenstein, MD, and Sankey V. Williams, MD.
REFERENCES
- 1.Kerr EA, Mittman BS, Hays RD, Siu AL, Leake B, Brook RH. Managed care and capitation in California: how do physicians at financial risk control their own utilization? Ann Intern Med. 1995;123:500–4. doi: 10.7326/0003-4819-123-7-199510010-00004. [DOI] [PubMed] [Google Scholar]
- 2.Eisenberg JM. The internist as gatekeeper. Ann Intern Med. 1985;102:537–43. doi: 10.7326/0003-4819-102-4-537. [DOI] [PubMed] [Google Scholar]
- 3.Wilkin D, Smith A. Explaining variation in general practitioner referrals to hospitals. Fam Pract. 1987;4:160–9. doi: 10.1093/fampra/4.3.160. [DOI] [PubMed] [Google Scholar]
- 4.Lasker RD, Shapiro DW, Tucker AM. Realizing the potential of practice pattern profiling. Inquiry. 1992;29:287–97. [PubMed] [Google Scholar]
- 5.Warren BH. An outcomes analysis approach to utilization management: quality assessment of appropriateness of specialty referrals. Utilization Management. 1994;9:34–8. doi: 10.1177/0885713X9400900107. [DOI] [PubMed] [Google Scholar]
- 6.Cochrane RA, Singhal H, Monypenny IJ, Webster DJT, Lyons K, Mansel RE. Evaluation of general practitioner referrals to a specialist breast clinic according to the UK national guidelines. Eur J Surg Oncol. 1997;23:198–201. doi: 10.1016/s0748-7983(97)92220-4. [DOI] [PubMed] [Google Scholar]
- 7.Nutting PA, Franks P, Clancy CM. Referral and consultation in primary care: do we understand what we're doing? J Fam Pract. 1992;35:21–3. [PubMed] [Google Scholar]
- 8.Franks P, Clancy C. Referrals of adult patients from primary care: demographic disparities and their relationship to HMO insurance. J Fam Pract. 1997;45:47–53. [PubMed] [Google Scholar]
- 9.Salem-Schatz S, Moore G, Rucker M, Pearson SD. The case for case-mix adjustment in practice profiling. JAMA. 1994;272:871–4. [PubMed] [Google Scholar]
- 10.Weiner JP, Starfield BH, Steinwachs DM, Mumford LM. Development and application of a population-oriented measure of ambulatory care case-mix. Med Care. 1991;29:452–72. doi: 10.1097/00005650-199105000-00006. [DOI] [PubMed] [Google Scholar]
- 11.Welch HG, Black WC, Fisher ES. Case-mix adjustment: making bad apples look good. JAMA. 1995;273:772–3. Comment. [PubMed] [Google Scholar]
- 12.Verrilli D, Welch HG. The impact of diagnostic testing on therapeutic interventions. JAMA. 1996;275:1189–91. [PubMed] [Google Scholar]
- 13.Langley GR, Minkin S, Till JE. Regional variation in nonmedical factors affecting family physicians' decisions about referral for consultation. Can Med Assoc J. 1997;157:265–72. [PMC free article] [PubMed] [Google Scholar]
- 14.Roland M, Morris R. Are referrals by general practitioners influenced by the availability of consultants? BMJ. 1988;297:599–600. doi: 10.1136/bmj.297.6648.599. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Webb S, Lloyd M. Prescribing and referral in general practice: a study of patients' expectations and doctors' actions. Br J Gen Pract. 1994;44:165–9. [PMC free article] [PubMed] [Google Scholar]
- 16.Langley GR, MacLellan AM, Sutherland HJ, Till JE. Effect of nonmedical factors on family physicians decisions about referral for consultation. Can Med Assoc J. 1992;147:659–66. [PMC free article] [PubMed] [Google Scholar]
- 17.Grol R, Whitfield M, De Maeseneer J, Mokkink H. Attitudes to risk taking in medical decision making among British, Dutch and Belgian general practitioners. Br J Gen Pract. 1990;40:134–6. [PMC free article] [PubMed] [Google Scholar]
- 18.Langley GR, Tritchler DL, Llewellyn-Thomas HA, Till JE. Use of written cases to study factors associated with regional variations in referral rates. J Clin Epidemiol. 1991;44:391–402. doi: 10.1016/0895-4356(91)90077-m. [DOI] [PubMed] [Google Scholar]
- 19.Calman NS, Hyman RB, Licht W. Variability in consultation rates and practitioner level of diagnostic certainty. J Fam Pract. 1992;35:31–8. [PubMed] [Google Scholar]
- 20.Coulter A, Noone A, Goldacre M. General practitioners' referrals to specialist outpatient clinics. BMJ. 1989;299:304–8. doi: 10.1136/bmj.299.6694.304. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Hanchak NA, Schlackman N, Harmon-Weiss S. US Healthcare's quality-based compensation model. Health Care Finance Rev. 1996;17:143–59. [PMC free article] [PubMed] [Google Scholar]
- 22.McArdle PJ, Whitnall M. The referral practice of general medical practitioners to the surgical specialties: implications for the future. Br J Oral Maxillofac Surg. 1996;34:394–9. doi: 10.1016/s0266-4356(96)90094-7. [DOI] [PubMed] [Google Scholar]
- 23.Elixhauser A, Andrews RM, Fox S. Clinical Classifications for Health Policy Research: Discharge Statistics by Principal Diagnosis and Procedure, Division of Provider Studies Research Note 17, Agency for Health Policy and Research. Rockville, Md: Public Health Service; 1993. AHCPR publication 93-0043. [Google Scholar]
- 24.Elixhauser A. Software and User's Guide. Healthcare Cost and Utilization Project (HCUP-3), Research Note 2. Rockville, Md: Public Health Service; 1996. Clinical Classifications for Health Policy Research, Version 2. AHCPR publication 96-0046. [Google Scholar]
- 25.Cowen ME, Dusseau DJ, Toth BG, Guisinger C, Zodet MW, Shyr Y. Case mix adjustment of managed care claims data using the Clinical Classification for Health Policy Research (CCHPR) method. Med Care. 1998;36:1108–13. doi: 10.1097/00005650-199807000-00016. [DOI] [PubMed] [Google Scholar]
- 26.McEvoy GK. AHFS 97 Drug Information. Bethesda, Md: American Society of Health-System Pharmacists; 1997. [Google Scholar]
- 27.Harrell Fe, Jr, Califf RM, Pryor DB, Lee KL, Rosati RA. Evaluating the yield of medical tests. JAMA. 1982;247:2543–6. [PubMed] [Google Scholar]
- 28.Harrell Fe, Jr, Lee KL, Matchar DB, Reichert TA. Regression models for prognostic prediction: advantages, problems, and suggested solutions. Cancer Treat Rep. 1985;69:1071–7. [PubMed] [Google Scholar]
- 29.Jollis JG, Ancukiewicz M, DeLong ER, Pryor DB, Muhlbaier LH, Mark DB. Discordance of databases designed for claims payment versus clinical information systems. Ann Intern Med. 1993;119:844–50. doi: 10.7326/0003-4819-119-8-199310150-00011. [DOI] [PubMed] [Google Scholar]