Physicians’ Tacit and Stated Policies for Determining Patient Benefit and Referral to Cardiac Rehabilitation

Abstract

Background

The benefits of prescribing cardiac rehabilitation (CR) for patients following heart surgery is well documented. However physicians continue to underutilize CR programs and disparities in the referral of women are common. Previous research into the causes of these problems has relied on self-report methods which presume that physicians have insight into their referral behavior and can describe it accurately. In contrast, the research presented here employed clinical judgment analysis (CJA) to discover the tacit judgment and referral policies of individual physicians.

Purpose

The specific aims were to determine 1) what these policies were, 2) the degree of self-insight that individual physicians had into their own policies, 3) the amount of agreement among physicians, and 4) the extent to which judgments were related to attitudes toward CR.

Method

Thirty-six Canadian physicians made judgments and decisions regarding 32 hypothetical cardiac patients, each described on five characteristics (gender, age, type of surgical procedure, presence/absence of musculoskeletal pain, and degree of motivation) and then completed the 19 items of the Attitude towards Cardiac Rehabilitation Referral instrument.

Results

There was wide variation among physicians in their tacit and stated judgment policies. Physicians exhibited greater agreement in what they believed they were doing (stated policies) than in what they actually did (tacit policies). Nearly one-third of the physicians showed evidence of systematic, and perhaps subliminal, gender bias as they judged women as less likely than men to benefit from CR. Correlations between attitude statements and CJA measures were modest.

Conclusions

These findings offer some explanation for the slow progress of efforts to improve CR referrals and for gender disparities in referral rates.

The decision to refer patients to cardiac rehabilitation (secondary prevention) programs following coronary revascularization procedures is vital to their recovery and sustained health. Coronary heart disease is the leading cause of morbidity and mortality world-wide (Roger et al., 2012). Research on cardiac rehabilitation (CR) programs reveals incontrovertible evidence of their morbidity and mortality benefits for patients (Goel, Lennon, Tilbury, Squires, & Thomas, 2011; Heran et al., 2011; Oldridge, Fischer, & Rimm, 1988; Smith et al., 2011; Suaya, Stason, Ades, Normand, & Shepard, 2009). Despite being internationally-endorsed (Balady et al., 2011; Balady et al., 2007; Corra et al., 2010; Piepoli et al., 2010; Smith et al., 2011), CR programs are grossly underutilized by providers; referrals of eligible patients continue to hover around only 18% (Chan et al., 2010). Several studies have shown troubling gender disparities in the referral of women into CR programs (see Benz Scott, Ben-Or, & Allen, 2002, for review). The question of why there is a gender bias influencing women’s ability to receive a referral to CR remains largely unexplored. The current study examines how physicians’ judgments of patient benefit and referral decisions are made and specifically examines the role that patient gender plays.

Previous research examining why physicians underutilize CR programs has relied on survey methods (e.g., Grace, Evindar, Abramson & Stewart, 2004; Grace, Grewal, & Stewart, 2008; Suter, Bona, & Suter, 1992) that presume that doctors have insight into their referral behavior and can describe it accurately. In contrast, the research described in this article is based on the methods of social judgment theory (Brehmer & Joyce, 1988) that derive implicit or tacit decision policies, in the form of statistical weights, by analyzing decisions made over a large number of hypothetical cases in which cues (patient characteristics) are varied. When social judgment theory methods have been applied in medical contexts, they are also referred to as clinical judgment analysis (CJA). A principle advantage of the CJA approach in medical decision-making is that tacit policies are inferred from actual choice behavior and so investigators are less dependent upon physicians’ self-report.

CJA has been used successfully to study judgment and decision-making in cardiovascular health, specifically physicians’ diagnostic decisions regarding chronic heart failure (Skaner, et al. 2000), judgments of patient risk for acute heart failure (Skaner, Strender & Bring, 1998), decisions to prescribe lipid-lowering agents and judgments of coronary heart disease (Dhami & Harries, 2001; Harries, Evans, Dennis, & Dean, 1996), accommodation decisions for stroke patients at discharge (Unsworth & Thomas,1993), medical students’ and residents’ assessments of cardiac risk (Tape & Wigton, 1989), and more recently, nurse practitioners’ judgments of patient risk for coronary heart disease (Beckstead & Stamp, 2007; Stamp, 2012). Until now however, CJA has not been applied to judgment and decision-making regarding CR. Key findings, common among studies employing CJA, are 1) there is considerable disagreement among providers when evaluting the same patients; providers tend to focus differentially on various patient characteristics, and 2) providers have rather modest levels of self-insight into their own judgment and decision-making processes.

A related line of research has established that attitudes can moderate judgments and decisions. Individuals holding more extreme attitudes toward an object (i.e., a type of person, event, behavior, etc.) tend to emphasize the differences between classes of such objects and to highlight the similarities within classes more so than do individuals holding more moderate attitudes (for reviews, see Hamilton, 1968, and Haslam & Turner, 1995). This type of interaction has become known as the accentuation effect. Beckstead (2005) demonstrated how the accentuation effect can alter performance in CJA tasks. Building on this work, we speculated that the attitudes that physicians hold regarding CR might be systematically related to their judgment and decision policies. We therefore included the 19 items of the Attitude towards Cardiac Rehabilitation Referral (ACRR) scale developed by Grace, Evindar, Abramson and Stewart (2004) to assess the potential accentuating effects of these attitudes.

In the present study we set out to discover the tacit and stated policies for judging patient benefit and for making CR referrals of each member of a sample of Canadian physicians. The specific aims are to determine 1) what these policies were, 2) the extent to which different physicians agreed and disagreed with one another, 3) the degree of self-insight that individual physicians had into their own judgment and decision-making as measured by the correspondence between their tacit and stated policies, and 4) the extent to which judgments and decisions were related to physicians’ attitudes toward CR.

Method

Physician Participants

Family physicians and cardiac specialists listed in the Canadian Medical Directory Online who treat patients indicated for CR were sent an email describing the study and inviting them to participate. A convenience sample of 51 physicians responded to an online questionnaire.

Measures

The online questionnaire contained the clinical judgment task, the ACRR, and basic demographic questions (gender, years of practice, and area of practice specialty). Details of the components are described below.

Clinical judgment analysis task

An orthogonal set of 32 patient profiles was constructed using a 2 × 2 × 2 × 2 × 2 factorial arrangement of five patient characteristics: the patient’s gender, age (55 years vs. 75 years), type of cardiovascular procedure (coronary bypass surgery, or percutaneous coronary intervention, PCI), the presence or absence of musculoskeletal pain, and either high or low interest/motivation to participate in CR. (Note that this design provides a balanced assessment of the influence of the patient’s gender in that each of the 16 male patient profiles had a corresponding female counterpart, identical on all other cue values.) Four profiles, selected at random, were repeated to assess reliability. Each physician was presented with the same 36 profiles (including 4 duplicates) in the same order.

For each profile physicians were instructed to indicate “How likely is it that this patient will benefit from CR?” using a 100-point scale, presented in increments of 5, with anchors: 0 = “no chance of benefit”, and 100 = “will definitely benefit”, and then asked “Would you refer this patient for cardiac rehabilitation, yes or no?” Although admittedly limited, we assessed the validity of these responses by asking physicians “Overall, how realistic were the patient descriptions you read?” on a 0 – 10 scale ranging from “not at all realistic” to “completely realistic”.

Assessment of tacit policies

Tacit judgment policies were obtained by applying multiple linear regression to each physician’s judgments of patient benefit. Tacit referral policies were obtained applying discriminant function analysis to their dichotomous referral decisions. Our initial intent had been to obtain tacit and stated policies for both, judgments of benefit, and referral decisions from each physician for comparison, however because 16 physicians indiscriminately referred all 32 patients, this was not possible. Instead we focus on judgments of patient benefit, and where possible comment on referral decisions. Analysis proceeded on an individual, or idiographic, basis. The relative influence of the cues was determined from standardized regression coefficients (βs). In order for these weights to be comparable across physicians, they were normalized by dividing each physician’s βs by the sum of the absolute values of their five βs and multiplying by 100. The multiple correlation coefficient from each of these regression models (labeled R_S in the parlance of CJA) quantifies cognitive control, or the degree to which the judge consistently applies his or her tacit policy. Individuals may have quite different polices (as reflected by different patterns among their cue weights), yet be similar in terms of the consistency with which each applies his or her policy (i.e., they may have similar R_S values). These features of CJA are of particular value in the study of medical decision-making because the tacit policies of individual physicians may be captured, assessed for consistency, and compared with those of other physicians making it possible to measure cognitive sources of agreement and disagreement within the medical community.

Assessment of stated policies

After judging the patient profiles, physicians were shown a list of the five cues used. They were then asked to indicate how important each cue was as they had formed their judgments, by allocating 100 points among the five cues. They were instructed to assign the most points to the cue(s) they relied on the most. These point allocations were treated as the explicit, or stated policies of the physicians, that is, as what each individual believed to be the relative influences that had determined his or her judgments. These stated policies were thus elicited by directly asking the physicians which cues they had used on the task they had just performed. The degree of correspondence with the tacit (statistical) weights is then a direct measure of self-insight on the task.

Assessment of attitudes towards cardiac rehabilitation referral

The online questionnaire also contained the 19-item ACRR scale (Grace, Evindar, Abramson & Stewart 2004; Grace, Grewal & Stewart, 2008). The ACRR assesses physician’s attitudes and beliefs about the efficacy of CR, referral norms, ease of the referral process, and the desire to manage patients independently. Response options were: 1 = strongly disagree, 2 = agree, 3 = neutral, 4 = agree, and 5 = strongly agree.

Procedure

Ethical considerations

Prior to data collection, the study was approved by the university’s institutional review board for research. Participants were assured of anonymity and confidentiality. No personally identifying information was collected. All data were stored in password protected files.

Data collection and analysis

The online questionnaire was constructed using Qualtrics Labs, Inc., online software (Version 12, www.qualtrics.com). Embedded in the questionnaire was JavaScript that allowed us to the measure response times (in msecs) which were used as a quality-control check on the data obtained. Participants logged into a secure server hosting the online questionnaire using their own computers located in their offices or homes. Data were collected between February and June of 2012. All analyses were conducted using SPSS version 19. Prior to aggregating any correlations or standardized regression coefficients for group comparisons, Fisher’s r-to-Z arc-sine transformation was applied; aggregated values were then transformed back to original metric for presentation. When assessing relationships between variables across individuals, Spearman’s correlation coefficient ρ (rho) was employed unless otherwise specified.

Results

Sample characteristics and data quality

Fifty-one online questionnaires were submitted. Forty-one of these had complete data. Three were excluded due to no variability in judgments (i.e., all 32 patient profiles were rated 100, and referred to CR), leaving 38 for analysis. Inspection of response time measures revealed that two individuals took less than 60 seconds to read and respond to the entire set of profiles and so were excluded from substantive analyses. Results are reported for 36 physicians, 21 (58%) of whom were general practitioners, eight (22%) specialized in cardiology, three (8%) were internists, one was an anesthesiologist, one an occupational health specialist, and two who did not indicate their specialty. Their average number of years in practice was 22.2, and the standard deviation was 10.4. Fifteen (42%) were female. With each of 36 physicians making benefit judgments and referral decisions for 32 patient profiles, our sample contains 2,304 responses for analysis. The time to complete the questionnaire ranged from 9 to 38 minutes, with an average of 16 minutes.

Before considering judgment policies, self-insight, and agreement among physicians, we address the quality of our data. First, reliability was estimated for each physician by correlating his or her judgments over the four duplicate cases. The mean of this test-retest correlation was reasonable (r = .830), however because it was based on only four duplicate pairs per physician, its 95% confidence interval was quite wide (−.869 to .999). Second, validity of the judgments obtained is supported by the fact that physicians rated the patient profiles as being realistic (M = 8.03, SD = 1.81, min = 4, max = 10).

There was considerable variability among physicians as they applied the judgment scale. Some used the full range (0 to 100) when judging the likelihood of patients benefitting from CR, while the majority used only a portion of it (responses for each physician are summarized in Table 1). The average judged likelihood of benefitting from CR across the entire sample was 66.1 and the average referral rate was 85%, with 16 physicians failing to differentiate among patient profiles (i.e., referring all 32). The correlation between average judgment of benefit and referral rate over physicians was .433, p = .008, suggesting that physicians who viewed CR as being more beneficial (on average, for the sample of 32 patients) tended to refer more of these patients. The correlation between standard deviations of benefit judgments and referral rates was −.501, p = .002, indicating that more discriminating physicians made fewer referrals. Male and female physicians did not differ in their average benefit judgments (64.1 and 68.3, respectively) nor in their referral rates (81% and 90%, respectively). The eight cardiac specialists did not differ from the other 28 physicians in their average benefit judgments (63.2 and 66.9, respectively) nor in their referral rates (81% and 86%, respectively). Neither was years in practice significantly related to these measures (−.040 and −.299, respectively).

Table 1.

Summary of Benefit Judgments and Referral Decisions.

Physician	Minimum	Maximum	Mean	SD	Referrals
1	40	100	63.1	21.8	100%
2	50	90	72.7	9.8	100%
3	60	100	92.8	11.4	100%
4	50	85	69.1	12.1	100%
5	50	100	77.0	17.3	88%
6	0	100	50.8	26.8	25%
7	5	85	48.0	37.5	100%
8	55	80	71.7	7.0	100%
9	65	85	73.8	6.5	100%
10	30	80	57.3	16.5	75%
11	45	100	70.9	19.4	91%
12	50	90	73.9	12.6	88%
13	20	100	49.5	27.7	78%
14	35	80	53.1	12.2	59%
15	50	90	65.0	12.7	100%
16	15	100	64.5	24.0	69%
17	60	100	80.9	10.1	72%
18	55	90	76.9	12.3	100%
19	5	90	46.7	26.8	78%
20	30	100	65.9	22.8	100%
21	20	85	52.2	18.3	72%
22	50	90	73.1	10.8	100%
23	50	90	68.1	9.7	81%
24	50	85	65.3	8.4	100%
25	35	90	72.7	12.6	97%
26	0	100	55.3	29.1	56%
27	65	100	83.6	10.3	81%
28	70	90	80.2	5.7	100%
29	45	85	66.7	12.6	94%
30	50	90	81.1	9.7	100%
31	15	85	56.3	25.1	66%
32	15	50	27.0	9.4	100%
33	80	100	90.0	9.8	100%
34	15	85	48.9	28.2	50%
35	65	90	81.7	7.8	97%
36	30	80	53.9	12.1	41%
Mean	39.6	90.0	66.1	15.7	85%
SD	21.3	9.9	14.0	8.0	20%

Note: Mins, maxs, means and standard deviations (SD) are over 32 likelihood (of benefit) judgments made by each physician (0 to 100 scale). Referrals is the number of patients referred out of 32 × 100. Physicians 29 through 36 are cardiac specialists.

Finally, before proceeding to examine cue weights, self-insight, and agreement, the appropriateness of the proposed linear model for describing tacit policies was assessed for each physician. The R_S values (i.e. the multiple correlation coefficients from the regression of each physician’s judgments on to cues) averaged .797 and ranged from .370 to .988, suggesting that the tacit judgment policies were adequately modeled for most physicians. Table 2 displays the R_S values for each physician. There were no significant differences in average R_S values between the eight cardiac specialists and the other 28 physicians, nor between male and female physicians; nor was years in practice related to R_S values.

Table 2.

Measures of Consistency, Insight and Agreement on Benefit Judgment Task.

Physician	RS	RI	ra	rstated	rm
1	.941	.938	.597	.667	.747
2	.886	.803	.508	.438	.652
3	.731	.435*	.336	.657	.566
4	.946	.897	.560	.627	.701
5	.943	.908	.568	.677	.703
6	.619	.602	.093	.236	.166
7	.944	.873	.559	.644	.710
8	.854	.829	.521	.685	.727
9	.831	.721	.450	.647	.621
10	.918	.840	.560	.588	.718
11	.807	.783	.517	.643	.736
12	.811	.784	.264	.415	.419
13	.849	.667*	.464	.361	.634
14	.810	.719	.309	.383	.459
15	.932	.911	.550	.667	.716
16	.756	.404*	.352	.233	.605
17	.667	.394	.091	.638	.221
18	.898	.372*	.516	.073	.679
19	.937	.886	.539	.481	.686
20	.910	.772*	.545	.502	.702
21	.908	.823	.532	.617	.692
22	.370	.323	.049	.218	.141
23	.808	.666	.429	.368	.632
24	.849	.712	.529	.490	.736
25	.702	.602	.385	.449	.677
26	.854	.598*	.539	.357	.726
27	.723	.443*	.197	.588	.288
28	.598	-----	.288	-----	.582
29	.831	.735	.499	.473	.707
30	.592	.537	.261	.640	.511
31	.922	.882	.595	.643	.749
32	.526	.481	.236	.438	.637
33	.977	.949	.597	.667	.731
34	.988	.950*	.623	.630	.747
35	.488	.282	.123	.420	.322
36	.558	.497	−.208	−.217	−.469
Mean	.797	.686	.405	.487	.572
SD	.155	.201	.196	.200	.248

Note: R_S is the consistency with which a physician applied his/her judgment policy, R_I is an index self-insight.

^*indicates R_I was significantly lower than R_S (p < .05). r_a is averaged agreement between a physician’s judgments and those made by all other physicians. r_stated is averaged agreement between a physician’s stated policy and those of the other physicians. r_m is averaged agreement between a physician’s tacit policy and the tacit policies of the other physicians. Physicians 29 through 36 are cardiac specialists.

Objective judgment weights (tacit policies)

Normalized cue weights for each physician are shown in Table 3. The magnitude of a cue weight shows the amount by which the judged likelihood of a patient benefitting from CR changed (on the 0 to 100 scale), when the cue was present in the patient. Positive values indicate an increase in judged benefit while negative values indicate a decrease. A value near zero indicates that the cue had little influence on the judgments. These tacit weights varied considerably among the sample of physicians.

Table 3.

Tacit Policy Cue Weights from Benefit Judgment Task.

Physician	Gender	Age	Procedure	Pain	Motivation
1	−6	−24	0	2	68
2	−5	−29	−13	−9	44
3	−12	−37	10	12	29
4	−2	−9	8	−23	58
5	−5	−33	3	8	51
6	−5	−21	54	−16	4
7	8	−6	−8	−5	73
8	−14	−2	−6	−2	76
9	13	−16	−3	−23	45
10	−2	−24	−10	3	61
11	−3	−13	3	−15	66
12	−5	22	23	19	31
13	−2	−3	0	34	61
14	−28	12	23	9	28
15	−14	−7	−3	−16	60
16	−9	4	15	27	45
17	−15	−44	26	−13	2
18	0	7	−13	−7	73
19	−1	−11	14	22	52
20	−5	0	−3	−20	72
21	5	−26	7	−14	48
22	−14	52	0	10	24
23	5	−28	−19	−2	46
24	−9	19	−6	−6	60
25	−5	−25	25	−1	44
26	−11	−21	−3	−11	54
27	−7	−65	−7	7	14
28	−41	−2	8	−8	41
29	1	−24	−5	−14	56
30	−2	13	−19	−22	44
31	−7	−24	2	1	66
32	−19	12	13	9	47
33	5	−5	5	−5	80
34	−6	−11	0	5	78
35	7	7	−12	44	30
36	14	9	37	−9	−31
Mean	−5.4	−9.8	4.1	−0.8	47.2
SD	10.6	21.6	15.6	16.0	23.9

Note: Cue weights are transformed standardized regression coefficients expressed relative to the sum of the absolute values of the five coefficients × 100. Cue coding key, Gender: female = 1, male = 0; Age: 1 = 75 yrs, 0 = 55 yrs; Procedure: 1 = by-pass, 0 = PCI, Pain: 1 = present, 0 = absent; Motivation: 1 = high, 0 = low. Physicians 29 through 36 are cardiac specialists.

Patient motivation was the most influential cue for 29 (81%) physicians, all of whom showed positive weights (M = 56.0, SD = 13.9) indicating that they had judged highly-motivated patients as more likely to benefit from CR than patients described as having low motivation. By contrast, the valences (+ or −) of the other cues were mixed and so the averaged values (shown in Table 2) should be interpreted with caution. Of particular interest, 27 physicians (75%) had negative weights (M = −9.4, SD = 8.9) for the patient gender cue indicating that most had judged female patients as less likely to benefit from CR than the male patients who were matched on the other cues. Only eight physicians (22%) had positive weights (M = 7.3, SD = 4.4) on the gender cue; one physician had a weight of zero. Twenty-six physicians (72%) had negative weights (M = −20.4, SD = 14.4) for the patient age cue indicating that they had judged older patients as less likely to benefit than younger patients and 10 (28%) had positive weights (M = 13.8, SD =14.7) indicating that they had judged older patients as more likely to benefit than younger ones. Twenty-one physicians (58%) judged patients with musculoskeletal pain as less likely to benefit (M = −11.4, SD = 7.0) from CR than patients without pain, while 15 physicians (42%) judged patients with pain as more likely to benefit (M = 14.1, SD = 12.7). The type of cardiac procedure cue elicited the most even split among physicians as they judged patient benefit. Fifteen physicians (42%) had negative cue weights (M = −8.6, SD = 5.5) indicating that they had judged bypass patients as less likely to benefit than patients who underwent PCI, while 17 physicians (47%) had positive cue weights (M = 16.3, SD = 13.9) indicating that they had judged bypass patients as more likely to benefit than PCI patients; three physicians had weights of zero on this cue.

Subjective judgment weights (stated policies) and self-insight

The subjective weights obtained via the point allocation procedure represent a physician’s stated policy; their degree of correspondence with the objective regression weights provides a direct measure of self-insight. This correspondence was examined in three ways. First, the correlation among the five pairs of cue weights was calculated for each physician. As the stated weights were always positive quantities (i.e., point allocations) they were correlated with the absolute values of the tacit weights. The mean of these correlations was r = .625, and they ranged from −.458 to .944. The wide range among these correlations is not surprising given that each is calculated on only five pairs of values.

Second, the stated cue weights were used in place of the regression weights to predict the 32 judgments previously made by each physician. When self-insight is perfect, that is when a judge actually uses the cues as he or she described using them, the multiple correlation coefficient (R_I) from this procedure will be equal to R_S (the multiple correlation coefficient obtained from the regression of his or her judgments onto the cues). For this analysis stated cue weights were assigned the same sign (+ or −) as their corresponding tacit weights. These R_I values averaged .686 and ranged from .282 to .950 (see Table 2 for all 36 values). R_I values did not differ between male and female physicians, nor between cardiac specialists and non-cardiac physicians, nor were they correlated with years in practice. For all physicians these values were smaller than their corresponding R_S indicating that no individual had perfect self-insight into exactly how he or she had used the cues in the judgment task. A paired-sample t-test confirmed that the average R_I was significantly less than the average R_S (t₍₃₄₎ = 8.169, p = .001). Tests of significance (one-tailed z-tests) on each pair of correlations revealed that R_I values were significantly lower than R_S values (ps < .05) for eight (22%) of the physicians (these are indicated with asterisks in Table 2).

Finally, self-insight was assessed by comparing the stated and tacit policy weights in cases where one or the other, but not both, was zero. Fifteen physicians indicated in their stated policies that they placed zero weight on at least one cue. Eleven of these physicians stated that patient gender had played no role in their judgments of patient benefit, yet all eleven had nonzero tacit weights for gender, and of particular interest, seven had negative weights indicating that they had judged female patients as less likely to benefit than males. Four stated that patient age had not influenced their judgments, but all four had negative tacit weights indicating that they had judged older patients as less likely to benefit than younger patients. Nine physicians stated that the type of surgical procedure did not play a role when judging benefit, however all nine had nonzero tacit weights for this cue. There was a near even split on the valence of this cue; five physicians had judged bypass patients as less likely to benefit than PCI patients, and four had judged bypass patients as more likely to benefit. There were no differences on the other two cues. Only five physicians stated that they had used cues for which their tacit weights were zero. Three of these physicians inaccurately allocated points to the type of procedure cue, one to the gender cue, and one to the age cue.

Agreement among physicians

Examining tacit and stated judgment policies

Agreement among physicians over their 32 judgments of patient benefit was determined in two ways, by calculating the pairwise Pearson correlations, r_a, across patient profiles, and by using intraclass correlation coefficients (ICC, Shrout & Fleiss, 1979). With 36 physicians, there were 630 pairwise correlations for analysis. These correlations among physicians’ judgments averaged .405 and ranged from −.369 to .948. The average pairwise correlation (r_a) for each physician’s judgments with those of the other 35 physicians is shown in Table 2. Physician #34 shows the highest value (.623) indicating that her judgments were most representative of the entire sample, while Physician #36 has a negative value (−.208) indicating that his judgments were the most discrepant in the sample. Interestingly, both these physicians were cardiac specialists.

The ICC index quantifies agreement within the sample as a whole; as employed here it indicates how interchangeable two physicians selected at random from our sample would be. Two types of ICCs are relevant; the ICC(3, 1) which assesses agreement without regard to mean differences among physicians was .376, and the ICC(2, 1) which penalizes for mean differences was .232.

Similar measures of agreement were calculated for physicians’ stated policies. One physician did not allocate points among the cues so analyses were based on only 595 pairwise correlations. The average correlation among physicians’ stated policies was r_stated = .487 and ranged from −.814 to 1.0. The average of the 35 pairwise correlations for each physician are shown in Table 2. A t-test comparing the average correlation among stated policies with that among actual judgments was significant (t₍₅₉₄₎ = 3.162, p = .002) indicating that there was more agreement among stated policies than among actual judgments. The ICCs for stated policies corroborated this difference: ICC(3, 1) = .511 and ICC(2, 1) = .366. In summary, the overall agreement among physicians was modest, although there was significantly more agreement in what they believed they were doing (i.e., among their stated policies) than there was in what they actually did (i.e., among their judgments).

Examining cognitive sources of disagreement

According to social judgment theory, the degree of agreement between two judges, r_a, is the degree to which the two judges successfully match their regression equations (r_m), weighted by the product of the intra-judge (R_S) consistencies (Naylor & Schenck, 1966). If a judge’s regression equation (tacit policy) is used to obtain predicted values of the judgments previously made, and the same is done for a second judge using his or her regression equation, r_m is the correlation between these two sets of predicted judgments and represents the agreement in their tacit policies, corrected for attenuation due to the inconsistency within each judge’s responses. For two judges, the relationship among their agreement, maximum possible agreement, and consistencies is then r_a12 = r_m12 × R_S1 × R_S2. Indeed if both judges formed their respective policies and used them with perfect consistency in making their judgments, inter-judge agreement (r_a) would be policy agreement (r_m).

We calculated r_m for all 630 pairs of physicians; these ranged from −.766 to .999 with an average of .572. The average of these correlations for each physician is shown in Table 2. While r_a shows how much the judgments made by a given physician agreed with those made by all of the others, r_m shows how much more agreement is theoretically possible if all physicians had applied their tacit policies with complete consistency. In other words, even after adjusting for inconsistencies within physicians, the degree of agreement between physicians is still far from perfect.

Relationships of judgments and decisions to attitudes

Correlations between responses to the 19 statements comprising the ACRR and measures from the CJA task were calculated. All attitude statements were scored such that higher values indicated greater agreement. Most of these correlations were small to moderate and very few reached statistical significance. For each CJA measure we report the range of correlations with attitude statements and highlight only those correlations that reached significance (p < .05).

Physicians’ attitudes were systematically related to their tacit weights of the patient motivation cue, but not to the weights of the other four patient characteristics. Responses to two attitude statements correlated with motivation cue weights: Clinical practice guidelines promote referral to CR, and My department/practice generally refers all eligible patients to CR as a standard of care (.346, p = .038, and .359, p = .032, respectively) indicating that physicians who strongly endorsed these statements were those who had placed the largest emphasis on patient motivation as they had judged how likely patients were to benefit from CR. Correlations with the 17 remaining attitude statements were nonsignificant and ranged from −.263 to .329. The ranges of correlations with the other tacit cue weights were: patient gender (range: −.318 to .327), patient age (range: −.303 to .235), type of cardiac procedure (range: −.283 to .293), and musculoskeletal pain (range: −.253 to .167).

Responses to only one statement, I can prescribe an exercise regimen for my patients myself, were significantly correlated (.379, p = .023) with the R_S values suggesting that physicians who were more consistent in their judgments of patient benefit may also be more confident when prescribing exercise to their patients. The range of correlations with the other statements was −.154 to .158. None of the responses to the attitude statements correlated significantly with R_I values (range: −.153 to .251).

Referral rates, calculated for each physician, were positively correlated with responses to two statements: My colleagues generally refer patients to CR, and My department or practice generally refers all eligible patients to CR as a standard of care (.429, p = .009, and .513, p = .001, respectively) suggesting that social context may exert some influence on referral rates. In contrast, referral rates were negatively correlated with responses to the statements I am not familiar with the CR programs in my area, and I prefer to manage my patients secondary prevention myself (−.364, p = .029; and −.361, p = .031; respectively) suggesting that efforts to familiarize physicians with local CR programs might increase referral rates, and that some physicians may experience the act of referring their patients to CR programs as somehow negative, perhaps viewing it as a form of relinquishing some control or independence. Responses to the remaining statements were not significantly correlated with referral rates (range for other statements: −.309 to .171). Mean judgments of benefit (calculated for each physician over 32 patients) were not significantly correlated with responses to any of the attitude statements (range: −.229 to .229).

Discussion

Comparison to other CJA studies

This investigation of physicians’ judgments of patient benefit and referral decisions using CJA has revealed findings broadly compatible with previous research using this method (Beckstead & Stamp, 2007; Harries, et al., 1996; Skaner, et al., 1998; Wigton, 1988; 2008). Wide variation among providers in their tacit and stated judgment policies, a fairly modest level of self-insight into their performance, and low agreement between physicians were found. Our findings are consistent with those reported in these previous studies and confirm that physicians are not always able to clearly articulate exactly how they are forming judgments or making clinical decisions.

The self-insight index R_I averaged .686 in our sample of physicians, which although a bit higher than the average .620 reported by Kirwan, Chaput de Saintogne, Joyce, Holmes, and Currey (1986) in their study of rheumatologists’ judgments of arthritis, is still far from ideal. One possible explanation for the higher self-insight in our sample is that the judgment task involved only five cues, whereas Kirwan et al., employed ten. As tasks involve more cues, it is conceivable that judges would have more information to keep track of and so self-insight might suffer. In a task involving eight cues Beckstead and Stamp (2007) found self-insight averaged only .510 among advanced nurse-practitioners when they had judged patient risk for heart disease, which is consistent with this explanation.

The modest degree of self-insight in our sample of physicians as they judged likelihood of patient benefit is an important finding. Of the eleven physicians who stated that gender had played no part in their judgments, seven (64%) had judged female patients as less likely than males to benefit from CR. Furthermore, seven of the eight physicians who showed the lowest self-insight (indicated in Table 2) also showed negative weights for the gender cue (only one of these is among the group of seven just mentioned). Thus, some physicians (some 37% of our total sample) may have been unaware that they had systematically judged female patients as less likely than male patients to benefit from CR. To the extent that such a subliminal gender bias may be at play among some physicians in practice settings, this would explain (in part) the gender disparity frequently seen in CR referrals (Benz Scott, et al, 2002).

The overall agreement among the physicians’ judgments was modest (r_a = .405), however it is not dissimilar to that found in other studies of physicians. For instance, Smith, Gilhooly, and Walker (2003) reported that agreement among physicians prescribing treatments for depression was .470 and Unsworth and Thomas (1993) found agreement of .530 among physicians and other rehabilitation specialists regarding their accommodation decisions for stroke victims. The modest agreement observed among our physicians can be attributed to two cognitive sources: imperfect consistency within individual physicians as they made their 32 judgments, and genuine differences among their judgment policies. Although our finding of modest agreement among physicians is nothing new to CJA researchers, ours is the first study (to our knowledge) which demonstrates empirically that physicians exhibit greater agreement in what they believed they were doing than in what they actually did. This finding has implications for task forces and other work groups attempting to create clinical guidelines via expert consensus.

Comparison to other methods for studying CR referral

The phrase “referral rate” has been used in different ways by different authors which makes comparisons across studies confusing if not impossible. First of all a rate is, by definition, some fraction or proportion of a given total, and as such, the choice of numerator and denominator in the calculation are crucial for determining the meaning of the resulting rate.

Second, and most importantly for understanding our findings, “referral rate” may be interpreted to mean three very different things: (1) most simply, the number of patients who are referred relative to the total number who are eligible and being considered for referral, regardless of who is doing the referring; (2) the number of patients who are referred by Provider X relative to the total number of eligible patients being considered by Provider X; and (3) the number of providers who, when examining Patient Y, would refer this patient relative to the total number of providers who examined Patient Y. In case 2 we are concerned with questions such as, does Provider X have a higher (lower) referral rate than some other provider, say Provider W? In case 3 we are concerned with questions such as, if two patients, Patient Y and Patient Z, were to be examined by several providers, would Patient Y be referred by more (fewer) of them than would Patient Z?

The CJA approach used in the current study allowed us to address questions arising from cases 2 and 3 because the same 32 patients were evaluated by 36 providers, which makes our results distinct when compared with those other studies. For instance, Barber, Stommel, Holmes-Rovner and McIntosh (2001) reported that CR referral rates vary by type of provider such that patients receiving care from a cardiac specialist are more likely to be referred. This claim warrants careful scrutiny; these authors note that 36 physicians were identified as having managed the study’s 1,475 patients while in hospital, however, no data on which of these physicians had made the decision to refer (or not to refer) each patient was obtained. The data that the authors did access showed that patients who, at the time of discharge, were scheduled for post-discharge appointments with community-based cardiac specialists, were more likely to have been referred to CR at discharge (13%) compared to patients scheduled for post-discharge appointments with community-based non-cardiac physicians (6%). The original statement is somewhat misleading, implying that cardiac specialists were more likely to refer a patient than were non-cardiac physicians, when in fact the study is mute as to who actually made the referral decisions, making it an example of Case 1 described above. In the current study we found that when evaluating the same patients, eight cardiac specialists on average, referred 26/32 = 81% of patients (range: 13 to 32) while 28 non-cardiac physicians on average referred 28/36 = 86% (range: 8 to 32).

Another example comes from Grace, et al. (2004) who used an approach similar to case 3 above. One-hundred-seventy-nine physician respondents read a single patient description and were asked whether or not they would refer the patient. Seventy-nine percent of physicians said that they would refer the patient. The authors also reported “family physicians were significantly less likely to refer patients to CR than were specialists” and provided values of 65% and 96%, respectively, in support of this statement. This claim is limited as only one patient was presented to each physician for consideration. An strength of their study design was to randomly distribute two versions of their survey that differed only in whether the patient was described as male or female; however, this between-groups manipulation did not produce a significant difference in the proportion of physicians who referred the patient. In contrast, the CJA approach in the current study contributes to the literature in that it did reveal that some individual physicians had judged female patients as less likely to benefit from CR, despite a lack of evidence that this may be the case, and may have done so unconsciously.

Finally, our results speak to the limited success of efforts to improve CR referral rates. Twenty-odd years ago Suter et al. (1992) reported that nearly 40% of physicians listed “lack of patient motivation” as an important reason for not referring patients. They questioned whether the benefits of CR were being denied to many patients simply because physicians believed that patients lacked motivation and so did not bother to discuss CR with them. In the current study we found that patient motivation was the most influential cue for nearly all of our physicians as they judged the likelihood of patients benefiting from CR. Our data also showed that the physicians who judged patients as more likely to benefit tended to refer more patients.

A key predictor of a patient’s decision to participate in CR is a strong endorsement from his or her physician (Ades, Waldmann, Polk, & Coflesky, 1992; Tsui, Shanmugasegaram, Jamnik, Wu, & Grace, 2012). To the extent that a physician’s endorsement is tied to the judged likelihood that the patient will benefit, and that judged likelihood of benefit is driven by the physician’s beliefs about the patient’s motivation, it may prove fruitful for researchers and policy makers to examine more closely why it is that physicians believe patients lack motivation, and if in fact this belief is ill-founded, how best to change it. On the other hand, if this belief is shown to be accurate, effort would be better spent on the more difficult problem of increasing patient motivation.

Limitations

The current study has some limitations. First, the convenience sample of physicians was small in size and may not be representative. However, physicians, as a group, are more homogeneous with regard to their knowledge, training, attitudes, and behavior, than the general population and therefore non-response bias may not be as crucial when studying physicians as it is in surveys of the general population (Kellerman, Herold, 2001). Second, the physicians made decisions about “paper”, rather than real, patients. Although the use of written patient descriptions has been criticized, (Gorman, Clover & Doherty, 1978; Jones, Gerrity & Earp, 1990) this method is practical and has been validated for medical decision-making (Kirwan, Chaput de Saintogne, & Joyce, 1990; Peabody, Luck, Glassman, Dresselhaus, & Lee, 2000).

Conclusions

In spite of these limitations, our findings suggest that physician’s descriptions of their judgment/referral policies may not reveal how these judgments are actually made in practice, and that physicians vary considerably in their judgments of which patients will benefit most from CR. If this is generally true, it would prevent true agreement and uniformity in the selection of patients appropriate for CR. Some evidence supporting this conclusion is offered by Beckie et al. (2009) who reported in detail on the numbers of women who were initially referred to a CR program, but whom upon closer examination, were found ineligible. Of 1,681 women referred, 716 were found ineligible, and of these 372 (52%) were ineligible due to diagnostic exclusion criteria, confirming that physicians in practice do not show consensus regarding who is most likely to benefit from CR. Over the past few decades, significant effort has been expended by the American College of Cardiology Foundation and the American Heart Association to improve patient enrollment and attendance in CR programs (Drozda et al., 2011; Thomas et al., 2010) and innovative, gender-specific programs have been shown to increase attendance among women (e.g., Beckie & Beckstead, 2010), but without an appropriate referral by a healthcare provider, all other barriers to participation are immaterial.