Risk assessment in diabetes management: how do general practitioners estimate risks due to diabetes?

Abstract

Objectives

To evaluate the ability of general practitioners (GPs) in Germany to estimate the risk of patients with diabetes developing complications.

Methods

An interview study using a structured questionnaire to estimate risks of four case vignettes having diabetes‐specific complications within the next 10 years, risk reduction and life expectancy potential. A representative random sample of 584 GPs has been drawn, of which 150 could be interviewed. We compared GPs' estimates among each other (intraclass correlation coefficient (ICC) and Cohen's (multirater‐) κ) and with risks for long‐term complications generated by the multifactor disease model “Mellibase”, which is a knowledge‐based support system for medical decision management.

Results

The risk estimates by GPs varied widely (ICC 0.21 95% CI (0.13 to 0.36)). The average level of potential risk reduction was between 47% and 70%. Compared with Mellibase values, on average, the GPs overestimated the risk threefold. Mean estimates of potential prolongation of life expectancy were close to 10 years for each patient, whereas the Mellibase calculations ranged from 3 to 10 years.

Conclusions

Overestimation could lead to unnecessary care and waste of resources.

Adequate care for people with chronic illnesses such as diabetes mellitus has become an increasing challenge for healthcare systems all over the world. In Germany, the proportion of people with diabetes is already high and is expected to increase in the near future. Most of the health and economic burden as well as the loss of quality of life associated with the disease can be ascribed to the development of late diabetic complications.

Therefore, the proper assessment of the patient's prognosis plays a central role in the management of diabetes. Disease management programmes, which aimed to achieve a substantial improvement in the care of patients with diabetes, request the determination of individual therapeutic goals by doctors and the introduction of adequate treatment regimens based on the best available medical evidence.

Only little is known about the ability of doctors to estimate individual health risks of their patients with diabetes, but existing studies support the assumption that there are deficits. Young‐Hyman et al¹ found that only 44% of care givers perceived the weight of infant patients to be a potential health problem, despite the fact that 57% of the children were obese and 12% super‐obese. Walker et al² showed that nearly 50% of doctors who were at a higher risk themselves reported an optimistic bias that they were less likely to develop diabetes than other people of the same age and sex.

The importance of a doctor's proper assessment of the risks of diabetes is unquestionable. For example, risk factors for cardiovascular diseases in patients with type 2 diabetes often remain untreated, despite the fact that the benefits of interventions are well established.³,⁴,⁵ Moreover, there is some evidence that a proper perception of risk factors by the patient can support health‐related behaviour, although some results in this field are contradictory.⁶,⁷,⁸,⁹ In this regard, personalised risk assessment and communication to the patient was shown to be more effective than generalised patient information.¹⁰,¹¹ Therefore, to introduce adequate treatment, as well as proper risk assessment and perception of risk reduction potential is essential to GPs. The aim of this study was to evaluate the ability of doctors to estimate the risk of patients with diabetes developing complications.

Methods

This study was conducted as a structured telephonic interview. GPs received (by fax/email) case vignettes of four patients with diabetes (table 1) approximately 1 week in advance. Approval by an ethical committee was not required because no real patient data were used.

Table 1 Characteristics of the case vignettes.

	Patient 1	Patient 2	Patient 3	Patient 4
Age (years)	46	62	53	52
Sex	Male	Male	Female	Female
Type of diabetes	Type 2	Type 2	Type 1	Type 2
Duration of diabetes since diagnosis (years)	Newly diagnosed	12	33	4
BP status	Hypertonic, treated with antihypertensives (systolic BP: 150 mm Hg)	Systolic BP: 190 mm Hg	Hypertonic, treated with antihypertensives (systolic BP: 170 mm Hg)	Hypertonic, treated with antihypertensives (systolic BP: 170 mm Hg)
Smoking habits	Smoker (20 cigarettes/day)	Smoker	Non‐smoker	Non‐smoker
Screening for diabetic eye disease	None performed	Performed without finding	Performed without finding	Performed without finding
Screening for diabetic kidney disease	None performed	Performed without finding	Microalbuminuria	Microalbuminuria
HbA1c value (%)	13.0	6.7	9.3	12.5
Blood lipid status
Total cholesterol (mg/dl)	274	205	136	203
HDL‐cholesterol (mg/dl)	25	47	44	31
Triglycerides (mg/dl)	226	129	101	715
Other conditions	No other pre‐existing cardiovascular disease known	No blood glucose self‐monitoring	Blood glucose self‐monitoring, four times daily	No blood glucose self‐monitoring

BP, blood pressure; HbA1c, glycosylated haemoglobin; HDL, high‐density lipoprotein.

In the interviews (January–February 2004), GPs were asked to estimate the probability of the occurrence of five diabetes‐related complications within the next 10 years, the proportion by which these risks may be reduced by permanently adjusting the patients to national guideline values and, correspondingly, the number of years by which the average life expectancy may be prolonged by such an intervention.¹² Additionally, characteristic factors with a potential impact on the prognostic ability of GPs such as sex, age, years of experience with patients with diabetes, proportion of patients with diabetes among total patients and diabetes‐related qualifications were also examined. Furthermore, GPs were asked to rate their certainty in prognosis on a scale from “very certain” to “very uncertain”.

Based on data of the Regional Associations of Statutory Health Insurance Physicians, a list of all 55 238 eligible GPs was prepared. Of these, a random sample of 584 (1.1%) GPs was drawn. In all, 231 (39.6%) agreed to take part, but of these 81 (13.9%) missed the time to complete the interview. The number of completed interviews was therefore 150 (25.7%).

We first compared doctors' assessments to study their agreement. The intraclass correlation coefficient (ICC) combines measures for the ability to differentiate between cases (4 vignettes×5 complications each = 20 cases) and agreement per case between raters. The ICC has a range from 0 to 1, higher values indicating better agreement. Values >0.7 are commonly accepted as “fair” agreement.¹³ As a second measure of agreement, Cohen's (multirater) κ was used.¹⁴ However, raw data were too sparse for the calculation of κ and data had to be dichotomised (risk assessment > or <50%). Analysis was performed using SPSS V.12.0 and an SPSS‐Macro by Nichols (nichols@spss.com) based on Siegel and Castellan.¹⁵

We then compared GPs' assessments with risks for long‐term complications generated by the multifactor disease model “Mellibase”.¹⁶ The model uses complex stochastic Markov processes to model probabilities of progression (transition probabilities) dependent on physiological input values of individual patients with diabetes (age, sex, haemoglobin A1c (HbA1c), blood pressure, cholesterol levels, smoking, and duration and history of illness). The model incorporates actual findings of published studies after assessing their methodological quality. It is based on a summary of data representing epidemiological and clinical evidence from studies such as the UK Prospective Diabetes Study and many other prospective clinical trials, together with Diabetes Register data and meta‐analyses appraising diabetes treatments as well as associations between HbA1c levels and a range of microvascular and macrovascular events. It is also based on data from the Framingham Heart Study, which assesses relationships between lipid profile and coronary heart disease.¹⁷,¹⁸

For estimates of the potential of individual risk reduction, the achievement of targets (normal range) for metabolic factors as recommended by the German national guideline for care of diabetes mellitus type 2 was assumed, while keeping other individual input factors constant.¹² Table 2 shows the Mellibase estimates for risks of complications as well as the potential risk reduction (relative risk reduction) for the four patients.

Table 2 Estimates for the 10‐year risk for complications and potential for risk reduction (calculated by Mellibase).

	Patient 1		Patient 2		Patient 3		Patient 4
	Risk (%)	Relative risk reduction (%)	Risk (%)	Relative risk reduction (%)	Risk (%)	Relative risk reduction (%)	Risk (%)	Relative risk reduction (%)
Myocardial infarction	34.9	76.4	32.2	50.3	13.4	54.1	24.9	67.9
Amputation	26.6	81.7	21.4	70.2	6.9	93.1	9.8	77.1
Blindness	7.1	96.7	0.3	21.0	4.9	65.3	5.0	96.8
Renal failure	0.6	41.3	0.8	3.2	22.7	22.0	11.5	0.0
Stroke	8.9	59.1	24.0	71.9	8.8	22.1	9.1	32.4

To find out whether agreement between doctors' estimates and Mellibase‐calculated estimates is influenced by doctors' characteristics, we performed an analysis of variance using a multivariate repeated‐measures model. Repeated measures represented deviations between doctors' estimates and Mellibase‐calculated estimates for the 20 cases.

Results

The sample comprised 60 (40.0%) female GPs and 72 (48.0%) GPs aged <50 years of age. Professional experience was <20 years in 58 (39.2%) GPs, 20–39 years in 86 (58.1%) GPs and ⩾40 years in 4 (2.7%) GPs. In all, 15 (10.0%) GPs had received specific training in diabetes care. A total of 73 (49.7%) GPs reported up to 10% cases of diabetes among their patients, 62 (42.2%) GPs had 11–30% and 12 (8.2%) GPs had >30%.

The overall ICC for risk rating was 0.21 (95% CI 0.13 to 0.36; based on 147 raters; three partially missing). Analysis of the complication‐specific ICCs showed results even below the ICC for overall rating. Among them, the highest agreement was found for risk assessment of renal impairment (0.16 (95% CI 0.06 to 0.73)). Even lower agreement was observed for risk assessment of leg amputation (0.13 (95% CI 0.04 to 0.68)), blindness (0.12 (95% CI 0.04 to 0.66)) and myocardial infarction (0.08 (95% CI 0.02 to 0.55)). Agreement only slightly above zero was observed for the rating of stroke (0.03 (95% CI 0.01 to 0.33)). All ICC values were far below the accepted threshold of 0.7 for “fair agreement”. This finding was supported by a value of 0.09 for the overall κ statistic, which also indicates a very low degree of agreement.¹⁹

The interviewed doctors generally highly overestimated patients' risks of developing late diabetic complications (p<0.001, t test), with only one exception (amputation, patient 2; table 3).

Table 3 Patients' risk: general practitioners' assessment in comparison with Mellibase‐calculated values (one‐sample statistics; t test).

	n	Doctors' estimates	SD	SEM	Calculated value (MB)	Overestimation by doctors against MB values (%)	Mean difference (test–reference)	95% CI of the difference	p Value
Patient 1
Myocardial infarction	150	61.580	25.542	2.086	34.9	76	26.680	22.559 to 30.801	0.000*
Amputation	149	38.195	25.705	2.106	26.6	44	11.595	7.433 to 15.756	0.000*
Blindness	149	30.872	25.307	2.073	7.1	335	23.773	19.676 to 27.869	0.000*
Renal failure	149	35.664	25.432	2.086	0.6	5844	35.064	30.947 to 39.182	0.000*
Stroke	150	48.653	25.220	2.059	8.9	447	39.753	35.684 to 43.822	0.000*
Patient 2
Myocardial infarction	149	42.946	24.604	2.016	32.2	33	10.746	6.763 to 14.729	0.000*
Amputation	149	18.023	18.745	1.536	21.4	−16	−3.377	−6.411 to −0.342	0.029**
Blindness	149	13.621	16.216	1.329	0.3	4440	13.321	10.696 to 15.946	0.000*
Renal failure	149	17.148	18.566	1.521	0.8	2044	16.348	13.342 to 19.353	0.000*
Stroke	149	38.389	24.966	2.045	24.0	60	14.389	10.347 to 18.431	0.000*
Patient 3
Myocardial infarction.	150	54.533	27.353	2.233	13.4	307	41.133	36.720 to 45.546	0.000*
Amputation	150	36.420	25.892	2.114	6.9	428	29.520	25.343 to 33.697	0.000*
Blindness	150	32.360	25.695	2.098	4.9	560	27.460	23.314 to 31.606	0.000*
Renal failure	150	42.833	28.646	2.339	22.7	89	20.133	15.512 to 24.755	0.000*
Stroke	150	48.367	26.840	2.192	8.8	450	39.567	35.236 to 43.897	0.000*
Patient 4
Myocardial infarction	150	52.220	24.821	2.027	24.9	110	27.320	23.315 to 31.325	0.000*
Amputation	149	31.537	23.528	1.928	9.8	222	21.737	17.928 to 25.546	0.000*
Blindness	150	27.780	23.770	1.941	5.0	456	22.780	18.945 to 26.615	0.000*
Renal failure	150	37.640	26.866	2.194	11.5	227	26.140	21.805 to 30.475	0.000*
Stroke	150	47.780	26.573	2.170	9.1	425	38.680	34.393 to 42.967	0.000*

MB, Mellibase.

*p<0.01, highly significant; **p<0.05, significant.

The median level of overestimation was more than threefold. Figure 1 illustrates the obvious mismatch between GPs' risk assessments and calculated values.

Figure 1 Percentage probabilities of risks (bars to the right) and risk reduction (bars to the left) for each complication: general practitioners' estimates (mean ±(1.96*SEM)) and calculated values.

According to the GPs, the potentials for risk reduction by permanently adjusting the patient to national guideline values were, on average, similar for all types of complications under consideration (fig 1). Moreover, there were only small differences in the average level of potential risk reduction between the patients (47–70% overall). In contrast with the GPs' estimation, Mellibase calculations revealed well‐structured patterns of risk reduction probabilities, which were characteristic for each patient (fig 1). As a result, the potentials for risk reduction were in part substantially overestimated as well as underestimated by the GPs when compared with the calculated values (t test, p<0.001, except stroke in patient 1 and myocardial infarction in patient 2). Accordingly, most of the two‐sided error bars in fig 1 did not overlap with the bars representing the calculated values.

The GPs consistently overestimated the potential prolongation of life expectancy on average (p<0.001). Mean estimates were close to 10 years for each patient, whereas Mellibase calculations ranged from 3 to 10 years (fig 2).

Figure 2 Avoidable reduction in patients' life expectancy: general practitioners' estimates (mean ±(1.96*SEM)) and calculated values.

Regarding perception of confidence, the estimates of those doctors who were “very uncertain” or “rather certain” diverged significantly more from Mellibase‐calculated values (p<0.05). All other factors characterising the GPs did not show an influence, not even the duration of professional experience or specialisation in diabetes. However, 50.6% of the GPs assessed themselves as “rather uncertain” or “very uncertain” with their prognoses.

Conclusions

The present study has shown that estimates for risks of patients with diabetes of developing late diabetic complications and their opportunities to prevent them by GPs in Germany are varied. Individual estimates as provided by the GPs generally covered the whole range of possible values (no risk up to 100% risk). On the other hand, there was uniformity in estimates of probabilities to prevent those complications (about 50% in every single situation). Therefore, it seems likely that the GPs' estimates were not or were only weakly guided by a common concept. However, considering the obvious complexity of such multifactorial risk estimates, the results are not surprising. They might rather demonstrate the limits of human intellectual capability to deal with uncertainty in complex situations.

Compared with estimates from the evidence‐based risk estimation system, doctors' risk estimates were generally much too high. The smallest degree of overestimation was nearly 100% for myocardial infarction but more than 10‐fold for renal failure. The ability to estimate possible risk reductions by adjusting patients to guideline levels was also poor, but oscillated between underestimation and overestimation because the doctors' estimates were always around “50%”. Further the GPs also overestimated the potential prolongation of life expectancy as a result of adjusting patients to guideline values. Half the GPs were uncertain with their prognoses and the analysis showed no differences between subgroups of doctors.

This study is limited to participation rate, which among German GPs is, in general, low. However, 39.6% of the sample intended to participate. We cannot comment on the direction of the selection bias because we did not conduct a non‐responder analysis and non‐responders did not provide personal data. We assume that the responders were more interested in evidence‐based medicine than non‐responders. Therefore, the results are assumed to overestimate the GP's ability to estimate risks.

However, with regard to our findings, the question on the impact of quality and outcome of care must be raised: what does this mean for treatment decisions aiming to prevent long‐term complications? One could argue that no major problem results because GPs nearly always overestimate the risks of complications. This might lead to more awareness and therefore to more instead of less activity, which in turn would not endanger care for the patient. However, it could lead to unnecessary care and waste of resources. The relatively uniform expectation that the risk experiencing complications will be reduced “by half” regardless of the specific situation could be an indicator of the perception that they cannot influence the process very much. Both the communication of high risk and the resulting polypragmasia do not support patients' motivation and compliance.

Furthermore, the study results should encourage policy makers, who assume that pay‐for‐performance policies are simple to implement, to reconsider their strategies. First, they should know that there is a wide variation between patients in interference of risk by medical doctors. Second, they should recognise the delimited clinical and financial benefits of disease management programmes for many patients affected with chronic diseases.

A systematic risk assessment is the cornerstone of any evidence‐based medical approach in the prevention of long‐term complications. Hofer et al²⁰ found that interventions of doctors and of specialists are not consistent with clinical trial evidence. The reason often is a biased risk perception. In our study too, neither experience nor specialisation of the doctors had any influence on the accuracy of their risk estimates; hence knowledge management tools might help improve the situation. Even though different risk assessment tools such as the reference system Mellibase or, for example, the UK Prospective Diabetes Study Risk Engine (limited to type 2 diabetes) predict slightly different risks, it is likely that their risk estimates will be more precise than those of GPs (http://www.accu‐chek.de/mellibase/de/content/accu_chek_mellibase/accu_chek_mellibase.html;http://www.dtu.ox.ac.uk). Therefore, as stated in the Introduction section, knowledge management tools and an accompanying detailed communication of individual risks will improve patients' health behaviour and their preventive efforts. Among the questions to be raised, it will be of great importance to know whether knowledge‐based support of risk estimations can improve effectiveness and efficiency in healthcare. Further studies should investigate potential improvements in therapeutic decisions, patient motivation and compliance, as well as treatment results when using such tools.

Abbreviations

HbA1c - haemoglobin A1c

ICC - intraclass correlation coefficient