Cost effectiveness of continuing professional development in health care: a critical review of the evidence

Continuing professional development (CPD) for healthcare professionals is an important strategic instrument for improving health. The Department of Health identifies CPD as a way of maintaining standards of care; improving the health of the nation; and recruiting, motivating, and retaining high quality staff.¹ To this end, direct NHS spending on continuing professional development in 1999-2000 was about £1bn ($1.6bn).²,³ If we regard CPD as any method to improve health professionals' skills the total resources devoted to it are probably much greater, particularly with the recent increased participation in response to the need for recertification and revalidation.⁴ To ensure the maximum gain from participation in CPD, these resources must be used efficiently.

To assess the efficiency of participating in CPD, economic criteria are needed. Resources for health care are scarce, and money spent on CPD could otherwise be used for direct patient care. These opportunity costs are explicitly considered in the economic methods of cost benefit analysis and cost effectiveness analysis. The literature contains various reviews of cost effectiveness analysis in both health care⁵,⁶ and education.⁷ Such articles explain why cost effectiveness analysis (or another method of economic evaluation) is essential and how such evaluation should be undertaken, and they clearly define the set of economic terms (such as cost benefit analysis, cost effectiveness analysis, rate of return, and opportunity cost) that need to be incorporated into this type of research. Casebeer et al highlighted the need for economic evaluation of CPD activities,⁸ but they emphasised the use of cost benefit analysis, which requires monetary values to be assigned to measures of effectiveness. Cost benefit analysis is generally used to ascertain whether an intervention should be undertaken. Cost effectiveness analysis is used to decide which interventions (out of a number of alternatives) should be undertaken.

However, cost effectiveness analysis in education research is rare.⁷,⁹,¹⁰ This is partly because of limited training for researchers, antipathy toward (economic) analysis that might constrain policy, and the dearth of significant results in studies of educational effectiveness.⁹ The quality of such research is also often poor: Clune found that only 1% of 541 “cost-effectiveness” studies of elementary and secondary education between 1991 and 1996 could be considered reliable, with strong design and analysis.¹⁰ In contrast, economic evaluation of healthcare technologies is increasing, and the methods for making such analysis are rapidly evolving.¹¹ (Even here, however, critical reviews identify a substantial number of weak cost benefit and cost effectiveness analyses.^12–14)

Summary points

• Continuing professional development for healthcare professionals must be cost effective to avoid a waste of resources, and economic studies of such interventions must therefore be of sufficient quality and quantity to allow conclusions to be drawn

• A systematic search for suitable economic studies found only nine studies, of varying quality and with ambiguous use of economic terms

• The review indicates that the evidence base simply does not allow any empirical conclusions to be drawn about the economic value of continuing professional development

• More cost effectiveness studies are urgently required, and greater attention must be paid to ensuring that methods of evaluation and analysis are appropriate

There is a sizeable literature on the effectiveness of CPD interventions (over 100 randomised controlled trials are thoroughly reviewed by Davis et al¹⁵,¹⁶), but the evidence on the cost effectiveness of CPD has not been systematically investigated. In this article we therefore investigate the quantity and quality of the evidence on the cost effectiveness of CPD for healthcare professionals.

Methods

Search strategy

The aim of our search strategy was to identify evaluations of CPD interventions that included some form of economic analysis. We searched the main bibliographic databases (Medline, CINAHL, Web of Science, BIDS, ERIC, University of York Centre for Reviews and Dissemination, and the Research and Development Resource Base in CME at the University of Toronto) from their inception and without any language restrictions. We then searched relevant papers for additional references and attempted to identify grey literature (via internet searches, for example). The key words we used in the search included one term relating to CPD and education (“[continuing] professional development,” “[continuing professional] medical education,” and “postgraduate medical education”) and one relating to costs (“costs,” “efficiency,” “cost-benefit,” and “cost-effectiveness”).

Inclusion criteria

We included studies in the review if they considered an educational intervention of CPD undertaken by health professionals. There are several different definitions of CPD, but we adopted an inclusive definition—“post-registration acquisition of skills or knowledge by healthcare professionals.” This definition includes formal programmes as well as use of guidelines, mentoring, and independent study.

We included any outcomes that measured the costs and effectiveness of the intervention, although we excluded papers described as costs studies and cost saving exercises that failed to report the cost of the intervention. Neither type of study allows an economic appraisal of the effectiveness of the intervention. We applied no methodological restrictions, such as restricting our review to studies that used randomised or controlled experimental techniques.

Review method of selected studies

We reviewed the studies in terms of their educational aims, the mode of education, and the results, and we evaluated each for the appropriate use of economic methods. Specifically, we used the six criteria of Udvarhelyi et al,¹² which were designed to assess health technologies. The economic methods for evaluating health technologies and educational interventions are sufficiently similar that this approach is legitimate.

Results

Results of search

Our search revealed nine papers, of which seven were cost effectiveness studies and two were cost benefit analyses (table 1).^17–26 This is obviously a meagre crop of literature with limited scope for generalisation.

Table 1.

Results of literature search for economic analyses of continuing professional development in health care

Study	Educational aim (effectiveness measure)	Intervention arms	RCT	Measure(s) of effectiveness	Method of representing cost effectiveness	Most cost effective arm
Studies reported as cost effectiveness analyses
Gomel et al (1998)17	Screening or treatment of alcoholism	Training with no support, minimal support, or maximal support	Yes	No of patients screened or advised	Average cost per outcome	Training plus no support
Kaner et al (1999)18	Screening or treatment of alcoholism	Guidelines alone, with training, or with training and support	Yes	No of patients screened or advised	Average cost per outcome	Guidelines with training and support
Modell et al (1998)19 Beech et al (1998)20	Screening for haemoglobin disorders	Nurse led training plus posters	Yes	No of screening tests	Cost per additional outcome	No comparison possible
Morrison (1999)21	Treatment of infertility	Guidelines with workshops and practice visits	Yes	No of pregnancies	Cost per additional outcome	No comparison possible
Morriss et al (1998)22	Treatment of somatised mental disorder	Seminar training	No	Reduction in costs or No of cases treated	Cost per additional outcome %	No comparison possible (but positive rate of return)
Steele et al (1989)23	Reduction in outpatient prescribing costs	Outreach visit with peer comparison feedback	Yes	Reduction in costs	Costs saved	No comparison possible (but positive rate of return)
Stevens et al (1997)24	Screening for cervical cancer	Face to face outreach plus materials	Yes	No of screening tests	Average cost per general practitioner	No comparison possible
Studies reported as cost benefit analyses
Rutz et al (1992)25	Treatment of depression	Group meetings	No	Costs saved from treating depression	Rate of return	Positive
Prashker and Meenan (1991)26	Specialty training	Rheumatology and gastroenterology v general internal medicine	No	Salary benefits from specialty training	Rate of return	Positive for gastroenterology, negative for rheumatology

RCT=randomised controlled trial. 

Results of review

Table 2 shows the results of our appraisal of the studies. On average, the studies met about half of the six criteria (a similar result to that reported by Udvarhelyi et al,¹² where the average healthcare technology study met three criteria). Specifically, most studies failed to apply a full appraisal of costs (probably the most important criterion), discounting, or sensitivity analysis, and only five studies provided a summary measure of cost effectiveness.

Table 2.

Quality assessment^* of economic analyses of continuing professional development in health care

Study	Statement of agency (who gets benefit or who pays)	Description of benefits of intervention	Full specification of costs	Use of discounting	Use of sensitivity analysis	Calculation of summary measure of efficiency	Total score (out of 6)
Gomel et al (1998)17	Yes	Yes	Yes	No	No	Yes	4
Kaner et al (1999)18	Yes	Yes	No	No	No	Yes	3
Modell et al (1998)19 Beech et al (1998)20	Yes	Yes	No	No	No	Yes	3
Morrison (1999)21	Yes	Yes	No	No	No	Yes	3
Morriss et al (1998)22	Yes	Yes	Yes	No	No	Yes	4
Steele et al (1989)23	Yes	Yes	No	No	No	No	2
Stevens et al (1997)24	Yes	Yes	No	No	No	No	2
Rutz et al (1992)25	Yes	Yes	Yes	No	Yes	No	4
Prashker and Meenan (1991)26	Yes	Yes	No	Yes	Yes	No	4
Total	9	9	3	1	2	5	3.2

^*Based on assessment method of Udvarhelyi et al.¹² 

Two of the papers applied a cost benefit analysis, allowing rates of return to the education programme to be calculated, although neither paper offered such calculations.²⁵,²⁶ Rates of return allow the (discounted) money costs and benefits of an intervention to be compared. To calculate the rate of return, the excess of benefits over costs is divided by the costs of the intervention. For these two studies, the rates of return were 39%²⁶ and greater than 10 000%.²⁵ We could also calculate rates of return for two of the cost effectiveness studies: these were 460% and 63%.²²,²³ Although these rates may be accurate for these particular interventions, they seem extremely high when compared with the base interest rate used by the Bank of England (currently 4%, which includes a premium that adjusts for inflation). Moreover, one of these studies²⁶ did not investigate what most health professionals would consider CPD; instead, it used the human capital model of education²⁷ to compare the economic gain arising from two specialties.

Ideally, the evidence base should allow for comparison of the results of papers that identify the same healthcare outcome. This is possible for four of the nine studies across two different healthcare outcomes. Given the appraisal in table 2, however, any inference drawn from these comparisons can be only tentative. Two studies compared the cost effectiveness of different training and support conditions for increasing screening for and treatment of alcohol addiction.¹⁷,¹⁸ The results are conflicting, as the first study found training with no physician support to be the most cost effective mode whereas the second found the training with support to be most cost effective.

The studies by Modell et al¹⁹,²⁰ and Stevens et al²⁴ addressed increasing the use of screening services. Modell et al found that effectiveness increased after formal general practitioner education, but so did costs. Knowledge of the availability of funding (or a cost benefit analysis) is therefore required to ascertain whether the education should be implemented. Stevens et al found no significant difference in effectiveness between the control and intervention groups (with increased costs for the intervention arm because of outreach visits). In this instance, the additional expenditure was clearly not cost effective. However, Modell et al included the resource implications (costs of additional screenings) in their analyses, whereas Stevens et al did not.

Finally, two papers seemed to use an inappropriate method of economic analysis. Morriss et al calculated the rate of return to seminar based education (460%), as the main aim of the study was to reduce direct healthcare costs.²² While the study itself offers a calculation of cost effectiveness, the aim of the research and lack of a comparison educational mode suggest that it should be reclassified as a cost benefit analysis.

Steele et al tested two different educational interventions to determine which was the most cost effective at reducing outpatient prescribing costs.²³ Given that the outcomes were therefore measured in monetary units (dollars saved), a cost benefit approach is appropriate, as cost effectiveness cannot be calculated. Although a rate of return to the outreach intervention can be calculated (peer comparison feedback showed no difference in prescribing costs and thus had a negative rate of return), the cost of the intervention was not fully included. Any rate of return estimate would therefore overstate the benefit of the intervention.

Conclusions

We reviewed the literature that assesses the cost effectiveness of CPD interventions in health care. Our literature search shows that economic evaluations of CPD are rare. Furthermore, the evidence that does exist is not consistent in its approach to costing or analysis. This leaves no scope for a full systematic review of a particular educational intervention, and a single trial of an intervention does not allow decisions to be made on the strength of the results. Overall, it is impossible to draw any feasible conclusions regarding the cost effectiveness of different modes of CPD for healthcare professionals.

The external validity of the existing studies may be impaired for many reasons. This raises concerns over the extent to which the results can be generalised and used to inform policy. Both costs and technologies change over time, along with differences in input prices and technologies across settings (and especially across countries). Also, the studies mainly focused on intermediate rather than final outcomes: any effects of CPD on patient health therefore have to be inferred from changes in the behaviour of physicians.

Interpreting the results is difficult since uniform methods of costing or analysis were not applied. The “ingredients” included in the costs analysis were not identified in a standard form: development costs and the opportunity costs of the participants needed to be included. In addition, the resource implications of an educational intervention (such as for additional screenings) would need to be included in a cost benefit analysis but not in a cost effectiveness analysis. Often, the evidence seemed to be directed at both forms of analysis, despite these analyses being methodologically and purposively distinct.

Many of the studies were inadequate to meet the objective of deciding which is the most cost effective mode of CPD. The aim of cost effectiveness analysis is to aid decisions between interventions, and thus studies need to test the relative cost effectiveness of two or more alternatives. Several of the studies compared just one educational mode with a control group of standard care. Were monetary values attached to the outcomes and a cost benefit analysis applied, a decision could be made on whether the intervention should be implemented. Otherwise, a value judgment is required to determine if, say, the cost per additional screening represents value for money. When two or more educational designs are compared cost effectiveness can be more readily understood if the educational goal is held constant. The cost effectiveness analysis will then discriminate between the types of CPD.

Implications of our study

The literature on the effectiveness of CPD continues to expand, but effectiveness is not a sufficient criterion for implementation. For scarce resources to be devoted to CPD, the relative cost effectiveness of different educational interventions must be established, and those offering the most value for money must be implemented. An investment in high quality evaluations would therefore reap health benefits for the public and ease policy makers' decisions about resource allocation. Cost effectiveness analysis must be applied to studies of educational effectiveness, and this should be possible with the methods detailed by Levin and McEwan⁷ or Drummond and Jefferson.⁵ At present, notwithstanding the substantial resource commitment to CPD, evidence on the cost effectiveness of CPD is completely inadequate.