Does the academic performance of psychiatrists influence success in the NHS Clinical Excellence Award Scheme?

Abstract

Objectives

Given the uncertainty about factors that influence receipt of Clinical Excellence Awards (CEA) and recent availability of advanced research metrics, we examined the factors that predict CEA success using a convenience sample of English psychiatrists.

Design

Observational study examining region, subspecialty, H-index, M-index, number of publications, years since registration and years in specialty.

Setting

ACCEA Nominal Roll, cross-referenced with data from the GMC's list of registered medical practitioners and Thompson's Web of Science database.

Participants

A total of 494 psychiatrists including 245 with national levels awards and a random sample with local level awards.

Main outcome measures

Receipt of local or national CEA awards in 2008 and 2009.

Results

Of those with national awards, 126 had university contracts and 119 NHS contracts. Across all staff, years since qualification in medicine and H-index were the dominant influences. For local awards we found that years worked in the specialty was the main predictor of a CEA award with a smaller contribution from H-index. For national awards to university staff (academics) years on the medical register and publication rate were significant predictors. For national awards to NHS staff (non-academics) H-index and total cites were predictive, but these were themselves related to age.

Conclusions

Progression in CEAs among psychiatrists is strongly influenced by age (years spent in specialty and years on the medical register) with an additional contribution from research productivity. Currently, research impact is crudely assessed in the CEA process. We suggest that CEA committees formally assess the impact of NHS-related research using standardized research metrics which are openly available. We also suggest that supporting organizations and local trusts adhere to the rules mandated by the ACCEA.

Introduction

Distinction Awards were introduced in May 1948 mainly to compensate consultants for the potential loss of private practice income when working for the newly formed NHS. Prior to 1990 they were generally awarded for life, with no review process, and with unclear eligibility criteria, a system was heavily criticized for inherent bias and lack of transparency.^1–3 In 1994 the Advisory Committee on Distinction Awards (ACDA) first offered to publish annual reports containing a breakdown of award holders by specialty, ethnicity and gender. In 2001 the government proposed a new Clinical Excellence Award (CEA) scheme which came into force in April 2004.⁴ The introduction of the CEA system brought some improvements; most notably clearer criteria, focus on evidence and since 2008 disclosure of award holders' personal statements. However there have been continuing difficulties with the new CEA approach. Possible issues include the undue influence of age and academic success; lack of transparency and lack of a true evidence-based approach. Guidelines suggest that applicants should ‘provide evidence’ and ‘demonstrate usage of evidence-based practice’ in four domains but submission of externally validated supportive evidence is specifically forbidden. There is also increasing concern regarding allocation of awards which has been reduced for new consultants. Nationally in the 2010 round, the Advisory Committee on Clinical Excellence Awards (ACCEA) received 1065 applications and awarded 317, down from 601 awards in 2009. Locally, financially minded trusts prefer to give as few awards as possible per year^a in small increments rather than conduct a full 0–12 dynamic evaluation of the applicant on each occasion. Notably to date there has been no review system for local awards and there is no procedure for trusts to downgrade awards. Indeed even after withdrawal of awards, pay remains protected.⁵ The main complication of this incremental approach is that over time awards will tend to favour repeated application (and hence seniority) over excellence per se. National statistics shows that a 59% of eligible NHS consultants are in receipt of a CEA, and thus considered to be performing ‘over and above contractual expectations’. Thirteen percent of eligible NHS consultants have a level 9 award or above. National data also show that the mean age for an award is 50 years for level 9 after a typical duration of 10 years as a consultant, however, it is important to note there is no age-based cut-off per se.

The current criteria for both local and national awards relate to the quality of NHS care, namely developing, delivering and managing a high quality service, contributing to the NHS through research and innovation and contributing to the NHS through teaching and training. Of these it is contributing to the NHS through research and innovation that is most measurable and less open to subjectivity in large part because advanced research metrics are easily available. Research by NHS clinicians and academics with honorary NHS contracts is extremely important to the development of improved treatments and better services in clinical practice. Thus research should certainly be contributing to the overall measure of excellence for outstanding clinicians as well as academics. Yet, fears have been raised that research productively weighs too heavily in CEA evaluation for NHS staff, leading to the generation of local rules down weighting the research and teaching domain by some NHS trusts.

In the past individuals' research productivity was difficult to measure and overly reliant upon self-report. In 1961 Garfield developed the Science Citation Index® (SCI)⁶ and its major innovation was that it included indexed references to earlier work cited in each listed paper. Later when this became available electronically (on CD-Rom in 1989 and online in the Bath Information and Data Services in 1991 and as MIMAS ISI Web of Science in 1999) it became possible to examine increasingly sophisticated measures of productivity. Such bibliometric measures of research productivity have become influential in peer review.⁷ In particular the impact factor (IF) has been widely used to rank journals but less appreciated is that these citation counts can also rank individuals and organizations.⁸ As a result research productivity is increasingly measured in terms of impact or importance. One of the most innovative metrics is the Hirsch Index or H-index.⁹ The H-index is a measure of the number of papers which have a certain citation rate. Its definition is not user friendly but it is conceptually simple (see below).¹⁰ It aims to measure both quantity (number of papers) and quality (impact, or citations to these papers as cited by others).¹¹ However there are some limitations with the H-index, most notably citations only accrue slowly over time so the H-index will favour older researchers who have published important papers some time ago. One way to facilitate comparisons between people with different lengths of careers is to divide the H-index (or citation count) by the number of years of activity (the so called M-index). Ideally this would be measured as the number of years since their first publication but could be measured by number of years in medicine or years in speciality. A second limitation of the H-index is that there is no correction for author contribution, which is whether an author is first, last or somewhere in between. Hirsch (2005) suggested it might be useful to normalise the H-index by a factor that reflects the average number of co-authors. One solution is to divide the number of citations for each paper by dividing the number of citations by the number of authors for that paper, and then calculate the H-index of the normalized citation counts.¹²

Given the underlying concerns about factors that influence CEA and recent availability of research metrics for all clinicians and academics with NHS contracts, we decided to examine to what extent academic performance predicts CEA success using a convenience sample of English psychiatrists.

Methods

We chose to look at a convenience sample of Psychiatrists in England who were recorded on either the 2008 or 2009 ACCEA database. We did not attempt to survey consultants not on the database and we could not extract older data due to incompleteness of the records. We used pooled data from the publically available ACCEA Nominal Roll (first published in full in July 2008 and updated in July 2009).¹³ We also used data from the GMC's List of Registered Medical Practitioners (also known as the ‘online register’). The GMC database provides information on a complete list of registered medical practitioners including the doctor's reference number, name, gender, year and place of primary medical degree, status on the Register, including whether the doctor holds a licence to practise, date of registration and entry on the GP/Specialist Registers. From this information we calculated the number of years in clinical practice and the number of years in psychiatric practice.

We collected data on research productivity for every consultant listed in the CEA Nominal Roll. The number of published items was examined using publications listed on Thompson Web of Science (see http://wok.mimas.ac.uk) up to January 2010. Using one database ensured uniformity of comparisons. From the same database we extracted the number of citations, which is the number of times published items had been officially cited by other Thompson Web of Science publications up to January 2010. We also used the official H-index. The H-index was automatically calculated by the Thompson Institute for Information (ISI) database and refers to N_p papers that have at least h citations each, and the other (N_p-h) papers have no more than h citations each. We attempted a correction for length of career using an adaptation of the M-index. The M-index was manually calculated by dividing the H-index by either registered years (M-index-a) or specialty years (M-index-b). A similar adjustment is possible for raw publications and number of citations. We also defined individuals with more than three publications as being ‘research active’.

In order to examine cross-sectional associations we used simple correlation analysis which shows relationships in a matrix. In order to examine the contribution of predictors to award success we treated each level of award 1–9 as a linear score and assigned 9, 10, 11, 12 to bronze, silver, gold and platinum. We used Microsoft Excel and StatsDirect 2.7.7. for analysis.

Results

Descriptive information

From the 2008 nominal roll there were 264 psychiatrists but only 249 were on the specialty register for psychiatry with complete data. There were 29 doctors in child and adolescent psychiatry, 36 in forensic psychiatry, 139 in general adult psychiatry and 45 in old age psychiatry. Subspecialty accreditations such as liaison psychiatry or substance misuse were not listed and therefore could not be analysed. Of these 249 award holders, 212 (85.1%) had local level awards. Using the 2009 data-set we extracted additional data on 284 psychiatrists who held a level 9 award or above. Two hundred and forty-five had a national award received between 1993 and 2009. Of these 245, 126 were medical academics employed by Universities and 119 were NHS clinicians many of whom had honorary contracts with Universities. Thus using non-overlapping data from 2008 and 2009 data-sets we had valid data on 494 psychiatrists currently employed in the NHS, about 14% of all 3500 practicing consultant psychiatrists. Descriptive data are shown in Table 1.

Table 1.

Univariate data by CEA level

CEA level	n	Years in specialty	Registered years	H-index (mean)	Publications (mean)	Citations (mean)
1	48	8.22	17.8	0.87	2.4	32.8
2	86	8.3	17.9	1.1	2.3	20.6
3	19	10.6	24.1	1.05	2.8	39.3
4	14	11.3	22.2	1.8	3.1	33.8
5	20	10.6	21.9	3.5	11.2	164.8
6	15	11.8	24.7	3.6	13.5	204.4
7	20	11.4	23.6	3.1	9.6	87.6
8	15	12.2	25.1	2.4	5.2	143.4
9	36	14.9	31.8	8.6	41.1	649.7
Bronze	133	12.4	24.4	7.8	30.9	537.3
Silver	55	13.3	28.4	15.6	60.3	1577.8
Gold	52	13.3	30.5	18.7	94.4	2183.4
Platinum	24	12.7	34.4	34.9	204.9	5295.5

Prediction of local CEA awards

Across all local award holders the mean number of publications listed in the Thompson ISI database was 6.4 (SD 18.3) and the mean number of citations was 105.4 (SD 489.5). We judged 30.2% to be research active by our definition. Linear correlation suggested strongest association with years in specialty and registered years (Table 2). Multiple regression showed that years in specialty and also H-index and citation rate were independently influential (Table 3). Research metrics and years in practice explained 34.7% (adjusted R²) of the variance in CEAs but this was largely due to years in specialty which had an adjusted R² of 31.7% when entered alone.

Table 2.

Correlation matrix for local level award holders

	CEA	H-index	Cites	Publications	REG_YRS	Spec_YRS	M-index A	M-index B	Cites-REG	Cited-Spec	Raw-REG	RAW-Spec
CEA	1.00
H-index	0.19	1.00
Cites	0.07	0.89	1.00
Publications	0.13	0.92	0.85	1.00
REG_YRS	0.38	0.15	0.11	0.06	1.00
Spec_YRS	0.57	0.14	0.05	0.08	0.58	1.00
M-index A	0.15	0.97	0.82	0.92	0.02	0.07	1.00
M-index B	0.11	0.96	0.89	0.89	0.09	–0.01	0.94	1.00
Cites-REG	0.06	0.91	0.99	0.89	0.07	0.03	0.87	0.91	1.00
Cited-Spec	0.04	0.86	1.00	0.82	0.09	0.01	0.79	0.88	0.98	1.00
Raw-REG	0.09	0.86	0.75	0.97	–0.04	0.01	0.92	0.84	0.83	0.73	1.00
RAW-Spec	0.08	0.90	0.86	0.98	0.01	-0.01	0.92	0.92	0.90	0.85	0.97	1.00

REG_YRS = years on the medical register; Spec_YRS = years in main medical specialty; Cites-REG = cites by years on the medical register; Cited-Spec = cites by years in main medical specialty; Raw-REG = number of publication by years in main on the medical register; RAW-Spec = number of publication by years in main medical specialty

Table 3.

Multiple regression of CEA status for local level award holders

Variable	Beta	T	Sig T
Cite_Rate	–0.39	–2.85	0.0048
H-index	0.49	3.61	0.0004
Year in medical specialty	0.51	8.94	0.0000

Multiple R = 0.60, R Square = 0.36, Adjusted R Square = 0.35, Standard error = 2.15

Prediction of national CEA awards for university employees

Across all national award holders the mean number of publications listed in the Thompson ISI database was 125.6 (SD 121.1) and the mean number of citations was 2868.3 (SD 3789.4). One hundred percent were research active. Linear correlation suggested strongest association with years on the medical register (Table 4). Multiple regression revealed two highly significant predictors (Table 5) namely years on the medical register; and publication rate. Raw number of publication was also influential. Together these explained 40.3% of the variance with the strongest effect from years on the medical register.

Table 4.

Correlation matrix for national level university employed award holders

	Award level	H-index	Cites	Publications	Medical years	Specialty years	M-index A	M-index B	Cites/Rate	Raw/Rate	Cites/spec	Raw/spec
Award level	1.00
H-index	0.35	1.00
Cites	0.32	0.94	1.00
Publications	0.35	0.86	0.83	1.00
REG_YRS	0.69	0.25	0.23	0.19	1.00
Specialty years	0.19	0.01	–0.01	0.06	0.26	1.00
M-index A	0.14	0.92	0.85	0.80	–0.09	–0.06	1.00
M-index B	0.30	0.91	0.88	0.75	0.17	–0.31	0.86	1.00
Cites/Rate	0.24	0.93	0.97	0.81	0.08	–0.03	0.92	0.87	1.00
Publications/Rate	0.22	0.80	0.75	0.94	–0.06	0.01	0.87	0.71	0.80	1.00
Cites/spec	0.27	0.89	0.97	0.74	0.19	–0.17	0.82	0.93	0.95	0.68	1.00
Publications /spec	0.34	0.83	0.80	0.94	0.14	–0.19	0.79	0.86	0.80	0.90	0.79	1.00

REG_YRS = years on the medical register, Spec_YRS = years in main medical specialty, = Cites-REG = cites by years on the medical register, Cited-Spec = cites by years in main medical specialty, Raw-REG = number of publication by years in main on the medical register, RAW-Spec = number of publication by years in main medical specialty

Table 5.

Multiple regression of CEA status for national level university employed award holders

Variable	Beta	T	Sig T
Years since registration	0.76	7.72	00001
Publication rate	0.95	3.31	0.0012
Number of Publications	–0.75	–2.58	0.0111

Multiple R = 0.63, R Square = 0.40, Adjusted R Square = 0.39, Standard error = 0.91

Prediction of national CEA awards for NHS employees

Across all national award holders the mean number of publications listed in the Thompson ISI database was 13.0 (SD 15.1) and the mean number of citations was 311.1 (SD 958.4). We judged 69% to be research active. Linear correlation was strongest for years on the medical register since qualification but multiple regression found that H-index and total cites were most influential with a smaller contribution from citation rate (Tables 6 and 7).

Table 6.

Correlation matrix for national level NHS employed award holders

	Award level	H-index	Cites	Publications	Medical years	Specialty years	M-index A	M-index B	Cites/Rate	Raw/Rate	Cites/spec	Raw/spec
Award level	1.00
H-index	0.41	1.00
Cites	0.36	0.30	1.00
Publications	0.33	0.83	0.32	1.00
REG_YRS	0.43	0.05	0.16	0.04	1.00
Specialty years	0.16	0.03	0.04	–0.04	0.26	1.00
M-index A	0.29	0.96	0.25	0.78	–0.17	–0.04	1.00
M-index B	0.37	0.97	0.29	0.81	0.01	–0.13	0.95	1.00
Cites/Rate	0.34	0.33	1.00	0.34	0.10	0.02	0.30	0.32	1.00
Publication/Rate	0.21	0.76	0.26	0.95	–0.20	–0.13	0.80	0.76	0.30	1.00
Cites/spec	0.35	0.31	1.00	0.33	0.15	0.01	0.26	0.30	1.00	0.27	1.00
Publication /spec	0.30	0.80	0.30	0.98	–0.01	–0.18	0.77	0.83	0.33	0.95	0.31	1.00

REG_YRS = years on the medical register, Spec_YRS = years in main medical specialty, Cites-REG = cites by years on the medical register, Cited-Spec = cites by years in main medical specialty, Raw-REG = number of publication by years in main on the medical register, RAW-Spec = number of publication by years in main medical specialty

Table 7.

Multiple regression of CEA status for national level NHS employed award holders

Variable	Beta	T	Sig T
Cite_Rate	–1.95	–2.42	0.0167
Cites (total number)	2.16	2.71	0.0076
H-index	0.33	3.94	0.0001

Multiple R = 0.45, R Square = 0.20, Adjusted R Square = 0.18, Standard error = 0.68

Discussion

Our findings reveal different predictors of award status according to the contract of the employee and the type of award. In general, years on the medical register and/or years in specialty showed the strongest correlation with award level. For psychiatrists in receipt of local awards it appears to be years worked in the specialty field that most determines level of CEA award with a small contribution from research productivity (H-index). For university employees it was years on the medical register and publication rate that was most influential. For NHS staff in receipt of national awards it was research alone that was most influential. After pooling all staff, years practising medicine and H-index were the dominant influences (data not shown).

It is notable that years in specialty and years on the medical register were the dominant factors underlying receipt of CEA for two groups. In fact total cites and H-index, the predictors in the NHS national award group are also themselves strongly related to age as both accumulate with time. Years on the medical register and to a lesser extent years in specialty are related to chronological age and we cannot discount age as a significant confounding factor as we did not have access to individuals' date of birth. It could be argued that length of career should logically be related to NHS excellence as performance ‘over and above contractual expectations’ might take time to accrue. Indeed research output tends to progress with career progression.¹⁴ Yet there is very little published evidence that measurable clinical excellence accrues with time and performance based comparisons of older versus younger doctors show mixed results.^15–18 Indeed there is an equally valid concern that performance may often deteriorate with time leading to calls from the GMC for revalidation. However, the GMC proposals for revalidation and an improved evidence-based continuing professional development infrastructure have been repeatedly delayed.^19,20 Our data suggest that individuals performing at an excellent clinical level and who are research active are unlikely to be rewarded by CEA progression without significant numbers of years in specialty. Data from the ACCEA suggest that successful CEA applicants will typically achieve a level 3 award after nine years in specialty. Similarly, a national level award is only likely after at least 12 years on the specialist register. This might be surprising as national award committees consider evidence covering the preceding five years, and it should be possible to progress to a national award from nine years in specialty given sufficient ability but we found only one example of a psychiatrist with a national award after less than 10 years in specialty. According to the ACCEA there should be no recommended minimum time before an application will be considered but several Royal Colleges (including the Royal College of Psychiatrists) have chosen to impose a self-regulated cap according to age such that applicants can only apply for national awards after 10 years as a consultant regardless of merit (personal communication). Such rules imposed by the Royal College of Psychiatrists may be one reason for a slightly lower than average success rate of psychiatrists, namely 26.2% compared to a national average of 30.6%.

Such strong association with age-related markers suggest that either excellence is strongly aged related or that the application process favours older applicants. The current system places strong emphasis on incremental evaluation, whereby applications in future years relate to performance since the last award. Local CEA committees have no mechanism to review previously allocated awards and published data from the ACCEA show that national awards are downgraded or removed only exceptionally. Thus previous merit is generally assumed and any new achievement or merit given as an increment. In our opinion this incremental system will favour multiple applications and in effect favour older applicants. An alternative system would be to review the overall merit of the applicant upon each application, so that any application could be graded up or down. Such reforms are under consideration by the Doctors and Dentists Review Body (DDRB) who are due to report in 2011.

We found that several measures of research performance, in particular H-index, citation rate and publication rate did have an influence on awards. Citations can be considered the gold standard measure of research impact, albeit measured cumulatively. The H-index is increasingly considered to be a robust measure of impact upon the field. The H-index has been found to have considerable face validity.⁹ Hirsch calculated the H-index of Nobel price winners and found 84% of them to have a H-index of at least 30. Bornmann & Daniel found that on average the H-index for successful applications for postdoctoral research fellowships was consistently higher than for non-successful applicants.²¹ It is notable that the H-index was influential in two groups evaluated here. Currently neither the CEA application process nor the evaluation process incorporates formal research metrics, however from 2010 applicants must list the raw number of peer-reviewed publications in the last five years. Nevertheless any peer assessor would have difficulty weighing the impact of these as well as their relevance to the NHS.

We wish to acknowledge several limitations in this analysis. First, we only had data on psychiatrists successful at at least level 1 and entered onto the 2008 or 2009 ACCEA nominal roll database. We estimate this to be approximately 14% of psychiatrists and offers no opportunity to examine those who were unsuccessful or those who did not apply for an award. Second, the published output for minor publications, books and chapters is not complete in any current medical database. However we maintain that use of the same database nevertheless ensures a fair comparison between individuals. Unfortunately we were unable to refine individual contribution to NHS research. For example, it would be interesting to see if service development research and audit has any particular impact on award status. Third, our analysis of predictors could not take into account any adequate measure of clinical ability. This might be improved if the CEA application forms were published in full, however even then, information on clinical performance is not collected in a quantative manner. Markers of clinical performance are currently controversial but in the future might include measures of satisfaction filled anonymously by treated patients after remission or might include peer measures such as results of 360 degree appraisals. Since such measures were not available to us, clinical acumen might be assumed to account for at least part of the variance in CEA status not explained by any of the factors listed here. Finally, we acknowledge we did not have access to date of birth, teaching performance or several other factors that could be influential.

The 2004 ACCEA restructure attempted to improve on the limitations of distinction awards but has rarely been examined scientifically. This lack of analysis may be because award status was not published in full before 2007. Prior to this only national level awards were published and then only as a list of names and specialties. In our opinion all CEA data including submitted forms and scoring should be openly available for scrutiny at both a local and national level. Many limitations in the ACCEA process still exist and await review from the DDRB. We conclude that current progression in NHS clinical excellence awards appears to be determined mainly by years since graduation and years spent in specialty with a smaller contribution from measures of research productivity and an unknown contribution from clinical acumen. In order to improve upon the current assessment process we suggest that the ACCEA formally adopts standardized research metrics which are openly available, ideally focusing on NHS-related research. We also suggest that local and national committees develop methods to measure clinical contribution more objectively and that local trusts and supporting organizations adhere to the rules set by the ACCEA.

DECLARATIONS

Competing interests

None declared

Funding

None

Ethical approval

Not applicable

Guarantor

AJM

Contributorship

AJM performed data collection, critical appraisal, analysis and interpretation; AJM and SH wrote the paper; all authors performed data extraction

Acknowledgements

The authors thank the staff of the postgraduate library at Leicester General Hospital

Reviewer

Michael Goldacre