The reliability of the McCabe score as a marker of co-morbidity in healthcare-associated infection point prevalence studies

Abstract

This study aimed to ascertain the reliability of the McCabe score in a healthcare-associated infection point prevalence survey.   A 10 European Union Member States survey in 20 hospitals (n = 1912) indicated that there was a moderate level of agreement (κ = 0.57) with the score. The reliability of the application of the score could be increased by training data collectors, particularly with reference to the ultimately fatal criteria. This is important if the score is to be used to risk adjust data to drive infection prevention and control interventions.

Introduction

The classic McCabe–Jackson criteria to predict the likelihood for survival of patients with gram-negative bacteraemia on the basis of the level of underlying disease was first developed based on observations of endotoxin tolerance in humans (McCabe and Jackson, 1962). In healthcare-associated infection (HAI) point prevalence surveys it is used as a subjective score of underlying illness severity. This simple method of classifying patients according to a prognosis of rapidly fatal (<1 year), ultimately fatal (1–4 years) and non-fatal (>5 years) has been shown to be a better predictor of survival than the APACHE II score (Perl et al., 1995). More recently it has been demonstrated to be a strong determinant of septic shock related mortality (Delodder et al., 2011). It has also been modified for use in intensive care units with the original three-group classification into a four-group model by splitting the ‘ultimately fatal’ prognosis into a ‘long-term’ (>6 months) and a ‘short-term’ prognosis (<6 months) (Fernandez et al., 2006).

With respect to HAI in the hospital setting it has been shown to correlate with the prevalence of HAI in Finland (Lyytikäinen et al., 2008) and France (Thiolet et al., 2007), and more recently in a pilot of the European PPS (Zarb et al., 2012), and with specific types of HAI in specific settings such as urinary tract infections in surgical settings (Medina et al., 1997). It is therefore a potentially important tool for risk stratification in infection prevention and control. However, little is known about the reliability of the measure itself.

As the McCabe score is seen as a subjective measure of clinical prognosis, the validity and reliability of this measure in studies of HAI is important to ascertain. We have previously (PPS) reported on the results of a pilot PPS validation study in Europe (Reilly et al., 2015). Here we report on the reliability of the recording of the McCabe score in data collected for the European PPS.

Methods

A validation study of the PPS for HAI was carried out in 10 European Union Member States. Each contributed data for a minimum of 200 patients from at least two hospitals. The McCabe score was collected from information in the medical records or through discussion with the medical staff in the ward.

The reference (gold standard), for the validation process was the ECDC-PPS protocol and codebook (ECDC 2011) applied by a validation team of at least one trained expert external from (and/or acting on behalf of) the national/regional PPS coordinating centre. This expert was accompanied by a hospital infection control team member who had undertaken the primary PPS data collection, for the purposes of access and orientation. Identical data collection was conducted by the validator. Patient notes, nursing notes, hospital information systems and clinicians were the data sources. Ethical approval was obtained from Glasgow Caledonian Ethics Committee and approval for data collection was secured by each member state to comply with local requirements however no personal identifying information was transferred to the authors from the local data collection teams.

Concordance between the validator and primary collectors were analysed using the kappa statistic (κ). The record matching and subsequent analyses were performed using bespoke software written ab initio in FORTRAN 95 and run under OpenVMS on a Compaq Alpha system. Interpretation of κ was (κ: 0.81–1.00 is excellent, 0.61–0.80 is good, 0.41–0.60 is moderate, 0.21–0.40 is fair/marginal, <0.2 is poor agreement; negative values are possible and also denote ‘poor’).

Results

Of the 3958 patient records, a total of 1950 were selected for validation in accordance with the calculated study sample size. Of those, 1912 were matched to the primary dataset, since it was not possible to link all patient records due to errors in data entry or missing data.

McCabe scores were recorded in both the primary and validation datasets in 1526/1912 (79.8%) of all recorded data. For the remainder around 16.7% were unknown, due to lack of information in the medical records or the clinician not being available to calculate the score, either by the primary data collector and/or the validation data collector and 3.5% were unmatched (primary/ validation) data. There was minimal variation in McCabe completion by European Union Member States. Table 1 presents the data by category of McCabe score. The McCabe score had moderate levels of agreement in the dataset where all responses were considered inclusive of the unknowns (κ = 0.57).

Table 1.

McCabe score by primary and validation data collection.

McCabe score	Validation
Primary data collection	Non-fatal	Rapidly fatal (within 1 year)	Ultimately fatal (within 5 years)	Unknown	Total
Non-fatal	961 (94%)	9 (8%)	49 (13%)	194	1213
Rapidly fatal	8 (1%)	81 (72%)	19 (5%)	2	110
Ultimately fatal	56 (5%)	23 (20%)	320 (82%)	15	414
Unknown	60	1	29	19	109
Total	1085	114	417	230	1846a

^a66 not matched.

Where the McCabe score was either classified by the data collector or provided by the clinician (n = 1526), the levels of agreement between the primary data collector and validator were: 94% (961/1025) for non-fatal, 72% (81/113) for rapidly fatal and 82% (320/388) for ultimately fatal (Table 1). The McCabe score had good agreement in the dataset when unknowns were removed from the analysis (κ = 0.78).

Discussion

The McCabe score is a useful predictor of risk for infection in selected settings in published studies (Perl et al., 1995; Medina et al., 1997; Fernandez et al., 2006; Thiolet et al., 2007; Lyytikäinen et al., 2008; Delodder et al., 2011; Zarb et al., 2012). However, this study is the first to formally assess its reliability in an HAI prevalence study in the hospital setting. The results show a moderate level of agreement overall when all data were taken account of, inclusive of unknowns by either the primary or validation data collector (κ 0.57).

Variation in agreement was noted by category of McCabe. Rapidly fatal had poorer levels of agreement (72%) than other categories, whereas non-fatal had high levels of agreement (94%). This variation indicates that some of the categories of McCabe may be easier to interpret than others and suggests that data collectors, perhaps especially if they are not clinicians, may have difficulties in assessing a patient’s short-term prognosis. Importantly, the McCabe score had good level of agreement in the dataset where both primary and validator data collectors had recorded the result (κ = 0.78).

The McCabe score requires abstractor judgement, usually involving verification with a clinician present on the ward. This means that data collectors may record ‘unknown’ when they are unsure of the score or unable to find the information required to calculate the score. These results indicate that investment in training of data collectors in the McCabe classification would be worthwhile to maximise data completion and enhance the reliability of the data. Reliable recording of this score will enable risk adjustment for infection prevention and control assessment.