Effects of diagnostic uncertainty and misclassification on hospital performance indicators for acute stroke care

ABSTRACT

The Stroke Improvement National Audit Programme (SINAP), now replaced by the Sentinel Stroke National Audit Programme, was established to provide comparative data on hospital performance indicators for stroke, but comparisons are only valid if case ascertainment is complete. In Gateshead we compared initial results from SINAP with those from a pre-existing hospital stroke register, which ran independently for 11 months after SINAP's introduction in 2010, as well as with Hospital Episode Statistics (HES) data. Of 315 confirmed acute stroke cases identified from the three combined data sources, 96 (30%) were omitted from SINAP and 51 (16%) were missed by HES. Of 478 suspected strokes in the combined datasets, 214 were either misclassified by HES or remained with uncertain diagnosis. These patients had much lower mortality and shorter hospital stays than those with confirmed stroke. This diagnostic uncertainty could be an important source of uncontrolled variation in, or even a potential target for manipulation of, hospital performance indicators for stroke.

Background

Over the past few years, hospitals in England have been encouraged to collect standardised data on the management of common conditions for national clinical audits. One such audit is the Stroke Improvement National Audit Programme (SINAP), which aims to describe the processes and outcomes of acute stroke care and enable the performance of different hospitals to be compared.¹ Such comparisons will only be fair if case ascertainment is complete and accurate, so that indicators are based on equivalent denominators.

At the Queen Elizabeth Hospital (QEH), Gateshead, a ‘stroke database’ (SDB), comprising a prospective register of all acute stroke admissions, was kept from 1997 until 2011, and used for both clinical management and research. The SINAP register was introduced in May 2010, but because it served a different purpose, the previous system continued independently until the end of March 2011; therefore, we had the opportunity to compare results.

There were obvious discrepancies in the number of cases registered during the 11-month overlap period between the SDB and SINAP registers; therefore, we decided to look in more detail at the reasons and the possible consequences.

Methods

The SDB used multiple sources of notification, but ascertainment depended to a large extent on the stroke team actively seeking out possible stroke cases, wherever they were admitted. All potential cases were given a register number, although some were subsequently reclassified as transient ischaemic attack (TIA; using the traditional 24-h cut-off for complete resolution), ‘uncertain’ (probably not stroke) or ‘other’ (definitely not stroke). Responsibility for collecting SINAP information was allocated to different staff by hospital management and there was limited direct interaction with the SDB during the overlap period. In cases where there were discrepancies between the SINAP and SDB information, the diagnoses were checked from admissions records and scan reports. In cases where there was still significant uncertainty, the diagnosis was checked from the discharge summary or case notes.

Given that SINAP focused only on the first 3 days of hospital care, information on deaths, length of stay and discharge destination was obtained from the SDB or the hospital information system.

In an attempt to establish a definitive list of true stroke cases, we then checked the Hospital Episode Statistics (HES) records of all acute admissions during the 11-month period with any stroke-related diagnostic code. Again, we checked brain scan reports, discharge summaries and medical notes of any cases where there appeared to be discrepancies in diagnosis, dates or discharge destination, although we did not attempt to get full details of other information.

During this two-stage case review and reclassification process, it became clear that there was no absolute diagnostic gold standard and that even the final list would contain substantial uncertainties. Thus, the tables presented here do not give estimates of sensitivity and specificity, but show how the original SINAP figures compare with the consensus diagnoses obtained after including data, first from the SDB (Table 1) and then from the HES (Table 2). We then examined the potential effects of these uncertainties and reclassifications on estimates of overall stroke fatality rates (Table 3), starting with all potential stroke cases and then eliminating those where the diagnosis was shown to be incorrect, remained uncertain or was ‘missed’ by at least one of the three data sources. The aim was to illustrate the range of possible mortality figures, rather than trying to estimate a single ‘correct’ value.

Table 1.

Comparison of SINAP AND SDB cases.

Table 2.

Comparison of HES with combined SINAP and SDB cases.

Table 3.

Causes and consequences of misclassification or diagnostic uncertainty.

Results

SDB–SINAP comparison

During the 11-month overlap period, 247 cases were registered in SINAP (235 strokes, four TIAs and eight other diagnoses), compared with 447 in the SDB (335 strokes, 52 TIAs and 60 uncertain or other diagnoses).

After review, four of the SINAP ‘stroke’ cases were reclassified as TIAs, 11 as uncertain and one as ‘other’ (subdural haematoma), whereas two of the four ‘TIAs’ and four of the eight ‘other diagnoses’ were reclassified as strokes. Two SINAP stroke cases had been missed by the SDB. Several SDB cases were also reclassified and duplicate records eliminated between the two databases, leaving a total of 450 cases (309 strokes, 66 TIAs, 48 uncertain and 27 other diagnoses). Table 1 compares the original SINAP figures with the revised and combined SDB/SINAP classification, and also shows the in-hospital death rate and average length of stay within each diagnostic category.

Among those with confirmed strokes, 63% had a focal lesion on computerised tomography or magnetic resonance imaging (MRI) scan, relevant to their clinical presentation, compared with 17% of TIAs and 10% of those with uncertain diagnosis. Of the 90 stroke cases missed or misclassified in the original SINAP data set, 57 (63%) had a relevant scan lesion but only 11 (12.2%) died; therefore, the in-hospital fatality rate was overestimated by 1.8% (11% in relative terms).

Late presentation was more common among the stroke cases missed by SINAP: 9% were admitted over 72 h after onset, compared with 2% of those on the SINAP register. Only 50% of the non-SINAP cases were admitted directly to the stroke unit, compared with 69% of the others.

Of the 309 confirmed stroke cases, 25 (8.1%) had primary intracerebral haemorrhage. Of these, 11 were not in the SINAP data set and three were misclassified as ischaemic strokes.

During the study period, 18 patients (16 in the ‘stroke’ and two in the ‘uncertain’ group) received intravenous thrombolytic treatment. Two of these cases were not recorded in the SINAP data returns.

Of those in the ‘uncertain’ group, 59% had had a previous stroke or TIA, compared with 28% in the stroke group and 41% in the TIA group.

Comparison with HES stroke episodes

From the HES records, we identified 338 in-patient episodes with a stroke-related diagnostic code (ICD10 codes I61, I63 or I64) among the first 12 diagnoses listed. Combining all three data sources produced a total of 509 potential stroke admissions, of which ten were duplicates and 21 did not have a primary stroke-related diagnosis in any of the datasets. Table 2 shows the remaining 478 possible stroke cases, 308 of whom had an acute stroke ICD code as primary diagnosis. Of the remaining 170 cases on the stroke register, 162 had no mention of stroke among all the HES diagnoses coded.

Of the 28 cases on the HES stroke list, missed by both SDB and SINAP, six had confirmed strokes (three of whom died without admission to the stroke unit), three TIAs, 16 diagnoses remained uncertain and three were eventually given non-stroke diagnoses.

Thus, although the three data sets agreed on the diagnosis of stroke in most cases, there were substantial disagreements and each source contained false positives and false negatives, as well as a large number of cases in which the diagnosis remained uncertain. Table 3 classifies these disagreements and uncertainties and indicates their potential effect on hospital mortality figures for stroke. The top row shows the total number of potential stroke cases initially considered and cases are subtracted sequentially for the reasons shown down the left hand side. The first column shows the numbers of cases excluded and column 2 the proportion dying in hospital within each group. Columns 3 and 4 show the number of cases remaining and the fatality rate among them, respectively.

Of the 478 potential stroke cases, 163 (34%) proved to have TIAs, uncertain or other diagnoses. A further 51 confirmed strokes (11%) were ‘missed’ by the HES. The SDB and SINAP combined missed fewer stroke cases compared with the HES, but had a higher false positive rate and a higher proportion of uncertain cases. Thus, nearly half the potential stroke diagnoses were subject to question or disagreement, and the estimated fatality rate ranged from 13.8% in the most inclusive set of 478 cases to 17.8% in the most restricted set, the 219 strokes correctly identified in the SINAP register.

Discussion

Complete prospective ascertainment of acute stroke admissions in a busy general hospital is not easy and it is not surprising that the newly introduced SINAP register missed a significant number of genuine stroke cases. There is no reason to think that the audit was done less efficiently at QEH than at other hospitals. The SDB, which had already been going for 14 years, was set up mainly for research and, therefore, was independent of hospital management, although it was later used for organising clinical follow-up and as an independent check on the HES stroke figures. SINAP was set up separately with management support and the two systems ran in parallel for 11 months until the SDB ended. The amount of detail required by SINAP on the prehospital and admission pathway (down to the level of ambulance job numbers) was greater than the initial data required for the SDB, and might have discouraged staff from registering cases where the diagnosis of stroke was not immediately clear.

Although the extent of disagreement between the SDB and the HES was less than had been found in previous comparisons, a considerable number of stroke cases were still missed by HES, even though coding was done at discharge, when the diagnosis should have been clearer. It might take some time to distinguish a stroke from a TIA or from various ‘stroke mimics’; therefore, many possible strokes were later reclassified in the prospective registers, but diagnostic uncertainty often seems to persist, accounting for the surprising number of cases still classified as ‘uncertain’, even after review.

Some of these patients might have functional symptoms, migraine or other stroke mimics, but many more have underlying cerebrovascular disease,² as shown by the high frequency of previous strokes and TIAs in this group. In such patients, the metabolic or haemodynamic stress of any systemic illness might cause symptomatic dysfunction in relatively ischaemic areas of brain (sometimes referred to as ‘MICROS’ (Metabolic Insults Causing Reactivation Of Stroke symptoms)³), which can lead to infarction. Many of these patients had multiple pathology, and it is debatable whether diagnostic uncertainty could be significantly reduced by diffusion-weighted MRI or other imaging techniques, even if they were more widely available.

Many of these cases of ‘threatened stroke’ are managed on acute geriatric or general medical units and few of them would be considered for thrombolytic treatment. Given that rapid admission to a stroke unit and the proportion of cases thrombolysed are key quality standards, linked with financial incentives and contributing to overall hospital ratings,⁴ there is a strong disincentive to classify these cases as ‘strokes’ if there is any room for doubt.

The same factors could encourage staff to classify some minor strokes as ‘TIAs’, although the benefits of this in terms of process indicators must be set against the inevitable increase in death and dependency rates and average length of stay among the remaining stroke cases.

The present study was done just after the introduction of the SINAP system; therefore, omissions and misclassifications were more likely to be attributable to the ‘learning curve’ than to any hidden incentives. Nevertheless, as time goes by, the tendency for hospitals to manipulate the figures to improve their league table positions is bound to increase and allowance should be made for this when interpreting the data for service audit or research.

SINAP has now been replaced by the Sentinel Stroke National Audit Programme (SSNAP), a more comprehensive mandatory audit of in-patient stroke care. The number of cases reported by each trust is compared with the expected number of stroke admissions based on the HES figures of the previous year, and trusts are now graded according to their ‘case ascertainment’. Although this should reduce deliberate under-reporting, minor strokes and uncertain cases are less likely to be included in such a detailed audit.

Even ignoring any systematic biases in classification, incomplete case ascertainment could have a major influence on the interpretation of differences in hospital performance indicators for stroke. Some of these, such as the apparent fall in mortality rates after the reorganisation of acute stroke services in London, have been widely publicised⁵ and could have important implications nationwide. Differences among centres participating in cluster-randomised trials, in the proportion of stroke admissions included, could also seriously affect the interpretation of results.⁶

Stroke accounts for a substantial proportion of the deaths used for calculating hospital standardised mortality ratios (HSMRs), the validity of which has provoked fierce debate since the scandals at Mid-Staffordshire and elsewhere.^7,8 Our results show that even HES data, on which HSMRs are based, cannot provide a gold standard denominator for stroke admissions. Whether via conscious manipulation or simply through variation in coding practice, this uncertainty could have a substantial effect on published hospital mortality rates.