Validity of physician-reported hospitalized infections in a US arthritis registry

Abstract

Objective. In safety studies, events reported as infections may be misclassified and, therefore, affect the validity of estimated risks associated with biologic agents. Using data from the Consortium of Rheumatology Researchers of North America (CORRONA), we evaluated hospitalized infection reports contributed by rheumatologists to establish their validity.

Methods. All patients hospitalized with infections from 2002 to 2007 reported to CORRONA were examined and compared with information from hospital discharge summaries and other confirmatory data. Infectious episodes were classified by two physicians as confirmed, empirically treated, possible or unlikely.

Results. Of 562 reported hospitalized infectious episodes, 9% were classified as unlikely and had minimal or no supporting evidence for infection, leaving 509 hospitalized infectious episodes. Of these, 53% of the infectious episodes were classified as confirmed, 15% empirically treated and 32% possible. The confirmation status of infectious episodes for younger or biologic-exposed participants was similar to older and biologic-unexposed participants.

Conclusion. More than two-thirds of hospitalized infections reported by rheumatologists were confirmed or had evidence that the physician was treating an infection. In almost all cases, there was at least modest evidence for an infection. Future studies should consider case definitions for infections or sensitivity analyses, or both, regarding the certainty of an infection to account for possible misclassification and reduce bias.

Introduction

The risk of adverse events (AEs), including infections, is an important consideration for physicians prescribing biologic agents for the treatment of inflammatory diseases. One potential concern with the assessment of infection risk is the validity of the reported event. In randomized controlled trials (RCTs), AEs are typically grouped using a classification system such as the Medical Dictionary for Regulatory Activities (MedDRA). The investigator judges causality for the study medication (e.g. probable, possible), but AE data are rarely reported or analysed this way. The regulatory definition of a ‘serious AE’ (SAE) is one that requires hospitalization, prolongs a hospitalization, is fatal or life threatening or results in a congenital abnormality. However, neither the MedDRA classification system nor the regulatory definition of a SAE constitutes a case definition. To satisfy a case definition for an infection, pre-defined criteria must be met; the Duke criteria for endocarditis is one such example of a case definition [1]. For infections that occur among RA patients, case definitions for infections have been developed [2, 3] but have been used only in a few studies. Moreover, limited research suggests that including infections with less certainty in safety analyses may mask a safety signal by introducing a substantial number of non-infectious events that have no association with the drug exposure [4]. The incidence rates of infectious AEs and SAEs would also be inflated if non-infectious events [e.g. an exacerbation of chronic obstructive pulmonary disease (COPD)] are misclassified as infections (e.g. pneumonia).

As RCTs provide the best evidence for efficacy, they are generally underpowered to detect SAEs, and data on comparative safety between agents are scant. For this reason, large-scale observational registries have been established in a number of countries to provide complementary information on the safety of biologic agents [5]; most of the data are contributed by rheumatologists. However, unlike the identification of infections using administrative claims data, where the performance of various algorithms has been examined [2, 3, 6], the validity of case identification from physician reports has been less well studied. We, therefore, evaluated suspected infections occurring in hospitalized settings reported by rheumatologists participating in the Consortium of Rheumatology Researchers of North America (CORRONA). We examined the positive predictive value (PPV) of infection reports from these rheumatologists compared with a gold standard of independent physician review of hospital medical records and other confirmatory data.

Methods

Cohort summary

Participants in CORRONA are recruited from community and academic medical practices across the USA. Recruitment began in 2002 and is ongoing; CORRONA has historically focused on RA, and more than 15 000 RA patients have now been enrolled. Details of the CORRONA cohort have been reported previously [7]. CORRONA is observational and treatments are not mandated, but a standardized evaluation including clinical measures (e.g. tender or swollen joint count) and other disease-specific assessments [e.g. patient and physician global assessment of disease activity, disability as measured by a modified Health Assessment Questionnaire (mHAQ), and use and dosing of various arthritis medications] are done at each visit, which occur at a median interval of 4 months. CORRONA is governed by local and centralized institutional review boards, and patients provided written consent for participation.

Identification and confirmation of infections

At each CORRONA visit, rheumatologists report whether their patient experienced an infection and, if so, what type. Information about hospitalization is also ascertained. After reporting an infection, rheumatologists are sent a follow-up confirmation form requesting additional details about the infection. At the time the physician is sent the form for infection confirmation, the physician is also asked to obtain and provide additional medical records (e.g. hospital discharge summary).

Using physician-reported infections reported from March 2002 to December 2007, infections were evaluated using a classification system developed by the investigators (Appendix 1, with additional details provided in the Appendix of [2]) and judged as confirmed, empirically treated (e.g. there was evidence that the physician was treating an infection, but no definitive information such as positive microbial cultures was identified), possible (i.e. there may have been an infection, but the evidence was limited) or unlikely (i.e. minimal or no supporting evidence for infection). Two internal medicine physicians (N.M.P. and A.J.) classified these infections; approximately half were reviewed in duplicate to assure agreement. Discordance was resolved by consensus in collaboration with a third physician (J.R.C.).

The site of infection, as well as microorganism data when available, was recorded for every infection. We defined an infectious episode as all infections that every person experienced during a single hospitalization. An overall confirmation status for each infectious episode was assigned as the highest confirmation status of all the individual infections. For example, a hospitalization where a confirmed pneumonia and a possible urinary tract infection occurred was classified as a confirmed infectious episode.

Analytic approach

Agreement between physician reviewers for infectious events classified as confirmed or empirically treated was reported as κ with 95% CIs. Using 2 × 2 contingency tables, the PPV of a rheumatologist-reported hospitalized infection was compared with a gold standard of a confirmed or empirically treated infection. As part of a sensitivity analysis, the gold standard was made more liberal and included confirmed, empirically treated or possible infections. Results were stratified by whether the physician reviewers had hospital discharge summaries or other primary medical records available to review, or whether lesser amounts of information (i.e. only the infection confirmation form) were available. PPVs for each site/type of infection were evaluated to determine whether some reported sites/types had higher PPVs than others. Results also were stratified by age and recent biologic exposure.

Results

Among 455 unique CORRONA participants, 562 hospitalized infectious episodes were reported. Of these, 64% had additional medical records available to review, which in most cases was at least a hospital discharge summary. Agreement between the two physician reviewers for the 257 medical records abstracted in duplicate was excellent (κ = 0.94, 95% CI 0.90, 0.98).

Overall, 9% (n = 53) of infectious episodes were classified as unlikely. In most cases, these had minimal or no supporting evidence for infection, leaving 509 hospitalized infectious episodes. Of these, 53% of the infectious episodes were classified as confirmed, 15% empirically treated and 32% possible. Infectious episodes for which primary medical records were available to review were significantly more likely to be classified as either confirmed or empirically treated (86 and 8%, respectively) than those for which only the confirmation form was available (4 and 25%, respectively; P < 0.0001 comparing combined endpoints). Among persons <60 years of age, infectious episodes were not more likely to be classified as confirmed or empirically treated compared with older persons (6% more likely among younger persons, P = 0.51). Similarly, infectious episodes among persons exposed to biologics in the prior 6 months were not more likely to be classified as confirmed or empirically treated compared with biologic unexposed (4% more likely among biologic exposed, P = 0.63).

A total of 606 unique infections were identified during the 509 hospitalized infectious episodes. Of these, 34% had identifiable organism(s); most had only a single organism identified, but 10% had two identifiable organisms and 2% had three identifiable organisms. The PPVs of the individual sites of infection and the proportion where at least one organism was recovered for infections classified as confirmed or empirically treated are shown in Table 1. The proportion classified as confirmed was 100% for bacteraemia and was <50% for conditions such as sinusitis and diverticulitis. As shown in Table 1, 70.8% of infections were classified as confirmed or empirically treated.

Table 1.

Site/type of infection and confirmation status of 606 hospitalized infections reported by CORRONA rheumatologists

Total infections, n (%)	Anatomic site	Confirmed, n (%)	Empirically treated, n (%)	Totala, n (%)	Identifiable organism, %a,b
174 (28.7)	RTI—pneumonia	101 (58)	29 (16.7)	130 (74.7)	17.7
93 (15.3)	Skin—cellulitis/soft tissue	52 (55.9)	12 (12.9)	64 (68.8)	45.3
74 (12.2)	UTI—kidney/urinary tract	48 (64.9)	9 (12.2)	57 (77)	61.4
44 (7.3)	Other	5 (11.4)	11 (25)	16 (36.4)	62.5
38 (6.3)	Sepsis	24 (63.2)	6 (15.8)	30 (78.9)	46.7
31 (5.1)	Abdomen—gastroenteritis	21 (67.7)	4 (12.9)	25 (80.6)	60
21 (3.5)	RTI—other	10 (47.6)	4 (19)	14 (66.7)	35.7
19 (3.1)	Bacteraemia	19 (100)	0 (0)	19 (100)	94.7
19 (3.1)	RTI—sinusitis	8 (42.1)	2 (10.5)	10 (52.6)	20
14 (2.3)	Osteomyelitis	9 (64.3)	2 (14.3)	11 (78.6)	54.5
14 (2.3)	Septic arthritis	10 (71.4)	3 (21.4)	13 (92.9)	69.2
12 (2)	Device-associated	10 (83.3)	0 (0)	10 (83.3)	90
11 (1.8)	Abdomen—diverticulitis	2 (18.2)	3 (27.3)	5 (45.5)	40
10 (1.7)	Abdomen—other	2 (20)	2 (20)	4 (40)	50
7 (1.2)	CNS—meningitis/meningoencephalitis	4 (57.1)	2 (28.6)	6 (85.7)	66.7
6 (1)	Abdomen—abscess	5 (83.3)	1 (16.7)	6 (100)	50
4 (0.7)	Skin—other	2 (50)	1 (25)	3 (75)	100
3 (0.5)	Abdomen—cholecystitis	0 (0)	0 (0)	0 (0)	0
3 (0.5)	Unknown	0 (0)	1 (33.3)	1 (33.3)	0
2 (0.3)	Genital tract infection	0 (0)	0 (0)	0 (0)	0
2 (0.3)	UTI—prostate	0 (0)	0 (0)	0 (0)	0
1 (0.2)	Abdomen-hepatitis	1 (100)	0 (0)	1 (100)	100
1 (0.2)	CNS—epidural abscess	1 (100)	0 (0)	1 (100)	100
1 (0.2)	CNS—other	1 (100)	0 (0)	1 (100)	100
1 (0.2)	Opthalmologic infection	1 (100)	0 (0)	1 (100)	100
1 (0.2)	RTI—empyema/lung abscess	1 (100)	0 (0)	1 (100)	100
606 (100)		337 (55.6)	92 (15.2)	429 (70.8)	45.2

^aConfirmed or empirically-treated infections. ^bTo satisfy this criterion, organisms must have been specifically mentioned by name in the information made available to the physician adjudicators. For that reason, a hospital discharge summary could have confirmed that the patient had bacteraemia or sepsis, but if the organism was not specifically mentioned, that infection would not have been considered to have an identifiable organism. RTI: respiratory tract infection; UTI: urinary tract infection.

Discussion

Among CORRONA participants, we found that a majority of rheumatologists reported infections that occurred during a hospitalization were either confirmed or had evidence that the physician was treating an infection. In almost all cases (∼91%), there was at least modest evidence for an infection. Assuming that results from CORRONA are applicable to other settings, this result should provide reassurance about the validity of hospitalized infectious events reported by rheumatologists participating in observational safety registries. We did not find significant differences in the infection confirmation status between patients recently treated and not treated with biologics. This result should attenuate the concern that physicians have a lower threshold to hospitalize biologic-treated patients for a suspected infection, which could bias safety analyses.

The implications of our work provide impetus for greater transparency and rigor in identifying SAEs, as there are some reported infections that may be misclassified. For example, a patient with chronic lung disease who is hospitalized for dyspnoea and has no fever, a normal white blood cell count and an abnormal but non-specific chest X-ray may be treated with antibiotics, oxygen, glucocorticoids, nebulized β-agonists and diuretics. A hospitalized infection may be reported, but there may be a little certainty that the patient experienced pneumonia rather than an exacerbation of their underlying COPD. Indeed, we showed that between a quarter and third of respiratory tract infections were possible or uncertain, often for these reasons. Some other sites/types of infections like diverticulitis (which could be diverticulosis treated with antibiotics) or cholecystitis are similarly problematic and may be misclassified as an infection. In our work, these were classified as ‘possible’ infections and not included in our primary endpoint. In analyses of the safety of anti-rheumatic disease agents, if there are a large number of these events relative to more certain infections, and assuming that these non-infectious events have no relationship with biologic therapies, an increased risk of infections associated with biologics may be obscured, as we have previously shown [4].

As a possible limitation of our work, we did not seek to confirm outpatient infections; it is likely that these have lower PPVs than those occurring in hospitalized settings. Moreover, we did not compute the negative predictive value of a lack of a reported infection, or sensitivity/specificity. This computation requires reviewing substantial amounts of data where no event was reported and has been shown to have a low yield [2]. As an additional consideration, we made the assumption that the hospital discharge summaries and other primary medical records were independent sources of information from the physician's initial infection report. Our expectation was that, in most cases, the rheumatologists would report the hospitalized infection to CORRONA at the time they saw the patient in their clinic, and only later would they obtain the hospital discharge summary and additional information that would be used to fill out the follow-up infection confirmation form. However, we recognize that the reporting rheumatologist may have had access to these primary data at the time that they reported the infection. They may also have had access to other clinical information that they did not make available to the physicians who adjudicated the infections. If so, this would have resulted in more infections being classified as ‘possible’, and thus our estimates of the proportion of infections classified as ‘confirmed’ or ‘empirically treated’ are likely conservative. Finally, although we have previously developed formal case definitions for both bacterial and opportunistic infections in RA patients that require satisfying varying combinations of clinical signs, symptoms, radiologic criteria, etc. [2, 3], applying them requires access to a large portion of the entire hospital medical record. In most cases, however, we had only the hospital discharge summary and selected supplemental data (e.g. microbial culture results); hence, applying the more detailed infection case definitions that we previously developed was not possible.

In conclusion, the majority of rheumatologist-reported infections occurring during a hospitalization was confirmed or had reasonable evidence for an infection. Confirmation status varied substantially by the site/type of infection. Although some RCTs employ adjudication committees to confirm AEs according to pre-specified case definitions, few rheumatology studies have employed this strategy. In future observational studies and RCTs, infection-related safety is likely to be enhanced through providing an assessment of the certainty of infections. The risk of infection associated with anti-rheumatic disease agents can then be evaluated in sensitivity analyses to ensure that results are robust to possible misclassification.

Appendix 1

Classification of infection based upon hospital discharge summaries, medical records, and physician follow-up confirmation form^a

Classification if hospital medical records were available to the physician adjudicators (note that reporting physician also had previously indicated that the patient had experienced an infection)

(i) Confirmed infection (any of these satisfy):

(a) diagnosis of infection(s) in hospital discharge summary, without any qualifier such as ‘probable’ or ‘possible’ and irrespective of any additional details of culture or radiological reports;

(b) no specific diagnosis of infection(s) in the hospital discharge summary, but mention of any positive culture report (includes bacterial, fungal or viral), unless labelled as a ‘contaminant’;a

(c) no specific diagnosis of infection(s) in the hospital discharge summary, but a description of clinical management of infection(s) with documentation of intravenous antibiotic administration.

(ii) Empirically treated infection:

(a) diagnosis of infection(s) in hospital discharge summary, with a qualifier suggesting uncertainty such as ‘probable’, ‘possible’, ‘provisional’, ‘rule-out’ or ‘working diagnosis’ and irrespective of any additional details of culture or radiological reports.a

(iii) Possible (any of these satisfy):

(a) description of clinical management in hospital discharge summary of an inflammatory condition that is often non-infectious (e.g. diverticulitis, cholecystitis, COPD exacerbation and aspiration pneumonitis) and without substantial evidence (e.g. positive culture data) for an infection;

(b) no information in the hospital discharge summary regarding any infection.

(iv) Unlikely:

(a) alternate, non-infectious aetiology was found to explain signs/symptoms of infection (e.g. pulmonary thromboembolism).

Classification if hospital discharge summary was not available to the physician adjudicators, and the patient's rheumatologist reporting the infection provided only the follow-up infection confirmation formb

(i) Confirmed infection (any of these satisfy):

(a) reporting physician took care of the patient during the hospitalization and had first-hand knowledge of the infection;

(b) reporting physician personally reviewed the hospital medical records (e.g. discharge summary) and physician also provided to the adjudicators additional documentation is provided showing either positive cultures or confirmatory radiologic findings consistent with infection.

(ii) Empirically treated infection (any of these satisfy):

(a) reporting physician reviewed hospital medical records (e.g. discharge summary) but provided no additional documentation to the adjudicators;

(b) reporting physician reviewed medical records from other physicians who took care of the patient in the hospital.

(iii) Possible (any of these satisfy):

(a) reporting physician was told by the patient that they were hospitalized with an infection, but no primary documentation was available to the physician or the adjudicators;

(b) reporting physician reviewed hospital medical records (e.g. discharge summary) and additional documentation was provided to the adjudicators showing either negative cultures or radiologic findings that were normal or inconsistent with infection.

^aDetailed description of the criteria used to define ‘contaminant’ organisms, classify radiologic findings, etc. is found in the Appendix of [2]. ^bThe infection confirmation form asked the reporting physician to describe what information he/she had access to in order to determine if there was an infection.