Validity of Self-report of Infections in a Longitudinal Cohort of Patients with Rheumatoid Arthritis Differs by Source of Report and Infection Severity

Abstract

Objective

We evaluated and compared the validity of patients' and rheumatologists' reports of infection compared to infections confirmed by medical record review.

Study Design and Setting

Reports of infections in 961 patients with rheumatoid arthritis (RA) from the Brigham and Women's Rheumatoid Arthritis Sequential Study (BRASS) were included over a two year period. BRASS is a longitudinal prospective cohort which collects detailed questionnaire data from patients semi-annually and their treating rheumatologists every year.

Results

Rheumatologist report of infection was more likely to be confirmed by medical record review than patient self-report (57.1% vs. 34.3% for definite or possible infections). Confirmation rates varied based on whether the participant received her primary care from the same network of health care providers. For participants with primary care “out of network”, between 7.0% and 23.1% of patient or rheumatologist reports were confirmed by medical record review versus between 16.1% and 41.7% for those with primary care “in network”.

Conclusion

The present study shows that relying strictly on patient or rheumatologist report of infection for a confirmed endpoint is not ideal, but useful in case finding. The confirmation rate is affected by a number of factors including severity and definition of the infection and limited by data availability.

Introduction

When a new drug is approved, there is typically substantial uncertainty as to adverse effects. Since little is learned about many aspects of drug safety during clinical trials designed to study treatment efficacy, post-marketing surveillance serves a critical role in elucidating the risks associated with new medications (1). Drug or disease registries supply important information for post-marketing surveillance, as these data provide a perspective on drug-related adverse events. Registries often rely on patient or physician reports of adverse events, and relatively little is known about the validity of those reports.

Some degree of erroneous reporting of adverse drug reactions by patients and physicians should be expected. In the setting of new medications, both patients and physicians may be hyper-vigilant, attributing events to the new treatment when the unknown (unobservable counterfactual) is that that an adverse event may have occurred in the absence of treatment. Studies designed to explore or uncover such associations must rely on some form of event reporting.

This study examined the accuracy of patient and rheumatologist reports of infections in a cohort of patients with rheumatoid arthritis (RA). Little is known about the validity of these reports, particularly among a patient population suffering from a chronic condition that requires regular monitoring and management of symptoms. We used patient and rheumatologist reports from a longitudinal prospective cohort study, the Brigham Rheumatoid Arthritis Sequential Study (BRASS), to evaluate and compare the validity of these reports for infection with those assessed from medical record review.

Methods

Study Population

BRASS participants receive care from rheumatologists at the Brigham and Women's Hospital in Boston. All participants are English-speaking, at least 18 years of age, not diagnosed with lupus or psoriatic arthritis, and provided signed, informed consent. Participants complete surveys every six months to gather data on demographic and lifestyle factors, past and current medication use, general health and function, and pain symptoms. Their rheumatologists also complete questionnaires annually, as well as perform and document medical examinations at those times. Although recruitment and follow-up are ongoing, reports of infection were obtained from the 961 participants enrolled in BRASS as of March 15, 2007.

Data collection

Every six months, participants reported infections experienced in the prior six month period by responding to the following: “Please mark the response for each of the infections requiring antibiotics that you have experienced in the past 6 months.” In addition to serious infections such as tuberculosis and sepsis, participants could specify “other”. Components of the BRASS participant surveys are administered by in-person interview by a research assistant, while other parts are self-administered. Rheumatologists were asked annually to report new infections in their patients in the preceding 12 month period in a self-administered questionnaire (“Has your patient developed any of the following co-morbidities, drug-related toxicities, or opportunistic infections requiring IV therapy in the past year?” with options including abscess, sepsis, herpes zoster, staph aureus, and “other”.) Self-reports of infection were obtained directly by answering the questions above (i.e. reporting recent use of antibiotics for treatment of infection), as well as indirect reports such as discontinuation of medication due to infection. These latter “indirect” reports were drawn from reported changes to medication since last visit with the participant reporting infection in response to ”If you stopped taking this medicine, why did you stop taking this medicine?” or the rheumatologist specifying infection as the reason for a change in medication. In the present study, we included participant reports from the 6, 12, 18, and 24 month surveys and rheumatologist reports from the 12 and 24 month surveys.

The first 24 months of follow-up in BRASS were assessed for reports of infection. Using the date of questionnaire return on which an infection was reported we evaluated the medical records over a six month period for patients and twelve months for rheumatologists for evaluation of infection. Two reviewers extracted data related to medications, signs and symptoms, laboratory findings, and radiological studies, as well as date of first symptom or, if unavailable, the first mention of the infection.

To evaluate additional measures of validity such as the negative predictive value of these reports, a sample of 51 participants were randomly selected from BRASS participants who never reported an infection during their follow-up nor had one reported by their rheumatologist. Using their BRASS date of enrollment, a random index date was generated up to 24 months later to replicate the distribution of responses in the entire BRASS survey. The index date was used as a proxy for the date of reported infection to review the participants' medical record in search of missed or unreported infections.

Defining infection

Three definitions of infection were considered in the present study ranging from strictest to least stringent using data abstracted from medical record review:

1. Definite infection according to reviewers or positive identification by previously validated infection algorithms(2);

2. Definite or possible infections according to reviewers or positive identification by previously validated infection algorithms;

3. Definite or possible infection according to reviewers or, positive by previously validated infection algorithms, or documented use of antimicrobial for an infection in the medical record. (see Figure 1)

Figure 1.

Graphical representation of the multilevel definitions of confirmed infection by medical record review.

* Definite impression by reviewers or positive identification by infection algorithms

^† Definite or possible impression by reviewers or positive by algorithm

^¶ Definite or possible impression, positive by algorithm, or reported use of antibiotic for an infection in the medical record.

Using the data abstracted from medical records, infections were defined using previously published algorithms(2). Briefly these algorithms rely on combinations of serologic evidence, antimicrobial treatment, radiographic findings, and clinical findings upon examination. For instance to define osteomyelitis the algorithm required histopathologic evidence from a bone biopsy or both imaging and “suggestive indirect features” (e.g. positive blood culture, draining soft tissue, temperature ≥ 38C). For unspecified infections and those not included in the algorithms, reviewers' impressions classified infections as definite, possible, or none. To improve the reliability of chart reviews, the two reviewers (JFS, MLS) had to independently agree on both the likelihood and type of infection during training sessions. They demonstrated nearly 100% concordance on the 30 training charts, and were able to obtain consensus on any discrepancies by reviewing the medical records in question together. The review protocol was modified to address potential areas of discrepancies and subsequent charts were reviewed by individual reviewers. All ambiguous cases were reviewed with board certified internists (DHS, EWK). Reports of upper respiratory infections were excluded, including cases of sinusitis and otitis media, on the grounds that these events are common, difficult to distinguish from other common ailments (e.g. allergic rhinosinusitis), and are generally not considered to be of substantial clinical importance.

Statistical analysis

We compared infections reported by study participants, their treating rheumatologists, or both, with medical record confirmation using 2×2 contingency tables. First we compared both the rheumatologist's and/or patient's report against definite confirmation of infection (definition #1), against definite or possible confirmation of infection (definition #2), and with either definite or possible infection, or rheumatologist's report of antimicrobial used for an infection in the medical record (definition #3). For each of these, positive predictive values were calculated. The contingency tables were also stratified by whether or not the BRASS participant's primary care physician (PCP) was in the Partners Healthcare network, since the reviewers were able to read the electronic medical records of the physicians in the Partners network. This stratification accounted for the possibility that those “in network” had more medical record data available to confirm their reports of infection. In addition, rheumatologist reports of infection were considered separately and the above measures of validity were calculated against the various confirmed infection definitions.

Results

Of the 961 participants in our initial study population, there were 937 self-reports of infection among 598 participants over the first 24 months of follow-up. One-third of the infections were reported at six months, and then tapered from there with 27% at 12 months, 21% at 18 months, and under 19% at the 24 month survey period. An additional 51 records were reviewed among participants with no reported infections on BRASS surveys, totaling 988 possible participant infections examined in the medical records (Table 1). The participants' treating rheumatologists reported 7.2% (n=67) of these infection reports. One hundred eighty six (19.9%) of the infection reports by both patients and rheumatologists were accompanied by reports of oral antibiotic use while 41 reports (4.4%) included physician notes of IV antibiotic treatment in the medical records.

Table 1.

Cross-classification of infection reports by confirmation by medical record review

		Medical record review Infection Definition #1*
		Yes	No	Total
Patient or Rheumatologist Report	Yes	103	834	937
	No	1	50	51
	Total	104	884	988
	PPV=11.0%, NPV=98.0%

^*Definite impression by reviewers or positive identification by infection algorithms

Using the strictest classification of infection requiring definite confirmation by reviewers or positive infection by algorithm (definition #1), 103 (PPV=11.0%) were confirmed by medical record review (Table 1). When the gold standard was expanded to include possible infections by reviewer's impression (definition # 2), 252 (26.9%) of the reported infections were confirmed. Of the 937 reports, 293 (31.3%) were confirmed as infections when the above gold standard was further expanded to include reported use of antibiotics (definition # 3). When validity was considered separately for those with information from their “in network” PCP record (n=410, 44% of infection reports) versus those with PCPs “out of network” (n=527, 56% of infection reports), positive predictive values were higher for those “in network” (Table 2). Positive predictive values (PPV) ranged from 7,0% to 23.2% for participants whose primary care was not within the Partners Healthcare network, and ranged from 16.1% to 41.7% for those with “in network” primary care. Infections reported by BRASS participants' rheumatologists were more likely to be confirmed by medical record review for all available definitions of infection (28.6% to 69.2%) (Table 3). When we examined the confirmation rate of infection subtypes, we found that infections such as sepsis were more likely to be confirmed (92.3% for definite confirmation vs. 32.2% definite confirmation of cellulitis) (Table 4).

Table 2.

Measures of validity by definition of infection confirmation for all BRASS participants and stratified by access to medical records from primary care physician (PCP).

		nconfirmed	PPV	95% CI
Definition #1*	All participants	103	11.0	(9.1, 13,2)
	PCP “in network” ‡	66	16.1	(12.7, 20.0)
	PCP “out of network” ‡	37	7,0	(4.8, 9.3)
Definition #2†	All participants	252	26.9	(24.1, 29.9)
	PCP “in network”	144	35.1	(30.5, 40.0)
	PCP “out of network”	108	20.5	(17.1, 24.2)
Definition #3¶	All participants	293	31.3	(28.3, 34.4)
	PCP “in network”	171	41.7	(36.9, 46.7)
	PCP “out of network”	122	23.2	(19.6, 27.0)

^*Definite impression by reviewers or positive identification by infection algorithms

^†Definite or possible impression by reviewers or positive by algorithm

^¶Definite or possible impression, positive by algorithm, or reported use of antibiotic for an infection in the medical record.

^‡Stratified by whether or not the participants' PCP is in the Partners Healthcare network

Table 3.

Positive predictive values of self-reported infection by the rheumatologist.

		nconfirmed	PPV	95 % CI
Definition #1*	PCP “in network”‡	17	43.6	(27.8, 60.4)
	PCP “out of network” ‡	8	28.6	(13.2, 48.7)
Definition #2†	PCP “in network” ‡	27	69.2	(52.4, 83.0)
	PCP “out of network” ‡	16	57.1	(37.2, 75.5)

^*Definite impression by reviewers or positive identification by infection algorithms

^†Definite or possible impression by reviewers or positive by algorithm

^‡Stratified by whether or not the participants' PCP is in the Partners Healthcare network

Table 4.

Positive predictive values by infection subtype, %

	Confirmation of Infection
Specific infection type reported (total)	Definite*	Definite or Probable†
Bronchitis (n=40)	52.5	62.5
Sepsis (n=13)	92.3	100
Cellulitis (n=90)	32.2	60.0
Osteomyelitis (n=19)	42.1	57.9
Pneumonia (n=49)	32,7	57,1

^*Definite impression by reviewers or positive identification by infection algorithms

^†Definite or possible impression by reviewers or positive by algorithm

Note: The remaining infections fell under a general “not otherwise specified” category, and yielded a low PPV (2.4%^* and 16.9%^†).

Discussion

In the present study we found that patient or rheumatologist report of infection was not always confirmed by medical record review. The confirmation rate was affected by a number of factors, including the amount of information available as represented by whether the participant's PCP was in the Partners Healthcare network. Furthermore, when rheumatologist reports were considered separately, we found that rheumatologist report yielded a higher positive predictive value compared to all reports, which were predominantly from patients.

The large number of unconfirmed self-reports might be explained by insufficient data to confirm infections by medical record review or by patient's misreporting due to misunderstanding what constitutes an infection. Both patient and rheumatologist reports were included in our definition of infection to maximize sensitivity. Reports of antibiotic use were considered potential evidence of infection. By including such a broad definition of possible infections, we lower the specificity and inflate the proportion of “false positives.” However, our aim was to cast a broad net in order to capture all true cases. We were generally successful in this respect, as evidenced by our low false negative rate: there was only one false negative report among the 51 participants we examined. The one missed infection observed was due to a wound complication resulting in significant necrotic debris which required both intravenous and oral antimicrobial treatment. When the dates were cross checked with questionnaire return dates, it appeared that the patient was no longer participating in BRASS and therefore did not complete the survey which may have captured this infection.

Infectious adverse events have been of interest in RA, particularly in association with treatments such as methotrexate (3, 4) and tumor necrosis factor antagonists (5, 6). Observational studies have utilized many different sources of information for infections including inpatient medical records, questionnaires, and healthcare utilization data. Each of these data sources allows for varying degrees of validation of endpoints(5, 7). A recent study using administrative claims to identify possible severe infections found varying confirmation rates depending on the treatment setting and number of diagnosis claims where medical records were used to confirm the infections(8).

We used medical records to confirm reports of infection from patients and their treating rheumatologists. Since rheumatologists are not always the primary care practitioners, they may not be notified when a patient has symptoms of infection or receives antibiotics. Although one strength of the data is that patients and physicians are both asked about infections, it does not deal with the differing definitions that patients and physicians have of what an infection is. Presumably patients are more likely to report upper respiratory infections and urinary tract infections, when a clinician may be focused more specifically on pneumonia, bacteremia, and osteomyelitis. This doesn't make either wrong, necessarily, but it does yield different results. In the design of the questionnaires for BRASS participants could report a handful of specific infections but could also report “other” which were classified as not otherwise specified and investigated. While these “other” infections might be a temporary nuisance to the patient, they may not be significant enough that patients went on to tell their rheumatologist about their cold. Available documentation on care other than by the BWH rheumatologist varied considerably, particularly for those participants with primary care “out of network”. For some, test results and correspondence between out of network PCP and BWH rheumatologist were included in their charts, while for others no details were available at all. Without access to these records, we cannot definitively say whether confirmation of infection would be greatly improved had out of network records been available. This limits a rheumatologist's ability to accurately report infections. It also limits our ability to confirm infections from rheumatologists' medical records and “in network” medical record review; This is supported by our finding that when a participant's PCP was in network with records available, we found that reports of infection were more likely to be confirmed (Tables 2 and 3). The availability of extensive records for 410 (44%) of the 937 infection reports, and the ability to examine all patients and rheumatologist reports during the first two years of follow-up, are strengths of the present study. However, the fact that over 50% of reports had their primary care outside of the Partners system is also a limitation.

The present study shows that relying strictly on patient's or rheumatologist's report of infection for a confirmed endpoint is not ideal. Such reports are very useful in case finding but require confirmation by record review. Although rheumatologist's report of infection led to improved predictive value, our patients' rheumatologists completing the annual surveys for this cohort are subspecialists and may not be aware of endpoints outside of their specialty. The high negative predictive value of patient and rheumatologist reports suggests that lacking a mention of infection may be an accurate method to determine patients who were very unlikely to have had an infection during follow-up.