Validation of influenza and pneumococcal vaccine status in adults based on self-report

SUMMARY

Self-report of polysaccharide pneumococcal vaccination is not thought reliable because of the increased risk of adverse events from inadvertent re-vaccination in elderly people. Some studies suggest a high sensitivity of self-report and hence a low risk of adverse events if vaccination is administered when medical records are unavailable. Self-report of pneumococcal and influenza vaccination in a sample of >64-year-olds in the United Kingdom was compared with information in their medical records. Self-report of pneumococcal vaccination, in contrast to some of the other studies had a low sensitivity. The findings here support the need for accurate knowledge of prior vaccine status before offering the polysaccharide pneumococcal vaccine. The study also confirms that self-report of influenza vaccination could be relied upon if rapid knowledge of uptake is required.

INTRODUCTION

Influenza vaccination has been recommended in those aged >64 years in many countries including the United Kingdom. Pneumococcal vaccination is also now recommended in all those aged >64 years in the United Kingdom. Knowledge of prior receipt of influenza or pneumococcal vaccination is useful for those charged with monitoring uptake or with deciding on administration of vaccine to individuals, e.g. by primary health-care staff or at hospital discharge. Accurate medical records of vaccination status are the gold standard, but, if reliable and valid, self-reports would be timely and less expensive sources of information in circumstances when medical records are not easily available.

Self-report of pneumococcal vaccination is not considered ideal because of the increased risk of adverse events in elderly people from inadvertent re-vaccination with the polysaccharide pneumococcal vaccine currently in use. Some studies in the United States have, however, suggested a high sensitivity of self-report and hence a low risk if inadvertent re-vaccination is administered when medical records are unavailable and, in turn, a low risk of adverse events [1, 2]. We examined whether this was the case in another setting and compared the findings with self-report of influenza vaccination.

METHODS

A cost-of-illness study in 65- to 84-year-olds was conducted in 10 UK research general practices in 1998 and 1999 [3, 4]. Only practices with fully computerized records in the MRC General Practice Research Framework were used to ensure full ascertainment [4]. Practice-based research nurses searched the computerized medical records every 2 weeks to identify patients with newly diagnosed pneumonia or influenza-like illness. A clinical diagnosis rather than an organism-specific diagnosis was acceptable given the original aim of assessing the cost-effectiveness of adult vaccination against influenza and pneumococcal disease. Based on information extracted by the research nurses from medical notes, patients with pneumonia were included if there was a clinical diagnosis of a lower respiratory tract infection with new focal signs requiring antibiotics. Patients with pneumonias due to viral or other known bacterial causes were excluded. Patients with general practitioner (GP)-diagnosed influenza or influenza-like illness were included. Apart from the two patient groups that had been noted as having had a clinical diagnosis of influenza-like illness or community-acquired pneumonia, a random sample of patients with no history of influenza-like illness or pneumonia was also obtained, frequency matched by 5-year age group, sex and practice to patients with such illnesses. The sampling frame used was the patient register (practice list) held by each practice.

Self-reported prior vaccine uptake was obtained from a postal questionnaire sent to the study participants. For pneumococcal vaccine patients were asked ‘Have you ever had a vaccine against pneumonia?’ If answering ‘yes’ they were also asked to indicate if vaccination was within the last 5 years or more than 5 years ago. For influenza vaccine, patients were also asked ‘Have you had the flu vaccine in the last 12 months?’ Self-reported information was compared with GP medical records of vaccinations given before the illness. Those patients where information on vaccine status was obtained from both self-report and medical records were included in the analysis of the validity of self-report.

Sensitivity, specificity, positive and negative predictive values and the kappa statistic were calculated using GP records as the ‘gold standard’. It was considered as very good agreement between both measures if the kappa value was 1·00–0·81. Good, moderate, fair, or poor agreement was if the value was 0·80–0·61, 0·60–0·41, 0·40–0·21 and <0·20 respectively.

RESULTS

The response rate was 72% in two of the three patient groups, out of 122 patients initially identified with pneumonia and 118 patients initially identified with influenza-like illness. In the sample of 184 patients with no history of influenza-like illness or pneumonia, over 80% were the first eligible to be sampled.

Influenza vaccine uptake in the past year according to medical records, including those with missing self-reported vaccine status, was 64% (56/88) in those having had pneumonia and 54% (46/85) in both those having had an influenza-like illness and those without either illness (99/184). Ever having received pneumococcal vaccine was 39% (34/88), 16% (14/85) and 22% (41/184) respectively.

Self-report of influenza vaccine uptake in the past year had very good agreement with GP records and was highly sensitive and specific (92·1%) (Table 1). Sensitivity was equal in all patient groups and the point estimates for specificity were slightly lower in those having had an influenza-like illness (87%, 95% CI 80–94) or pneumonia (82%, 95% CI 74–91) compared to those who had not (94%, 95% CI 91–97) but with overlapping 95% confidence intervals, consistent with no difference between the three groups of patients (Table 2).

Table 1.

Validity of self-reported vaccination status

CI, Confidence interval; PPV, positive predictive value; NPV, negative predictive value.

Table 2.

Validity of self-reported vaccination status in patients with a history in the past year of (a) influenza-like illness, (b) community-acquired pneumonia, (c) none of these illnesses

CI, Confidence interval; PPV, positive predictive value; NPV, negative predictive value.

Self-report of ever having had pneumococcal vaccination had only moderate agreement with GP records and high specificity (Table 1). Compared to 94% sensitivity for recall of prior influenza vaccination, however, sensitivity of recall of prior pneumococcal vaccination was only 64%, and did not vary between the three patient groups (Table 2). If limited to a history of pneumococcal vaccination in the last 5 years only, sensitivity increased somewhat to 75%, while agreement remained moderate. The kappa statistic was similar at 0·58 (95% CI 0·47–0·69) as was the percent agreement (86%).

DISCUSSION

The findings here support the caution expressed by GPs and other health-care workers, regarding accurate knowledge of prior vaccine status before offering the polysaccharide pneumococcal vaccine in order to reduce the risk of adverse events. In comparison, the high sensitivity and specificity of self-reported influenza vaccine uptake in the past year confirms the reliability, seen in other settings [1, 2, 5, 6], of self-report when rapid knowledge of influenza vaccine status or coverage is required.

In the absence of a proper ‘gold standard’ measure of vaccine uptake, medical records are commonly used as the ‘silver standard’. Computerized GP records used here have been shown to be highly sensitive in the UK setting [7] but it is possible that information on lack of vaccination may be less complete. This sample of the population came from research practices that are likely to have data with higher levels of completeness and accuracy than other practices. Thus, any reduction in sensitivity and specificity of self-reports is, therefore, likely to be closer to the truth than would be expected in other primary health-care settings.

A source of bias, that might make sensitivity less than it should be, is the use of information on self-report based on questionnaire data. Self-report with prompting by a health-care worker might be expected to improve sensitivity. The observed difference in the sensitivity of patient self-report of influenza compared to pneumococcal vaccine is, however, apparent despite the limitations of the ‘silver standard’ used here and the use of self-report from questionnaires.

The low sensitivity for self-reported pneumococcal vaccine is not surprising as it is not given annually and is, therefore, easy to forget. The risk of relying on self-report for pneumococcal vaccination in this context is the inadvertent re-vaccination of those who mistakenly think they had not received vaccination. A similarly low sensitivity (75% if given in the past 5 years) was also noted in another very recent US study, but medical record information was only available on 27% of the eligible sample [8]. In another study from Australia, with medical record information available only on 72% of the sample [9], sensitivity was 81%. In contrast, in a US population aged >64 years sensitivity of postal questionnaire-based self-reported pneumococcal vaccine uptake was high at 97% and specificity was low (53%) [1]. With low specificity (hence a proportion thinking they had received pneumococcal vaccine when they in fact had not) the problem is the risk of missed opportunities to provide vaccination.

A probable explanation for the low specificity in one of the US studies is the incompleteness of medical records from other potential vaccine providers. Specificity rose from 53% to 74% after excluding patients living outside the catchment area of the hospital with no medical records of vaccine status [1]. Up to 35% of patients in another study, were reported as having influenza vaccinations at locations other than the health-care centre [6]. In contrast GPs in the United Kingdom provide most of the ambulatory care of patients and are usually the sole providers of vaccinations for elderly people. Elderly patients in the United Kingdom are also less likely to move to another practice than other patients with few using private services for vaccination. It is, therefore, likely that the GP medical records used in this study are more complete than in the United States. The reason for the low sensitivity of self-reported pneumococal vaccine seen here is less easy to explain but may be because pneumococcal vaccine is relatively less well known than in the US study. Whatever the reasons, the risks of re-vaccination with the current polysaccharide pneumococcal vaccine need to be balanced against the benefit of the vaccine when medical records are not easily available.

Re-vaccination with this vaccine is associated with greater frequency of local complications. In a cohort study 1% of 900 first-time vaccinees had severe arm restriction where they were unable to raise their arm above their heads compared to 5% of 500 re-vaccinees [10]. About 1% in both groups developed high fever lasting 2 days but none required hospital care. In a US trial of re-vaccination, 10% developed severe erythema, similar in effect to cellulitis of the arm, restricted to older women with loose upper arm skin and poor muscle tone [11]. No increase in hospitalization rates was noted in another re-vaccinated cohort of just over 1000 compared to 66 000 primary vaccinated elderly US Medicare patients [12].

As the offer of pneumococcal vaccination to all people aged >64 years of age gets underway in the United Kingdom the sensitivity of self-report is likely to increase. In the meantime, the findings here of a substantial minority failing to recall receiving pneumococcal vaccination when in fact they had been vaccinated (e.g. in those without recent influenza-like illness or pneumonia, self-report had a negative predictive value of ∼90%) support the checking of records for prior pneumococcal vaccination status to minimize risk of local adverse events.

DECLARATION OF INTEREST

None.