Validity of Self-Assessment of Skin Reaction After Smallpox Vaccination

SYNOPSIS

Objective

Smallpox vaccinees should be evaluated for the presence of a major skin reaction (“take”) one week after vaccination, but this could prove to be logistically infeasible in the context of an emergency mass-vaccination campaign. We validated a tool for self-evaluation of the vaccination site for presence of take.

Methods

We conducted a prospective, double-blinded, paired-measurement validation study of 174 non-naive adult vaccinees and their physician evaluators. Subjects provided paired, blinded, independent assessments of take 7–9 days after vaccination.

Results

Overall, vaccinees and evaluators agreed on 157 of 174 (90.2%) take assessments. Sensitivity of the tool was 99.1%, and specificity was 75%. The positive predictive value of self-assessment was 87.2% and the negative predictive value was 98%. Specificity of the tool and measures of agreement were significantly modified by age, education, and occupation. When adjusted for the expected take rate among a population including naive vaccinees, positive predictive value and overall agreement increased significantly.

Conclusions

Self-assessment may be a feasible option for evaluation of take in the event of mass smallpox vaccination. The predictive values and overall agreement of the tool are satisfactory, and can be expected to increase when used in a largely naive population.

The local skin reaction visible over the site of a recent smallpox vaccination represents the clinical gold standard of successful vaccination, and serves as a surrogate correlate of immunity to disease.¹ A major reaction, or “take,” is characterized by a pustular lesion or an area of induration or congestion surrounding a central lesion, which can be a scab or an ulcer.² The rate of successful take among vaccinees has been reported to range from 53% to 99%, depending on age, prior vaccination status, vaccination technique, vaccine virus strain, and degree of vaccine dilution.^3–8 The World Health Organization (WHO) and the U.S. Centers for Disease Control and Prevention (CDC) recommend that both first-time vaccinees and non-naive vaccinees be instructed to return 6–8 days following vaccination for evaluation of take.^1,4,9–12 Some first-time or distantly vaccinated individuals may not show signs of a sufficient skin reaction, despite appropriate vaccination technique. In such cases, the lack of take is assumed to indicate that the individual is not immune to smallpox, and revaccination is indicated.^2,12,13 The recommendation for clinical evaluation of take poses a significant logistic constraint, as vaccinees must be seen by a health professional twice within 6–8 days. This could be especially problematic in the context of mass vaccination, when great numbers of vaccinees are expected to be processed within days, and the need to re-examine each would severely tap available medical resources. The availability of a simple tool that would allow valid, reliable self-assessment of take would permit self-screening by the vaccinees themselves. While the use of such a tool has been suggested as part of the U.S. vaccination response strategy for a smallpox outbreak,¹² the validity and practicality of its use have yet to be evaluated.

Routine childhood vaccination against smallpox was discontinued in Israel in 1980, but vaccination of military recruits continued until 1996.¹⁴ In September 2002, Israel initiated a national vaccination program against smallpox, targeting previously-vaccinated “first responders” among medical and emergency workers.¹⁵ Vaccination was limited to those who had been vaccinated in the past, while so-called naives were not eligible for vaccination within this program. Given the national history of childhood and military smallpox vaccination in Israel, persons born after January 1, 1978, were considered naive, and thus ineligible for inclusion. Vaccination was carried out simultaneously in the civilian and military sectors, according to guidelines published by the Ministry of Health.¹⁶ By mid-2003, some 21,000 people were vaccinated within the framework of this effort. We used this opportunity to validate a novel tool for self-evaluation of the smallpox vaccination site for the presence of take.

METHODS

Study population

We studied a consecutive sample of non-naive adults over the age of 24 who were vaccinated within the framework of the military first-responder program. This component targeted medical officers, public health and epidemiology teams, special medical response teams, and Homefront Command units.

Vaccination technique

All subjects received chick embryo Lister strain vaccinia vaccine delivered at a concentration of 1–2 ×10⁷ pock-forming units (PFU)/ml. Scarification was achieved using a multiple puncture technique, with a standard, beveled, non-bifurcated, 23G disposable needle inserted briskly 15 times through a 20-microliter vaccine droplet applied to the skin over the deltoid region. The vaccination site was then covered with a semi-permeable adhesive dressing. All subjects were screened for inclusion criteria, exclusion criteria, and medical contraindications,^10,16 and provided written consent to vaccination.

Self-evaluation tool

Subjects were instructed to report for physician verification of take 7–9 days after vaccination, in accordance with Ministry of Health guidelines.¹⁶ Immediately prior to physician examination, subjects were asked to fill out a simple self-assessment questionnaire, and to circle one of five color photographs which, in their opinion, most closely resembled the local skin reaction over the area of vaccination (Figure). Several physician evaluators participated in the study. All were trained in take recognition by the vaccination campaign coordinators, and were instructed to identify as “positive take” skin lesions that, upon physical examination, resembled photographs 3–5 in the questionnaire. Evaluators remained blinded to subjects' self-assessment, and recorded their own, independent assessment of take in a separate file.

Figure.

Self-assessment tool

Demographic variables

Information on subjects' gender, age, level of education, rank, and occupation was collected from computerized personnel records to examine the effects of these variables on the validity of the self-assessment tool.

Data analysis

We calculated the overall sensitivity, specificity, false-positive and false-negative rates, and positive and negative predictive values of self-assessment, using the physician evaluation result as the gold standard for comparison. We also computed the kappa statistic and percent of agreement, which measure the degree of concordance between self-assessment and physician assessment of the same individuals (a kappa value >0.75 indicates excellent agreement).^17,18 All measures of validity and agreement were then stratified by demographic variables and analyzed using χ² or Fisher exact tests of significance, to compare measures across variable categories. Multivariate logistic regression models were fitted for two separate outcome measures: overall subject-physician agreement (i.e., does subject agree or disagree with physician's determination), and subject-physician agreement limited to cases where take was absent (i.e., does subject agree or disagree with physician's determination that there was no take). This second model measures the tendency of a subject without take to provide a true-negative rather than a false-positive self-assessment. For inclusion in the regression models, demographic (exposure) variables were categorized and added in a forward stepwise fashion. Finally, we recomputed positive and negative predictive values and overall agreement of the self-assessment tool for a projected naive population, assuming identical sensitivity and specificity inherent to the tool, but adjusting for the higher clinical take rates expected among never-vaccinated subjects.^3–7 Statistical analysis was conducted using PEPI and SPSS computer software.^17,19

RESULTS

One hundred seventy-four subjects were studied, of whom 143 (82%) were males. The median age was 35. More than half of the subjects (58.7%) attended school for >12 years, 31.1% completed a 12-year high school education, and 10.2% completed fewer than 12 years of formal education. Some 41.3% held college degrees. Half of the study subjects were officers, and 61.1% held professional occupations, whereas 38.9% held nonprofessional and unskilled jobs, such as drivers and clerks. The prevalence of take among the study population, as determined by physician examination, was 63.2%.

Vaccinees and evaluators agreed on the outcomes of 157 of the 174 paired observations (90.2%). Measures of self-assessment validity are shown in Table 1. Overall sensitivity was high (99.1%), and specificity was 75%. The positive predictive value of self-assessment in the study population was 87.2% and the negative predictive value was 98%. The overall kappa statistic for all observations was 0.78.

Table 1.

Presence of take as assessed by vaccinees and physicians, and measures of self-assessment validity

When stratified by demographic variables, category-specific measures of sensitivity and positive and negative predictive values did not differ significantly. This was not the case for specificity, which was significantly modified across all demographic categories other than gender (Table 2). Specificity was markedly greater among younger and more highly educated subjects, as well as among officers and professionals. A similar pattern was seen for the stratified kappa and agreement calculations (Table 3). No gender difference was detected, but both kappa and percent of agreement were modified by age, education, and occupation.

Table 2.

Specificity of self-assessment by demographic variables

Table 3.

Kappa statistic and overall percent agreement of self-assessment, by demographic variables

Multivariate logistic regression models were fitted to test the demographic variables' ability to predict agreement outcomes. In the overall subject-physician agreement model, no demographic variable remained significant. In the specificity model, which tested subject agreement with the evaluator's determination that take was absent, occupation remained a highly significant predictor of concordance (odds ratio [OR]=10.0, 95% confidence interval [CI] 2.6, 38.3; p=0.008). This indicates that when take was in fact absent, professionals were 10 times more likely than nonprofessionals to provide a true-negative (as opposed to a false-positive) self-evaluation.

Positive and negative predictive values are strongly affected by the prevalence of the condition in question among the population under study—in this case take rates among non-naive vaccinees. Clinical take was present in 63.2% of our study subjects. Since take rates are expected to be much higher among naive vaccinees, we recomputed positive predictive value, negative predictive value, and overall agreement of the self-assessment tool to estimate its performance in a mass vaccination campaign designed to include naives, assuming a conservative overall clinical take rate of at least 90%.^3–8 Under these conditions, positive predictive value and overall agreement increased significantly compared to the values seen in the current study (97.3% vs. 87.2%; p=0.001 and 96.7% vs. 90.2%; p=0.02, respectively). Negative predictive value decreased in the projected population, but this difference was not statistically significant (90.2% vs. 98.0%; p=0.13).

DISCUSSION

While the potential need for widespread, emergency, population-based smallpox vaccination has recently refocused our attention on this long-eradicated disease, various issues relating to exigent vaccination of the masses have paradoxically remained under-studied. Specifically, several practical and logistical considerations continue to represent the major obstacles to successful and efficient mass vaccination.^8,12,20 While all smallpox preparedness plans include the issues of vaccine production, distribution, and administration, few have addressed the practical aspects of post-vaccination assessment of take, and the logistical considerations necessary for providing re-vaccination for non-responders. It is crucial that this key planning issue receive due attention now, in order to allow for the timely design of appropriate responses.

One solution to the logistical problem of summoning vaccinees back to the vaccination site for take evaluation may be self-assessment. It is not self-evident, however, that laypersons will understand what it is that they are being asked to assess, or that they will have the confidence to do so. As is evident from the results of our study, self-assessment is a feasible option, although it performs more poorly among older vaccinees, the less educated, and nonprofessionals.

The high sensitivity of our self-assessment tool indicates that it can be expected to meet its main logistical goal of keeping vaccination sites free and available for vaccination, unburdened by week-old vaccinees presenting for verification of take. Our results show that vaccinees can reliably identify take if it is present, suggesting that self-assessment can prevent unnecessary return to vaccination sites by sufficiently vaccinated people. The tool's lower specificity, however, indicates that some insufficiently vaccinated people may falsely identify their skin reactions as positive take. These vaccinees, under the impression that they have responded to the vaccine, will not present for re-vaccination. While this segment of the population does not signify an added logistical burden, it does represent a small but well defined segment of the population that may remain susceptible to disease due to insufficient vaccination. The predictive values and vaccinee-evaluator agreement of the tool are satisfactory, and can be expected to increase when used in a largely naive population.

The gold standard used in this study was the evaluation provided by the examining physician. Although several physician evaluators participated in this study, there was likely little inter-observer variation, since all evaluators were similarly trained in take recognition by the vaccination campaign coordinators, and based their evaluations on the resemblance of the lesions to one of the five color photographs depicted on the assessment questionnaire. Like other smallpox vaccine studies published in recent years, the current study was conducted on a selected population, comprised of military personnel with a specific over-representation of medical professionals. Thus, their ability and willingness to self-assess take may not represent those of the general population. We were able, however, to study a heterogeneous group of vaccinees, which permitted analyses of performance by gender, age, education, and occupation. While the self-assessment tool may present a practical alternative to trained evaluation, the generic prototype we tested may not be optimal for all population groups. Specific tools may need to be developed for poor-performance groups by replacing the photograph panel or by clarifying the self-analysis instructions. It is also possible that regardless of the tool's characteristics, certain population groups may be incapable of self-analysis. It is important to verify this possibility in order to permit effective contingency planning for professional assessment of take for these groups.

We envision various novel uses of self-assessment tools for the management of take verification and for evaluation of population immunity in emergency vaccination situations. For example, vaccinees can be instructed to visit an Internet website one week after vaccination, and to click on an image that most closely resembles the appearance of the vaccinee's take. Depending on the image selected, the vaccinee would either see a pop-up message reassuring that vaccination was successful, or, if the image selected indicated no take, would be referred to a list of health department sites for re-vaccination. Alternatively, the photograph panel can be broadcast on local television stations, accompanied by instructions to call a toll-free, on-screen telephone number. A recorded message would instruct the caller to press the touch-tone number that corresponds to the photograph that most closely resembles the caller's take. Callers indicating the presence of take would hear a reassuring message, while those indicating the absence of take would hear a list of locations for re-vaccination. Finally, if health departments can cooperate with cellular telephone service providers, vaccinees calling from mobile phones to report their take status could be informed in real-time about the re-vaccination site closest to their location. Regardless of the method employed, we urge planners to consider the integration of self-assessment into all mass smallpox vaccination contingency plans. While self-assessment appears to be a feasible alternative to centralized take evaluation, further investigation is required to better identify population groups at risk for poor self-assessment performance, and to seek alternative solutions for them.