Self-Report of Tobacco Use Status: Comparison of Paper-Based Questionnaire, Online Questionnaire, and Direct Face-to-Face Interview—Implications for Meaningful Use

Abstract

Identifying tobacco use status is essential to address use and provide resources to help patients quit. Being able to collect this information in an electronic format will become increasingly important, as the Centers for Medicare and Medicaid Services has included the assessment of tobacco use as part of its Stage 1 Meaningful Use criteria. The objective was to compare the accuracy of online vs. paper assessment methods to ascertain cigarette smoking status using a face-to-face structured interview as the gold standard. This was a retrospective analysis of a stratified opportunity sample of consecutive patients, reporting in 2010 for a periodic health evaluation, who completed either a scannable paper-based form or an online questionnaire and underwent a standardized rooming interview. Compared with face-to-face structured interview, the overall observed agreement and kappa coefficient for both methods combined (paper and online) were 97.7% and 0.69 (95% confidence interval (CI) 0.51–0.86) . For the online form they were 97.4% and 0.61 (95% CI 0.33–0.90), and for the paper form they were 97.9% and 0.75 (95% CI 0.54–0.96). There was no statistically significant difference in agreement between the online and paper-based methods (P=0.76) compared with a face-to-face structured interview. Online assessment of tobacco use status is as accurate as a paper questionnaire, and both methods have greater than 97% observed agreement with a face-to-face structured interview. The use of online assessment of tobacco use status has several advantages and more widespread use should be explored. (Population Health Management 2014;17:185–189)

Introduction

Because of the widely recognized importance of tobacco use as a health threat to individuals and populations, health care providers are compelled to ask about and document tobacco use status during outpatient visits. This documentation is usually the trigger for providing counseling and other evidence-based interventions recommended for tobacco dependence treatment. Collecting, maintaining, summarizing, and acting upon this information is important to improve the health of individual patients, population health, and measurement of the quality of care provided by organizations and programs. With the growing use of electronic medical records, this assessment is and will increasingly be captured in electronic format.

The Centers for Medicare and Medicaid Services (CMS) has focused on tobacco use assessment as one of the 25 Stage 1 Meaningful Use objectives.¹ Meaningful Use is a national initiative designed to encourage adoption and standardized use of electronic health record (EHR) technology. It is legislated under the Health Information Technology for Economic and Clinical Health (HITECH) Act, which was enacted under the American Recovery and Reinvestment Act of 2009. The performance of both individual practitioners and hospitals on multiple measures will be assessed using data stored in a certified EHR.² Furthermore, financial incentives are being offered in the early years of the program to acknowledge that this process and the technology to support it will be expensive.³ Therefore, health care organizations must find a method of capturing these data that is consistent, cost-effective, and user friendly to allow analysis and meaningful use of the data. Optimally, the data capture should require minimal manual effort and not be intrusive to the daily clinic workflow.

Mayo Clinic implemented an electronic capture of a structured set of clinically important “pre-visit” questions in 1995—the Current Visit Information form or CVI.^4,5 This was originally a scannable paper questionnaire that was computer interpretable. Since that time the self-reported tobacco use status has been identified in more than 1 million patients and recorded in the electronic medical record. The study team previously evaluated the accuracy of self-reported data from the Mayo CVI questionnaire and found it to be clinically useful and valid.^6,7 In 2010, Mayo Clinic implemented an online version of the scannable paper CVI (eCVI) that could be completed by patients prior to an appointment. The eCVI offers some advantages over paper in its ability to use branching logic (for instance, if a patient reports never having used tobacco, what type and how much tobacco is being used will not be asked) and does not allow for empty fields or “unknown status.” In addition, as part of a standardized rooming process used by the Department of Medicine at Mayo Clinic,⁵ a clinical assistant asks each patient a structured set of questions including current medications, allergies, and smoking status.

Although self-report of tobacco use status has been found to be a valid approach,^8,9 the validity of an online assessment of tobacco use is not well established. Therefore, the study team conducted this study to compare these 3 methods: online (eCVI), handwritten/scannable form (CVI), and structured face-to-face interview with the latter being considered the gold standard method.

Methods

This was a retrospective analysis of a stratified opportunity sample of consecutive patients, reporting in 2010 to the Mayo Clinic Division of Preventive and Occupational Medicine for a periodic health evaluation, who completed either a CVI or an eCVI and underwent a standardized rooming interview by a trained clinical assistant.

After excluding patients without general clinical research authorization, 235 consecutive patients who completed the paper questionnaire and 235 consecutive patients who completed the online questionnaire were included. Only patients who smoked cigarettes were included (with or without other forms of tobacco) because the Stage 1 Meaningful Use criteria address only cigarette smoking.

The standardized rooming process used by the Department of Medicine at Mayo Clinic⁵ included asking a structured set of questions including current medications, allergies, and smoking status. The answers were subsequently entered into the medical record. Data were abstracted from the electronic medical record by a trained abstractor. These data included smoking status, age, sex, and educational level. Twenty randomly selected chart abstractions were reviewed by 2 investigators to ensure accuracy and consistency.

The CVI and eCVI reported smoking statuses were compared with face-to-face interview results. The Cohen's kappa coeffient was used to describe the chance adjusted agreement. A kappa>0.75, 0.40–0.75, and <0.40 were considered excellent, fair, and poor, respectively. The observed agreement among different age groups, sex, and education levels were compared by using Pearson's chi-square test. A 2-tailed P value less than 0.05 was considered statistically significant. All analyses were performed using SAS version 9.3 software (SAS Institute Inc., Cary, NC).

Results

Patient selection and inclusion are depicted in Figure 1. A total of 470 patients were included in the analysis (235 in each arm). The smoking prevalence was 10% in both the eCVI and CVI groups. The results of the overall, eCVI, and CVI analyses are displayed in Tables 1–3, respectively. The overall observed agreement combining both the eCVI and CVI compared to the face-to-face interview was 97.7% with a kappa of 0.69 (95% CI 0.52–0.86). The observed agreement between the eCVI and face-to-face interview was 97.4% with a kappa 0.61 (95% CI 0.33–0.90). The observed agreement between the CVI and face-to-face interview was 97.9% with a kappa of 0.75 (95% CI 0.54–0.96). There was no statistical difference in agreement between the eCVI and CVI (P=0.7603). When stratified by age, sex, and education, the only significant difference was noted in the age group filling out the eCVI who were aged 80–89; however, there were only 2 individuals in this age group yielding an observed agreement of 50%. This difference was not present when analyzed using all those older than age 65.

FIG. 1.

Flowchart of patient inclusion.

Table 1.

Overall Agreement (eCVI and CVI) vs. Face-to-Face Interview

				95% CI
eCVI and CVI	Number	% Agreement	Kappa	Lower	Upper	P value
Overall	470	97.7	0.69	0.52	0.86
Age
20–29	2	100.0				0.60
30–39	20	95.0	0.64	0.01	1.00
40–49	99	97.0	0.71	0.40	1.00
50–59	140	98.6	0.83	0.59	1.00
60–69	147	97.3	0.59	0.22	0.95
70–79	50	100.0
80–89	12	91.7
Age
28–49	121	96.7	0.69	0.42	0.98	0.60
50–59	140	98.6	0.83	0.59	1.00
60–64	86	96.5	0.56	0.12	1.00
65+	123	98.4	0.49	−0.11	1.00
Sex
Male	298	97.3	0.68	0.47	0.89	0.75
Female	172	98.3	0.72	0.42	1.00
Education*
Some high school	4	100.0				0.39
High school, GED	24	91.7	0.47	−0.13	1.00
Some college	71	98.6	0.79	0.40	1.00
College graduate	143	97.9	0.39	−0.16	0.94
Postgraduate study	222	97.7	0.77	0.58	0.96

CVI, Current Visit Information; eCVI, online CVI questionnaire; CI, confidence interval; GED, general equivalency diploma.

^*Six individuals did not answer this question on the CVI.

Table 2.

Agreement Between eCVI and Face-to-Face Interview

				95% CI
eCVI	Number	% Agreement	Kappa	Lower	Upper	P value
Overall	235	97.4	0.61	0.33	0.90
Age
20–29	0					0.0006
30–39	14	100.0
40–49	59	96.6	0.48	−0.13	1.00
50–59	71	100.0
60–69	71	95.8	0.39	−0.15	0.92
70–79	18	100.0
80–89	2	50.0
Age
28–49	73	97.3	0.65	0.20	1.00	0.1989
50–59	71	100.0
60–64	47	93.6	0.38	−0.15	0.91
65+	44	97.7
Sex
Male	160	97.5	0.65	0.34	0.97	1.0000
Female	75	97.3	0.49	−0.01	1.00
Education
Some high school	1	100.0				0.5446
High school, GED	9	88.9	0.61	−0.06	1.00
Some college	32	96.9	0.65	0.02	1.00
College graduate	70	98.6	0.66	0.04	1.00
Postgraduate study	123	97.6	0.56	0.11	1.00

CVI, Current Visit Information; eCVI, online CVI questionnaire; CI, confidence interval; GED, general equivalency diploma.

Table 3.

Agreement between CVI and Face-to-Face Interview

				95% CI
CVI	Number	% Agreement	Kappa	Lower	Upper	P value
Overall	235	97.9	0.75	0.54	0.96
Age
20–29	2	100.0				0.2761
30–39	6	83.3
40–49	40	97.5	0.84	0.55	1.00
50–59	69	97.1	0.83	0.59	1.00
60–69	76	98.7	0.79	0.40	1.00
70–79	32	100.0
80–89	10	91.7
Age
28–49	48	95.8	0.73	0.37	1.00	0.5154
50–59	69	97.1	0.73	0.38	1.00
60–64	39	100.0
65+	79	98.7	0.66	0.04	1.00
Sex
Male	138	97.1	0.70	0.42	0.98	0.6513
Female	97	99.0	0.85	0.57	1.00
Education*
Some high school	3	100.0				0.6505
High school, GED	15	93.3
Some college	39	100.0
College graduate	73	97.3
Postgraduate study	99	98.0	0.86	0.68	1.00

CVI, Current Visit Information; CI, confidence interval; GED, general equivalency diploma.

^*Six individuals did not answer this question on the CVI.

Discussion

The observed agreement between eCVI and CVI compared with the face-to-face structured interview was excellent. The only statistically significant difference between the 2 methods was in those aged 80–89, but the number of individuals in this age group was too small to draw any inferences. Further study will be needed to determine if the validity of technology-assisted self-entry of medical data is adversely affected by advancing age. In all other groups the observed agreement between the eCVI and CVI was equivalent, and there was no statistical difference between the 2 methods of assessing tobacco use status.

Prior studies have shown face-to-face self-report of tobacco use status to be valid.^8,9 Other studies have shown that paper and computer-aided self-assessment of behaviors, including substance abuse, are valid^10–17; however, to the study team's knowledge, this is the first study to assess the validity of an online and paper assessment of tobacco use.

Limitations of this study include the retrospective observational design and self-reporting of smoking status. Bias from these limitations likely is minimized by the standardized rooming process, including the face-to-face assessment of smoking status that was in place during the time frame of this study. In addition, self-report of tobacco use status has been found to be a valid approach to determining tobacco use status,^8,9 and there is no reason to suspect that this would not hold true in this population.

The results of this study suggest that an online questionnaire is a valid method to assess tobacco use status and should be an acceptable alternative to a paper survey or face-to-face interview to fulfill the CMS Meaningful Use criteria. Meaningful Use criteria require that tobacco use be captured as structured data.¹⁸ Assessing tobacco use status online has several potential advantages including the collection of already structured data that is searchable and easily reported to satisfy Meaningful Use criteria. In addition, this would increase practice efficiency by unloading tasks that are typically performed by medically trained personnel, as well as standardizing the assessment so that there is no variability between patients in how tobacco use status is assessed. These benefits also hold true for other Meaningful Use criteria that include the requirement for collection of structured data. In 2014 providers will not only have to assess tobacco use, but they will be required to provide cessation counseling intervention to those who are identified as tobacco users.¹⁸ Online assessment could have benefit here as well because online assessment has the ability for automatic branching logic that could direct identified tobacco users to receive some type of tobacco cessation intervention. The ultimate goal is to decrease the number of individuals who are using tobacco. Now, armed with the information to correctly identify tobacco users through various methods including online assessment, we can begin to leverage the technology to satisfy not only CMS Meaningful Use criteria, but also—and more importantly—encourage the treatment of tobacco use and decrease the adverse health effects caused by tobacco in the population at large.

Conclusion

Online assessment of tobacco use status is as accurate as a paper questionnaire, and both methods have greater than 97% observed agreement with a face-to-face structured interview. The use of online assessment of tobacco use status has several advantages and more widespread use should be explored.

Author Disclosure Statement

Drs. Steffen, Murad, Hays, Newcomb, Molella, and Hagen, and Mr. Cha declared no conflicts of interest with respect to the research, authorship, and/or publication of this article. This study was supported by internal funding from the Division of Preventive Medicine, Mayo Clinic, Rochester, MN.