Responsiveness of a Brief Measure of Lung Cancer Screening Knowledge

Abstract

Our aim was to examine the responsiveness of a lung cancer screening brief knowledge measure (LCS-12). Eligible participants were aged 55–80 years, current smokers or had quit within 15 years, and English speaking. They completed a baseline pretest survey, viewed a lung cancer screening video-based patient decision aid, and then filled out a follow-up posttest survey. We performed a paired samples t-test, calculated effect size, and calculated absolute and relative percent improvement for each item. Participants (n=30) were primarily White (63%) with less than a college degree (63%), and half were female (50%). Mean age was 61.5 years (standard deviation [SD]=4.67) and average smoking history was 30.4 pack-years (range=4.6 – 90.0). Mean score on the 12-item measure increased from 47.3% correct on the pretest to 80.3% correct on the posttest (mean pretest score=5.67 vs. mean posttest score=9.63; mean score difference=3.97, SD=2.87, 95% CI=2.90, 5.04). Total knowledge scores improved significantly and were responsive to the decision aid intervention (paired samples t-test=7.57, p<.001; Cohen’s Effect Size=1.59; Standard Response Mean [SRM]=1.38). All individual items were responsive, yet two items had lower absolute responsiveness than the others (item 8: “Without screening, is lung cancer often found at a later stage when cure is less likely?” pretest correct=83.3% vs. posttest=96.7%, responsiveness=13.4%; and item 10: “Can a CT scan find lung disease that is not cancer?” pretest correct=80.0% vs. posttest=93.3%, responsiveness=13.3%). The LCS-12 knowledge measure may be a useful outcome measure of shared decision making for lung cancer screening.

Introduction

Smoking persists as the number one risk factor for lung cancer [1, 2]. In 2016, approximately 158,000 deaths are expected in the US from lung/bronchus cancer, which account for over one quarter of all cancer deaths [2]. The National Lung Screening Trial (NLST) observed a 20% reduction in lung cancer deaths for heavy smokers 55 to 74 years of age who received three annual screenings with low-dose computed tomography (LDCT) compared to standard screening chest x-rays [3]. Yet, lung cancer screening carries risks, such as false-positive findings leading to unnecessary invasive procedures, radiation exposure, and overdiagnosis [1, 4]. Prompted in large part by the NLST findings, the U.S. Preventive Services Task Force (USPSTF) updated its guidelines by providing a Grade B recommendation for lung cancer screening with LDCT for those 55–80 years of age who have a 30 pack-year smoking history and currently smoke or have quit within the past 15 years. Those who choose to undergo lung cancer screening should have an annual LDCT until they have not smoked for 15 years or develop a substantial comorbidity [5]. In its national coverage determination, the Centers for Medicare & Medicaid Services (CMS) includes an unprecedented requirement: a patient counseling and shared decision making visit with use of one or more patient decision aids prior to referral for lung cancer screening [6]. A shared decision making approach helps patients learn about the trade-offs of lung cancer screening and discuss their values and priorities with their physician, [1, 7] with the goal of making informed, values-concordant decisions[7–14].

The USPSTF recommendation and CMS national coverage determination have been a catalyst for the development of patient decision aids and broader shared decision making interventions for lung cancer screening. As more decision aids are being developed, there is momentum to create certification standards to ensure the quality and effectiveness of decision aids [9, 15]. One concept supported by certification standards is knowledge, a necessary component of decision quality [8–13]. Previously, we developed a brief measure, the LCS-12, to characterize knowledge of lung cancer screening among smokers and former smokers and to serve as an outcome measure for studies evaluating shared decision making interventions[16]. The objective of this study is to examine the responsiveness of the LCS-12 to capture changes in patient’s knowledge after viewing a video-based patient decision aid about lung cancer screening. Evaluation of item responsiveness was part of a measure development study (ClinicalTrials.gov ID: NCT02282969) and was conducted to inform a randomized control trial to evaluate a patient decision aid (Clinicaltrials.gov ID: NCT02286713).

Methods

In prior work, we developed the LCS-12 based on a consensus process where key facts were identified as essential knowledge for making an informed decision about lung cancer screening. The 12-item measure showed good test-retest reliability (ICC=.84) and discrimination between high and low performers, although item difficulty was high for several items [16]. We retained several high difficulty items due to their coverage of key concepts in making informed screening decisions, such as positive predictive value of an abnormal screening result.

For this investigation, we examined the responsiveness of the total score and individual items, including items with high difficulty. We assessed knowledge scores among current and former smokers before and after they viewed a video-based patient decision aid. The patient decision aid is described in detail in previous work [7]. The video-based patient decision aid was narrated from text written at an 8^th grade reading level. It provides an overview of lung cancer risk, the experience of a LDCT scan, and the magnitude of the benefits and harms of screening, and it encourages users to consider their personal values in making a decision about screening [7].

Participants were recruited from: (1) local community locations and social media, (2) the Tobacco Treatment Program at The University of Texas MD Anderson Cancer Center, (3) or had previously indicated a willingness to participate in other studies through lung cancer screening or smoking cession programs at MD Anderson. Eligible participants were aged 55 to 80 years, current smokers or former smokers who had quit within the last 15 years, English speaking, and had no history of lung cancer. Surveys were sent and data were collected via the secure Research Electronic Data Capture (REDCap ) tool [17]. An online survey link was sent to those who agreed to participate. Following a waiver of signed consent, participants who indicated consent through REDCap completed a demographic questionnaire and baseline knowledge pretest survey, viewed the short video-based patient decision aid about lung cancer screening, and completed a follow-up knowledge posttest survey which was the same as the baseline knowledge pretest.

Measure responsiveness refers to a measure’s ability to detect absolute change in a particular item and the magnitude of the change [18–20]. We used a paired samples t-test for the full measure score to assess responsiveness. For individual items, responsiveness was calculated to show the absolute change between the percent correct on the pretest and posttest [20]. We also calculated the relative percent change between scores for individual items on the pretest and posttest. To further assess the magnitude of change in knowledge scores, we calculated Cohen’s Effect Size and the Standard Response Mean (SRM) as proposed by Husted et al. [20]. Cohen’s Effect Size was calculated as the mean pretest and posttest total score difference divided by the standard deviation of the mean pretest score (mean difference/mean pretest standard deviation [SD]) [20–23]. The SRM was calculated as the mean pretest and posttest total score difference divided by the mean difference standard deviation (mean difference/mean difference SD) [20, 24]. Participants were provided a $50 gift card for their participation. Statistical analyses were completed using SPSS. This study was approved by MD Anderson's Institutional Review Board.

Results

Participants (n=30) were primarily White (63%) with less than a college degree (63%), and half were female (50%) (Table 1). They had a mean age of 61.5 years (SD=4.67) and had smoked for an average of 30.4 pack-years (SD=18.87, range=4.6 – 90.0). Mean total score on the 12-item measure increased from 47.3% correct on the pretest to 80.3% correct on the posttest (mean pretest score=5.67 vs. mean posttest score=9.63; mean score difference=3.97, SD=2.87, 95% CI=2.90, 5.04) [24]. Knowledge scores improved significantly from the pretest to posttest (paired samples t-test=7.57, p<0.001). Cohen’s Effect Size was 1.59 and the SRM was 1.38, both indicating a large responsiveness to change [20–22, 24, 25].

Table 1.

Demographic Characteristics.

Demographic Characteristic n=30	Total n (%)
Age in years, Mean (SD)	61.5 (4.7)
Gender (woman)	15 (50.0)
Race/ethnicity
Black/African American	9 (30.0)
Hispanic/Latino	1 (3.3)
White	19 (63.3)
Other	1 (3.3)
Education
High school graduate/G.E.D.	7 (23.3)
Some college/trade school	12 (40.0)
College degree or more	11 (36.7)
Health Insurance
Medicare	10 (33.3)
Medicaid	4 (13.3)
From job or private insurance	11 (36.7)
Other	5 (16.7)
Smoking Status
Current Smoker	20 (60.7)
Former Smoker	10 (33.3)
Pack Years (n=29), Mean (SD)	30.4 (18.9)

Note: All results are presented as number and percentage unless noted otherwise. Abbreviations: SD=standard deviation; G.E.D=general education development.

No participant selected all the correct responses to item 3, “When should someone stop being screened for lung cancer? (Check all that apply); Answer choices: 1—You quit smoking more than 15 years ago (correct response); 2—Your last CT scans shows you don’t have cancer (incorrect response); 3—You have other health problems that may shorten your life (correct response); 4—You are not able or willing to be treated for lung cancer (correct response); 5—I don’t know.” However, over half (57%) of participants selected at least one correct response, and were accordingly categorized as answering the question correctly. Most items demonstrated an absolute responsiveness of 20% or greater (Table 2). At the pretest, two items were answered correctly by approximately 80% of participants, suggesting ceiling effects for items 8 and 10, respectively: “Without screening, is lung cancer often found at a later stage when cure is less likely?”, and “Can a CT scan find lung disease that is not lung cancer?”, (item 8 pretest correct=83.3% vs. posttest=96.7%, responsiveness=13.4%; item 10 pretest correct=80.0% vs. posttest=93.3%, responsiveness=13.3%).

Table 2.

Question Items and Responsiveness for Lung Cancer Screening Measure.

Item	Item difficulty* (%)		Responsiveness** (%)	Percent Improvement† (%)**
	Pretest	Posttest
1. What percentage of lung cancer deaths are caused by smoking?	23.3	43.3	20.0	85.7
2. Where does lung cancer rank as a cause of cancer death in the US?	33.3	96.7	63.3	190.0
3. When to stop screening (at least 1 correct)	56.7	86.7	30.0	52.9
4. How many people with an abnormal CT scan will have lung cancer?	16.7	40.0	23.3	140.0
5. Can a CT scan suggest that you have lung cancer when you do not?	40.0	96.7	56.7	141.7
6. Can a CT scan miss a tumor in your lungs?	73.3	93.3	20.0	27.3
7. Will all tumors found in the lungs grow to be life threatening?	70.0	96.7	26.7	38.1
8. Without screening, is lung cancer often found at a later stage when cure is less likely?	83.3	96.7	13.4	16.0
9. How much does screening for lung cancer with a CT scan lower your chances of dying from lung cancer?	6.7	40.0	33.3	499.7
10. Can a CT scan find lung disease that is not cancer?	80.0	93.3	13.3	16.7
11. Can a CT scan find heart disease?	56.7	90.0	33.3	58.8
12. Is radiation exposure one of the harms of lung cancer screening?	33.3	90.0	56.7	170.0

^*Item Difficulty: Proportion of correct responses

^**Responsiveness (absolute) = (posttest percent correct score) – (pretest percent correct score)

^†Percent Improvement (relative) = (posttest correct score – pretest correct score)/(pretest correct score)*100

Discussion

The LCS-12 showed good responsiveness with significant improvements in scores after participants viewed video-based patient decision aid. All items were responsive, including those with high item difficulty identified in the original development study (e.g., item 9, mortality benefit of lung cancer screening). While all items showed participant knowledge improvement, some were more responsive than others. Items with high baseline scores (e.g., item 8 on the mortality benefit of screening and item 10 on secondary findings) showed less improvement, but we suggest these items be retained because they reflect important concepts in making informed screening decisions about whether to get screened for lung cancer [16].

Overall score improvement is consistent with previous work describing the importance of knowledge as an outcome measure for evaluating decision aids [13, 26]. We found total score as well as individual items, supporting reliability and validity of the measure [16, 18, 19].

These findings should be interpreted within the context of study limitations. Participants were recruited from those who had previously indicated interest in lung cancer screening or were part of a tobacco cessation program. They may be more interested in or knowledgeable about lung cancer screening. Yet, the measure was responsiveness to knowledge gained by viewing a video-based patient decision aid about lung cancer screening. Additionally, not all items show strong responsiveness. While items 8 and 10 have lower responsiveness and may show ceiling effects, the domains they cover (secondary findings, and mortality benefit of screening benefit) are important for informed decision making about lung cancer screening per expert review [16].

In conclusion, the LCS-12 measure is responsive and can be used in descriptive studies or serve as an outcome measure in intervention studies where improvement of lung cancer screening knowledge is a goal. The Patient Protection and Affordable Care Act includes provisions for the certification of decision aids. Providing evidence that a decision aid improves decision quality will likely be a key component of the certification process. This measure may be useful in certifying decision aids about lung cancer screening by showing that they improve knowledge, one essential component of decision quality. It will be useful in evaluating the quality of shared decision making visits using decision aids as required by CMS [5, 6, 8–13]. Future work assessing the feasibility of this measure can clarify strategies for incorporating lung cancer screening shared decision making into the clinical setting.