Measuring the Impact of Quantitative Information on Patient Understanding: Approaches for Assessing the Adequacy of Patient Knowledge about Colorectal Cancer Screening

Abstract

Background. Guidelines recommend that decision aids disclose quantitative information to patients considering colorectal cancer (CRC) screening, but the impact on patient knowledge and decision making is limited. An important challenge for assessing any disclosure involves determining when an individual has “adequate knowledge” to make a decision. Methods. We analyzed data from a trial that randomized 213 patients to view a decision aid about CRC screening that contained verbal information (qualitative arm) versus one containing verbal plus quantitative information (quantitative arm). We analyzed participants’ answers to 8 “qualitative knowledge” questions, which did not cover the quantitative information, at baseline (T0) and after viewing the decision aid (T1). We introduce a novel approach that defines adequate knowledge as correctly answering all of a subset of questions that are particularly relevant because of the participant’s test choice (“Choice-Based Knowledge Assessment”). Results. Participants in the quantitative arm answered a higher mean number of knowledge questions correctly at T1 than did participants in the qualitative arm (7.3 v. 6.9, P < 0.05), and they more frequently had adequate knowledge at T1 based on a cutoff of 6 or 7 correct out of 8 (94% v. 83%, P < 0.05, and 86% v. 71%, P < 0.05, respectively). Members of the quantitative group also more frequently had adequate knowledge at T1 when assessed by Choice-Based Knowledge Assessment (87% v. 76%, P < 0.05). Conclusions. Patients who viewed quantitative information in addition to verbal information had greater qualitative knowledge and more frequently had adequate knowledge compared with those who viewed verbal information alone, according to most ways of defining adequate knowledge. Quantitative information may have helped participants better understand qualitative or gist concepts.

Trial Registration: ClinicalTrials.gov ID# NCT01415479

Highlights

• Patients who viewed quantitative information in a decision aid about colorectal cancer screening were more knowledgeable about nonquantitative information and were more likely to have adequate knowledge according to a variety of approaches for assessing that, compared with individuals who viewed only qualitative information. This result supports the inclusion of quantitative information in decision aids.

• Researchers assessing patient understanding should consider a variety of ways to define adequate knowledge when assessing decision quality.

Introduction

Challenges and Previous Assessments of “Adequate Knowledge”

Possessing adequate knowledge is an essential part of making an informed medical decision. If patients do not adequately understand basic facts about their options—the risks, benefits, and alternatives—then consent is ethically questionable.¹ While subjective measures of knowledge assess how a person feels about their level of understanding (e.g., Decision Conflict Scale²), objective measures of knowledge assess the individual’s understanding of specific facts. Objective measures play an essential role in some approaches to assessing the quality of a medical decision, such as the Multi-dimensional Measure of Informed Consent (MMIC)³ and the Colorectal Cancer Decision Quality Instrument.⁴

Researchers have developed tests to assess objective knowledge, usually composed of multiple-choice and true-false questions covering key facts. Sometimes these tests are designed to determine if the participant’s knowledge level is “adequate.”^4–6 One central challenge for formulating and evaluating knowledge tests is that an adequate level of knowledge must be achievable without knowing everything that health professionals know. Legal and ethical theories of informed consent have attempted to define what subset of information must be disclosed to a patient or potential research participant.¹ The reasonable person standard, for example, requires disclosure of all information that would be material to the decision for a reasonable person.¹ The standard is importantly vague, though, due to the terms material and reasonable.

There is no simple answer about what information should be disclosed or what questions should be used to test whether a patient or potential research participant has adequate knowledge.^7–10 There are 3 key issues that make this determination challenging.

First, the designer of a knowledge test must decide what facts or topics should be covered. As mentioned above, a vast number of facts are relevant to any informed choice, and only a subset can be disclosed during informed consent or evaluated on a knowledge test.

Second, there are many ways to write a question to assess comprehension of a given fact or topic, and how a question is written will affect how many individuals will answer it correctly. For example, patients have a higher chance of answering a true-false question correctly than a multiple-choice question with 4 options or a fill-in-the-blank question about the same fact.

Third, a designer of a knowledge test must decide how many of the questions must be answered correctly for the subject to count as having adequate knowledge. An argument could be made for requiring 100% correct, since the questions were chosen because they are considered essential to an informed choice. On the other hand, many knowledge tests judge patients to have adequate knowledge even if they only get 80% or even fewer correct. Every test in school, after all, has a passing score lower than 100%.

Knowledge Assessment in Colorectal Cancer Screening

This article examines these issues in relation to colorectal cancer (CRC) screening and the design of decision aids. While CRC screening is recommended by the United States Preventive Services Task Force,¹¹ American Cancer Society,¹² and others^13,14 for all people ages 45 to 75 y, people with average risk can choose from a variety of approved tests, most commonly colonoscopy or stool testing (FIT or Cologuard).

Colonoscopy provides a direct visual examination of the colon, can remove most polyps found, and is performed just once every 10 y if normal. However, it is also an invasive procedure that uses sedation, must be performed in a hospital setting, and requires the patient to drink a strong laxative to clean out the colon the day before the procedure. Stool tests are noninvasive and can be performed at home, although they are conducted annually, require handling stool, can fail to identify polyps and cancers, and positive stool tests require a follow-up colonoscopy.¹¹

Multiple studies have examined patient understanding of their options, often before and after these patients have reviewed a decision aid about CRC screening.¹⁵ For instance, a recent review identified 9 studies that measured knowledge before and after a patient viewed a self-administered decision aid (i.e., outside of a clinic visit).¹⁶ A vast number of other papers have measured lay individuals’ knowledge of CRC screening in a wide variety of settings,¹⁷ although a smaller number aimed to identify whether individuals had adequate knowledge. We selected 3 papers that did this as part of an attempt to evaluate the quality of decision making following the MMIC approach³: these 3 papers were selected to display the range of possible approaches,^4–6 since a more systematic review is beyond the scope of this article.

The knowledge tests used by these studies (Table 1) display great variability. While the authors do not explicitly discuss the reasons for choosing which facts to cover, their design of individual questions and the passing scores suggest important value judgments. Such judgments are independent of the standard evaluation of questions on surveys or questionnaires, such as psychometric properties, face validity, ceiling or floor effects, and ability to discriminate between different groups with presumed different levels of knowledge.⁴

Table 1.

Knowledge Questions from Previous Studies of Colorectal Cancer (CRC) Screening Organized by CRC Screening Fact or Knowledge Area

Category	Question
Baseline bowel cancer risk without screening	How many men/women out of 1000 will die from bowel cancer over the next 10 y without screening? (Smith, question 1)
	Out of every 100 people, about how many will get colon cancer some time in their lives? (Sepucha, question 2)
	Out of every 100 people, about how many will die of colon cancer? (Sepucha, question 10)
Risk reduction in bowel cancer deaths by screening (Baseline bowel cancer risk with regular screening)	How many men/women out of 1000 will die from bowel cancer over the next 10 y with screening? (Smith, question 2)
	How does regular testing for colon cancer change the chances that a person will die from colon cancer? (Sepucha, question 7)
Possibility of false positives	Do all men/women who have an abnormal bowel cancer screening test result have bowel cancer? (Smith, question 3)
	Does having a colon cancer test result that is not normal always mean that a person has colon cancer? (Sepucha, question 3)
	If the stool test detects blood, there is a 100% chance of CRC (incorrect). (Denters, question 10)
Numeric risk of experiencing a false positive	How many men/women out of 1000 will have an abnormal result, go for extra tests, and find out they do not have cancer? (Smith, question 4)
Possibility of false negatives	Do you think the bowel cancer screening test will find every bowel cancer? (Smith, question 5)
	Which colon cancer test is least likely to miss a cancer? (Sepucha, question 8)
	If the results of a colon cancer test are normal, is it possible that a person could still have colon cancer? (Sepucha, question 9)
	If a person has CRC, there is a 100% chance the stool test will detect this (incorrect). (Denters, question 11)
	The follow-up investigation [to a positive stool test] (a colonoscopy) is in almost 100% of cases correct in detecting CRC. (Denters, question 14)
Awareness of age guidelines	At what age do doctors usually recommend people start getting regular tests for colon cancer? (Sepucha, question 1)
	Participation in the screening program is obligatory for persons between the ages of 50 and 75 years. (Denters, question 18)
Risks associated with colonoscopy	How often do serious problems, such as serious bleeding or a tear in the colon, happen as a result of a colonoscopy? (Sepucha, question 4)
	The follow-up investigation (a colonoscopy) can cause a colonic perforation or bleeding (correct). (Denters, question 12)
Frequency of screening tests	For a person with an average risk for colon cancer, which test do doctors recommend be done every year? (Sepucha, question 5)
	For a person with an average risk for colon cancer, which test do doctors recommend be done every 10 years? (Sepucha, question 6)
	The stool test has to be repeated every 2 years (correct). (Denters, question 9)
Facts about colon cancer	Inheritability	CRC can be hereditary. (Denters, question 4)
	Possibility of being asymptomatic	In the absence of symptoms, participation in not useful. (Denters, question 7)
		A mass screening program can detect CRC before it becomes symptomatic. (Denters, question 1)
	Miscellaneous	The presence of blood in the stool can be a sign of CRC. (Denters, question 8)
		CRC is one the most prevalent cancers in the Netherlands. (Denters, question 5)
		Younger persons have a higher chance of being diagnosed with CRC than older persons. (Denters, question 6)
		CRC has a better chance of survival when detected in an early stage. (Denters, question 2)
		Persons can die from CRC when not treated on time. (Denters, question 3)
Information regarding screening procedures and follow-up	Positive stool test follow-up	If the stool test detects blood, a follow-up investigation is necessary to check for the presence of CRC. (Denters, question 15)
	Miscellaneous	If the colonoscopy detects precursors of CRC, these can almost always be removed in the same procedure. (Denters, question 16)
		After removal of precursor lesions, regular checkups of the bowel are not necessary. (Denters, question 17)
		If a perforation or bleeding occurs, surgery is the only treatment option. (Denters, question 13)

The knowledge tests we reviewed cover a wide range of topics. The only topics that are covered by all 3 tests are the possibilities of a false positive and false negative, although only 1 asks the specific false-positive rate.⁵ Two out of the 3 tests ask the specific (quantitative) risk of CRC with and without screening, with 1 test inquiring about 10-y mortality⁵ and another about lifetime incidence and mortality.⁴ Two of the 3 knowledge tests ask about the frequency of specific screening tests, the ages at which screening is recommended, and colonoscopy’s potential complications.^4,6 One test includes multiple questions about screening procedures and follow-up that are not covered by the other knowledge tests.⁶ Questions on different tests about the same topics are framed differently, sometimes as true-false and sometimes as multiple choice.

The reviewed tests also use very different passing scores, with minimal explanation of reasons for this choice. Using a “competency-based approach,” Smith et al.⁵“decided a priori that participants were considered to have ‘adequate knowledge’ if their total score was 6 or more out of 12,” or 50%. Denters et al. grouped their 18 knowledge questions into 2 categories (general knowledge and screening specific items) and relied on an expert panel to decide:

“that knowledge can be considered adequate if at least two-thirds of knowledge items had been answered correctly (total knowledge score ≥ 12) under the condition that at least half of the items on general knowledge (general knowledge score ≥ 4) and at least two-thirds of screening specific items (screening specific knowledge score ≥ 8) were answered correctly.”⁶

Sepucha et al.⁴ defined adequate knowledge as having a “knowledge score at or above the mean knowledge of the decision aid group.”

Clearly, these differences in knowledge tests can have a major impact on whether an individual counts as having adequate knowledge or not: the same person could pass one test and fail another. For studies that compare decision aids or other interventions by appeal to the percentage of individuals achieving adequate knowledge, these differences in knowledge tests might affect a study’s findings. A closer examination of the justification for choosing specific questions and setting a passing score appears essential to improving the normative evaluation of decision support and patient choices.

Rationale for analysis

In this study, we report a secondary analysis of a randomized trial conducted by our team that measured the impact of adding quantitative information to a decision aid regarding CRC screening.¹⁸ The study also measured the impact of adding a nudge to increase the chance that the patient will choose FIT for screening, but that part of the study is not the focus of this secondary analysis. Widely used guidelines recommend that decision aids should include quantitative information about probabilities or frequencies specifying baseline risk, risk reduction, and possible negative consequences, to support informed choice and shared decision making.¹⁹ There is limited evidence, however, that disclosure of quantitative information of this sort improves decision quality.^20,21 Previous studies have shown that disclosure of quantitative information increases the accuracy of patient perceptions, measured as correct descriptions of the probabilities that were disclosed, but these studies have not analyzed the impact on other parts of patient knowledge²² or have not found an effect on nonquantitative knowledge.⁵

A previous publication has described the study’s outcomes involving intent to be screened, test preference, decision conflict, and uptake¹⁸ but did not analyze whether presenting quantitative information improved patient knowledge. In this secondary analysis, we examine the impact of disclosing quantitative information on patients’ knowledge of facts other than the specific probabilities that were disclosed (“qualitative knowledge”). We also examine the impact of using different approaches for determining whether patients have “adequate knowledge.”

Furthermore, we propose and apply a novel approach we call “choice-based knowledge assessment.” As mentioned above, few patients achieve perfect scores (100%) on knowledge tests. Researchers respond by setting a “passing score” for being judged to have adequate knowledge, such as 50%, 66%, or 80%, even though there is no theoretical basis for any of these cutoffs. Furthermore, relying on an overall “passing score” does not reflect the observation that some questions may be particularly relevant for a patient’s choice. In the choice-based knowledge assessment, we have identified a subset of the knowledge questions that appear to be particularly relevant for specific choices (not getting screened, getting screened with FIT, or getting screened with a colonoscopy), and patients count as having adequate knowledge only if they answer all of the questions in this subset correctly.

The ethical justification for this approach is that if we allow some patients who get less than 100% correct on a knowledge test to count as having adequate knowledge (as previous researchers do), we should, at the very least, expect patients to understand certain knowledge items that are particularly relevant to the patient’s choice. This approach is supported by the reasonable person standard commonly used in the informed consent literature.¹ In their discussion on the nature of understanding, ethicists Beauchamp and Childress commented that “patients and subjects usually should understand, at a minimum, what an attentive health care professional or researcher believes a reasonable patient or subject needs to understand to authorize an intervention [emphasis added].”^1(p132) Importantly, Beauchamp and Childress indicated that the required content for the patient’s understanding follows directly from the specific intervention they are choosing, not the full range of possible interventions. Some knowledge about nonselected interventions may be necessary, of course, but the required type or level of understanding may shift based on which intervention the patient is choosing. Beauchamp and Childress further said that

“understanding need not be complete, because a grasp of central facts is generally sufficient. Some facts are irrelevant or trivial; others are vital, perhaps decisive. In some cases, a person’s lack of awareness of even a single risk or missing fact can deprive him or her of adequate understanding.”^1(p132)

Which facts are vital and required and which are trivial may partly depend on which intervention the patient is choosing, and this is an idea we explore for CRC screening in this article.

Methods

Study Setting

This project is a secondary analysis of a study that was conducted from August 2011 to February 2013 at 5 primary care sites in the Indiana University Health system. The original study was approved by the Indiana University Institutional Review Board. The methods and primary outcomes of this study have been reported elsewhere.¹⁸

Inclusion and Exclusion Criteria

Participants consisted of male and female adults aged 50 to 75 y who were nonadherent to CRC screening guidelines, defined as having no colonoscopy in the past 10 y, no sigmoidoscopy in the past 5 y, or no FIT in the past year.^11,12 Patients were excluded from the study if 1) they had an elevated risk for CRC, 2) were undergoing workup for possible CRC, 3) did not read English, or 4) had been advised by a health care professional to avoid CRC screening.

Study Procedure

Participants completed a baseline, self-administered survey (T0) in the presence of a research assistant. Participants then viewed a decision aid based on their assignment to 1 of 4 experimental groups: basic information (control), quantitative, nudge, and quantitative + nudge.

All participants saw the same basic information about CRC and CRC screening tests, consisting of a module that included a video produced by the American Cancer Society followed by 4 PowerPoint slides. The basic information (control) group saw only this module.

After viewing the basic information module, the quantitative group viewed 4 slides that presented lifetime CRC incidence and mortality without screening, lifetime CRC mortality with regular screening with colonoscopy and FIT, and the frequency of positive FIT (requiring follow-up colonoscopy) and of serious complications from colonoscopy.

The nudge group also started by viewing the basic information module but then viewed 3 slides with phrases meant to increase uptake of FIT. Finally, the quantitative+nudge group viewed the basic information module followed by the slides from the quantitative arm and then those from the nudge arm.

Immediately after viewing the decision aid, participants completed a postintervention survey (T1). Six months later, participants were contacted to repeat the postintervention survey and assess screening uptake (T2). More information on the study procedure and decision aids used can be found elsewhere.¹⁸

Measures of Knowledge

The T0, T1, and T2 questionnaires include a knowledge test containing 8 true-or-false questions that were constructed through a consensus process among the members of the research team (see Table 2). We refer to these questions as “qualitative knowledge” questions, to distinguish them from 4 additional “quantitative knowledge” questions that were asked regarding average lifetime CRC incidence and mortality and risk reduction provided by regular colonoscopy and FIT. In our assessment of whether patients had adequate knowledge, we focused entirely on answers to the qualitative knowledge questions. We took this approach because one of the goals of the study was to assess the effect of presenting quantitative information in a decision aid on knowledge and decision making, beyond recall of those numbers.

Table 2.

Qualitative Knowledge Questions in our Study

Question (True/False)	Correct Answer
1. Doctors recommend that people should start having colon tests at age 40.	False
2. There is only one test available for checking the colon.	False
3. Regular stool testing can lower your chance of dying from colon cancer.	True
4. If you have a stool test and everything is normal, you don’t need to get another one for 5 years.	False
5. You can have colon cancer and not have any symptoms.	True
6. If you have a stool test and blood is found, you will need to have a colonoscopy.	True
7. If you have a colonoscopy and everything is normal, you will need to have your next one in 10 years.	True
8. You only have to worry about getting colon cancer if someone in your family has had it.	False

Knowledge was measured in several different ways. We calculated the mean number of questions correct per arm at T0 and T1 to assess the overall effect of the decision aids on knowledge. We also assessed the percentage of patients in each arm, at T0, T1, and T2, who achieved adequate knowledge, defined as achieving a “passing score” on the knowledge test, following previous studies.^4–6 For this, we use the cutoffs of 5 correct, 6 correct, and 7 correct out of 8.

Choice-Based Knowledge Assessment

We also used an alternative, novel way of determining adequate knowledge that identifies essential questions based on an individual’s planned screening behavior after viewing the decision aid. The patient is considered to have adequate knowledge only if they answer all of the questions of this subset correctly. Table 3 shows the selection of the essential questions for each possible screening intention. For instance, consider question 5, which assesses comprehension that it is possible for a person to have colon cancer even if they do not have symptoms. This fact is essential for a patient who chooses not to be screened: if they think that their lack of symptoms means that there is no chance of having CRC, they may forego screening based on a false belief that they are completely safe. On the other hand, failing to understand this fact does not seem as important for a decision made to be screened with colonoscopy or FIT. Since the patient in that case has chosen to be screened, they are clearly not acting on a false belief that they cannot have colon cancer.

Table 3.

Choice-Based Knowledge Assessment: Essential Questions to Determine Knowledge Adequacy, Based on Test Intent (T1)

Test Intent (T1)	Essential Questions to Determine Knowledge Adequacy (T1)
No intention to undergo screening of any sort	Q2. There is only one test available for checking the colon.
	Q3. Regular stool testing can lower you chance of dying from colon cancer.
	Q5. You can have colon cancer and not have any symptoms.
	Q8. You only have to worry about getting colon cancer if someone in your family has had it.
Intend to undergo FIT	Q2. There is only one test available for checking the colon.
	Q4. If you have a stool test and everything is normal, you don’t need to get another one for 5 years.
	Q6. If you have a stool test and blood is found, you will need to have a colonoscopy.
	Q7. If you have a colonoscopy and everything is normal, you will need to have your next one in 10 years.
Intend to undergo colonoscopy	Q2. There is only one test available for checking the colon.
	Q3. Regular stool testing can lower you chance of dying from colon cancer.
	Q7. If you have a colonoscopy and everything is normal, you will need to have your next one in 10 years.

Similar reasoning applies to question 8, which tests whether the patient knows that they can get colon cancer even if nobody in their family had it. This fact is essential to having adequate knowledge for those who choose not to be screened but not for those who intend to be screened. In contrast, question 2—knowing that there is more than 1 screening test available to check the colon—is essential knowledge for patients no matter what they chose: if they chose colonoscopy, they should know about the availability of FIT, and vice versa. If they chose not to be screened, they should be aware of both the available choices, since one might be more palatable to the patient.

Statistical Methods

We combined the 2 groups that viewed only verbal information (basic information group and nudge group) to form the “control group” and the 2 groups that viewed verbal information and quantitative information (quantitative group and nudge+quantitative group) to form the “quantitative group.”

We determined test preference for T1 based on patient responses to questions on the original questionnaire.¹⁸

To determine if there were significant differences in knowledge between the 2 groups at T1 after viewing the decision aid (adjusting for T0), we used analysis of covariance regression models, linear models for mean number correct, and logistic models for adequate knowledge, defined either by passing score (yes/no) or essential knowledge (yes/no). Our level for significance was set at a P value less than 0.05. Analyses were conducted in SAS version 9.4 (Cary, NC).

Results

Data on participant demographics can be found elsewhere.¹⁸ Briefly, most of the participants were White and female, and there were no significant differences for demographic variables or outcomes between groups at baseline (T0). In total, we analyzed 213 participants; 105 were included in control arm (C) and 108 included in the quantitative arm (Q).

Results from knowledge as mean number of questions correct are reported in Table 4. Knowledge increased for both arms after viewing the decision aids from T0 to T1. Viewing quantitative information resulted in significantly higher scores at T1 in mean number of qualitative knowledge questions answered correctly, adjusting for T0 (2.81 [Q] v. 2.64 [C], P < 0.05).

Table 4.

Mean Qualitative Knowledge at T0 and T1 by Arm

Variable	Group	n	T0 ( $\overline{\mathit{x}}$ ± s)	T1 ( $\overline{\mathit{x}}$ ± s)	Change (T1–T0) ( $\overline{\mathit{x}}$ ± s)	P Values (T1–T0)a	Model Estimate (Standard Error; T1–T0)
Qualitative knowledge	C	105	4.25 ± 1.70	6.89 ± 1.25	2.64 ± 1.82	0.0055	−0.37 (0.13)
	Q	108	4.49 ± 1.78	7.30 ± 0.85	2.81 ± 1.68

C, control arm; Q, quantitative arm.

^aP value for arm from the model: T1 outcome = arm + T0 outcome + site + age + gender.

The percentage of patients who answered a predetermined number of questions correctly can be found in Table 5. The percentage of patients who answered 5 out of 8, 6 out of 8, or 7 out of 8 correctly increased from T0 to T1 in both arms. In addition, for a cutoff of 6 out of 8 and 7 out of 8, viewing quantitative information resulted in a significantly greater percentage of patients achieving adequate knowledge at T1 (94% [Q] v. 83% [C], P < 0.05, and 86% [Q] v. 71% [C], P < 0.05, respectively) but not for 5 out of 8 (100% [Q] v. 94% [C], P > 0.05).

Table 5.

Adequate Knowledge Defined as ≥5, ≥6, or ≥7 out of 8 by Arm

	Adequate Knowledge Defined as:
	≥5 out of 8			≥6 out of 8			≥7 out of 8
Arm	T0 (%)	T1 (%)	P Valuea (T0, T1)	T0 (%)	T1 (%)	P Valuea (T0, T1)	T0 (%)	T1 (%)	P Valuea (T0, T1)
C (n = 105)	47.6	94.3	0.9510	22.9	82.9	0.0190	8.6	71.4	0.0096
Q (n = 108)	50.0	100		33.3	94.4		13.9	86.1

C, control arm; Q, quantitative arm.

^aP value for arm from the logistic model: T1 knowledge = arm + T0 knowledge + site + age group + gender.

Table 6 displays the percentage of patients having adequate knowledge based on the choice-based knowledge assessment. Using this model, significantly more participants who viewed quantitative information were found to have adequate knowledge at T1 when compared with those who did not view quantitative information (87% [Q] v. 76% [C], P < 0.05).

Table 6.

Adequate Knowledge Defined by Choice-Based Knowledge Assessment, by Arm

Arm			Number of Participants Satisfying Requirement at T1	P Valuea
C (n = 104)	Has essential knowledge for preferred test		79/104 (76.0%)	0.0322
Q (n = 101)	Has essential knowledge for preferred test		88/101 (87.1%)
Adequate Knowledge, by Arm and Test Intent
	Test Intent (T1)
C	No screen	Has essential knowledge for no screen	22/34 (64.7%)
	Colonoscopy	Has essential knowledge for colonoscopy	31/34 (91.2%)
	FIT	Has essential knowledge for FIT	26/36 (72.2%)
Q	No screen	Has essential knowledge for no screen	17/24 (70.8%)
	Colonoscopy	Has essential knowledge for colonoscopy	38/41 (92.7%)
	FIT	Has essential knowledge for FIT	33/36 (91.7%)

C, control arm; Q, quantitative arm; FIT, fecal immunochemical test.

^aP value for arm from the logistic model: T1 essential knowledge = arm + site + age + gender.

Discussion

Major Findings

In this study, we found that viewing quantitative information about CRC screening led to improved knowledge about facts that did not involve recall of the specific quantitative information that was disclosed. This is the first study to show a positive effect on “qualitative knowledge” from disclosing quantitative information. This increase in qualitative knowledge also resulted in higher percentages of patients having adequate knowledge to make an informed choice according to multiple, but not all, ways of assessing knowledge, including a novel approach to determining adequate knowledge that we call choice-based knowledge assessment.

Quantitative Information Increases Qualitative Knowledge

Previous research has shown that presenting quantitative information about risk (such as the specific frequency of a medication side effect) may affect patients’ risk impressions and behavior, compared to if they view just qualitative statements of risk.^22,23 On knowledge tests, patients who have seen quantitative estimates of their risk also do better at answering questions about their specific risk level than do patients who were not told that specific risk level.²² But previous studies have not shown an impact on the retention of nonquantitative information. For example, a previous study found that individuals who viewed a decision aid about CRC screening that included quantitative information did better on numeric questions in a knowledge test but not on nonnumeric (“conceptual”) questions than did individuals who viewed a pamphlet that did not include quantitative information.⁵

In contrast, participants in the current study who viewed quantitative information in addition to the standard qualitative (nonquantitative) information about CRC screening did better on a knowledge test that assessed only the qualitative information, compared with those who viewed just the qualitative information.

One possible explanation is that the quantitative information reinforced the qualitative information. For instance, one of the knowledge questions is a true/false question that “Regular stool testing can lower your chance of dying from colon cancer” (correct answer: true). While the qualitative module viewed by both groups states that colon cancer screening lowers the chance of dying from colon cancer, the quantitative module includes an icon chart showing that out of 1000 average-risk people who never get a stool test, about 30 will die from colon cancer, and that number goes down to 6 per 1000 for those who get annual stool tests. While the knowledge test did not ask participants to recall the numbers, seeing these numbers and the icon chart may have reinforced the message that tests “lower your chance of dying from colon cancer.”

Similarly, although the qualitative module says that people can get colon cancer even if they do not have a family history, the quantitative module explains, again using an icon chart, that out of 1000 people with average risk who never get screened, 60 will get colon cancer in their lifetime, potentially reinforcing this message. This quantitative information could have led more participants to correctly respond “false” to the true/false question, “You only have to worry about getting colon cancer if someone in your family has had it.” This finding emphasizes the potential importance of “gist” implications of quantitative information, emphasized by dual process accounts of risk communication.^24,25 Smith et al. also cited a dual process account, the fuzzy trace theory,²⁶ in explaining their knowledge assessment, although in a different way. They include quantitative questions (e.g., “How many men/women out of 1000 will die from bowel cancer over the next 10 years without screening?”) but accept answers that are close to the right number but not exactly correct.

It is possible that simply repeating the qualitative statements of the key facts could have had a similar effect, although simple repetition of information is not a reliable way to improve retention, and spacing between repetitions appears to be essential.²⁷ In addition, more detailed decision aids have not generally outperformed shorter ones at increasing knowledge.^28(202) Any proposed mechanism must remain speculative, however, since the study was not designed to identify mechanisms and did not have the power to detect differences in the frequency correct for individual questions. Future studies of the impact of quantitative information should eliminate the difference in the number of repetitions of information and be properly powered to better investigate the impact on individual questions.

Altering Thresholds for Adequate Knowledge Led to Varying Results

Our findings also highlight the importance of differences in methods for determining whether a patient has adequate knowledge. First, modifying the cutoff for passing the knowledge test changes the percentage of patients who count as having made a decision with adequate understanding, a key aspect in the ethical evaluation of informed choice. For instance, just 79% of patients count as having adequate understanding if we use a cutoff of 7 correct out of 8, while 97% have adequate understanding if we use a cutoff of 5 out of 8. The choice of cutoff also may affect the findings of the study. Using a threshold of 6 or 7 correct out of 8, we found that providing quantitative information increases the proportion of patients who achieve adequate understanding to make, but not if the cutoff was 5 out 8. Without a clear rationale for using one cutoff or another, the implications of the study cannot be fully appreciated.

Our survey of the literature on CRC screening decision making found significant variation and sparse justification for methods of measuring adequate knowledge, suggesting the need for more extensive analysis and discussion than is currently given in the design and reporting of decision-making research. Others have similarly called for more careful attention to the process of defining and using knowledge tests in assessing quality of informed consent.^7,10,16

Choice-Based Knowledge Assessment

The impact of varying thresholds for adequate knowledge led us to create and pilot the use of our choice-based knowledge assessment, in which a patient counts as having adequate understanding only if they answer all of a subset of questions correctly that were matched to their intended behavior. Interestingly, and similar to more traditional approaches, this analysis found that a significantly higher proportion of patients who viewed quantitative information had adequate knowledge.

One advantage of the choice-based knowledge assessment is that the threshold for determining adequate knowledge (100% correct for the subset selected) makes clear sense, since the questions are selected as being particularly important based on the patient’s intended behavior. Another strength is that it appears reasonable to tailor the knowledge assessment based on the decision the patient has actually made, rather than every possible decision.

We recognize that this approach is in some ways less demanding and in some ways more demanding for determining that a patient has adequate knowledge: it is less demanding since a patient can be counted as having adequate knowledge based on answering fewer questions correctly (i.e., a subset of all the knowledge questions that count as “essential”), whereas it is more demanding since the patient must answer all the essential questions correctly. Of note, using the choice-based knowledge approach to determine the adequacy of knowledge for individual patients does not justify reducing the amount of information provided by the decision aid, since it has to provide the essential information for all of the choices. Furthermore, the knowledge test should cover all the essential information for all options, since the patient’s choice cannot be predicted ahead of time.

The choice-based knowledge approach to assessing adequate knowledge also faces some challenges. First, there is considerable difficulty in deciding which questions (which facts and information) are the “essential” ones for a given decision. Further research assessing the merits and applicability of various questions based on test preference will be needed. Second, it is difficult to determine an individual’s intended behavior in order to guide the choice of which knowledge questions are most important for a patient.²⁹ Third, it may be unclear whether the essential questions should be determined by the patient’s intention at T1, as we did here, or on which test they actually received at 6 mo. As in other approaches in which a patient’s intention is assessed, it may be useful to assess the level of commitment of the individual to that choice.

Of note, some commentators have recently questioned whether the ethical evaluation of consent requires evaluation of patient understanding, particularly for research participation.^7,30,31 Some of these commentators have considered a return to focusing on what is disclosed to participants, and how clearly it is presented, rather than measuring objective knowledge.⁹ This would actually be a return to the focus of traditional rules such as the reasonable person standard (the basis in fact for choice-based knowledge assessment), which focus on disclosure rather than understanding.¹

Limitations

This study looked at only a single randomized trial, related to 1 screening initiative (CRC screening) and was a secondary analysis. Furthermore, half of the participants also viewed a “nudge” toward FIT in the original study, and it is unknown what impact that could have on knowledge.

Conclusions

We described considerable variability in how researchers have measured patient knowledge and determined the adequacy of knowledge in studies of CRC screening. Researchers assessing patient understanding should consider various ways to define adequate knowledge when assessing quality of decision making, including considering the choice-based knowledge assessment method introduced here.

In our secondary analysis of a previously concluded study, we found that varying the method for assessing participant knowledge had considerable effects on the percentage of participants who were judged to have adequate knowledge. We also found that patients who viewed quantitative information in addition to verbal information had greater qualitative knowledge and more frequently had adequate knowledge compared with those who viewed verbal information alone, according to most ways of defining adequate knowledge. Quantitative information may have helped participants better understand or retain qualitative or gist concepts.