Development of KROHL, a tool for evaluating oral health knowledge

Abstract

Objectives

The authors aim to evaluate a scale assessing oral health knowledge, the KROHL (Knowledge Related to Oral Health Literacy) including the inter-rater reliability for scoring open-ended questions, internal consistency of the hypothesized scales, the discriminant validity of the resulting scale and its relationship to existing measures of oral health literacy.

Methods

The questionnaire was administered via face-to-face interviews to 144 volunteers recruited in the waiting areas of clinics throughout the NYU College of Dentistry.

The KROHL questionnaire evaluates oral health knowledge by asking open-ended questions about the appearance, cause, treatment and prevention of caries, gum disease, oral cancer, tooth loss and malocclusion. Those 20 questions were scored to produce scale scores. Demographic information, a self-reported measure of HL and the CMOHK (Comprehensive Measure of Oral Health Knowledge) were also collected.

Data were analyzed using Pearson correlation coefficients and principal components analysis, by computing Cronbach's alpha and Cohen's kappa, and by comparing group means with ANOVA.

Results

Kappa indicated good to excellent agreement among raters for the full and the individual subscales of the KROHL. Cronbach's alpha indicated good consistency of the full scale score, but not the individual scales. The mean (SD) KROHL score was lower in the patient group than in the dental students (13.3 (5.9) vs. 26.1 (4.7), p < .001), and varied directly with education level within the patients. KROHL scores were unrelated to existing measures of health literacy.

Conclusions

The KROHL scale is an innovative, reliable and valid tool to assess overall oral health knowledge and provide information to customize educational interventions. Further research is needed to determine the validity and reliability of the scale in multiple settings.

Innovation

The innovation of the KROHL tool of assessment of oral health knowledge lies in its ability to scale depth of knowledge within the domains of identification, causes, prevention, and treatment for the most common oral conditions.

1. Introduction

Personal Health literacy (HL) is defined as the degree to which individuals have the ability to find, understand, and use information and services to inform health-related decisions and actions for themselves and others. [1].

The pathway of how HL influences health, first proposed by Paasche-Orlow and Wolf [2], was followed by evidence showing that HL affects health outcomes, impacting knowledge, self-efficacy and health behaviors [3]. Individuals with limited oral HL have been shown to have lower levels of oral health knowledge, lower self-efficacy levels, use preventive services less frequently, and suffer from increased levels of oral disease burden and severity [4,5,6]. These factors likely contribute to oral health disparities [5,6,7].

Health knowledge has been defined as disease-related information that is content and context specific [8]. A link between HL and specific knowledge has been established for chronic conditions like diabetes, asthma, hypertension, congestive heart failure and HIV [9]. Not knowing may affect one's ability to recognize signs and symptoms of disease that, in turn, may explain why individuals with limited HL delay seeking treatment. It also affects patient-provider interactions and self-management [2,3]. Therefore, it has been suggested that interventions which promote disease-specific knowledge will improve HL and health outcomes [3].

The most common oral diseases (dental caries, periodontal disease, tooth loss and oral cancer), although largely preventable, are among the most ubiquitous diseases worldwide and disproportionally affect those experiencing social and economic inequalities. These diseases greatly affect quality of life because of pain, eating difficulty, infections, loss of self-confidence and shame [11,12]. In addition, they share modifiable risk factors with other important non communicable diseases (e.g., cardiovascular disease, cancer, chronic respiratory disease and diabetes). [10,11].

With the purpose to further study the role of oral health literacy and other contributing factors on oral health among the diverse population of patients seeking treatment at NYU College of Dentistry, we started by reviewing the current tools of assessment. Evidence from systematic reviews on oral HL tools indicate there is no gold standard. Importantly, most of the tools evaluate a limited number of skills, primarily word recognition, followed by reading and comprehension and numeracy. Further, they lead to a negative bias when used among non-English individuals. [13,14,15]. Instead, our interest is focused on the evaluation of the oral health knowledge aspect of health literacy. One existing tool, the Comprehensive measure of oral health knowledge (CHOHK) [16], evaluates oral health knowledge in general, and we sought to extend this methodology to the evaluation of disease specific knowledge. The aim of the study was to develop and test the psychometric properties of the Knowledge Related to Oral Health Literacy scale (KROHL scale). The scale was designed to assess knowledge specifically related to caries, periodontal disease, oral cancer, tooth loss and malocclusion in four domains—identification, causes, treatment, and prevention. The importance of a reliable assessment of oral health knowledge is threefold: to provide practical, specific patient information in the healthcare setting, to evaluate oral health educational interventions, and to be a research tool to study health equity [17].

2. Materials and methods

2.1. Research design

This study employed a cross-sectional evaluation of patients, dental students and dental faculty for the purpose of describing the psychometric properties of the KROHL, a tool in development for the assessment of specific aspects of oral health knowledge.

2.2. Participants

2.2.1. Patients

Potential participants were recruited from waiting areas of dental clinics at the college. To be eligible, individuals needed to be English speakers at least 18 years old. Some were patients, but accompanying individuals, e.g., parents, were also eligible. A total of 144 participants were enrolled in the study.

2.2.2. Students

All senior dental students at the college were eligible and invited to participate. Of the total of 45 student volunteers, 33 females and 14 males completed the questionnaire. This group was considered trained responders and were used to evaluate discriminant validity.

2.2.3. Faculty

For the reliability studies, 5 members of the clinical faculty at the college were recruited. They represented a range of experience (from 6 to 38 years since graduation) and gender (2 males and 3 females).

The protocol was approved by the Committee on Activities Involving Human Subjects (IRB-FY2017-852).

2.3. Measures

2.3.1. KROHL

The KROHL questionnaire was designed to evaluate the most prevalent and impactful oral conditions: caries, periodontal disease, oral cancer, tooth loss and malocclusion. Within each, the knowledge domains of ‘identification and symptoms’, ‘causes’, ‘prevention’ and ‘treatments’ were evaluated. Specifically, these open-ended questions were asked for each of the 5 conditions, resulting in 20 responses:

• 1.“What is _______ and what do you see and feel while having ______ (insert caries, periodontal disease, oral cancer, tooth loss or malocclusion)?”
• 2.“What causes ______ (insert caries, periodontal disease, oral cancer, tooth loss or malocclusion)?”
• 3.“How do you prevent ______ (insert caries, periodontal disease, oral cancer, tooth loss or malocclusion)?”
• 4.“How do you treat _____ (insert caries, periodontal disease, oral cancer, tooth loss or malocclusion)?”

Individuals were encouraged to respond to each ad libitum.

Scoring of the KROHL. The current scoring system is based on the most frequent words, and their synonyms, that participants used to answer a particular question following the pre-established rules. For example: to evaluate caries prevention knowledge, we look for statements referencing three content areas: oral self-care practices, dietary considerations, and professional dental care. Thus, an answer like “brushing with fluoridated toothpaste, not eating candy and going for checkups twice a year” received full score, while less complete answers received partial scores. Answers not including any of these content areas, or an answer of ‘I don't know’, were scored zero. Full details for the scoring of each item are available upon request.

Scoring was done by the developer of the scale (CS) or someone that she trained. While not necessary, all raters here were faculty of the NYU College of Dentistry. Each question was graded using a 4-point scale: good (2 points), fair (1 point), minimal (0.5 point) or given no points (0) if the response was ‘I don't know’ or incorrect. The final KROHL score sums the 4 individual KROHL questions for each of the 5 conditions (caries, periodontal disease, oral cancer, tooth loss and malocclusion). Because everyone answered the tooth loss definition/see and feel question by rephrasing the statement, that non-differentiating item was excluded from scoring.

Inter-rater reliability. To determine the reliability of the scoring system, 5 faculty members at the college who were not involved in the study rated 20 records that were randomly selected from among the 144 patients achieving moderate scores. Judges were given an operator's manual with the instructions to apply only those criteria to their judgements (available upon request). Cohen's kappa was then computed between the ratings made by each those judges and ratings made by a member of the development team.

2.3.2. CMOHK

The 23 multiple choice questions of the CMOHK [16] evaluate general knowledge of oral health, dental caries prevention and management, periodontal disease prevention and management, and oral cancer prevention and management. Scores were determined by summing the number of correct items. The CMOHK has demonstrated good psychometric characteristics in different settings [18,19,20,21]. CMOHK categorized individuals as having poor (0–11), fair (12–14) or good levels (15–23) of oral health knowledge.

2.3.3. Self-reported health literacy measure

We included the Brief health literacy screeners (BHLS): “How often do you have someone help you read health related materials?”,“How confident are you filling out medical forms by yourself?” and “How often do you have problems learning about your medical condition because of difficulty understanding written information?”

These three questions, developed by Chew [22] are known to accurately identify patients with limited HL. Responses are selected from a 5-point Likert scale using the options never, occasionally, sometimes, often or always. Questions 1 and 3 were reverse scored. The total score ranges from 3 to 15, with higher scores indicating higher literacy. A score above 9 indicates adequate literacy. The internal consistency of the three items was high (Cronbach's α = 0.84), and this tool has shown a good discriminant validity in multiple studies [23,24]. In analysis, we considered each item individually.

2.3.4. Demographics

The survey queried demographic characteristics like age, gender and educational level as well as self-rated oral health.

2.4. Procedure

A research team member determined eligibility and explained the purpose of the study to those patients who qualified. Oral consent was obtained from individuals that agreed to participate. The questionnaire was read aloud to participants via face-to-face interviews in a private office space by trained interviewers, who recorded answers by hand. A printed copy of the survey was offered to respondents so they could see the questions. They were reminded it was acceptable to answer “I don't know”. The interview lasted approximately 20 min.

2.5. Data management and analysis

Collected responses were entered into an Excel file and imported into IBM SPSS (v28, IBM Corp, Armonk, NY), where all analysis was done. Quantitative responses were summarized using percentage and frequencies, means and SD. Internal consistency was evaluated using the Cronbach alpha statistic, and by use of a principal components analysis (PCA). Interrater reliability was calculated as kappa. Effects of age, sex, and education on KROHL scores was evaluated with 1-way ANOVA, which was also used to compare the clinic sample with a group of senior dental students. We hypothesize that KROHL scores will be directly proportional to education, and that the periodontal and cancer subscale scores will be inversely proportional to age, as these conditions generally affect older individuals. Associations between the KROHL scale and other variables and measures were computed as Pearson correlation coefficients. Unless otherwise noted, statistically significant implies p < .05.

3. Results

144 volunteers were enrolled in the study. All participants completed all the questionnaires. While responses to the KROHL questions were collected until ‘exhaustion’, the vast majority of the respondents provided a single response.

Patient demographics. Of the 144 participants, 48.6% were male. Their age ranged from 18 to 72 y and averaged 37.6 y (SD = 15.4 y). The level of education was 27.5% high school (HS), 43.7% college and 28.9% post-college.

3.1. Descriptive statistics and internal consistency

Average KROHL scores are shown in Table 1. The average for the full scale of 13.4 falls well short of the maximum score of 38. One sees best knowledge on the Caries subscale and poorest knowledge of Malocclusion. The full scale and Malocclusion subscales showed acceptable to excellent levels of internal consistency. In the latter case, this seems to be driven by the fact that respondents knew little about any aspect of malocclusion. By contrast, respondents may have known some things assessed on the other scales, but knowledge was not consistent across the various items within each condition. This is further illustrated by results shown in Supplemental Table 1.

Table 1.

Descriptive statistics for the full scale and subscales of the KROHL (N = 144).

Scale	Mean	SD	Min	Max	% ‘0’⁎⁎	Alpha
Full⁎	13.4	5.88	0	30	0.5	0.78
Caries	4.0	1.47	0	7	2.1	0.50
Periodontal Disease	3.1	1.76	0	7	8.5	0.60
Oral Cancer	2.6	1.90	0	6.5	12.2	0.74
Tooth Loss⁎	2.5	1.31	0	6	5.3	0.54
Malocclusion	1.2	2.15	0	8	56.1	0.90

^⁎TL subscale had a total of 3 items; others had 4.

^⁎⁎either wrong or IDK.

PCA (Principal Component Analysis) was performed on the 19 individual items composing the full scale. Results showed a 5-factor solution accounted for 59.3% of variance. While the varimax rotated solution basically segregated items from each of the 5 subscales (save Caries, where prevention and treatment items loaded more strongly on other factors), suggesting that knowing about one aspect of each condition generally went along with knowing about others within that domain. The strongest loadings were seen on the Malocclusion scale, supporting the straightforward interpretation of summed items on the Malocclusion scale, but less so in the other domains.

3.2. Inter-rater reliability

Kappa interrater agreement was obtained based on the full scale and also for the individual conditions (see Table 2). Generally, good to excellent agreement was shown by each rater on each of the subscales. In an exception, poor levels of agreement (0.41 and 0.29) were seen for caries scoring by R2 and R3. Those 2 raters were asked to re-score the caries questions, after reminding them to follow the instructions. The second attempt showed clear improvement (0.86 and 0.85). Both raters admitted not having carefully followed the instructions the first time.

Table 2.

Kappa statistics for the full scale and subscales of the KROHL.

Scale	N	Rater1	R2	R3	R4	R5
Full⁎	315	0.92	0.72/0.86	0.68/0.85	87	0.80
Caries+	79	0.92	0.41/0.55	0.29/0.40	0.85	0.74
Gum Disease	68	0.89	0.67	0.61	0.84	0.77
Oral Cancer	76	0.96	0.93	0.94	0.93	0.93
Tooth Loss⁎, ⁎⁎	39	0.88	0.88	0.77	0.82	0.89
Malocclusion	35	0.90	0.73	0.81	0.85	0.55

+ number after ‘/’ is kappa after remediation.

^⁎65 instances of ‘I don't know’ or similar excluded.

^⁎⁎TL subscale had a total of 60 items; others had 80.

3.3. Discriminant validity

3.3.1. Effects of age, education, and sex

As shown in Table 3, KROHL full scale, Caries, Oral cancer and Malocclusion subscale scores varied with education. Generally, while increasing education was associated with increasing scores on these scales, it was the post-college group that most distinguished itself. Knowledge of gum disease and tooth loss were similar in all education strata. KROHL scores were statistically similar in men and women. While scores on the Periodontal disease subscale tended to increase with age (r = 0.18, p = .04), other scales were not related to age.

Table 3.

Effects of education on KROHL subscale and total scores.

		N	Mean⁎	Std. Deviation
Scale	Education level
Caries total	HS	39	3.3a	1.33
	College	62	4.0a,b	1.35
	Post-college	41	4.6b	1.43
Periodontal disease total	HS	39	2.705a	1.85
	College	62	3.113a	1.69
	Post-college	41	3.427a	1.75
Oral cancer total	HS	39	2.0a	1.60
	College	62	2.5a,b	1.79
	Post-college	41	3.2b	2.15
Tooth loss total	HS	39	2.4a	1.17
	College	62	2.4a	1.31
	Post-college	41	3.0a	1.31
Malocclusion total	HS	39	0.7a	1.80
	College	62	0.8a	1.81
	Post-college	41	2.2b	2.62
KROHL total	HS	39	11.2a	4.94
	College	62	12.9a	5.09
	Post-college	41	16.4b	6.54

^⁎Unlike superscripts indicate different means (p < .05) using Tukey HSD.

Trained responder results.

We enrolled 45 senior dental students whose data further tested the validity of the scoring system. As shown in Table 4, dental students' knowledge exceeded that of patients on the full scale and each subscale of the KROHL. This was particularly evident for the Malocclusion scale, and least true of the caries scale. The students had similar scores on each subscale, but like the patients, were least knowledgeable about the Malocclusion subscale. These data demonstrate the ability of KROHL to discriminate those with known differences in oral health knowledge.

Table 4.

Compare mean KROHL scores in the patient sample and a sample of dental students.

Measure	Dental student?	N	Mean	Std. Deviation	p-value
Caries scale	No	144	4.0	1.47
	Yes	45	5.8	1.42	<0.001
Periodontal disease scale	No	144	3.1	1.76
	Yes	45	5.6	1.38	<0.001
Oral cancer scale	No	144	2.6	1.90
	Yes	45	5.6	1.42	<0.001
Tooth loss scale	No	144	2.5	1.32
	Yes	45	4.5	1.18	<0.001
Malocclusion scale	No	144	1.2	2.15
	Yes	45	4.6	1.30	<0.001
KROHL total	No	144	13.3	5.88
	Yes	45	26.1	4.66	<0.001

3.4. Relationship of KROHL to existing tools

The cohort averaged about 17 points on the CMOHK. Like the KROHL, scores tended to increase with education (Table 5) and were unrelated to sex. Neither sex nor education influenced mean scores on questions related to confidence with forms, help reading health materials or help understanding health material.

Table 5.

Effects of education on ancillary measures of health literacy and the CMOHK.

Measure	Education level	N	Mean	Std. Deviation
Confident w/forms	HS	39	3.3	1.39
	College	62	3.1	1.66
	Post-college	41	3.6	1.76
	Total	142	3.3	1.62
Help reading	HS	39	1.7	1.03
	College	62	1.8	1.07
	Post-college	41	1.6	1.09
	Total	142	1.7	1.06
Help understanding	HS	39	1.7	0.73
	College	62	1.9	1.15
	Post-college	41	1.6	0.81
	Total	142	1.7	0.96
Total CMOHK	HS	24	14.8	3.47
	College	34	17.5	3.97
	Post-college	27	17.4	3.38
	Total	85	16.7	3.81

KROHL scores were largely independent of CMOHK scores (r = 0.20, p = .07). This suggests that the KROHL assesses different information than the CMOHK.

KROHL total score and scores on the Oral cancer, Tooth Loss and Malocclusion scales tended to increase with ratings of confidence with forms (r = 0.19 to 0.24, p < .02). KROHL total scores and scores on the Oral cancer and Malocclusion scales tended to increase with decreased ratings of needing help reading (r = −0.18 to −0.24, p < .04). KROHL total scores and scores on the Caries, Oral cancer and Tooth Loss scales also tended to increase with ratings of understanding written material (r = 0.18 to 0.21, p < .04). These results suggest that KROHL scores tend to increase in those with higher levels of self-reported comfort consuming health information.

4. Discussion and conclusion

4.1. Discussion

KROHL was conceptualized as a tool to evaluate five specific oral health conditions (caries, periodontal disease, oral cancer, tooth loss and malocclusion) within four domains of knowledge (definition and identification, etiology and risk factors, prevention and self-care, and treatment).

In our sample of 144 participants the KROHL average score was 13.4 out of 38 possible points. The highest score was seen for caries knowledge and the lowest score for malocclusion. The low score on malocclusion could be explained by lack of familiarity with the term, as stated by several participants. As a result, we plan to evaluate the phrases ‘misaligned teeth’ or ‘abnormal bite’ for substitution.

Results from the content analysis, coefficient alpha, indicated that item consistency for the full scale was good, and justifies the creation of a single summary score for the description of oral health knowledge. On the other hand, only one of the individual disorder areas produced a coherent scale. That is: knowledge of definition/identification, cause, prevention and treatment for any disorder other than malocclusion were not consistent enough to justify adding the various elements together to produce a single summary measure. In the case of malocclusion, consistency reflected uniform lack of familiarity with the condition, and so adding these items together could make sense. We think items on the individual disorder should be considered individually. Thus, if a particular patient evidenced unfamiliarity regarding the prevention of caries, intervention could be targeted on improving that domain. In this way, a consistent evaluation of patient knowledge is available, and areas of remediation are made evident.

Supporting the concept that HL skills are considered content and context specific and influenced by age and stage in life [25] we were expecting to find that younger individuals may have less knowledge of periodontal disease or tooth loss, as they tend to happen later in life. Our findings showed only periodontal disease knowledge tending to increase with age (r = 0.18, p = .04) whereas the other scales were not related to age. This may be explained in part by the level of education of our cohort.

In the process of developing the KROHL scale, we identified that the answers to the definition/identification of tooth loss question did not provide useful information and it was eliminated from the final calculation. It was important to determine if, besides in the tooth loss category, each question provided relevant and unique information or, perhaps, there were overlaps. Findings from each of the domain of knowledge questions within each of the five conditions indicate that knowing about some aspect of the condition is related to other domains within the condition although the strength of the relationship suggests a benefit/value of evaluating each aspect separately. Within the domain of definition and identification it is important to note that the vast majority of the answers included exclusively see/feel descriptors.

Good to excellent agreement was demonstrated when the scoring system was used by non-team faculty with varying degrees of clinical experience when adhering to the scoring system.

KROHL total scores increased with education level, but that distinction was less evident for periodontal disease and tooth loss. These findings may help support the relevance of assessing disease specific knowledge, but it needs further corroboration.

While the student sample showed significantly better knowledge of each of the conditions as well the total KROHL (Table 4) than the patient sample, the students' scores were far from perfect. The lack of perfect or near-perfect scores intuitively expected from the students' group may be attributed to the attention the students' devoted to this task. Our impression was that those with more elaborated responses achieved better scores, but that motivation was often incomplete. Future work might evaluate whether an incentive (like a gift card) would further increase student scores.

In this study, we also administered 2 other tools: the CMOHK questionnaire and the BHLS. The CMOHK scores categorize individuals as poor, fair or good for conceptual oral health knowledge. Regardless of the level of education, all study participants were categorized as having good conceptual oral health knowledge on the CMOHK and these scores were largely independent of the KROHL score. In contrast, KROHL scores were lowest in individuals with HS education, intermediate in those who attended college and highest in those with a post-college education. We attribute this finding to the difference between the general knowledge assessed by the CMOHK and the specific knowledge assessed by the KROHL, but it may also be explained by the difference of testing using open-ended (recall skills) questions vs. multiple choice (recognition skills).

Similar to what participants reported while being evaluated for comprehension skills [26], we were concerned that using open-ended questions could make individuals feel “uncomfortable” or” judged” so we gathered feedback from the first 4 participants by requesting their opinion of the questionnaire. We then added the statement: “it is not expected you will know all the answers” to the interviewers' script and to reiterate the fact that was perfectly OK to answer “I don't know”. We also closely monitored the responses looking for incomplete questionnaires or dropouts.

4.2. Innovation

The innovation of the KROHL tool of assessment of oral health knowledge lies in its ability to scale depth of knowledge within the domains of identification, causes, prevention, and treatment for the most common oral conditions.

4.3. Limitations and conclusions

The KROHL scale showed the ability to point to specific gaps in knowledge of the most common oral condition: caries, periodontal disease, oral cancer, tooth loss and malocclusion. Identification of those gaps provides concrete and practical information to help on the development of individualized educational interventions at the point of care and could be used to evaluate the longitudinal effect of health educational interventions at the population level.

The study about interventions to reduce the effect of low literacy [27,28] consistently showed the benefits of improving professional communication skills, the reduction of situational demands and organizational complexity, and provided compelling evidence for the adoption of what it is known as universal HL precautions. Plain language communications, teach back methodology, information available in multiple languages and formats are some examples of the implementation of those precautions in what is also referred as organizational HL [1]. At the individual level it is important to recognize that even persons with high literacy levels can experience challenges under circumstances that require specific knowledge and can vary depending on the context. Changes in specific knowledge are metrics commonly used to report the results of HL interventions at the point of care, making KROHL a suitable tool to evaluate educational interventions to improve OHL.

Limitations from our findings relate to the size and characteristics of the population.

A larger-scale study with other cohorts, for example, a private patient sample, or first-time patients versus those that already received treatments, will help construct a more robust scale. In addition, the current approach of collecting information from open ended questions may limit its use in the clinical setting. To address this issue, we are working on a shorter version to include the “most relevant” items, according to potential user's feedback, to be followed by validity and reliability assessments, as well as using the knowledge gained from the KROHL scale to develop multiple choice items.

In conclusion

• •Oral health knowledge tends to be unique to a domain within a condition except in the case of malocclusion,
• •KROHL scores increase with education level, especially among post college individuals.
• •Periodontal disease knowledge tends to increase with age.
• •KROHL scores increase, as expected, in those with oral health training

Future work should address the question of whether oral health knowledge about a specific condition is related to increased preventive behaviors or improved oral health.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

Supplemental Table 1

Rotated principal components analysis of the 19 KROHL scale items