Oral health screening: feasibility and reliability of the oral health assessment tool as used by speech pathologists

Abstract

Background: The aim of this study was to investigate the feasibility and reliability of the Oral Health Assessment Tool (OHAT) as used by speech pathologists, to become part of a comprehensive clinical swallowing examination. Methods: A multicentre study in 132 elderly subjects was conducted by speech pathologists. The inter-rater, test–retest and intra-rater reliabilities of the OHAT were assessed in R statistics, version 3.0.1. Intraclass correlation coefficients (ICCs) were used for the total OHAT, and Kappa statistics were used for the individual categories. Results: Total OHAT scores showed good inter-rater (ICC = 0.96), intra-rater (ICC ≥ 0.95) and test–retest (ICC ≥ 0.78) agreement. The inter-rater Kappa statistics were almost perfect (κ ≥ 0.83) for seven of the eight individual categories of the OHAT and perfect for ‘dental pain’ (κ = 1.00). The test–retest Kappa statistics indicated excellent agreement for ‘natural teeth’ and ‘dentures’ (κ ≥ 0.86). The intra-rater per cent agreement was excellent for all categories except ‘gums and tissues’. Conclusions: This is the first study to examine the feasibility and reliability of the OHAT as used by speech pathologists. As the results showed both good feasibility and reliability, the OHAT has the potential to add to the clinical swallowing examination. However, future research investigating actual referral strategies and adaptation of care strategies following assessment with OHAT is needed.

INTRODUCTION

Oral health care and speech pathologists

The oral health of patients with dysphagia is concerning, particularly in elderly patients1., 2., because when poor oral health and oral diseases are combined with the presence of swallowing and feeding problems, poor functional status, underlying diseases and an increasing age, the risk of aspiration pneumonia is highest2., 3.. The importance of maintaining adequate oral health has long been recognised by speech pathologists (SPs), who primarily evaluate the motor and sensory functioning of the oral cavity structures involved in speech and swallowing⁴. Based on their professional knowledge of oral anatomy and physiology⁵, they pay particular attention to oral health and dentition⁶, which are essential for swallowing and speech production⁷.

Screening, assessment and diagnosis of swallowing disorders are activities within the scope of practice for SPs8., 9.. Swallowing disorders resulting from oropharyngeal dysfunction¹⁰, known as oropharyngeal dysphagia, can be caused by oral abnormalities, such as dental malocclusion, as well as oral-motor dysfunction⁹. Typical clinical swallowing examinations include an evaluation of dentition⁶ and oral-motor function6., 11., 12.; however, despite the importance of oral health in the prevention of aspiration pneumonia, especially in individuals with dysphagia¹³, there is no standardised method for screening the oral health of patients as part of such an examination.

Aspiration pneumonia is defined as the development of pneumonia after the aspiration of colonised oropharyngeal material into the larynx and lower respiratory airways14., 15., and it occurs in dysphagic patients, who are at increased risk for oropharyngeal aspiration¹⁴. Poor oral health is an important contributing factor to the development of aspiration pneumonia2., 13., 14., 15., 16., 17.. Therefore, oral care in dysphagic patients is essential, and oral health-care interventions have the potential to diminish the risk of aspiration pneumonia17., 18. and its associated elevated mortality risk18., 19..

Oral health and dental status are also critical during the preparatory phase of swallowing, particularly for mastication¹. Maintaining functional units (i.e. pairs of opposing mandibular and maxillary teeth, particularly natural teeth) is crucial for masticatory function1., 20.. The degree of masticatory function not only determines food selection21., 22. but also influences nutritional status21., 22., 23., 24..

Although the provision of daily oral hygiene support is primarily considered the responsibility of nursing staff16., 25., SPs are also in a position to detect oral ailments during their routine assessments6., 16.. The literature reports that declines in oral-health status often go unnoticed until oral health becomes visibly poor¹⁶. Oral care is often poor in dysphagic patients, whether these patients reside in a hospital or in rehabilitation or residential facilities¹. Therefore, a multidisciplinary approach to enhancing the quality of oral health has been suggested²⁶, and SPs can provide valuable contributions because of their existing scope of practice8., 9..

SPs have expertise in communication management8., 9. and are trained to be effective managers of communication difficulties experienced by patients who may have cognitive impairment. Such impairment may lead to behavioural responses that are often deemed by care staff to be uncooperative and seen as refusal of oral assessment and care. The dental literature reports a variety of communication strategies that may be utilised to assist in the completion of an oral assessment or dental examination²⁷, and SPs who are experienced in the domain of communication management8., 9. are also suitable professionals for performing oral health screenings.

Oral Health Assessment

As reported in previous studies27., 28., reliable and valid Oral Health Assessment Tools (OHATs) have been developed for use by non-dental professionals such as nurses, personal care attendants and allied health or medical professionals25., 28., 29., 30., 31., 32.. Although they are called ‘assessment tools’, they should actually be considered ‘screening instruments’ because they differ from dental examinations performed by qualified dentists²⁷. These tools are meant to screen the oral health status of a patient to make appropriate and timely referrals to a dentist or a dental hygienist²⁸. To be in accordance with the terminology used in the literature27., 28., 30., in this study, we use the term ‘assessment’; however, this term refers to an oral health screening that addresses patients’ dental needs²⁷. The Kayser-Jones Brief Oral Health Status Examination (BOHSE)25., 30. and the subsequently developed OHAT²⁸ are two instruments with proven validity and reliability that can be used by residential care staff for various patients, including those with cognitive impairment. The OHAT was initially adapted from the BOHSE by Chalmers et al.²⁸ and then subsequently modified by the Halton Region’s Health Department33., 34.. The success of the OHAT is that it requires minimal training25., 28., 35., 36. and it is therefore a feasible instrument for SPs to use for oral health screening. Furthermore, the psychometric properties of the modified OHAT have not been investigated previously; no data regarding its use by SPs are available. The purpose of this study was to investigate (i) the feasibility and (ii) the inter-rater reliability, test–retest reliability and intra-rater reliability of the modified OHAT as used by SPs.

METHODS

Oral Health Assessment Tool

As previously reported in the initial study by Chalmers et al.²⁸, the OHAT consists of eight categories (‘lips’, ‘tongue’, ‘gums and tissues’, ‘saliva’, ‘natural teeth’, ‘dentures’, ‘oral cleanliness’, and ‘dental pain’) with three possible scores (0: healthy, 1: some changes present and 2: unhealthy condition)²⁸. Scoring of each category is based on structured observation with clear operational definitions28., 33., 34., 37., 38., 39.. A score of 1 or 2 for any of the specifically marked categories (starred and underlined) mandates referral to an oral health professional (dentist, dental hygienist or denturist)33., 34.. The total score is the sum of the various subscores. Based on the screening results, staff members can determine whether patient needs can be met by daily oral care based on the development of an Oral Hygiene Care Plan40., 41., or if referral to an oral health professional33., 34. should be instituted. The materials required to perform the screening include only clean gloves and an adequate light source (daylight or artificial)²⁸. In this study, the modified OHAT tool was used33., 34. (Appendix I).

Appendix I.

An oral health assessment tool for dental screening (developed by Chalmers et al.²⁸ and modified by the Halton Region’s Health Department33., 34.).

	Resident:________________________________Completed by: _________________			Date: __/__/__
Category	0 = healthy	1 = changes	2 = unhealthy	Category scores
Lips	Smooth, pink, moist	Dry, chapped, or red at corners	Swelling or lump, white/red/ulcerated patch; bleeding/ulcerated at corners⋆
Tongue	Normal, moist, roughness, pink	Patchy, fissured, red, coated	Patch that is red and/or white, ulcerated, swollen⋆
Gums and tissues	Pink, moist, smooth, no bleeding	Dry, shiny, rough, red, swollen around 1–6 teeth, one ulcer/sore spot under dentures⋆	Swollen, bleeding around seven teeth or more, ulcers, white/red patches, generalized redness under dentures⋆
Saliva	Moist tissues, watery and free-flowing saliva	Dry, sticky tissues, little saliva present, resident thinks they have dry mouth	Tissues parched and red, very little or no saliva present; saliva is thick, rope-like, resident complains of dry mouth⋆
Natural teeth	No decayed or broken teeth/roots	1–3 decayed or broken teeth/roots⋆	Four or more decayed or broken teeth/roots, or very worn down teeth, or less than four teeth with no dentures⋆
Yes/No
Denture(s)	No broken areas or teeth, dentures regularly worn, and named	1 broken area/tooth or dentures only worn for 1–2 hours daily, or dentures not named, or loose	More than 1 broken area/tooth, dentures missing or not worn due to poor fit, or worn only with denture adhesive⋆
Yes/No
Oral cleanliness	Clean and no food particles or tartar in mouth or dentures	Food particles/tartar/plaque in 1 or 2 areas of the mouth or on small area of dentures or halitosis (bad breath)	Food particles/tartar/plaque in most areas of the mouth or on most of denture(s) or severe halitosis (bad breath)⋆
Dental pain	No behavioural, verbal or physical signs of pain	Verbal and/or behavioural signs of pain, such as pulling at face, chewing lips, not eating, aggression⋆	Physical signs of pain (swelling of cheek or gum, broken teeth, ulcers), as well as verbal and/or behavioural signs (pulling at face, not eating, aggression)⋆
Referral to an oral health professional: □ yes □no				Total score: /16

An asterisk * and underline indicates referral to an oral health professional (i.e., dentist, dental hygienist, denturist) is required.

Subject recruitment

Subjects were recruited based on specific criteria. The inclusion criteria were: (i) staying in residential care settings (assisted living facilities and nursing homes); or (ii) being hospitalised in an acute geriatric department. A consecutive sample was used to recruit subjects in the acute geriatric department and nursing homes. To recruit subjects in the assisted living facilities, a convenience sample was used due to practical reasons. A large variation in dental status in all these settings was expected42., 43.. Institutionalised and home-bound elderly are among the most dentally neglected subjects and have poorer oral health compared with elderly individuals living independently25., 44.. Age and cognitive ability were not used as exclusion criterion, as the OHAT was specifically developed for use in elderly patients with varying degrees of cognitive impairment. The level of cognitive impairment was determined from a participant’s chart review based on their medical diagnosis or their Mini Mental State Examination (MMSE) score⁴⁵. If the MMSE score was unavailable, the principal investigator, who is experienced in cognitive disorders, determined the level of cognitive impairment through extensive observation of language comprehension, executive functioning, attention and consciousness. Three nursing homes, two assisted living facilities and one acute geriatric department in a general hospital participated in the study. There were some differences in the provision of oral care between these institutions. Subjects residing at nursing homes were dependent on a delegated nurse for oral health care or from any other available nurse. Subjects in the acute geriatric department also received oral care from any available nurse, whereas subjects residing at the assisted living facilities were mainly responsible for their own oral health care.

Ethical approval was granted by two independent ethics boards, namely the Committee for Medical Ethics of the University Hospital of Antwerp and the Ethics Committee of the H. Hart Hospital of Roeselare-Menen (B300201215080). The study was conducted in full accordance with the World Medical Association Declaration of Helsinki. Prior to the start of the study, verbal and written consent was obtained from all subjects or from the legal representative or the appropriate directors of nursing if the patient could no longer provide written consent. The consent procedure for the study and the test–retest reliability evaluation was approved by the two previously mentioned ethics committees. Subjects with incomplete baseline data were not withheld from the test–retest evaluation.

Procedures and measures

The study was divided into two major parts: the preparatory study and the actual investigation. The flow chart presented in Appendix II shows the different steps of the study.

Appendix II.

Flow chart to define the different steps of the study.

Part 1: Preparatory study

The preparatory study consisted of two phases.

Phase 1.

A 3-hour training session with visual instruction was delivered to three SPs by the principal investigator using publicly available visual training resources28., 33., 34., 37., 38., 39.. The SPs had extensive experience in dysphagia management. The training was followed by trial assessments of three subjects in the acute geriatric department. The scoring of each category was discussed until agreement was reached in accordance with the visual training resources of the OHAT28., 33., 34., 37., 38., 39.. To facilitate OHAT use, a manual with descriptors of the different scores for each category was provided.

Phase 2.

Trial assessments of 17 subjects in a nursing home were performed to determine whether the SPs experienced difficulties in assigning a score for a certain category, to determine whether the SPs felt confident in completing the screening by means of the manual and the publicly available visual training resources28., 33., 34., 37., 38., 39., and to facilitate time registration while performing the screening. Each subject was simultaneously screened by the three SPs who independently completed the OHAT. After the completion of all trial assessments, the results were compared. Between Phase 2 of the preparatory study and the start of the actual investigation, there was an interval of 2 weeks to allow further practise with the OHAT tool. The SPs were required to perform the screening on their family members or acquaintances to become familiar with the scoring.

Part 2: Actual investigation

Two weeks after the preparatory part of the study, the actual study was performed over the following 14 weeks (Appendix II). All three SPs went simultaneously to all facilities. During the actual investigation, the feasibility, inter-rater reliability and test–retest reliability were assessed. Following the actual study, the intra-rater reliability was assessed using videotapes.

Feasibility of the OHAT.

The feasibility of the OHAT was defined based on the time required to complete the OHAT, the ability to score the categories of the OHAT and possible problems in administering the OHAT. All SPs completed a semi-structured questionnaire at three time points, namely, at the end of Phase 2 of the preparatory study, at baseline, and at the end of the actual study; this questionnaire was similar to the original questionnaire from the study by Chalmers et al.²⁸ with the addition of a few open-ended questions. As in the original questionnaire²⁸, a four-point Likert scale, ranging from ‘strongly disagree’ to ‘strongly agree’, was applied to rate the statements. The questionnaire is presented in Appendix III. For each participant in the actual study, the SPs were asked to register the time taken to complete the screening evaluation.

Reliability of the OHAT.

Reliability of the OHAT was measured by evaluating the inter-rater reliability, test–retest reliability and intra-rater reliability.

Inter-rater reliability.

One-hundred and thirty-five subjects were screened by three SPs, simultaneously, but independently, to evaluate the inter-rater reliability of the OHAT. As in Phase 2 of the preparatory study, each SP was blinded to the scores assigned by the other SPs. The subjects were screened in a sitting or supine position.

Test–retest reliability.

Based on advice from a medical statistician, test–retest reliability was assessed in 46 subjects. These subjects were randomly selected from the two nursing homes and one assisted living facility because individuals at these facilities tend to have a more stable health status. Two SPs re-evaluated the selected subjects during a second screening, 2 weeks later. This interval was sufficient to avoid memory effects and the occurrence of genuine oral health status changes in the subjects⁴⁶.

Intra-rater reliability based on videotapes.

Intra-rater reliability was also investigated based on independent videotape ratings by two SPs at three different time points, with at least 14-day intervals. Ten subjects were randomly selected to be videotaped at a frontal angle with the subject’s head and mouth in the frame to enable observation of the oral cavity. For every subject, a 10-minute recording was performed. The subjects were asked to open their mouth. The camera zoomed in on particular parts of the mouth (i.e. the lips, tissues, tongue, dentures and natural teeth), moving from one side to another. Afterward, dentures were removed and the camera was zoomed in on the upper and lower sides of the dentures to obtain a clear view of oral hygiene. Videos were chosen over photographs, because videotapes have proven to be useful for allowing multiple raters to observe the same performance⁴⁶. The rating of the videotapes was performed in a random order, 14 days after completing the actual study.

Data analysis

Population characteristics (i.e., age and gender) were assessed for normality using the Shapiro–Wilk test and QQ-plots. Differences in age, gender and the presence of cognitive impairment according to place of residence were assessed using the Kruskal–Wallis test (age) and the chi-square test (gender and cognitive impairment). The distribution of the scores for individual categories at baseline was assessed for the three different settings, and the Kruskal–Wallis test was used to detect significant differences between these settings. The frequency distribution of the total OHAT score was evaluated at baseline. Floor and ceiling effects associated with the total OHAT score were considered to be present if more than 15% of the subjects achieved the lowest or highest possible score⁴⁷. To evaluate the inter-rater reliability of the OHAT, the intraclass correlation coefficient (ICC) and a two-way random-effects model with measures of absolute agreement (ICC_{absolute agreement})47., 48. were used for the total OHAT scores. The inter-rater reliability of the individual OHAT categories was assessed using Fleiss Kappa. The ICC with a one-way random-effects model with measures of absolute agreement was used to assess the test–retest stability, an evaluation of intra-rater reliability⁴⁶, and the intra-rater reliability based on individual evaluations of the videotapes at different times. For individual categories, Cohen’s Kappa and Fleiss Kappa were, respectively, used to assess the test–retest and intra-rater reliabilities based on videotapes. The associated 95% confidence intervals were calculated using a bootstrap of 1,000 samples. As suggested in previous studies⁴⁶, ICC values higher than 0.75 are indicative of good reliability, whereas values lower than 0.75 represent poor-to-moderate reliability. Kappa statistic values of <0.00 were interpreted as indicating poor agreement, 0.00–0.20 as indicating slight agreement, 0.21–0.40 as indicating fair agreement, 0.41–0.60 as indicating moderate agreement, 0.61–0.80 as indicating substantial agreement and > 0.80 indicating almost perfect⁴⁶ or excellent agreement⁴⁹. Because Fleiss and Cohen’s Kappa depend heavily on the observed marginal frequencies, these scores can be misleading and should be treated with caution. To aid interpretation, we therefore additionally reported the per cent agreement for each individual category. The statistical analyses were performed using r 3.0.1 (R Foundation for Statistical Computing, Vienna, Austria) and spss 20.0 (IBM, SPSS, Inc., Chicago, IL, USA), with P <0.05 considered significant. Only complete screenings were included in the data analysis.

RESULTS

Of the 135 subjects, 132 completed the screening. Three subjects (two in nursing homes and one in the acute geriatric department) were excluded because of dementia-associated behavioural problems. In total, 70 subjects were recruited from nursing homes, 30 from assisted living facilities and 32 from the acute geriatric department. The demographic characteristics of the subjects are presented in Table 1. No significant differences (P > 0.05) in age and gender were observed with regard to the place of residence. Significant differences (P < 0.001) in cognitive status were found between the three settings. Significantly more nursing home residents had cognitive impairment. Among the subjects from the nursing homes and the acute geriatric department, the presence of cognitive impairment was based on a medical diagnosis. However, for seven (23.3%) of the 30 subjects residing at the assisted living facilities, the presence of cognitive impairment was based only on a comprehensive evaluation by the principal investigator. The OHAT score distribution at baseline for the individual categories is shown in Table 2. The majority of the subjects scored 0 in the categories ‘lips’, ‘saliva’, ‘dental pain’ and ‘gums and tissues’. With regard to ‘oral cleanliness’ and ‘tongue’, more than half of the subjects scored 1, and most subjects scored 2 in the categories ‘natural teeth’ and ‘dentures’. Irrespective of the place of residence, the score distribution was similar between the categories ‘lips’, ‘tongue’, ‘gums and tissues’ and ‘dental pain’, as shown in Table 3. The scores for ‘saliva’, ‘natural teeth’ and ‘dentures’ were different between subjects in the acute geriatric department compared with subjects in the other settings. More than half of the hospitalised subjects scored 2 for ‘natural teeth’ and ‘dentures’, and more subjects (37.5%) scored 1 for ‘saliva’ compared with the subjects in the other settings. Half of the subjects residing in nursing homes scored 2 for ‘oral cleanliness’, in contrast with a majority score of 1 for subjects at the other facilities. Differences in the score distribution were only significant (P < 0.05) for ‘saliva’ between the hospitalised patients and the subjects residing in nursing homes and for ‘oral cleanliness’ between the subjects in nursing homes and the subjects residing in assisted living facilities. Figure 1 shows the frequency distribution and corresponding mean and median of the subjects’ total OHAT scores. None of the subjects scored higher than 10, and no floor and ceiling effects were present.

Table 1.

Demographic characteristics of the subjects (n = 132)

Characteristics		Nursing home	Acute geriatric department	Assisted living places
		(n = 70)	(n = 32)	(n = 30)
Age*	Mean (SD)	83.4 (7.2)	84.3 (7.3)	86.2 (7.1)
	Median (min–max)	84.5 (63–101)	85.5 (62–100)	87.5 (63–101)
Gender*	Male, n (%)	18 (25.7)	11 (34.4)	8 (26.7)
	Female, n (%)	52 (74.3)	21 (65.6)	22 (73.3)
Cognitive impairment**	Presence, n (%)	54 (77.1)	14 (43.8)	7 (23.3)
	Absence, n (%)	16 (22.9)	18 (56.3)	23 (76.7)

Values are given as mean and standard deviation (SD) and median (min–max) for age and as n (%) for gender and for cognitive impairment. Analyses were performed using the Kruskal–Wallis test for equality of means for age and the chi-square test for gender and cognitive impairment.

^*P > 0.05.

^**P < 0.001.

Table 2.

Distribution of the scores at baseline for the individual categories of the Oral Health Assessment Tool (OHAT) for all subjects (n = 132)

Category	Score 0		Score 1		Score 2
	n	%	n	%	n	%
Lips	112	84.8	16	12.1	4	3.0
Tongue	37	28.0	94	71.2	1	0.8
Gums and tissues	69	52.3	55	41.7	8	6.1
Saliva	104	78.8	28	21.2	0	0
Natural teeth	44	33.3	21	15.9	67	50.8
Dentures	56	42.4	2	1.5	74	56.1
Oral cleanliness	17	12.9	68	51.5	47	35.6
Dental pain	121	91.7	10	7.6	1	0.8

Table 3.

Percentage distribution of Oral Health Assessment Tool (OHAT) scores at baseline for the individual categories regarding the place of residence (n = 132)

Category	Nursing home			Acute geriatric department			Assisted living places
	Score 0	Score 1	Score 2	Score 0	Score 1	Score 2	Score 0	Score 1	Score 2
Lips	85.7	12.9	1.4	78.1	15.6	6.3	90.0	6.7	3.3
Tongue	31.4	67.1	1.4	28.1	71.9	0.0	20.0	80.0	0.0
GandT	55.7	40.0	4.3	50.0	46.9	3.1	46.7	40.0	13.3
Saliva*	88.6	11.4	0.0	62.5	37.5	0.0	73.3	26.7	0.0
NT	37.1	12.9	50.0	18.8	18.8	62.5	40.0	20.0	40.0
Dent	44.3	2.9	52.9	28.1	0.0	71.9	53.3	0.0	46.7
OC*	11.4	38.6	50.0	12.5	59.4	28.1	16.7	73.3	10.0
DP	90.0	8.6	1.4	90.6	9.4	0.0	96.7	3.3	0.0

Analyses were based on the Kruskal–Wallis test for the score distribution, irrespective of the place of residence. Dent, dentures; DP, dental pain; GandT, gums and tissues; NT, natural teeth; OC, oral cleanliness.

^*P < 0.05.

Figure 1.

Frequency distribution of the total OHAT scores.

Feasibility of the OHAT

At baseline, the mean time (standard deviation; range) to completion of the OHAT was 2.45 (1.05; 0.42–6.20) minutes. All SPs thought that the OHAT was simple to use and quick to administer. One SP reported that the category ‘oral cleanliness’ was difficult to score at times. Subjects with dentures were easier to score than subjects with natural teeth. The questionnaire results showed that at both baseline and the end of the actual study, all three SPs ‘strongly agreed’ with nearly all the formulated statements. However, at baseline, one SP only ‘agreed’ with the statement related to the ‘dentures’ category. Comparing the questionnaire results from baseline with those from the end of the preparatory study, greater diversity was found in the self-perceived ability to score the various categories of the OHAT. During the preparatory part of the study, the SPs ‘agreed’ with most of the formulated statements, and two SPs ‘strongly agreed’ with the statements related to ‘natural teeth’, ‘dentures’ and ‘dental pain’. However, one SP ‘disagreed’ with the statement “able to complete the ‘dentures’ category” and another SP ‘disagreed’ with the statement related to ‘oral cleanliness’. None of the statements was rated as ‘strongly disagree’. An analysis of the answers to the open-ended questions revealed the following results: (i) there was a lack of information on pairs of teeth in the chewing position; (ii) sufficient visual resources were available to complete the OHAT; and (iii) the manual was seen as an important contribution because it leads to higher consensus in score assignment. At the end of the actual study, the manual was no longer necessary because of familiarity with the scoring. The presence of referral possibilities at the bottom of the scoring sheet was judged as useful.

Inter-rater reliability

The ICC value for the total OHAT score was 0.96 [95% confidence interval (95% CI) = 0.95–0.97], indicating very good inter-rater reliability⁴⁶. The inter-rater reliabilities of the individual OHAT categories and the advice for referral to an oral health professional are shown in Table 4. Kappa statistics neared perfect values (κ ≥ 0.83) for seven of the eight individual categories and the need for referral, and achieved perfect agreement for ‘dental pain’ (κ = 1.00).

Table 4.

Inter-rater reliability data [per cent agreement and Fleiss Kappa (three raters)] for individual categories and for referral to an oral health professional (n = 132)

Category	Per cent agreement (95% CI†)	Fleiss Kappa (95% CI†)
Lips	0.97 (0.94–0.99)	0.88 (0.80–0.96)
Tongue	0.95 (0.92–0.98)	0.89 (0.81–0.96)
GandT	0.91 (0.87–0.94)	0.83 (0.76–0.90)
Saliva	0.99 (0.96–1.00)	0.95 (0.90–1.00)
NT	0.98 (0.96–1.00)	0.97 (0.94–1.00)
Dent	0.98 (0.96–1.00)	0.97 (0.93–1.00)
OC	0.92 (0.89–0.96)	0.87 (0.81–0.93)
DP	1.00 (1.00–1.00)	1.00 (1.00–1.00)
Refer OHP	0.98 (0.96–1.00)	0.93 (0.84–1.00)

Dent, dentures; DP, dental pain; GandT, gums and tissues; NT, natural teeth; OC, oral cleanliness; Refer OHP, referral to an oral health professional (i.e. dentist, dental hygienist or denturist).

^†95% confidence interval.

Test–retest reliability

The stability of scores over time, the intra-rater reliability (represented by Cohen’s Kappa statistic) and the test–retest per cent agreement for the various categories and for the decision to refer in 46 subjects are shown in Table 5. The reliability data are provided separately for each rater. The ICC for the total OHAT score were 0.81 (95% CI = 0.68–0.89) and 0.78 (95% CI = 0.64–0.87) for raters 1 and 2, respectively, indicating good reliability⁴⁶. The test–retest and intra-rater Kappa statistics indicated almost perfect agreement for the categories ‘natural teeth’ and ‘dentures’. A substantial level of agreement was reached for ‘oral cleanliness’ for rater 1, whereas the agreement was only moderate for rater 2 (κ = 0.55). The test–retest and intra-rater Kappa statistics were moderate for tongue and saliva for both raters. With regard to ‘gums and tissues’, moderate scores were obtained by one rater, and the other rater only achieved slight agreement (κ = 0.15). ‘Lips’ showed fair agreement (κ = 0.38) and ‘dental pain’ showed slight agreement (κ = 0.14) for both raters. Regarding referral, the agreement was substantial for rater 1 (κ = 0.69) and was fair for rater 2 (κ = 0.39).

Table 5.

Test–retest reliability (per cent agreement and Kappa statistics for individual categories and for referral to an oral health professional; n = 46)

Category	Rater 1		Rater 2
	Per cent agreement	Cohen’s Kappa	Per cent agreement	Cohen’s Kappa
	(95% CI†)	(95% CI†)	(95% CI†)	(95% CI†)
Lips	0.89 (0.80 to 0.98)	0.38 (−0.10 to 0.79)	0.89 (0.78 to 0.96)	0.38 (−0.10 to 0.78)
Tongue	0.78 (0.67 to 0.89)	0.44 (0.14 to 0.69)	0.80 (0.70 to 0.91)	0.51 (0.23 to 0.76)
GandT	0.72 (0.59 to 0.83)	0.42 (0.15 to 0.65)	0.59 (0.46 to 0.74)	0.15 (−0.10 to 0.42)
Saliva	0.91 (0.83 to 0.98)	0.52 (−0.05 to 0.90)	0.91 (0.83 to 0.98)	0.45 (−0.07 to 0.85)
NT	0.93 (0.85 to 1.00)	0.89 (0.75 to 1.00)	0.91 (0.83 to 0.98)	0.86 (0.70 to 0.97)
Dent	0.96 (0.89 to 1.00)	0.91 (0.76 to 1.00)	0.93 (0.85 to 1.00)	0.87 (0.70 to 1.00)
OC	0.83 (0.70 to 0.93)	0.69 (0.47 to 0.88)	0.74 (0.61 to 0.87)	0.55 (0.30 to 0.76)
DP	0.85 (0.74 to 0.93)	0.14 (−0.14 to 0.55)	0.85 (0.74 to 0.93)	0.14 (−0.14 to 0.55)
Refer OHP	0.93 (0.87 to 1.00)	0.69 (0.18 to 1.00)	0.93 (0.87 to 1.00)	0.39 (0.22 to 1.00)

Dent, dentures; DP, dental pain; GandT, gums and tissues; NT, natural teeth; OC, oral cleanliness; Refer OHP, referral to an oral health professional (i.e. dentist, dental hygienist or denturist).

^†95% confidence interval.

Intra-rater reliability based on videotapes

The ICC for intra-rater reliability for the total OHAT score, based on ratings of 10 videotapes, showed good reliability for both raters [rater 1: ICC = 0.95 (95% CI = 0.88–0.99); rater 2: ICC = 0.96 (95% CI = 0.89–0.99)]. Table 6 shows the per cent agreement for the individual categories and for referrals. As almost no variance was observed in the scoring, the Fleiss Kappa could not be calculated.

Table 6.

Per cent agreement for individual categories and for referral to an oral health professional (n = 10)

Category	Rater 1	Rater 2
	Per cent agreement (95% CI†)	Per cent agreement (95% CI†)
Lips	0.93 (0.80–1.00)	1.00 (1.00–1.00)
Tongue	1.00 (1.00–1.00)	0.93 (0.80–1.00)
GandT	1.00 (1.00–1.00)	0.73 (0.53–0.93)
Saliva	1.00 (1.00–1.00)	1.00 (1.00–1.00)
NT	1.00 (1.00–1.00)	1.00 (1.00–1.00)
Dent	0.93 (0.80–1.00)	1.00 (1.00–1.00)
OC	1.00 (1.00–1.00)	1.00 (1.00–1.00)
DP	0.85 (0.74–0.93)	1.00 (1.00–1.00)
Refer OHP	1.00 (1.00–1.00)	1.00 (1.00–1.00)

Because of the lack of significant variance, Kappa was not calculated. Dent, dentures; DP, dental pain; GandT, gums and tissues; NT, natural teeth; OC, oral cleanliness; Refer OHP, referral to an oral health professional (i.e. dentist, dental hygienist or denturist).

^†95% confidence interval.

DISCUSSION

Daily oral health care in long-term institutions is often viewed as a nursing task¹⁶. However, based on the scope of practice8., 9. and the educational background⁵of SPs, they play a supplementary role in oral health promotion¹⁶ and in the evaluation of oral health and dentition⁶. During a clinical swallowing examination, SPs inspect the oral cavity and dentition; however, a standardised assessment of oral health is typically not implemented. Therefore, the aim of this study was to evaluate the feasibility and reliability of the OHAT as used by SPs.

Feasibility

This study demonstrates that the OHAT is a feasible instrument and is quick and simple to administer (with no need for special equipment). At the end of the preparatory part of the study, a few uncertainties in completing the tool were reported by the SPs, which were subsequently addressed and resolved. The short training period between the end of the preparatory study and baseline was sufficient to increase the self-confidence among the SPs in completing the tool. During the actual study, the SPs reported that they were able to complete the assessment. Thus, the self-perceived ability to complete all OHAT categories did not change significantly during the actual study, although differences occurred in the need for relying on the visual training resources. However, the initial need of the SPs to consult the manual showed a progressive decrease after 14 weeks of practice with the OHAT. As a result of methodological differences, it was not possible to compare the questionnaire results from our study with those of the study by Chalmers et al.²⁸.

Reliability

The inter-rater reliability and test–retest reliability of the total OHAT score were adequate for this study. In evaluating the individual categories of the OHAT, a high level of inter-rater reliability was achieved with almost perfect agreement in seven of the eight individual categories and the need for referral, as well as with perfect agreement for ‘dental pain’. Although the reliability was good for this study, only three raters were involved. Future studies should evaluate whether the reliability data are different when applying the OHAT to a larger sample of SPs. The inter-rater reliability data were higher in this study than in the previous study by Chalmers et al.²⁸, in which different types of nurses were involved as raters²⁸. Differences in educational background, the availability of visual training resources that could easily be accessed and the manual could be possible explanations for the higher reliability between the raters in this study. Chalmers et al.²⁸ did not use the Fleiss Kappa to evaluate the inter-rater reliability for all subjects because the inter-rater agreement was only assessed for two raters. In this study, three raters were involved, which increases the likelihood that the measurements obtained at the same time by the raters represent the subjects’ true oral health status. However, this study was limited by not examining the concurrent validity of the tool as determined by comparing the OHAT results obtained from SPs with a dental examination completed by a qualified dentist. To use the OHAT optimally among SPs and to establish the high sensitivity (=the presence) and high specificity (=the absence of the target condition)⁴⁶ of the tool, concurrent validity warrants further assessment, which will be conducted in the next phase of this research. Regarding the test–retest reliability, excellent agreement was only reached for ‘natural teeth’ and ‘dentures’. The test–retest results showed lower levels of agreement for ‘gums and tissues’, ‘lips’, ‘dental pain’ and the overall referral decision. The lower levels of agreement may be explained by possible changes in oral health50., 51.. In particular, spontaneous changes in oral health within 2 weeks cannot be excluded50., 51.. In this study, the participating subjects did not receive any recommendations within the 14-week period to improve their oral health care to minimise alterations in the test conditions. However, the subjects did not alter their oral care habits during the actual study, and standard oral care was performed if present. The study was limited by not examining the type of oral care practices the subjects received or applied themselves. The test–retest results from this study could not be compared with the study results of Chalmers et al.²⁸ because of the different methodologies. Additionally, the intra-rater reliability was assessed using videotapes. Despite a high ICC for both raters for the total OHAT score and a high percentage of agreement in the individual categories, caution is required in interpreting the results for the individual categories. In fact, it was not possible to calculate the Fleiss Kappa because of limited variance in the scores. Moreover, the use of very small samples (only 10 videotapes) may yield misleading results from a proportion-based Kappa-statistic⁴⁶. However, videotapes offer the advantage of allowing the assessment of identical aspects of oral health status and reducing stress that influences clinical presentation⁵². Videos also provide dynamic images and have been applied as a medium for dental health education in previous reports53., 54..

Scoring and interpretation of the OHAT

Regarding the distribution of the subjects’ total OHAT scores, a score of 5 was most frequently obtained. This total score may indicate the severity of the oral health status. However, each item should be considered separately because referral could be determined based on a single aberrant category. Therefore, the clinical significance of the total score should be questioned, as this factor has not been evaluated in previous studies. Further investigation should focus on the correlation between the total score and differences in the severity of oral health status as well as the need for referral. The mean total OHAT score was higher in this study than previously reported mean total OHAT scores²⁸, possibly because the three SPs judged oral health status more strictly than did other providers. In particular, the categories ‘dentures’ and ‘natural teeth’ were scored as ‘unhealthy’ for the majority of the subjects in this study, whereas a larger proportion of the subjects in the study by Chalmers et al.²⁸ scored ‘healthy’ for the same categories. This discrepancy in scoring necessitates further validation of the OHAT tool. Therefore, further research should focus on the accuracy of the OHAT tool when administered by SPs and nurses compared with a dental examination by a qualified dentist.

Type of residence and oral health

The score distribution for individual categories across the places of residence revealed that hospitalised patients had worse dental status (dentures and natural teeth) and more dry tissues compared with other subjects. Although there were no statistically significant differences between the residential care settings for the categories ‘dentures’ and ‘natural teeth’, the finding that dental status was worst in the hospitalised patients was corroborated by Pajukoski et al.⁴³. This finding may be explained by differences in concomitant diseases and polymedication, rather than by the nature of a patient’s illness⁴³. However, we did not perform an investigation of possible underlying conditions and etiological factors. Consequently, an irrefutable explanation is also lacking for the finding that a significant difference was found for ‘saliva’ between the hospitalised patients and the subjects residing in nursing homes. Oral cleanliness was worse in nursing homes, which could be attributed to the greater cognitive impairment of the inhabitants, resulting in difficulty performing oral hygiene. Behavioural difficulties associated with dementia, such as refusal to open the mouth, are seen as especially challenging tasks for oral care providers²⁷. Additionally, greater accumulations of dental plaque and calculus have been found on natural teeth and dentures in patients with dementia²⁷. Our study was limited by the lack of a standardised cognitive assessment battery in all subjects to evaluate their cognitive abilities. However, the presence of cognitive impairment was obvious in the hospitalised patients and the nursing home residents, as it was determined based on medical diagnoses. Caution is needed when interpreting the results of the subjects from the assisted living facilities, as the presence of cognitive impairment lacks a true medical diagnosis. In the assisted living facilities, where most of the ‘more independent’ subjects provide oral care themselves and had lower rates of cognitive impairment, oral cleanliness was better. However, approximately half of the elderly living in those assisted living facilities needed referral to an oral health professional because of the condition of their natural teeth or dentures. The differences in the nature of the oral categories needing intervention may ultimately lead to establishing oral health care intervention programmes. However, replication of this study with equally balanced groups is recommended. Based on the quantitative and descriptive interpretations of this study, we suggest the following adjustments to the OHAT to improve the effectiveness of this assessment tool. A description of the number of pairs of teeth in the chewing position, as in the original BOHSE25., 30., may provide additional information on mastication and food selection. Uncertainty in evaluating oral cleanliness could be resolved by incorporating additional illustrations in the publicly available visual training resources. However, the inter-rater reliability of this category was good. The category ‘dental pain’ assesses pain not only as a consequence of dental problems but also as a result of ulcers anywhere in the mouth. Therefore, assigning a different name to that category could be considered, further obviating the need for constant referral to the manual or available visual training resources of the OHAT. Future studies should evaluate whether the systematic implementation of an oral health assessment tool, such as the OHAT, during a swallowing examination would promote oral and dental care.

CONCLUSION

The findings of this study show that the OHAT is a feasible and reliable oral health assessment tool that can be used in clinical practice by SPs to screen oral health in a standardised manner in elderly dysphagic subjects. Future research is necessary to evaluate whether the implementation of an oral health assessment tool within a swallowing examination can promote oral health care awareness in daily clinical practice. We suggest exploring whether an oral health assessment tool used by SPs could have the potential to improve oral care management in dysphagic subjects.

Acknowledgements

This study did not receive any financial support. Many thanks to Griet Vercruysse, Emke Vercruysse, Hannelore Simpelaere and all participants, staff and Directors from WZC Marialove Heestert, WZC De Nieuwe Seigneurie Rumbeke, WZC Ter Berk Anzegem, Serviceflats Regina Coeli Brugge, Serviceflats De Beiaard Harelbeke and Geriatric Department of AZ Delta Campus Menen.

Conflicts of interest

The authors have no conflicts of interest to disclose.

Conflicts of interest

Appendix III Questionnaire to evaluate the feasibility of the OHAT

Based on the original questionnaire, developed by Chalmers et al.²⁸

• 1.
  Statements (Instruction: Please rate how strongly you agree or disagree with each of the formulated statements. Choose one of the following options: ‘strongly disagree’, ‘disagree’, ‘agree’ or ‘strongly agree’)

  • a
    I feel knowledgeable and prepared to use the OHAT.
  • b
    Using the OHAT improves my ability to detect dental pain and problems in residents’ mouths.
  • c
    I had enough time to learn about the OHAT before it was implemented.
  • d
    I am able to complete the ‘lips’ category of the OHAT.
  • e
    I am able to complete the ‘tongue’ category of the OHAT.
  • f
    am able to complete the ‘gums and tissues’ category of the OHAT.
  • g
    I am able to complete the ‘saliva’ category of the OHAT.
  • h
    I am able to complete the ‘natural teeth’ category of the OHAT.
  • i
    am able to complete the ‘dentures’ category of the OHAT.
  • j
    am able to complete the ‘oral cleanliness’ category of the OHAT.
  • k
    I am able to complete the ‘dental pain’ category of the OHAT.
• 2.
  Open-ended questions

  • a
    Do you experience difficulties when applying the Oral Health Assessment Tool?
    Yes/No, explanation:
    Do you think the intended section to refer to a dental professional is a necessary part of the oral health assessment tool?
    Yes/No, explanation:
  • b
    Do you have sufficient visual resources to perform the OHAT?
    Yes/No, explanation:
  • c
    Do you need the manual to perform the OHAT?
    Yes/No, explanation: