The design of a valid and reliable questionnaire to measure osteoporosis knowledge in women: the Osteoporosis Knowledge Assessment Tool (OKAT)

Abstract

Background

Osteoporosis knowledge is an important contributor to improving exercise and calcium intake behaviour. However, there are few validated instruments for measuring osteoporosis knowledge levels. The aim of this study was to design a valid and reliable instrument to measure osteoporosis knowledge in Australian women.

Methods

A 20 item instrument with true, false and don't know responses was drafted, based on the Osteoporosis Australia Osteoporosis Prevention and Self-management course and the information leaflet "Understanding Osteoporosis". The scoring range was 1 to 20. This was administered to a 467 randomly-selected, healthy women aged 25–44 years. Questionnaire performance was assessed by Flesch reading ease, index of difficulty, Ferguson's sigma, inter-item and item-total correlations, Cronbach's alpha and principal component factor analysis.

Results

Flesch reading ease was higher than desirable at 45, but this was due to the use of the word osteoporosis in many items. Of the individual items 17 had an index of difficulty less than 0.75. The questionnaire had a Ferguson's sigma of 0.96, a Cronbach's alpha of 0.70 and factor analysis consistent with only one factor (osteoporosis knowledge) being measured. Levels of osteoporosis knowledge were low with a mean score of 8.8 out of 20 which suggests the OKAT may be sensitive to change.

Conclusions

The OKAT for measuring osteoporosis knowledge has good psychometric properties in Australian 25–44 year old females. While it should be applicable to other Caucasian populations, this will require confirmation by further research.

Background

Osteoporosis is a major and growing public health problem in both sexes but particularly in women [1,2]. It is responsible for approximately 75,000 fracture events each year in Australia in those over 60 years of age at a cost to the community of $774 M [3]. It is estimated that the proportion of women with osteoporosis increases from 15% in those aged 60 to 64 years up to 71% in those over 80 years of age. In the United States, 10 million people already have osteoporosis and 18 million more have low bone mass [4]. Physical activity and adequate calcium intake are both important for the prevention of osteoporosis [4]. There is evidence suggesting that osteoporosis knowledge is one contributor to osteoporosis preventive behaviour, though this is not a clear-cut relationship. Cross-sectional studies have varied in whether they have found an association between levels of osteoporosis knowledge and osteoporosis preventive behaviours [5-9]. Prospective studies similarly have been conflicting with some studies demonstrating increases in osteoporosis knowledge and concurrent improvements in osteoporosis preventive behaviours [10-12] but others demonstrating changes in knowledge but not behaviour [13,14]. The limited research examining the ways in which osteoporosis knowledge might affect behaviour have demonstrated a relationship. In one study, causal analysis was used to demonstrate that osteoporosis knowledge was an important contributor to improving exercise and calcium intake behaviour [15]. Another study utilizing the precaution adoption process model found that women who were in the never-engaged stage of change (ie unaware of the health problem and of the precaution recommended to reduce the risk of experiencing the problem) had the lowest levels of osteoporosis knowledge and that knowledge levels tended to be higher in women at higher stages of change [16].

Few studies have reported levels of osteoporosis knowledge in random, population-based samples [13,17-19]. None of these have utilized validated instruments to measure osteoporosis knowledge and information available about the psychometric properties of the tools used was limited. None have specifically examined knowledge in women in the 25–44 year age range. Therefore, standardized osteoporosis knowledge levels in the general population and specifically in the 25–44 year age range remain unclear.

Measurement of osteoporosis knowledge is itself problematic and this may be part of the reason for the variation in the results of the studies described previously. There are three partially or fully validated instruments in the literature to measure osteoporosis knowledge [20-22]. We required an instrument in which all items were relevant to the Australian population. For example, two items in the Facts on Osteoporosis Quiz [20] referred to comparisons to African-American women, which were not relevant to populations outside the USA. The planned use of the instrument was to study women who had reached peak bone mass, but were premenopausal, so the instrument needed to have been fully validated in the 25–44 year age range. As the instrument was to be administered as part of a large study, it had to be suitable for self-administration, and easy to code the results. None of the existing instruments satisfied all these criteria.

The aim of this study was therefore to describe the development of and psychometric properties of an instrument to measure osteoporosis knowledge for use in a population-based random sample of 25–44 year old women.

Methods

The study was carried out in Southern Tasmania, Australia as part of an ongoing study examining the effects of lifestyle factors on bone mineral density in women aged 25–44. The population of the region in 2000 was 194,389 of which, 28,839 were women aged between 25 and 44 years of age [23]. Subjects were randomly selected in this age range from the 2000 electoral roll. Subjects were excluded if they had previously had measurement of bone density, had thyroid disease, renal failure, malignancy or rheumatoid arthritis, had a history of hysterectomy, were on hormone replacement therapy, were pregnant or planning pregnancy within 2 years of study entry, or were lactating. Ethics approval was obtained from the Royal Hobart Hospital Ethics Committee. All subjects gave written informed consent.

Osteoporosis knowledge was measured in subjects at baseline from April to November 2000. The survey instrument was based on knowledge content common to the Osteoporosis Prevention and Self-management course (OPSMC) and an information leaflet produced by Osteoporosis Australia "Understanding Osteoporosis". The OPSMC is a chronic disease self-management course developed by the Arthritis Foundation of Victoria and utilized by Osteoporosis Australia. Osteoporosis Australia is the peak body involved in community-based osteoporosis education in Australia. The OPSMC is a small group education program that aims to increase knowledge, improve confidence and awareness and self-management of osteoporosis prevention with an emphasis on promoting appropriate lifestyle change. Educational methods include lectures, discussion, brainstorming, demonstration and small group work. The information leaflet covers information on osteoporosis including its definition, natural history of bone strength, risk factors and preventive behaviours, including physical activity and calcium intake.

The items were selected from the common material in the two interventions by a consultant rheumatologist and researcher who has specialist expertise in osteoporosis. The knowledge instrument aimed to measure a broad range of osteoporosis knowledge items that would be applicable to the Australian setting and to avoid items that would be difficult to adapt to other settings in which osteoporosis demographics might be different. It is a 20 item questionnaire, with each item having true, false and don't know options (see Additional file: 1). The analysis was performed by scoring 1 for a correct response and 0 for an incorrect or don't know response. The total score could range from 0 to 20. Face validity was also assessed using a panel of 20 people from our institution. These included research nurses, research assistants and administrative staff.

Other factors measured in the protocol were height by stadiometer (Leicester height measure, Invicta Plastics Ltd, Oadby, England) and weight by a single set of calibrated scales (Heine, Dover NH USA). Body mass index was calculated (weight/ht²). Questionnaire assessment was also made of smoking history (current/former/never), number of children, family history of osteoporosis and/or fracture, as well as fracture history in the subject, education level (4 point scale: less than grade 10, up to grade 10, completed grade12, tertiary), employment status of main financial provider in the household (employed or unemployed), hours of employment of the respondent (0, less than or equal to 20 or >20 hours per week) and marital status (6 categories).

Statistics

The psychometric properties of the osteoporosis knowledge questionnaire were assessed by examining:

1. The Flesch reading ease. This assesses readability based on the average number of syllables per word and the average number of words per sentence. Scores range from 0 to 100. Standard writing averages approximately 60 to 70 [24]. The higher the score, the greater the number of people who can readily understand the document.

2. The index of difficulty. This is defined as the proportion of patients answering the item correctly and is calculated by: number of correct responses/total number of responses [25]. An item with an index of difficulty higher than 0.75 is deemed to be poor as it is too frequently answered correctly.

3. Item discrimination [25]. This tests how well an item discriminates between people who have a low and high osteoporosis knowledge score. For each item, a D-value is calculated by subtracting for each item the proportion of respondents answering correctly in the lowest quartile from those answering correctly in the highest quartile, aiming for a mean D-value of 50%.

4. Discriminatory power was measured by Ferguson's sigma. This has a minimum of 0 if all subjects get the same score, and a maximum of 1 if the subjects are equally divided among all possible scores as is desirable.

5. Inter-item correlation matrix. This was performed to check for negative correlations and to screen for items with consistently weak correlations with other items (r < 0.09, based on a sample size of 467, and p > 0.10).

6. Item-total correlations. The correlation of an item with the remainder of the scale with that item omitted is the item-total correlation. A correlation of <0.20 is considered poor [26].

7. Cronbach's alpha. This is a measure of inter-item consistency, and ranges from 0.0 to 1.0. If omitting an item increases Cronbach's alpha significantly, then excluding the item increases the homogeneity of the scale, which is desirable.

8. Factor analysis. Principal components factor analysis ascertains whether the underlying factors identified statistically within data collected by a survey instrument are consistent with the theoretical factors one was aiming to measure with the instrument. It also assesses whether the loading of individual items on the identified factors are consistent with the premise on which the survey instrument was constructed, though in the case of dichotomous items, these loadings need to be interpreted with caution [27]. In the case of questionnaires assessing knowledge, avoiding dichotomous items is problematic. Factor analysis has been used before in this context [22]. In this study, we would expect only one main factor (namely osteoporosis knowledge) with items loading above 0.3 [28] on this factor.

All analyses were performed in Stata version 7 (Stata Corporation, Texas, USA), except for the Flesch reading ease which was calculated in Microsoft Word 2000(Microsoft Corporation). A p-value < 0.05 (two tailed) was regarded as statistically significant.

Results

A total of 467 women were recruited (response rate 53%). The characteristics of subjects are given in table 1. Due to small numbers in the lowest educational level and in some categories of marital status, these data are presented as three categories of education and two of marital status (married or defacto and other). The average age of participants was 37.8 years, and there was a wide spread of participants across educational levels and levels of employment. The average baseline osteoporosis knowledge score was 8.8 out of a possible 20 (s.d 3.3, range1 to 17).

Table 1.

Characteristics of participants (n = 467).

Characteristic	Mean (SD) or %
Age	37.8 (5.4)
Height	163.1 (6.4)
Weight	69.6 (13.6)
BMI	26.1 (4.8)
Education level, %
Grade ten or less	33
Completed grade 12	21
University or other tertiary institution	45
Main financial provider unemployed, %	6
Number of children, median, (range)	2, (0–5)
Family history osteoporosis, %	17
Family history of fracture, %	62
History of fracture, %	29
Currently smoking, %	17
Ever smoked, %	49
Married or de facto, %	72

Table 2 gives psychometric characteristics of the OKAT. The questionnaire had a Flesch reading ease of 45. If the word osteoporosis were removed from the questions or substituted with the term "thin bones", the reading ease rose to 65 and 68, respectively.

Table 2.

Psychometric characteristics of the OKAT

Characteristic	Result
Flesch Reading Ease	45
D-value	44%
Discriminatory power	0.96
Cronbach's alpha	0.69

The index of difficulty for most items was satisfactory (between 0.12 and 0.66). Items 1 (Osteoporosis leads to an increased risk of bone fractures.), 4 (Osteoporosis is more common in men.) and 10 (Any type of physical activity is beneficial for osteoporosis.) scored above 0.75, indicating that most subjects answered these questions correctly.

The mean D-value for the questionnaire was 44%. Ferguson's sigma for the questionnaire is 0.96 which is very high.

There were no negative inter-item correlations. Items 9 (From age 50, most women can expect at least one fracture before they die.), 7 (A fall is just as important as low bone strength in causing fractures.) and 10 performed poorly when inter-item correlations were examined. These items had less than 50% of correlations with other items above 0.09. All other items had more than 50% of correlations above 0.09 and most had more than 75% and so were satisfactory. Item-total correlations are shown in table 3. Items 1, 7, 9 and 10 had an item-total correlation of less than 0.20. However, eliminating these items iteratively altered Cronbach's alpha by less than 1.5%. The changes in Cronbach's alpha resulting from elimination of each individual item in turn can be seen in table 3. For the 20 item questionnaire Cronbach's alpha was 0.69 and this increased only to 0.71 if all four items were excluded.

Table 3.

Psychometric Properties of the OKAT by Item

Item Number	Index of Difficulty	Item Discrimination (%)	Item-total correlation	Cronbach's alpha*	Factor Loading
1	0.97	12	0.17	0.69	0.21
2	0.37	63	0.33	0.68	0.42
3	0.43	63	0.34	0.68	0.43
4	0.91	21	0.22	0.69	0.27
5	0.60	67	0.37	0.67	0.46
6	0.20	0.47	0.33	0.68	0.41
7	0.39	35	0.13	0.70	0.15
8	0.27	45	0.27	0.68	0.32
9	0.26	24	0.14	0.70	0.14
10	0.80	17	0.03	0.70	0.03
11	0.12	28	0.21	0.69	0.27
12	0.66	47	0.26	0.68	0.32
13	0.27	53	0.32	0.68	0.37
14	0.67	61	0.34	0.68	0.41
15	0.52	53	0.28	0.68	0.34
16	0.50	70	0.40	0.67	0.46
17	0.24	48	0.30	0.68	0.36
18	0.06	17	0.22	0.70	0.26
19	0.30	60	0.35	0.68	0.42
20	0.29	50	0.27	0.68	0.33

*gives Cronbach's alpha with each item omitted in turn.

Principal factor analysis generated only one factor with an eigenvalue above 1 (Factor one – eigenvalue 2.3). Items loaded from 0.026 to 0.46 on this factor. Items 1, 4, 7, 9, 10, 11 and 18 loaded less than 0.3. Elimination of these items iteratively from the factor analysis did not alter the distribution of eigenvalues. There was still only one factor with an eigenvalue greater than one and the distribution of the loadings of the items on this factor was unchanged. Iterative elimination of the items with low loadings on factor analysis from item-total and Cronbach's alpha analyses caused changes of less than 1.5%. When items 1,4,9, 11 and 18 were eliminated there were decreases in alpha, rather than the desired increases.

Discussion

The OKAT performed satisfactorily on virtually all components of the analysis.

The questionnaire had a lower than preferred Flesch reading ease. This was due to the use of the word osteoporosis in 12 of the 20 items. If this word was removed or substituted for by "thin bones" the Flesch reading ease became highly acceptable. As the word osteoporosis was the most accurate description of the disease about which knowledge was being measured and is widely recognised in the general population [17], its use was retained and the resulting decrease in Flesch reading ease accepted.

The questionnaire had a satisfactory index of difficulty. Item discrimination was satisfactory and Ferguson's sigma was close to the ideal value of 1.0. The three items with a high index of difficulty were retained as it was considered they related to core information about osteoporosis namely, the definition of osteoporosis, female preponderance and the physical activity requirements needed for prevention. Though factor analysis must be interpreted cautiously when analysing dichotomous variables, the fact that the analysis generated only one factor with an eigenvalue above 1 is consistent with osteoporosis knowledge being the main factor being measured by the questionnaire, and this provides some support for the construct validity of the instrument. Certainly, if more than one underlying factor had been found, we would have had cause to question construct validity.

The elimination of the items performing poorly on item-total correlation and principal factor analysis changed Cronbach's alpha by less than 1.5% and had minimal effect on the factor analysis outcome, indicating that the elimination of these items had little effect on the psychometric qualities of the questionnaire. The 20 item Cronbach's alpha of 0.69 was satisfactory, particularly given that in order to achieve high discriminatory power, a scale must include very easy items as well as very difficult items, which tends to decrease the internal consistency of the scale [26]. The elimination of individual items that performed less well did not alter the overall psychometric properties of the questionnaire and would have reduced the breadth of knowledge covered by the questionnaire ie affected content validity. These items evaluated knowledge of the meaning of osteoporosis, its prevalence, the adverse health outcomes of osteoporosis and of the physical activity requirements for prevention. Previous studies [20,21] have described the problem of clinically pertinent items not performing ideally under psychometric testing, but still being important for the overall context of the instrument. We dealt with this problem in a similar manner to those studies. If the items contained what we considered to be core knowledge we retained them, but only after (1) examining the psychometric properties with and without the items to ensure that we did not adversely affect the reliability of the questionnaire and (2) describing the reasons for retention and the reasons for considering removing the item.

Baseline levels of osteoporosis knowledge measured using this instrument were low, with the average score of 8.8 being 44% of the possible maximum score. Most other studies of osteoporosis knowledge levels have shown low levels, but these have been in highly selected populations or have not used validated instruments to measure osteoporosis knowledge [6,8,10,12,14,29-32]. One study showed knowledge levels of 78% of the maximum score in 16–59 year old Norwegian women but the instrument was not validated [18]. As the average score using our instrument was low, there is scope for the instrument to be sensitive to change, which is a property that will be valuable if using the questionnaire to assess changes in knowledge with interventions. However, sensitivity to change remains to be tested.

This study has a number of potential limitations. While the sample was randomly selected, selection bias is possible due to the moderate response rate. Indeed proportion of current smokers in the sample is lower than the Tasmanian prevalence of daily smoking in females aged 25–44 in 1998 of 29% [33] and the proportion of women in married or in a de facto relationship is slightly higher than the Tasmanian proportion of 64%. However, the spread of education levels and the unemployment rate approximates the overall population figures for these variables. The range of demographics covered within the sample means that the validation has occurred over a reasonably heterogeneous group and so the questionnaire appears suitable for use in women aged 25–44 across a range of demographics. Although the osteoporosis knowledge instrument is based on the OPSMC content and the osteoporosis information leaflet, it covers a broad spectrum of osteoporosis knowledge and may be suitable for use to assess the impact of other osteoporosis educational interventions or to assess osteoporosis knowledge levels in young female Caucasian populations. The OKAT would require validation in other populations and some modifications might be necessary to reflect regional variations in osteoporosis demographics.

Conclusions

The OKAT for measuring osteoporosis knowledge has good psychometric properties in Australian 25–44 year old females. While it should be applicable to other Caucasian populations, this will require confirmation by further research.

Competing interests

None declared.

List of Abbreviations

OKAT: Osteoporosis Knowledge Assessment Tool

OPSMC: Osteoporosis Prevention and Self-management Course

Authors' Contributions

TW performed data analysis and drafted the paper. SF participated in the design of the study and implemented the OPSMC and provided comments on the paper. BO participated in the design of the study and provided input into the paper. GJ participated in the design of the study, coordinated the study and provided advice on data analysis and the writing of the paper.

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1471-2474/4/17/prepub