Validation of a questionnaire to measure sexual health knowledge and understanding (Sexual Health Questionnaire) in Nepalese secondary school: A psychometric process

Abstract

Background:

School-based sex education has the potential to prevent unwanted pregnancy and to promote positive sexual health at the individual, family and community level.

Objectives:

To develop and validate a sexual health questionnaire to measure young peoples’ sexual health knowledge and understanding (SHQ) in Nepalese secondary school.

Materials and Methods:

Secondary school students (n = 259, male = 43.63%, female = 56.37%) and local experts (n = 9, male = 90%, female = 10%) were participated in this study. Evaluation processes were; content validity (>0.89), plausibility check (>95), item-total correlation (>0.3), factor loading (>0.4), principal component analysis (4 factors Kaiser's criterion), Chronbach's alpha (>0.65), face validity and internal consistency using test-retest reliability (P > 0.05).

Results:

The principal component analysis revealed four factors to be extracted; sexual health norms and beliefs, source of sexual health information, sexual health knowledge and understanding, and level of sexual awareness. Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy demonstrated that the patterns of correlations are relatively compact (>0.80). Chronbach's alpha for each factors were above the cut-off point (0.65). Face validity indicated that the questions were clear to the majority of the respondent. Moreover, there were no significant differences (P > 0.05) in the responses to the items at two time points at seven weeks later.

Conclusions:

The finding suggests that SHQ is a valid and reliable instrument to be used in schools to measure sexual health knowledge and understanding. Further analysis such as structured equation modelling (SEM) and confirmatory factor analysis could make the questionnaire more robust and applicable to the wider school population.

INTRODUCTION

Age appropriate sex education offers young people the opportunity to explore emotions, feelings, and personal values, and to gain knowledge about sexually transmitted infections (STIs) and learn about their reproductive biology.[1] It is well-established that school-based sex education has the potential to prevent unwanted pregnancy and to promote positive sexual health at the individual, family, and community or health system level.[2,3] Poor sexual and reproductive health is the major cause of morbidity and mortality among young people. As a population, they are at risk of unwanted pregnancies, STIs and coerced early sexual relationships.[4,5,6] Developing countries represent a large proportion of young people in the world. However, very little is known about their knowledge and understanding of sexual health.[7,8,9,10]

In many countries, secondary schools are the sources of sexual health information for young people. However, there are wide variations and standardizations among schools, in terms of topics covered; teaching method; the role of the teacher; training for teachers; levels of student participation; and the age at which topics are taught. Many schools deliver sex education in a classroom setting, which may not help pupils to examine their attitudes and values or to explore facts and feelings in relation to their everyday life.[11] Some schools implement group discussions, role-play, and quizzes as an informal way of delivering sex education, which may encourage pupils to participate and make the information more relevant to their experiences.[12,13] A good quality evaluation of research on sex education may help schools to identify what level of support they will need from local authorities in the development and delivery of their policies.[14] A gradual and varied approach to sex education with evaluation such as asking for teenagers’ opinions provides information with respect to which strategies should be adopted.[15,16]

In Nepal, pupils are taught basic sex education at higher secondary level (grade 9 and 10) using the textbook health, population, and the environment.[17] The purpose of the sex education is to provide information about sex and sexual health issues and to develop the skills and knowledge to make informed decisions. However, it is frequently reported that teaching sexual health in school is often very poor, which is directly associated with teacher's embarrassment, lack of knowledge, and poor teaching techniques.[13] Nepalese schools evaluate the sex education program in less challenging, factual and biological issues whereas the broader issues such as feelings and relationships are often being overlooked.[18] There is a need of research to address how Nepalese schools can effectively encourage young people to choose healthier sexual lifestyles.[19] To conduct such research satisfactorily, it is essential to use valid questionnaire items. Some questionnaires used elsewhere are found to be very useful and helpful however, their reliability and validity in the Nepalese context remains questionable.[18,20,21,22] Therefore, this study aimed to pilot the questionnaire to evaluate Nepalese sexual health education programs in secondary schools.

Pilot studies are a type of feasibility study, which are “small-scale trials, performed in preparation for the major study”[23] or to pretest a research question.[24,25] Conducting a pilot study provides a warning about where the questionnaires instruments are inappropriate or too complicated for participants. Well-conducted pilot studies can be very useful to others embarking on projects using similar methods or instruments. This is particularly important because pilot studies can be “time-consuming, frustrating, and fraught with unanticipated problems.”[26] Well-designed and well-conducted pilot studies can offer information about the best research process and can occasionally provide insights to likely outcomes. In this study, we used the guidelines suggested by Peat et al. to improve the internal validity of the questionnaire.[27]

METHODS

Questionnaire

Considering social, cultural, and demographic characteristics of Nepal, a draft Sexual Health Questionnaire (SHQ) was developed on the basis of sexual health research conducted elsewhere.[18,22,28,29] Nevertheless, the questionnaire was based on the explicit objectives of the Nepalese school-based sex education program. A total of 52 questionnaire items were developed and distributed to sexual health experts for content validity. Remaining questionnaire items were further tested to pupils for plausibility, validity, and reliability. These later items were phrased using a typical 5-point Likert scale ranging from strongly disagree (1) to strongly agree (5). The 5-point Likert scale has a middle neutral point, which provides the respondent more room to think about which side they would like to go for the answers.[30,31] For the statistical significance, 5-point Likert scale tends to show the answer as a normal distribution toward a larger middle portion.

Polarities of the questions were checked, and items were phrased so as to avoid vagueness, biases, double-barreled questions, double negatives, and protests. The questionnaire determined information about pupils’ knowledge and understanding of sexual health. The methods used to validate the SHQ were translational validity (content validity and face validity), construct validity (principal component analysis [PCA]) and reliability (internal consistency and test-retest). The detail statistical model is given below [Figure 1].

Figure 1.

A statistical model to validate the sexual health questionnaire

Field work

During the period between November 2010 and February 2011, a pilot study was carried out among 268 respondents in Nepal using a self-administered questionnaire. The study included 259 pupils from three secondary schools in Makwanpur district and nine purposively chosen sexual health experts across Nepal. Details about sociobackground variables such as gender, age, educational level, ethnicity, and family's educational status were obtained from them. Respondents were told that their anonymity would be assured in the study. The main researcher input all valid questionnaire data using the SPSS V 19.0 [IBM Corp.: Armonk, NY]. The data were cleaned after double entry and discrepancies were resolved with manual checking. Ethical approval was obtained from District Education Office, Makwanpur prior to conducting this study.

All participants signed a written informed consent before participation, and each of them was free to discontinue participation at any time. To maximize the response rate, all questionnaires were delivered and collected face-to-face by the main researcher. The respondents filled in the questionnaires by themselves and any unclear questions were explained on the site without inducement.

Translational validity

Content validity

To estimate the validity of the questionnaire items, the content validity index (CVI) was used.[32] It reflects a complete range of attributes under study and it is usually undertaken by seven or more experts.[23,33] In the first instance, 52 questionnaire items were distributed among nine sexual health experts to ascertain whether the content of the questionnaire was appropriate and relevant to the study purpose. Each expert rated the relevance of each item on the SHQ using a 4-point Likert scale (1 = not relevant, 2 = somewhat relevant, 3 = relevant, 4 = very relevant). The 4-point Likert scale has a tendency to over-scale and exaggerates the answers. There is no middle point for the experts to either choose or think about what they should choose.[30,31,34] It is a kind of forced side ordinal measure, which is intentionally used to allow the experts choose a side to some degree. The level of endorsement was set to >0.89 (8/9 = 0.89) to establish the CVI. Any item not meeting the above requirement was dropped from further considerations.

Face validity

Face validity specifies the questionnaire appears to be appropriate to the study purpose and fit to the content area. It is the easiest validation process to carry out. It evaluates the appearance of the questionnaire in terms of feasibility, readability, consistency of style and formatting, and the clarity of the language used.[33,35] Face validity thus is a form of usability rather than reliability. An evaluation form was developed to determine the face validity of the questionnaire in terms of the clarity of the wording, the likelihood the target audience would be able to answer questions, and the layout and style. A total of 24 pupils from grade 10, who have already been to sex education session in secondary school in Hetauda, were randomly selected to complete the face validity form on a 4-point Likert scale; strongly disagree = 1, disagree = 2, agree = 3, and strongly agree = 4.

Plausibility check for evaluations

A total of 210 pupils from grade nine were given 38 questionnaire items mainly related to sexual health knowledge and understanding. These items were distributed to the participants in a classroom setting. Participants’ details were obtained that included their name, class, age, gender, school name and section. This was performed particularly to check any missing data and for the follow-up purpose. The questionnaires were collected after an hour and were considered for further analysis. Inconsistencies in the data (e.g. missing, wrong answer) within the questionnaire are unreasonable and impossible entries. It was avoided using plausibility check to reduce erroneously. The item acceptance level was ≥95% for all items.

Overall reliability

The overall reliability of the remaining questionnaire, obtained from 210 pupils, was examined in a 5-point Likert scale: 1 = strongly disagree, 2 = disagree, 3 = neither, 4 = agree and 5 = strongly agree, using the item-total correlations and Cronbach's alpha. This indicated how well the items fitted together conceptually.[33,36] All items with item-total correlations <0.30 and items greater than overall alpha 0.913 were excluded.

Construct validity

Exploratory factor analysis

An exploratory factor analysis (EFA) is used to provide score validity evidence through the examination of the internal structure of study measures. Such evidence can increase the utility of the instruments in the evaluation and ultimately increase the creditability and efficacy of assessment. An EFA using the principal component analysis (PCA) produces linear combinations of the original variable to generate the axes. In this study, respondents’ actual scores on the given questionnaire items were optimally weighted and then summed to compute their scores on a given component. The general form for the formula on the first component in a PCA is:

where C₁ is the respondents’ score on principal component one (the first component extracted), b_1p is the regression coefficient for observed variablepused to create principal component one and X_p is the respondents’ score on observed variable p. PCA describes the degree to which the items in the instrument relate to the relevant theoretical construct.[33,37] The sampling population for the PCA was 210 pupils, the same population as that used for plausibility check and were randomly selected from three secondary schools in Makwanpur. The analysis (rotation: Varimax with Kaiser normalization) was conducted with data from all participants on the acceptable items, at a minimum factor load r = 0.40.[38] To perform the factor analysis, variables should correlate fairly well, and this was checked using Pearson's correlation coefficients and the significance of the coefficients. The proportion of common variance in the variable is communalities, and this was checked (>0.5) for the given samples.[39] The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy and Bartlett's test of Sphericity were used to check whether the data were suitable for factor analysis. KMO > 0.70 have been characterized as middling and >0.90 as marvelous. Bartlett's test of Sphericity tests the null hypothesis that there is no relationship between the questionnaire items.[40]

Reliability

Internal consistency reliability

Internal consistency reliability looks at the inter-item correlations within an instrument. It indicates how well the items fit together theoretically.[36] This study used Cronbach's alpha correlation coefficient, which is commonly known as a measure of the internal consistency of a psychometric test score.[36,41] The following equation was used to generate data set from Cronbach's alpha:

Where K is the number of components, σ²x is the variance of the observed total test scores and σ²y_i is the variance of component i for the current sample of respondents. For both the total score and the resulting subscales, internal consistency was assessed with the Cronbach's alpha coefficient.

Test-retest reliability

This type of reliability is estimated by administering the same instrument to the same sample of respondents on two different occasions. The assumption is that there is no substantial change in the construct under study between the given 2 sampling time points.[33] The duration of time between the two tests is critical. However, a high correlation between the scores at the 2 times point indicates that the instrument is stable over time.[42] It is identified that the shorter the interval the higher the correlation, the longer the interval the lower the correlation.[35] Very long test intervals could affect the outcomes as a result of changes in participants’ attitude or their environment.[33,43] There is no specific indication about the best time interval to allow between the test and the retest. This is the researcher, who needs to consider factors such as the effects of time on knowledge to make an appropriate decision about the time interval between the tests.[44]

The questionnaire was administered to 25 pupils from grade 10, aged 14 to 18 years randomly selected from one of those three secondary schools in Hetauda municipality. They completed the questionnaire on two different occasions; at baseline and 7 weeks later. The data were ordinal with 4-point Likert scale (strongly disagree to strongly agree) and the scales were not continuous. Thus, a nonparametric statistical test was carried out using the Wilcoxon nonparametric rank correlations.[45,46]

RESULTS

A total of 273 respondents were invited to participate in the study; 268 consented and returned (97.6%) the questionnaire [Table 1]. Participants were secondary school students (n = 259) from Hetauda, Makwanpur in Central Nepal, and sexual health experts (n = 9) from different areas in Nepal. Pupils from grade 9 (n = 210) responded to the questions for factor analysis and reliability to calculate internal consistent reliability (Cronbach's alpha). Out of 49 pupils contacted from grade 10, 24 were given questionnaires related to face validity. The other 25 pupils were given the questionnaires for test-retest reliability analysis at two stages: One at the 1^st week and the other 7 weeks later. The total mean age of pupils (grade 9 and 10) was 15.38 (standard deviation [SD] =1.00) years; for grade 9 it was 15.09 (SD = 0.9) years and for grade 10 it was 15.6 (SD = 1.07) years. The overall age ranged from 14 to 18 years (SD = 1.0). Altogether 9 sexual health experts participated in this study, of which males comprised 90% (8) [Table 1].

Table 1.

Socio-demographic information of the respondents

Item acceptance criterion

Nine sexual health experts reviewed 52 questionnaire items. Eight questions could not meet the level of endorsement ≥0.89 (8/9 = 0.89) required to establish a CVI. These questions were: Do you live at home? Who do you live with? How many other family members live in the same household as you? What is your father's occupational status? What is your mother's occupational status? Boy and girl should have sex before they are married? You should be in love before having sex? and I would never consider my girlfriend having an abortion. These questions were excluded from further analysis [Table 2].

Table 2.

List of 28 nonvalidated questionnaires from a total of 52 questionnaires items indicating with symbols^(a–f)

Furthermore, a plausibility check for evaluation was performed from 38 questionnaire items (excluding 6 sociobackground variables) collected from 210 grade 9 pupils. The result showed that 3 items were not plausible for reliability analysis: Parents as a source of sexual health information, older relatives provide sexual health information, and religious leaders provide sexual health information. Thus, these items were also removed from further analysis.

Item-total correlation

The remaining 35 questions were further examined to assess their overall reliability using the item-total correlations and Cronbach's alpha. The initial result showed that 6 questionnaire items were not reliable for further analysis. These were: A person can suffer from an STI but may have no symptom, if boys and girls agree there is nothing wrong to have sex, a majority of my friends have already had sex, I get sexual health advice from boyfriend/girlfriend, menstruation is controlled by the nervous system, and father's chromosome determines baby's sex (item-total correlation <0.30). Hence, all these 6 questionnaire items were removed, and the analysis was rerun before considering them for further tests.

Correlation co-efficient and significance

To perform the EFA, variables should correlate fairly well, but not perfectly. The remaining 29 questionnaire items were looked for the correlation coefficient. It was observed that one item: Sexual abuse harms adolescents’ future had value 0.93, which is >0.90 (the acceptance level). Similarly, four other items: TV/films as a source of sexual health information, condom can not prevent HIV infection, young people should know that others have right to say no and it is women's responsibility to use contraception had a significant value >0.05 (acceptance level ≤0.05). So these five items were also removed from the further analysis [Table 2].

Exploratory factor analysis

A PCA was performed (eigenvalues >1) considering the remaining 24 questions. In the first instance, 3 questions: Early marriage leads to early pregnancy, unsafe abortion among girls leads to maternal death and school teachers as a source of sexual health information showed a minimum value of communalities (<0.5). Hence, these questions were removed one-by-one, considering smaller values first and bigger values at the end. The total variance suggested six factors to be extracted, which accounted for cumulative 74.48% rotation sums of squared loadings. However, the scree plot showed between two and six factors would be more reasonable to develop. These two tests suggested retaining a different number of factors. So as an alternative, the analysis was run with four factors to be extracted with varimax rotation. Once again, four questions: Parents do not understand young people's feelings, books/magazines as a source of sexual health information, sexual intercourse invites sex-related problems and condom is used correctly and consistently for safer sex purpose showed communalities <0.5 [Table 2]. So, these questionnaire items were also dropped one-by-one, except one question, condom is used correctly and consistently for safer sex purpose since this was an indicator question in the main research questionnaire item.

The KMO measure of sampling adequacy (0.841) and Bartlett's test of Sphericity (P < 0.001) were satisfactory to run the factor analysis for remained 18 questions. The rotation sum of squared loadings was 68.25% with four factors, and the scree plot [Figure 2] indicated an excellent point of inflexion on the curve (>1). Residuals, which are the differences between the observed correlations (initial) and correlations based on the model (later), should be <0.5. The final output showed that there were only 43 (28%) residuals >0.5. More than 50% of residuals >0.5 probably could give grounds for concern. A rotated component matrix (factor loading ≥0.4) showed the initial 18 questionnaires composed of four factors [Table 3].

Figure 2.

Scree plot suggesting four factors to be extracted

Table 3.

Validated 18 questions on sexual health questionnaire; with factor loadings (>0.5), mean (SD), item-total correlation (>0.3), Cronbach's alpha (>0.65), K–S test for normality check for test re-test data (P<0.05) and test re-test reliability (asymptotic significant)

All these 18 questionnaire items were significantly loaded on four factors, which uncovered the latent structure of a set of variables. It reduced the attribute space from a larger number of variables to a smaller number of factors using the linear combinations of variables to explain sets of observations. Factor-1 is “sexual health norms and beliefs,” which is the strongest factor (Cronbach's alpha = 0.93). There were four items loaded on this factor: I believe in abstinence which means to avoid sex, female should be considered as untouchable during the menstruation, a girl losses her dignity if she has sex before marriage and girl can suggest boy to use condom if he is suffered from STI.

Six items loaded on factor-2, which is “sources of sexual health information” (Cronbach's alpha = 0.84). These included: Friends/peers, relatives of similar age such as brothers/sisters/cousins, health professionals such as doctor/nurse, visits from outside such as health facilitators, youth workers and chemist/pharmacy.

Four items related to “sexual health knowledge and understanding” loaded on factor-3 (Cronbach's alpha = 0.83). These were: Unsafe sexual behavior is adolescent's current problem, STIs might cause infertility, loneliness occurs as a result of emotional change and it is better to have only one sex partner for sexual relationship. Factor-4 is the “sexual health awareness level” (Cronbach's alpha = 0.78) and four items loaded on this factor. These were: Parental counseling is important for young people's sexual health development, fertilization is a natural process that takes place in fallopian tube, adolescent experience growth of height during physical change and condom is used correctly and consistently for safer sex purpose.

Reliability

The overall alpha for the questionnaire item was 0.90, and none of the questions would substantially affect reliability if they were deleted [Table 3]. The overall corrected item-total correlation was also >0.3 for all the questions. The reliability of each factor was computed along with corrected item-total correlations, and these were all above 0.3. The overall alpha for each factor were 0.93 (factor 1), 0.84 (factor 2), 0.83 (factor 3) and 0.78 (factor 4). None of the question in the respective factors would increase the reliability if they were deleted. This suggested that all the questionnaire items in each factor were positively contributing to the overall reliability. The overall alpha for each factor was also >0.65 (the acceptance level), and so indicated a good reliability. The test-retest reliability of the questionnaire was undertaken by administrating the questions to 25 pupils, 20 males (80%) and 5 females (20%). The mean age was 15.60 years (SD = 1.08), with age ranges from 14 to 18 years. They all completed the questionnaires on two different occasions; at baseline and 7 weeks later.

Test re-test reliability

The Kolmogorov–Smirnov test of normality (for test and re-test data) showed all significant values (P < 0.05). This suggested that the data were not normally distributed; thus the Wilcoxon nonparametric test was computed. The result showed all the P > 0.05 [Table 3]. This informed that there were no significant differences in the responses to the questionnaire items between 2 time points.

Face validity

A total of 24 pupils from grade 10 were reached for face validity of which 22 were females and 2 were males. The mean age was 15.75 years, ranging from 14 to 18 years (SD = 1.07). The 16-year-old had the highest level of participation 10 (41.7%), followed by the 15-year-old 8 (33.3%). Most of the respondents agreed on clarity of the wording of the questionnaire items (98.85%). Only five responses showed disagreement, which accounted for 1.15% of the total responses. 90.28% of the respondents agreed that the target audience would be able to answer the question. Similarly, 91.67% respondents replied that questionnaire's layout and style would be acceptable to the intended target audience.

DISCUSSION

Construction and validation of the SHQ have led to a number of problems with the extension, amendment, and production. This study was able to develop a reliable and valid SHQ to measure the sexual health knowledge and understanding of young people in secondary school. However, this result must be interpreted with considerations as it is not free from the limitations. First, the participants were selected from only one area, which may not reflect the diversification of the samples for the proposed study. Second, the data were collected through self-report measures, which raise questions about the truthful response for sensitive issues such as sexual attitude and behavior. Moreover, participants have completed the survey in the classroom setting where other participants were completing the survey. The presence of other fellows in the room and participants’ perceptions of their level of privacy may have further compromised the accuracy of the self-reported data.[47,48,49]

The development of the questionnaire is a complex analytic process, and it is very important to look at the accuracy of the measurement while undertaking such procedure. In this study, the draft version of the piloting questionnaire was a combination of various heterogeneous questions used in other countries and context. Hence, it was necessary to identify the underlying subscales with satisfactory internal consistency using factor analysis, which is a rigorous and appropriate method. In the first instance, the data were checked for content validity to assess whether the questionnaires were relevant to the notion of the sexual health knowledge and understanding described in the study.[33,46] Plausible items were omitted using plausibility checks for evaluations to reduce errors in the data set as previously done in a German study.[50] The overall reliability improved the fitness of the questionnaire items in each scale. Another study has shown that the reliability of self-reported sexual health knowledge and practice varies with a variety of socioeconomic factors.[51] Some other reliability studies have also found that the longer recall intervals result in either underreporting or inaccurate reporting since a more elaborate reconstruction of events rather than a simple scanning of the more recent event is required.[52,53] In this study, factor analysis was conducted to measure the internal consistency forming the scale; however, five items did not load. This reflects the relative heterogeneity of the different items that are integrated in the whole inventory.[46]

In the PCA, a communalities check was carried out to look at the total variance among the variables. This did not invalidate the total score (68.25%) and so increased the utility of the subscales in different socioeconomic conditions. The result was four factors derived: Sexual health norms and beliefs, sources of sexual health information, sexual health knowledge and understanding and sexual health awareness level. This result is similar to the previous Australian study, which also identified four factors; however, it was related to the spirituality and relationships.[54] The reliability of each factor was satisfactory, and none of the items in the respective factors would increase the reliability if items were deleted. This suggested that all the items in each factor positively contribute to the overall reliability.

Face validity was satisfactory among the majority of the respondents and was useful to provide important information about sexual health knowledge and understating. This, therefore, indicates that the questionnaire items are both acceptable and feasible for use in the school setting.[55] The test-retest reliability was satisfactory to good, in spite of the heterogeneity and a reasonable test-retest interval. This notified that the test-retest reliability is consistent with the concept of character traits, and there were no significant differences in the responses to the items between 2 times points. However, this would need to be confirmed with a repeated measures analysis over a longer period of time.

Sexual health knowledge and understanding has been recognized as an important aspect of population health in Nepalese schools. Nevertheless, health educators and promoters find it difficult to measure as it is highly prejudiced and often embarrassing. This article has attempted to report the psychometric validation of the SHQ to measure sexual health knowledge and understanding, according to specific definition and context. This study could be further explored by undertaking convergent and discriminant validity to examine the similarities and differences of the SHQ with other sexual health knowledge and understating questionnaires. Nevertheless, the validation of this questionnaire provides the opportunity for Nepalese schools to explore and meet the challenges posed by young people, and to enhance their knowledge and understanding on sexual health.

CONCLUSION

Sexual health knowledge of young people is considered as an important aspect of the healthy population in Nepalese society. This research article describes the psychometric validation of the SHQ. Questionnaire validation is an expensive and time-consuming process. Health educators, policy makers, and researchers demand for improved and relevant tool to evaluate the sex education program, and this questionnaire can help them to conduct such research. There are also chances that the content of the questionnaire may discriminate pupils, especially those who know about sexual health and those who don’t know about it. Hence, it is strongly advised to consider the social and cultural circumstances before it can be used either in Nepal or in other countries. This validated final questionnaire has already been used by the main author in a randomized intervention trial in Nepalese schools to measure young peoples’ sexual health knowledge and understanding.

Financial support and sponsorship

This study was supported by a grant to the first author from Aberystwyth International Postgraduate Research Studentship (AIPRS) and Arthur Trott Fund in Wales.

Conflicts of interest

There are no conflicts of interest.