Skip to main content
NCBI home page
Wiley Open Access Collection logoLink to Wiley Open Access Collection
. 2024 Jul 14;36(3):264–274. doi: 10.1111/2047-3095.12482

Role of the health literacy assessment in healthcare: Content validation of “Health Literacy Behaviour” nursing outcome

Alba Correro‐Bermejo 1, Pilar Bas‐Sarmiento 2,3,✉, José Manuel Romero‐Sánchez 2,3, Olga Paloma‐Castro 2,3, Miriam Poza‐Méndez 2,4, Martina Fernández‐Gutiérrez 2,3
PMCID: PMC12282502  PMID: 39004610

Abstract

Purpose

To validate the content of the “Health Literacy Behaviour” nursing outcome (NO).

Methods

A content validation study was conducted during 2022. Each indicator was included in the NO, and its response levels were operationally defined. The initial version of each indicator and its response levels were refined and validated through the Delphi method. A panel of health literacy (HL) and nursing taxonomies experts evaluated the content through two content validity indicators. The content validity of the NO and each index was determined by calculating the content validity index (CVI). A qualitative analysis of the recommendations provided by the experts was carried out to improve the understanding of the indicators and their levels.

Findings

A total of 108 experts participated in this study. Mostly females with more than 10 years of professional experience. The results demonstrated a high CVI of the indicators and the NO “Health Literacy Behaviour.” All indicators achieved excellent (CVI ≥ 0.80) relevance and clarity. The CVI universal average method (CVI‐p) of the NO achieved an excellent result of 0.90.

Conclusions

The indicators included in the NO “Health Literacy Behaviour” have content validity.

Implications for nursing practice

These findings provide evidence‐based indicators to measure the patient's actions to obtain, process, and understand information about health and disease, interact with the health system, and make informed health decisions. The validation of this NO would identify populations with low HL, allowing the health of this community to be promoted. Health literacy should be a priority objective of health management and policies.

Keywords: health care, health literacy, Nursing Outcomes Classification, outcome assessment, Standardized Nursing Terminology, validation study

INTRODUCTION

Health literacy (HL) has been defined as “the knowledge, motivation, and competencies to access, understand, appraise, and apply information, to make judgments and decisions in terms of healthcare, disease prevention, and healthy behaviours, to maintain and promote quality of life throughout the life course” (Sørensen et al., 2012). Therefore, HL is a social determinant of health, which depends on individual abilities and skills, the cultural context, and the health system, and it is a step before patient empowerment (Nutbeam & Lloyd, 2021; Sørensen et al., 2021).

Numerous studies have shown how low HL levels are associated with low participation in health promotion and disease detection activities and inadequate management of chronic diseases (Cabellos‐Garcia et al., 2018; Figueroa Saavedra et al., 2020), a misunderstanding of complex treatments and errors in taking medication (Cabellos‐Garcia et al., 2018; McDonald & Shenkman, 2018), a higher rate of hospitalization, re‐hospitalization, and misuse of emergency services (McDonald & Shenkman, 2018), scarce use of preventive services and, ultimately, an increase in morbidity and mortality (Bas‐Sarmiento et al., 2015; Fernández‐Gutiérrez et al., 2018; Figueroa Saavedra et al., 2020). Similarly, low HL represents an obstacle for patients with chronic diseases to learn details of their disease and treatment, making it difficult to control their signs and symptoms (Bas‐Sarmiento et al., 2015). Therefore, it should be noted that HL is a crucial factor to consider in patient care because it affects the ability to make decisions and supports the self‐management of health (Poza‐Méndez et al., 2023).

Numerous studies show low levels of HL in the world (Duong et al., 2017; Prince et al., 2018), which have become even more evident during the pandemic. The lack of HL in the population in the face of COVID‐19 was considered a public health problem worldwide, where a majority of the adult population confirmed having problems and not having enough skills to care for their health and that of others (Instituto de Salud Carlos III, 2022; Zarocostas, 2020).

BACKGROUND

One of the roles of nursing is to create a health‐literate society that takes an active role in research, education, and health promotion (Cobo Sánchez, 2019; Correro‐Bermejo et al., 2023). Strategies to educate the population should be incorporated into individual patient care plans to become part of the routine clinical practice of nurses (Cobo Sánchez, 2019).

In 2008, the intervention 5515 Health Literacy Enhancement was incorporated into the fifth edition of the Manual Nursing Interventions Classification (NIC), which proposes, among other activities, to determine the HL status of the patient through an evaluation (Bulechek et al., 2013). Years later, in 2016, the North American Nursing Diagnosis Association (NANDA) approved the diagnosis “00262 Readiness for Enhanced Health Literacy” (Herdman & Kamitsuru, 2018). Subsequently, in 2018, the nursing outcome (NO) Health Literacy Behavior (2015) (Moorhead et al., 2018) was incorporated into the Nursing Outcomes Classification (NOC). This NO is associated with 21 indicators that allow the evaluation and promotion of the patient's autonomy to make decisions about their health, involving them in activities that promote health and disease prevention, training them in self‐care, and achieving therapeutic adherence (Correro‐Bermejo et al., 2023). Its inclusion in the taxonomy and nursing care plans allows nurses to provide more individualized and efficient care. Nurses must have tools that facilitate the systematic, objective, and reliable evaluation of the HL level of the population. In the same way, the need to determine the importance of the scores of the indicators of an NO has been detected (Paloma‐Castro et al., 2017). However, publications on NO remain scarce (Dias Emidio et al., 2020), especially on HL. Consequently, it is required to provide clinical evidence for effective nursing practice validation of the NO Health Literacy Behaviour and its indicators.

Content validity is defined as “logical judgment about the correspondence between the trait or the learning characteristic of the evaluated person and what is included in a test.” It aims to determine if the proposed items or questions reflect the content domain (knowledge, abilities, or skills) to be measured (Urrutia Egaña et al., 2014). In order to carry out more precise content validation, some authors agree that it is necessary to have an intentional sample of experts, know and analyze the characteristics of said sample, and be clear about the dimensions that are to be evaluated in the said instrument (Lynn, 1986; Urrutia Egaña et al., 2014). Additionally, studies suggest that the most appropriate way to collect data to support content validity is by conducting qualitative research (MacDermid, 2021).

This study aims to validate the content of the NO Health Literacy Behaviour (2015) for use in individualized care plans.

METHODS

Design

An exploratory and cross‐sectional study was carried out in 2022 to validate the content of the NO “Health Literacy Behaviour (2015)” (Moorhead et al., 2018), belonging to the NOC taxonomy.

This study is based on a recently published protocol (Correro‐Bermejo et al., 2023).

Sample/participants

A non‐probabilistic convenience sampling was conducted among HL and nursing taxonomy experts to assess the face and content validity of the indicators (Polit & Tatano, 2008). The sampling strategy aimed to ensure that participants met the following inclusion criteria: (a) minimum clinical experience of at least 2 years; (b) a minimum of 5 years of experience in development and use in the area of nursing taxonomy; (c) to participate or have participated in research activities and have scientific academic productions in areas related to taxonomies, instrument validation, or HL; (d) academic experience in the field of at least 2 years. To participate as an expert, the expert must meet criterion (c), as well as one more (a, b, or d) (Seganfredo & Almeida, 2010).

There is still no consensus in the literature on the number of experts recommended to carry out the content validation of an NO (Oh & Moorhead, 2019). The optimal size for a Delphi group is estimated between a minimum of 6 and a maximum of 30 participants (Bellido‐Vallejo et al., 2016; Landeta Rodríguez, 1999). In the present work, expert panels of more than 20 experts were formed.

The experts were recruited by email through professional nursing colleges, nursing associations and societies, and Spanish universities. They completed an anonymous survey through Google Forms.

Measurement and variables

The following grouped variables were considered: (a) sociodemographic variables—sex, age, position in the institution where they work, level of study, academic degree, whether they have a doctoral degree, and years of professional experience; (b) indicators of NO Health Literacy Behaviour (2015) (Moorhead et al., 2018); (c) an open‐ended question—“Write a proposal for new indicators or changes/suggestions to indicators already proposed.” A 5‐point Likert‐type measurement scale was used to evaluate each indicator. Responses ranged from 1 (“no knowledge”) to 5 (“extensive knowledge”).

Indicators of NO Health Literacy Behaviour (2015) and each level of their 5‐point Likert‐type measuring scale were operationally defined.

Data collection

Data collection was developed from January to December 2022. A web‐based platform to facilitate the participation of experts was deemed appropriate as it is cost‐effective and efficient (Gill et al., 2013). The online survey consisted of three sections: The first section contained a description of the study, the informed consent, and the survey instructions, including a clear definition for each choice category of the ranking scale; the second section collected sociodemographic data; and the third section contained the indicators (operationally defined) included in the NOC outcome to assess the relevance and clarity of each indicator. Finally, within each indicator was a free‐text section for experts to provide feedback, opinions, and comments.

Three phases of consensus were conducted using the Delphi technique (Bellido‐Vallejo & Pancorbo‐Hidalgo, 2017; Bellido‐Vallejo et al., 2016). In the first and second rounds of consensus, the relevance of each indicator and its definition were measured. An indicator will be relevant if “it is essential or important, that is if it should be included.”

Following Escobar‐Pérez and Cuervo‐Martínez (2008), the relevance and clarity of each indicator and its definition were measured in the third round of consensus. An indicator will be clear if it is “easily understood, that is, its syntactic and semantics are adequate.” Both clarity and relevance were measured with a 4‐point Likert scale, where 1 means “not clear/irrelevant”; 2 = unclear/low relevant; 3 = “fairly clear/fairly relevant”; 4 = totally clear/totally relevant.

Data analysis

A univariate descriptive analysis was performed to determine the sample distribution for each variable studied. The characterization variables were summarized using descriptive statistics, expressing qualitative variables in frequency and percentages and quantitative variables in mean and standard deviation (SD).

For the content validity of the indicators included in the NOC outcome “Health Literacy Behaviour (2015),” the approach advocated by Lynn (1986) was used. The content validity index (CVI) was calculated (Almanasreh et al., 2019) both at the individual indicator level (I‐CVI) and the average of the content validation index of all the indicators.

I‐CVI was computed as the number of experts rating 3 or 4 “relevance” for each indicator divided by the total number of experts. The CVI of all the indicators was calculated in two methods; one was the CVI universal agreement method (CVI‐au), and the second was the CVI universal average method (CVI‐p). CVI‐au was calculated by adding all indicators that achieve a relevance rating of 3 or 4 by the experts divided by the total number of indicators, whereas CVI‐p was calculated by taking the sum of the I‐CVIs divided by the total number of indicators (Almanasreh et al., 2019). The clarity index was computed as the number of experts rating 3 or 4 “clarity” for each indicator divided by the total number of experts.

Taking into account the size of the expert panel and according to the bibliography consulted, the relevance/clarity of the indicators was considered good if the CVI were more significant than or equal to 0.80 (Almanasreh et al., 2019; Lynn, 1986). Those indicators that did not reach these scores were reviewed and reformulated based on the feedback collected until a final version was agreed upon.

Qualitative analysis used thematic analysis following the methods of Braun and Clarke (2006). The themes identified were grouped by indicator. Each recommendation was coded according to the expert number. Two nurses and a psychologist (all PhD) were responsible for the manual analysis using office software. Each made a detailed division of the content of the interviews to group them according to the indicators under study. For the semantic analysis of the content of the recommendations given by experts, the software Sphinx iQ3 v.8.2.2 was used.

A data matrix was created, and data were processed statistically using SPSS, version 22 (IBM).

Ethical consideration

This research was carried out following the principles established in the current revised version of the Declaration of Helsinki and following the applicable legal requirements for biomedical research, Law 14/2007 on Biomedical Research. The Cadiz Ethics Committee approved this research with register number 56.23 (code: PEIBA 0686‐N‐23). Likewise, informed consent was included in the first section of the online survey. In legal terms, Spanish Organic Law 5/2018 on Personal Data Protection guaranteed the anonymity of the participants and the database, with no personal data that can be used to identify them.

RESULTS

Panel of experts

Table 1 shows the sociodemographic characteristics of the sample for each consensus round.

Table 1.

Sample sociodemographic profile.

Panel of experts round 1 (51 n) Panel of experts round 2 (27 n) Panel of experts round 3 (30 n)
n % n % n %
Sex
Female 41 80.4 20 74.1 23 76.7
Male 10 19.6 7 25.9 7 23.3
Educational level
Bachelor's degree/degree 13 25.5 5 18.5 11 36.7
Specialty 2 3.9 – – – –
University expert or unofficial Master's degree 7 13.7 2 7.4 3 10
Official Master's degree 10 19.6 6 22.2 3 10
Doctorate 19 37.3 14 51.9 13 43.3
Degree
Nursing 43 84.3 23 85.2 21 70
Anthropology – – – – 1 3.3
Nursing and occupational therapy 1 2 1 3.7 – –
Nursing and psychology 1 2 1 3.7 – –
Nursing and anthropology 5 9.7 1 3.7 4 13.3
Psychology 1 2 1 3.7 1 3.3
Nursing and physiotherapy – – – – 1 3.3
Nursing and philosophy – – – – 1 3.3
Nursing and linguistics
Job
Specialized care 14 27.5 7 25.9 11 36.7
Primary care 12 23.5 2 7.4 4 13.3
Researcher professor 19 37.2 13 48.2 10 33.3
Others 6 11.8 5 18.5 5 16.7
Professional experience
<5 years 22 43.1 9 33.3 6 20
5–10 years 4 7.8 5 18.5 3 10
10–20 years 11 21.6 4 14.8 3 10
>20 years 14 27.5 9 33.3 18 60
Open in a new tab

In the first round, the sample consisted of 51 experts, with a mean age of 37 years (SD = 12.065). The total number of women was 80.4%, mainly with undergraduate and graduate studies. Of the sample, 37.3% had a teaching/research profile, and 51% worked in the care setting. A large majority (82.4%) had experience in the last 5 years of the development/use of nursing taxonomies.

In the second consensus round, a sample of 27 experts with a mean age of 39 (SD = 12.32) was obtained. Overall, 74.1% were women, 51.9% had doctoral studies, and most were nurses. In this case, 48.2% had a teaching and research profile. Of the sample, 33.3% had professional experience of more than 20 years.

Finally, in the third round of consensus, the sample consisted of 30 experts with a mean age of 46 (SD = 12.258). Overall, 76.7% were women, the majority being nurses, and 43.3% with doctoral studies. It should be noted that 33.3% had a teaching and research profile, and 60% had professional experience of more than 20 years.

Content validity result

Initially, each indicator and its response levels were defined.

Table 2 shows the I‐CVI for each indicator. The expert panel evaluated the 21 indicators that showed an acceptable I‐CVI regarding relevance and clarity. I‐CVI values calculated for the 21 indicators ranged from 0.74 to 0.96.

Table 2.

Agreement rate of expert penalties and content validity index.

Round 1 Round 2 Round 3
Relevance Relevance Clarity Relevance
Indicators * CVI Mean SD Expert agreement (n = 51) * CVI Mean SD Expert agreement (n = 27) * CVI Mean SD Expert agreement (n = 30) * CVI Mean SD Expert agreement (n = 30)
201,501 identify personal health needs 0.92 3.29 0.610 47 0.96 3.44 0.577 26 0.86 3.23 0.774 26 0.96 3.50 0.682 29
201,502 obtain reputable information relevant to health 0.90 3.24 0.619 46 0.92 3.48 0.643 25 0.86 3.23 0.774 26 0.93 3.47 0.730 28
201,503 verbalize understanding of written information relevant to health 0.92 3.27 0.666 47 0.92 3.48 0.643 25 0.80 3.13 0.860 24 0.90 3.23 0.728 27
201,504 verbalize understanding of verbal information relevant to health 0.92 3.37 0.613 47 0.96 3.56 0.641 26 0.83 3.30 0.837 25 0.90 3.37 0.765 27
201,505 verbalize understanding of visual information relevant to health 0.82 3.16 0.809 42 0.96 3.67 0.555 26 0.80 3.23 0.898 24 0.86 3.17 0.791 26
201,506 verbalize understanding of recommended medication 0.84 3.225 0.717 43 0.88 3.52 0.7 24 0.93 3.43 0.817 28 0.93 3.53 0.730 28
201,507 verbalize understanding of recommended treatment 0.90 3.27 0.635 46 0.92 3.44 0.641 25 0.96 3.37 0.669 29 0.96 3.63 0.669 29
201,508 evaluate information relevant to personal health 0.88 3.24 0.710 45 0.81 3.11 0.892 22 0.90 3.33 0.758 27 0.90 3.50 0.682 27
201,509 acknowledge patient rights 0.84 3.25 0.717 43 0.85 3.41 0.844 23 0.80 3.23 0.935 24 0.83 3.27 0.907 25
201,510 acknowledge patient responsibilities 0.84 3.25 0.717 43 0.74 ** 3.04 1.091 20 0.80 3.23 0.898 24 0.80 3.30 0.877 24
201,511 complete health‐related documents 0.76 ** 3.08 0.845 39 0.88 3.44 0.698 24 0.96 3.57 0.679 29 0.93 3.53 0.730 28
201,512 identify personal healthcare preferences 0.90 3.29 0.642 46 0.85 3.37 0.839 23 0.93 3.37 0.718 28 0.93 3.53 0.730 28
201,513 identify personal healthcare preferences 0.86 3.25 0.688 44 0.88 3.56 0.698 24 0.90 3.47 0.776 27 0.86 3.43 0.817 26
201,514 identify preventive services 0.80 3 0.748 41 0.85 3.33 0.734 23 0.90 3.37 0.765 27 0.90 3.43 0.774 27
201,515 share questions 0.86 3.25 0.688 44 0.81 3.41 0.797 22 0.80 3.30 0.877 24 0.93 3.50 0.731 28
201,516 share concerns 0.94 3.35 0.594 48 0.92 3.41 0.636 25 0.83 3.27 0.828 25 0.93 3.50 0.731 28
201,517 access healthcare services congruent with needs 0.86 3.16 0.703 44 0.92 3.52 0.753 25 0.93 3.50 0.731 28 0.90 3.37 0.765 27
201,518 use personal support system 0.80 3.16 0.731 41 0.88 3.41 0.694 24 0.83 3.23 0.817 25 0.90 3.33 0.711 27
201,519 apply health information to personal situation 0.92 3.20 0.633 47 0.92 3.48 0.643 25 0.80 3.27 0.907 24 0.90 3.43 0.774 27
201,520 make informed decisions about health care 0.96 3.37 0.631 49 1 3.67 0.480 27 0.86 3.33 0.802 26 0.93 3.67 0.711 28
201,521 share decisions regarding health care 0.90 3.33 0.712 46 0.92 3.41 0.636 25 0.90 3.33 0.758 27 0.83 3.30 0.837 25
Open in a new tab
*

Content validity index (CVI) = (the number of experts rating an item ≥3)/(the total number of experts), CVI ≥ 0.80 is acceptable.

**

CVI < 0.80 is not acceptable.

In the first round, the indicator number 11 obtained a CVI of 0.76; in the second round, the indicator number 10 obtained a CVI of 0.74. In both cases, as the ICV result was close to 0.8 and no expert requested the removal of these indicators, it was decided to reformulate based on the experts’ recommendations.

Finally, the results of the CVIs for relevance and clarity indicated that 100% of the respondents agreed with the content improvements made in the third round.

Table 3 shows the CVI‐au and the CVI‐p. The CVI‐au provides the degree of agreement among the experts to classify the indicators as “fairly and fully appropriate.” The CVI‐p indicates the average content validation index of all indicators.

Table 3.

Agreement rate of expert penalties and content validity index.

Number of indicators with a relevance rating of 3 or 4/Total number of indicators CVI‐au
Round 1 21/21 1
Round 2 21/21 1
Round 3 21/21 1
Average of the CVI‐i of all the indicators CVI‐p
Round 1 18.34/21 0.87
Round 2 18.75/21 0.89
Round 3 18.91/21 0.90
Open in a new tab

Note: Content validity index universal agreement method (CVI‐au); content validity index universal average method (CVI‐p).

Qualitative analysis

In the first round, a qualitative analysis of the recommendations suggested by the experts to improve the definition of the indicators and their levels was carried out. Supplementary Material 1 shows the changes made in rounds 1 and 2.

Following the qualitative analysis of the recommendations made by the experts, the definitions of indicators 4, 6, 8–10, 15, 17, and 18, and the levels of indicators 1, 3–18, and 20 were modified. For example, the definition and levels of indicator 13 were modified based on the following recommendations:

Expert‐1: I would put health care providers instead of service providers (sounds more like material supplies).

Expert‐2: There is an indicator that says analyse, I would put the verb tense the same as the others. Instead of health care providers, I would put health care professionals directly.

Expert‐3: Regarding the indicator “Identifies health service providers,” it is difficult to think, even in the most marginal communities, that they do not know how to identify health professionals, so levels 1 and 2 could be put together in a single level 1, displacing the rest of the levels and adding a 5th level aimed at knowing if they can distinguish between specialists within a health profession.

For indicator 11, the definition and levels were modified due to the low I‐CVI obtained, and taking into consideration the comment of one of the experts: “In 5th level, include that it is capable of raising doubts or clarifying questions (it is indicative that we are not just signing for the sake of signing)”.

A second round was passed to the same experts invited in the first round to verify that the changes introduced were appropriate. Following this second consultation, minor modifications were made to the definitions of indicators 9, 10, and 19. The definition of indicators 9 and 10 included the link to the laws listing patients’ rights and duties. In the case of indicator 19, an example was added.

In addition, at the request of several experts, minor changes were made to the levels of indicators 4–6, 8, 12, 13, 15–19, and 21 to clarify and distinguish the different levels.

DISCUSSION

Tools must be available to facilitate the systematic, objective, and reliable assessment of this NO by professionals, to carry out the much‐needed individualized nursing interventions to achieve the NO, and subsequently allow the results to be evaluated. In this scenario, the NO, Health Literacy Behaviour, can guide nurses and researchers on which aspects of HL management are essential to assess. This content validation process allows the improvement of the nursing language and the execution of the planning and implementation of interventions in a guided way to identify which vulnerable populations need special care. In this way, it ensures the execution of appropriate actions for the individual. Therefore, this study aimed to validate the content of the Health Literacy Behaviour NO for use in individualized care plans.

Several studies identify the lack of a nursing gold standard to help us determine the effectiveness of nursing interventions (Morilla‐Herrera et al., 2011; Silva et al., 2015). In the case of HL, numerous validated tools are used to assess the HL level of the population, such as the Health Literacy Survey European Questionnaire 16 (HLS‐EU‐Q16) (Pelikan et al., 2014). Since 2018, the NOC Health Literacy Behaviour (2015) has opened the possibility for nursing to assess the HL of their patients and incorporate it into care plans to measure the effectiveness of their interventions. However, validation of its content is essential for it to be effective (da Costa Ferreira et al., 2021). In the present work, as in other studies (Oh et al., 2023; Silva et al., 2015), the validation has given rise to a NO with some indicators more valid and reliable.

Several studies note that one of the essential characteristics of content validation is that the indicators are defined to avoid controversy when interpreting them (da Costa‐Ferreira et al., 2021; Mokkink et al., 2016). There is research showing how having operationalized indicators increases agreement among nurses (Chantal Magalhães da Silva et al., 2017). Furthermore, Martín‐Dorta et al. (2021) suggested that operational definitions are essential for studies that address nursing taxonomies, as these provide us with basic information that describes what is to be measured and how it can be done. For this reason, the definitions of the NO Health Literacy Behaviour (2015) indicators are elaborated based on scientific evidence in the present work. Subsequently, modifications are made based on experts’ recommendations, as in the case of Bellido‐Vallejo and Pancorbo‐Hidalgo (2017).

For content validation studies to be reliable, the group of experts must be specialized in the topic to be dealt with, in this case, experts in nursing taxonomies and HL. Following the evidence recommendations (Bellido‐Vallejo et al., 2016), we tended to maximize the sample of experts. In the first round, we obtained a sample of 51 experts; in the second, 27 experts; and in the third, 30 experts.

Although there are various methods to perform content validation, such as the Ferhing method (da Costa‐Ferreira et al., 2021; Oh & Moorhead, 2019) or the CVI (Martín‐Dorta et al., 2021), one of the advantages of using the latter is that it is easy to calculate. In addition, it allows for determining the relevance (Devriendt et al., 2012) and clarity of each of the measured NO indicators, thus determining whether the indicator needs to be eliminated or reformulated. However, as indicated by MacDermid (2021), it should be noted that the quantitative information obtained in content validation must be completed with the contribution of qualitative analysis. This technique allows an in‐depth exploration of the evaluated indicators (Morgan, 1996). Thus, in this study, we start with the opinions and feedback of the experts to carry out the indicators’ reformulations, definitions, and levels.

Limitations

The NO and their indicators have been validated in Spanish. The results and recommendations of this study cannot be generalized to the NOC outcome in other languages and other settings, although the NOC has been translated into 11 different languages at this time. In order to obtain credible evidence of internationally validated NOC outcomes, further validation studies based on specific cultures are recommended. The next step is to use the validated NO in clinical settings and then evaluate the effects of using the NOC outcome.

Implications for nursing practice

Validation of this instrument will provide evidence‐based indicators to measure how patients obtain, process, and understand information about health and illness, how they interact with the health system, and how they make informed decisions about their health. This will allow a deeper understanding of patients’ needs and facilitate identifying populations with low HL. By identifying these communities, nurses can direct specific efforts to promote health more effectively, thus making HL a priority objective in health management and policies.

Once validated, this instrument can be implemented by nurses in various areas of health care, including chronic patient consultations and other health units. Its application will facilitate the early identification of patients with a low level of HL, which will allow the implementation of appropriate measures to improve it. This will improve patients’ quality of life and self‐care and guide necessary interventions to address their needs more effectively.

The integration of this instrument into clinical practice will directly influence nursing interventions. By providing a more complete understanding of patients’ health needs and capabilities, nurses will be able to offer a more personalized and effective approach to their care, thereby optimizing health outcomes. This means that the use of this instrument will not only improve the effectiveness of nursing interventions but will also contribute to more patient‐centered care and, ultimately, to an improvement in the overall quality of healthcare. In fact, this study has contributed to filling a gap in the nursing care plan (a specific nursing intervention—NIC—requires results nursing—NOC—to determine the patient's AS status through an evaluation).

CONCLUSIONS

This study provides evidence of the content validity of 21 indicators of the Spanish version of the NO, Health Literacy Behaviour (2015), included in the sixth NOC. The 21 indicators and each level of their 5‐point Likert‐type measuring scale were operationally defined. Two indicators were reformulated throughout the validation process based on expert opinions.

The validation of this NOC outcome would facilitate nurses in establishing individualized and efficient care interventions and identifying populations with low HL.

The results support the need for studies assessing the accuracy of NO indicators to guide nurses’ decisions in selecting indicators sensitive to nursing interventions.

AUTHOR CONTRIBUTIONS

Pilar Bas‐Sarmiento and Martina Fernández‐Gutiérrez: Conceptualization and supervision. Alba Correro‐Bermejo; Martina Fernández‐Gutiérrez; Pilar Bas‐Sarmiento; and José Manuel Romero‐Sánchez: Methodology; formal analysis; data curation; writing (original draft). Alba Correro‐Bermejo; Miriam Poza‐Méndez; and Olga Paloma‐Castro: Writing (review and editing). The authors have consented to publication and have checked the final document.

CONFLICT OF INTEREST STATEMENT

The authors declare no conflicts of interest.

FUNDING INFORMATION

The authors declare no funding sources.

ETHICS STATEMENT

The Cadiz Ethics Committee approved this research with registre number 56.23 (code:PEIBA 0686‐N‐23).

Supporting information

Supporting Information

IJNT-36-264-s001.docx (16.3KB, docx)

ACKNOWLEDGMENTS

The authors would like to thank all the professionals who participated in the expert panel. They would also like to thank the University Research Institute for Sustainable Social Development (INDESS) for supporting the translation of the manuscript.

Correro‐Bermejo, A. , Bas‐Sarmiento, P. , Romero‐Sánchez, J. M. , Paloma‐Castro, O. , Poza‐Méndez, M. , & Fernández‐Gutiérrez, M. (2025). Role of the health literacy assessment in healthcare: Content validation of “Health Literacy Behaviour” nursing outcome. International Journal of Nursing Knowledge, 36, 264–274. 10.1111/2047-3095.12482

REFERENCES

  1. Almanasreh, E. , Moles, R. , & Chen, T. F. (2019). Evaluation of methods used for estimating content validity. Research in Social & Administrative Pharmacy: RSAP, 15(2), 214–221. 10.1016/j.sapharm.2018.03.066 [DOI] [PubMed] [Google Scholar]
  2. Bas‐Sarmiento, P. , Fernández‐Gutiérrez, M. , Poza‐Méndez, M. , Pelícano, N. , & Ruiz, P. (2015). New European challenges: Health literacy intervention in diverse communities. Paper presented in The 2nd European Health Literacy Conference, Aarhus, Denmark.
  3. Bellido‐Vallejo, J. C. , & Pancorbo‐Hidalgo, P. L. (2017). Cultural adaptation and psychometric evaluation of the Spanish version of the nursing outcome “pain control” in primary care patients with chronic pain. Pain Management Nursing, 18(5), 337–350. 10.1016/j.pmn.2017.04.001 [DOI] [PubMed] [Google Scholar]
  4. Bellido‐Vallejo, J. C. , Rodríguez‐Torres, M. C. , López‐Medina, I. M. , & Pancorbo‐Hidalgo, P. L. (2016). Psychometric testing of the Spanish version of the pain level out‐ come scale in hospitalized patients with acute pain. International Journal of Nursing Knowledge, 27(1), 10–16. 10.1111/2047-3095.12070 [DOI] [PubMed] [Google Scholar]
  5. Braun, V. , & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. 10.1191/1478088706qp063oa [DOI] [Google Scholar]
  6. Bulechek, G. M. , Butcher, H. K. , Dochtermar, J. M. , & Wagner, C. M. (2013). Nursing interventions classification (6th ed.).Elsevier. [Google Scholar]
  7. Cabellos‐García, A. C. , Martínez‐Sabater, A. , Castro‐Sánchez, E. , Kangasniemi, M. , Juárez‐Vela, R. , & Gea‐Caballero, V. (2018). Relation between health literacy, self‐care and adherence to treatment with oral anticoagulants in adults: A narrative systematic review. BMC Public Health [Electronic Resource], 18(1), 1157. 10.1186/s12889-018-6070-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Chantal Magalhães Da Silva, N. , De Souza Oliveira‐Kumakura, A. R. , Moorhead, S. , Pace, A. E. , & Campos De Carvalho, E. (2017). Clinical validation of the indicators and definitions of the nursing outcome “tissue integrity: Skin and mucous membranes” in people with diabetes mellitus. International Journal of Nursing Knowledge, 28(4), 165–170. 10.1111/2047-3095.12150 [DOI] [PubMed] [Google Scholar]
  9. Cobo Sánchez, J. L. (2019). Health literacy and care: Much more than health education. Metas de Enfermería, 22(5), 3. 10.35667/MetasEnf.2019.22.1003081427 [DOI] [Google Scholar]
  10. Correro‐Bermejo, A. , Fernández‐Gutiérrez, M. , Poza‐Méndez, M. , & Bas‐Sarmiento, P. (2023). Content and clinical validation of the nursing outcome “health literacy behaviour”: A validation protocol. Healthcare, 11(4), 481. 10.3390/healthcare11040481 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. da Costa Ferreira, N. , Takao Lopes, C. , Moorhead, S. , & Gengo E Silva Butcher, R. D. C. (2021). Content validation of the nursing outcome knowledge heart failure management: Brazilian nurses' opinions. International Journal of Nursing Knowledge, 32(3), 206–214. 10.1111/2047-3095.12312 [DOI] [PubMed] [Google Scholar]
  12. Devriendt, E. , Van Den Heede, K. , Coussement, J. , Dejaeger, E. , Surmont, K. , Heylen, D. , Schwendimann, R. , Sexton, B. , Wellens, N. I. H. , Boonen, S. , & Milisen, K. (2012). Content validity and internal consistency of the Dutch translation of the safety attitudes questionnaire: An observational study. International Journal of Nursing Studies, 49(3), 327–337. 10.1016/j.ijnurstu.2011.10.002 [DOI] [PubMed] [Google Scholar]
  13. Dias Emidio, S. C. , Moorhead, S. , Oliveira, H. C. , Herdman, T. H. , Oliveira‐Kumakura, A. R. D. S. , & Carmona, E. V. (2020). Validation of nursing outcomes related to breastfeeding establishment. International Journal of Nursing Knowledge, 31(2), 134–144. 10.1111/2047-3095.12256 [DOI] [PubMed] [Google Scholar]
  14. Duong, T. V. , Aringazina, A. , Baisunova, G. , Nurjanah, Pham, T. V. , Pham, K. M. , Truong, T. Q. , Nguyen, K. T. , Oo, W. M. , Mohamad, E. , Su, T. T. , Huang, H. L. , Sørensen, K. , Pelikan, J. M. , Van Den Broucke, S. , & Chang, P. W. (2017). Measuring health literacy in Asia: Validation of the HLS‐EU‐Q47 survey tool in six Asian countries. Journal of Epidemiology, 27(2), 80–86. 10.1016/j.je.2016.09.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Escobar‐Pérez, J. , & Cuervo‐Martínez, A. (2008). Validez de contenido y juicio de expertos: Una aproximación a su utilización. Avances de Medición, 6(1), 27–36. [Google Scholar]
  16. Fernández‐Gutiérrez, M. , Bas‐Sarmiento, P. , Albar‐Marín, M. J. , Paloma‐Castro, O. , & Romero‐Sánchez, J. M. (2018). Health literacy interventions for immigrant populations: A systematic review. International Nursing Review, 65(1), 54–64. 10.1111/inr.12373 [DOI] [PubMed] [Google Scholar]
  17. Figueroa Saavedra, C. S. , Otzen Hernández, T. F. , Figueroa Delgado, V. , Sanhueza, P. , & Candia, J. (2020). Association between health literacy and adherence to medications among older adults. Revista Médica de Chile, 148(5), 653–656. 10.4067/S0034-98872020000500653 [DOI] [PubMed] [Google Scholar]
  18. Gill, F. J. , Leslie, G. D. , Grech, C. , & Latour, J. M. (2013). Using a web‐based survey tool to undertake a Delphi study: Application for nurse education research. Nurse Education Today, 33(11), 1322–1328. 10.1016/j.nedt.2013.02.016 [DOI] [PubMed] [Google Scholar]
  19. Herdman, T. H. , & Kamitsuru, S. (Eds.). (2018). NANDA International, Inc. nursing diagnoses: Definitions and classification 2018–2020 (11th ed.). Elsevier Health Sciences. [Google Scholar]
  20. Instituto de Salud Carlos III . (2022). Monitoring the behavior and attitudes of the population related to COVID‐19 in Spain (COSMO‐SPAIN): WHO Study. COSMO Spain. Accessed 7 May 2024. https://portalcne.isciii.es/cosmo-spain/ [Google Scholar]
  21. Landeta Rodríguez, J. (1999). El método Delphi: Una técnica de previsión para la incertidumbre. Ariel. [Google Scholar]
  22. Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–386. 10.1097/00006199-198611000-00017 [DOI] [PubMed] [Google Scholar]
  23. MacDermid, J. C. (2021). ICF linking and cognitive interviewing are complementary methods for optimizing content validity of outcome measures: An integrated methods review. Frontiers in Rehabilitation Sciences, 2, 702596. 10.3389/fresc.2021.702596 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Martín‐Dorta, W. J. , Brito‐Brito, P. R. , & García‐Hernández, A. M. (2021). Development and content validation of the CEECCA questionnaire to assess ability to communicate among individuals with aphasia based on the NANDA‐I and NOC. Healthcare, 9(11), 1459. 10.3390/healthcare9111459 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. McDonald, M. , & Shenkman, L. J. (2018). Health literacy and health outcomes of adults in the United States: Implications for providers. Internet Journal of Allied Health Sciences and Practice, 16(4), 1–5. https://nsuworks.nova.edu/ijahsp/vol16/iss4/2 [Google Scholar]
  26. Mokkink, L. B. , Prinsen, C. A. C. , Bouter, L. M. , de Vet, H. C. W. , & Terwee, C. B. (2016). The COnsensus‐based Standards for the selection of health measurement INstruments (COSMIN) and how to select an outcome measurement instrument. Brazilian Journal of Physical Therapy, 20(2), 105–113. 10.1590/bjpt-rbf.2014.0143 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Moorhead, S. , Johnson, M. , Maas, M. L. , & Swanson, E. (2018). Nursing outcomes classification (NOC). Measurement of health outcomes. Elsevier. [Google Scholar]
  28. Morgan, D. L. (1996). Focus groups. Annual Review of Sociology, 22, 129–152. 10.1146/annurev.soc.22.1.129 [DOI] [Google Scholar]
  29. Morilla‐Herrera, J. C. , Morales‐Asencio, J. M. , Fernández‐Gallego, M. C. , Cobos, E. B. , & Romero, A. D. (2011). Utility and validity of indicators from the Nursing Outcomes Classification as a support tool for diagnosing Ineffective Self Health Management in patients with chronic conditions in primary health care. Anales del sistema sanitario de Navarra, 34(1), 51–61. 10.4321/s1137-66272011000100006 [DOI] [PubMed] [Google Scholar]
  30. Nutbeam, D. , & Lloyd, J. E. (2021). Understanding and responding to health literacy as a social determinant of health. Annual Review of Public Health, 42, 159–173. 10.1146/annurev-publhealth-090419-102529 [DOI] [PubMed] [Google Scholar]
  31. Oh, H. , & Moorhead, S. (2019). Validation of the knowledge and self‐management nursing outcomes classification for adults with diabetes. CIN: Computers, Informatics, Nursing, 37(4), 222–228. 10.1097/CIN.0000000000000495 [DOI] [PubMed] [Google Scholar]
  32. Oh, H. , Moorhead, S. , & Chae, S. (2023). Validation of nursing outcomes classification: Knowledge and self‐management for cardiac disease. CIN: Computers, Informatics, Nursing, 41(9), 655–664. 10.1097/CIN.0000000000000998 [DOI] [PubMed] [Google Scholar]
  33. Paloma‐Castro, O. , Romero‐Sánchez, J. M. , Paramio‐Cuevas, J. C. , Pastor‐Montero, S. M. , Del Carmen Sánchez‐Dalda, M. , Rozadillas‐Sanmiguel, E. , & Moreno‐Corral, L. J. (2017). Development and psychometric evaluation of a questionnaire based on the nursing outcomes classification to determine the knowledge of parents on breast‐feeding: Research protocol. International Journal of Nursing Knowledge, 28(2), 100–108. 10.1111/2047-3095.12101 [DOI] [PubMed] [Google Scholar]
  34. Pelikan, J. M. , Röthlin, F. , Ganahl, K. , & Boltzmann, L. (2014). Measuring comprehensive health literacy in general populations: Validation of instrument, indices and scales of the HLS‐EU study. 6th Annual Health Literacy Research Conference, Bethesda, USA. http://www.bumc.bu.edu/healthliteracyconference/files/2014/06/Pelikan‐et‐al‐HARC‐2014‐fin.pdf
  35. Polit, D. E. , & Tatano, C. (2008). Nursing research: Generating and assessing evidence for nursing practice. Lippincott Williams & Wilkins. [Google Scholar]
  36. Poza‐Méndez, M. , Bas‐Sarmiento, P. , Erahmouni, I. , & Fernández‐Gutiérrez, M. (2023). Assessment of health literacy among migrant populations in Southern Spain: A cross‐sectional study. Nursing Open, 10(4), 2600–2610. 10.1002/nop2.1520 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Prince, L. Y. , Schmidtke, C. , Beck, J. K. , & Hadden, K. B. (2018). An assessment of organizational health literacy practices at an academic health center. Quality Management in Health Care, 27(2), 93–97. 10.1097/QMH.0000000000000162 [DOI] [PubMed] [Google Scholar]
  38. Seganfredo, H. D. , & Almeida, M. A. (2010). Produção de conhecimento sobre resultados de enfermagem [Knowledge production about nursing outcomes]. Revista Brasileira De Enfermagem, 63(1), 122–126. 10.1590/s0034-71672010000100020 [DOI] [PubMed] [Google Scholar]
  39. Silva, N. C. M. , Oliveira, A. R. S. , & Carvalho, E. C. (2015). Knowledge produced from the outcomes of the “Nursing Outcomes Classification–NOC”: integrative review. Revista gaucha de enfermagem, 36(4), 104–111. 10.1590/1983-1447.2015.04.53339 [DOI] [PubMed] [Google Scholar]
  40. Sørensen, K. , Levin‐Zamir, D. , Duong, T. V. , Okan, O. , Brasil, V. V. , & Nutbeam, D. (2021). Building health literacy system capacity: A framework for health literate systems. Health Promotion International, 36(1), i13–i23. 10.1093/heapro/daab153 [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Sørensen, K. , Van Den Broucke, S. , Fullam, J. , Doyle, G. , Pelikan, J. , Slonska, Z. , & Brand, H. , & (HLS‐EU) Consortium Health Literacy Project European . (2012). Health literacy and public health: A systematic review and integration of definitions and models. BMC Public Health [Electronic Resource], 12, 80. 10.1186/1471-2458-12-80 [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Urrutia Egaña, M. , Barrios Araya, S. , Gutiérrez Núñez, M. , & Mayorga Camus, M. (2014). Optimal method for content validity. Educación Médica Superior, 28(3), 547–558. http://scielo.sld.cu/scielo.php?script=sci_arttext&pid=S0864‐21412014000300014&lng=es&tlng=es [Google Scholar]
  43. Zarocostas, J. (2020). How to fight an infodemic. Lancet, 395(10225), 676. 10.1016/S0140-6736(20)30461-X [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supporting Information

IJNT-36-264-s001.docx (16.3KB, docx)

Articles from International Journal of Nursing Knowledge are provided here courtesy of Wiley

RESOURCES