Abstract
Background:
Occupational Health Services (OHS) are obliged to follow the principles of evidence-based health care. However, there needs to be tools to measure this. Therefore, we developed and validated a questionnaire for evaluating OHS practitioners’ attitudes, competence, and organisational support to perform evidence-based practice (EBP-OHS) following the JBI Model of Evidence-Based Healthcare.
Methods:
The questionnaire’s content validity was assessed by 12 experts in the field. Then, an opportunity sample of 524 OHS practitioners completed the questionnaire. We examined the questionnaire’s psychometric properties using exploratory factor analysis and subjected it to construct validity and reliability testing.
Results:
The content validity index of the chosen items was 0.78 or higher. Exploratory factor analysis revealed that the measure’s construct validity was adequate (KMO 0.9). Principal component factor analysis supported a three-factor structure (all eigenvalues 1.3 or more), which explained 60.3 % of the total variance. Aligned with these three factors, the EBP-OHS consists of three domains: Organisational support (seven items), OHS practitioners’ competence (six items) and OHS practitioners’ attitudes (two items). The scale’s reliability is good (Cronbach alpha 0.88).
Conclusions:
The EBP-OHS is a valid tool for measuring occupational health services’ evidence-based practice and enabling the implementation of research into practice. It embodies the phases of evidence transfer and implementation described in the JBI Model of Evidence-Based Healthcare and translates them into concrete measurable activities in the OHS context.
Keywords: Questionnaire, Evidence-Based Practice, Evidence-Based Health Care
1. Introduction
Evidence-based healthcare, formulated by the Joanna Briggs Institute (JBI), consists of five phases: global health, evidence generation, synthesis, transfer, and implementation [1]. The JBI model of Evidence-Based Healthcare (EBHC) was developed within the field of nursing science to establish a shared vision and language for the people generating and for those implementing evidence into practice. Although the model was developed within the nursing field, it is nonetheless very relevant and applicable within all healthcare. In this JBI Model, evidence-based practice (EBP) occurs during the phase of evidence implementation. According to the model [2], evidence-based practice involves “…giving consideration to the best available evidence; the context in which the care is delivered; client preference; and the professional judgement of the health professional”. At occupational health service (OHS) practices, evidence implementation means utilising information synthesised from research to inform decision-making involving workers and their health and safety. In addition to evidence-based practice and decision-making at the level of individual service users, OHS organisations also cater to enterprises as clients.
Only a few studies have focused on evidence-based practice and its provision in the occupational healthcare context specifically. Attitudes towards EBP have been recognised as positive [3–5]. Additionally, previous evidence has shown that providing evidence-based practice in the occupational health context requires strong organisational and management support, both of which have been typically lacking [3]. Managers should be able to create a culture that supports EBP and enables adequate competence, support, and resources for employees [6]. To achieve the required organisational and managerial support in OHS, there is a need to evaluate the present state of evidence-based practice and recognise targets for development.
The range of published instruments for measuring evidence-based practice in healthcare has been thoroughly mapped with two systematic reviews [7, 8]. They both provided detailed descriptions only of the instruments their authors considered to be of the highest quality. Both reviews concluded that most researchers used the Fresno scale [9]. The Fresno scale tests the respondent’s ability to frame a research question, search for evidence to answer it, understand the hierarchy of evidence, interpret its magnitude and internal and external validity, and grasp basic statistical and methodological concepts [10]. The tool effectively evaluates how well the respondents can obtain and interpret scientific evidence.
On the other hand, measuring attitudes towards EBP is a crucial feature of a few other scales, for example, the EBPQ [11] and the Quick EBP-VIK [12], both of which have been developed to be used only with nurses. Similarly, the scale developed by Heselmans et al. [4] focuses on Flemish occupational health physicians’ attitudes toward evidence-based occupational health and clinical practice guidelines. In other words, previous tools focus either 1) on the ability to obtain and understand scientific evidence or 2) on healthcare workers with particular job titles. Moreover, previous tools are not built on an operationalised practical understanding of EBHC, nor do they gauge the support respondents receive from their employers. This study aimed to develop and psychometrically test a new questionnaire for evaluating OHS practitioners’ attitudes, competence, and organisational support to perform evidence-based practice (EBP-OHS). The questionnaire is built on a sound theoretical footing provided by the JBI Model of Evidence-Based Healthcare, specifically its phases of evidence transfer and implementation at an organisational level. It also applies equally to nurses and physicians and acknowledges the specific requirements of the occupational healthcare arena.
2. Methods
The study included three phases of questionnaire development: item generation (phase 1), expert evaluation (phase 2) and psychometric testing (phase 3).
In phase 1, we developed the items of the questionnaire based on earlier research, and the instrument’s development proceeded inductively [13]. Relevant earlier work consisted of a study conducted by the Finnish Nursing Research Foundation in which they first developed and used a questionnaire to assess EBP in primary care in Finland [14] and another study that developed further an earlier questionnaire [4] focusing on attitudes towards evidence-based practice in OHS. We evaluated the contents of these two tools and included - and modified when necessary - items that we deemed relevant in the Finnish OHS context. The third main component of the new scale, in addition to the core extracted from the two previous questionnaires, was the process of EBHC and the concept of EBP as outlined in the JBI model [1, 2]. The development process resulted in a questionnaire with 15 Likert-scale items measuring occupational health professionals’ attitudes, competence, and organisational support for EBP. Responses are given using a 5-point Likert scale with the response options: “Fully agree”, “Agree”, “Somewhat disagree”, “Fully disagree”, and “I cannot say”. During practical testing with OHS professionals, we supplemented the core items with background questions, open questions, and questions measuring information sources used, consistent practices in OHS and methods to develop and support EBP.
In phase 2, we tested the scale’s content validity in two steps. First, ten OHS or EBHC experts evaluated the scale using a purpose-built form designed to assess the relevance and clarity of each item. The expert group comprised five occupational health nurses, four occupational health physicians and one Finnish Nursing Research Foundation expert. Two specialised occupational health physicians commented on the scale in the second step. We computed a content validity index (I-CVI) [15] using item relevance ratings from our content experts. Items achieving an I-CVI of 0.78 or higher for three or more experts can be considered evidence of good content validity [15].
In phase 3, the questionnaire was completed by an opportunity sample of 524 Finnish physicians and nurses working in OHS. This sample size is sufficient given that most prior EBP assessment instruments have been developed using a sample ranging from 101 to 500 participants [8]. The questionnaire was available for respondents online from November 2020 to August 2021.
Following the principle of informed consent, we informed respondents in the cover letter about the purpose of the study, the respondents’ anonymity and voluntariness, the confidentiality of data and the contact information of the researchers and partner organisations cooperating with the study. We also made a data privacy notice publicly available that was compliant with EU GDPR requirements. According to the ethical principles of research conducted with human participants in Finland, there was no need to obtain ethical approval for this study [16].
We estimated the questionnaire’s construct validity using exploratory factor analysis (EFA). First, we used the Kaiser-Mayer-Olkin Measure of Sampling Adequacy to test the suitability of the data for EFA. A KMO value 0.9 confirmed that the sampling was adequate (0.8–1) for EFA. We performed a principal component analysis to examine the construct validity of the 15 Likert-scale items measuring occupational health professionals’ attitudes, competence, and organisational support. We determined the emerging factors by using the eigenvalues generated by the PCA, with the lowest acceptable value being 1. We evaluated the items using communalities with values between 0 and 1, where higher values indicate more communality.
We used Cronbach’s alpha to evaluate the scale’s reliability, also called internal consistency. A high value of Cronbach’s alpha (0.7 or higher) indicates internal consistency [15, 17].
3. Results
Based on expert evaluation (n=12) and content validity index (I-CVI), we determined the scale’s content validity to be good. Following the expert ratings, we dropped items that failed to reach a CVI-I level of 0.78 or higher (data not shown). The recruited experts confirmed the scale is based on a sound theoretical footing of evidence-based health care in the OHS context. The experts had suggestions regarding the clarity of some items, so they were reworded.
We obtained an opportunity sample of 524 respondents from the scale’s target population. Fifty-five percent of the respondents (n=289) were occupational health nurses, 32% (n=166) were occupational health physicians, and the remaining 13% (n = 69) were experts and supervisors. See Table 1 for a description of the respondents.
Table 1.
Participant characteristics of our in-development questionnaire respondents.
| Variables | % (N) |
|---|---|
| Age | |
| 20-29 years | 9.9 (52) |
| 30-39 years | 25 (131) |
| 40-49 years | 25.6 (134) |
| 50-59 years | 26.7 (140) |
| 60-69 years | 12.8 (67) |
| Workplace | |
| Private OHS company | 78.4 (411) |
| Public OHS company | 9.5 (50) |
| Employer-owned OHS | 6.7 (35) |
| OHS owned jointly by several employers | 2.9 (15) |
| Working role | |
| Occupational health physician | 25.8 (135) |
| Occupational health physician role combined with supervisory or expert tasks | 5.9 (31) |
| Occupational health nurse | 52.1 (273) |
| Occupational health nurse role combined with supervisory or expert tasks | 3.1 (16) |
| Supervisor or expert role | 13.2 (69) |
| Work experience in OHS | |
| Less than five years | 28.8 (151) |
| 5-10 years | 22.9 (120) |
| 11-20 years | 29.6 (155) |
| Over 20 years | 18.7 (98) |
| Work experience in supervisory, development or expert tasks | |
| No experience | 54.8 (287) |
| Less than five years | 21.8 (114) |
| 5-10 years | 11.6 (61) |
| 11-20 years | 8.4 (44) |
| Over 20 years | 3.4 (18) |
OHS: Occupational Health Service.
Using principal component analysis, we identified three factors explaining 60.3% of the total variance. The communalities varied between 0.37–0.79, indicating that the items measured the factors reliably. Factor loadings varied between 0.51–0.84. Three factors emerged from the factor analysis: Organisational support (seven items), OHS practitioners’ competence (six items) and OHS practitioners’ attitude (two items) (see Tables 2 and 3).
Table 2.
Factor matrix.
| Item | F1 | F2 | F3 | Communalities |
|---|---|---|---|---|
| Evidence-based practice is a strategic objective for our organisation | .758 | .072 | .239 | .636 |
| Our organisation values evidence-based practice | .791 | .129 | .294 | .729 |
| My organisation encourages staff to use evidence-based information | .819 | .103 | .249 | .743 |
| Staff is regularly offered internal training about up-to-date, evidence-based information in occupational health care | .664 | .273 | -.010 | .516 |
| Staff is regularly offered external training about up-to-date evidence-based information in occupational health care | .613 | .248 | -.164 | .464 |
| Within my organisation, new evidence-based information is shared actively and for all employees | .754 | .248 | .017 | .630 |
| When new research evidence emerges, my organisation evaluates the concordance of existing guidance and practices with the new evidence | .665 | .236 | -.047 | .500 |
| I use evidence-based information regularly in my work | .289 | .672 | .225 | .586 |
| I can choose evidence-based OHS for an individual client | .100 | .805 | .175 | .689 |
| I can choose evidence-based OHS for workplaces | .230 | .776 | -.082 | .662 |
| I can justify to my clients the choices I make in OHS by using evidence-based information | .237 | .734 | .181 | .627 |
| I make use of treatment guidelines in my work | .084 | .511 | .312 | .366 |
| I use evidence-based information to justify the need for change in OHS | .253 | .509 | .211 | .368 |
| I think it’s important that OHS activities and guidance are evidence-based | .112 | .214 | .815 | .722 |
| I think it’s important that my own activities as an OHS professional are evidence-based | .058 | .276 | .844 | .793 |
OHS: Occupational Health Service.
Table 3.
Factor model of three factors.
| Factor name | Items | Communalities | Explained variance | Eigen-values | Factor loading |
|---|---|---|---|---|---|
| Organisational support | 7 | 0,464–0.743 | 38.8 | 5.8 | 0.613–0.819 |
| OHS practitioners’ competence | 6 | 0.366–0.689 | 13.1 | 2.0 | 0.509–0.805 |
| OHS practitioners’ attitude | 2 | 0.722–0.793 | 8.4 | 1.3 | 0.815–0.844 |
Table 4 presents the factor structure and items of the 15 Likert-scale items that achieved a Cronbach’s alpha of 0.88, showing good reliability.
Table 4.
Items of the EBP-OHS.
| Organisational support | Evidence-based practice is a strategic objective for our organisation |
| Our organisation values evidence-based practice | |
| My organisation encourages staff to use evidence-based information | |
| Staff is regularly offered internal training about up-to-date evidence-based information in occupational health care | |
| Staff is regularly offered external training about up-to-date evidence-based information in occupational health care | |
| Within my organisation, new evidence-based information is shared actively and for all employees | |
| When new research evidence emerges, my organisation evaluates the concordance of existing guidance and practices with the new evidence | |
| OHS practitioners’ competence | I use evidence-based information regularly in my work |
| I can choose evidence-based OHS for an individual client | |
| I can choose evidence-based OHS for workplaces | |
| I can justify to my clients the choices I make in OHS by using evidence-based information | |
| I make use of treatment guidelines in my work | |
| I use evidence-based information to justify the need for change in OHS | |
| OHS practitioners’ attitude | I think it’s important that OHS activities and guidance are evidence-based |
| I think it’s important that my own activities as an OHS professional are evidence-based |
OHS: Occupational Health Service.
4. Discussion
We developed and psychometrically tested a 15-item measure of evidence-based practice in OHS. The results revealed that the scale has good psychometric properties. Based on PCA, the measure consists of three domains. The first domain of the scale, ‘Organisational support’, consists of seven items that focus on the organisation’s role in EBP and how it supports and encourages staff to perform EBP. This includes organisational values towards EBP and practical activities, such as providing training (at or outside the workplace) about evidence-based information and what to do with it.
Additionally, the first domain incorporates the perspective of quality assurance, involving information sharing and ensuring alignment of guidance and practices with new evidence. Organisational support is linked to management in creating a culture for EBP and enabling competence, support, and resources. Previous studies have also highlighted the significance of organisational and managerial support, often lacking in the occupational health context [6, 20]. Drawing on these findings, measuring support at the organisational level can facilitate recognising managerial or organisational development needs in this area.
The second domain, ‘OHS practitioners’ competence,’ comprises six items. This domain describes the role of employees, what they do, and their proficiency in performing EBP. Regarding concrete actions, the domain items assess the use of evidence-based information at work and how it is applied to justify necessary changes in OHS. Regarding competence, the domain measures respondents’ ability to choose evidence-based services for individual clients and workplaces. The third domain focuses on OHS practitioners’ attitudes and consists of two items. These items measure how important it is to the employees that occupational health services and their professional activities are based on evidence. Previous studies have identified positive attitudes towards EBP [4, 20]. Attitudes are pivotal in achieving truly evidence-based practice and require regular evaluation. Identifying OHS professionals’ competence and attitudes facilitates the evaluation of their educational needs.
The EBP-OHS scale can measure the extent of organisational support for EBP in occupational health services and employee competence and attitudes toward EBP. The scale is divided into two levels: the organisational and employee levels. The organisational level (organisational support) is built upon the JBI Model of Evidence-based Healthcare, and its evidence transfer segment. Evidence transfer refers to disseminating knowledge to individual health professionals and systems [1, 18]. The evidence transfer segment includes education, systems integration, and active dissemination.
On the other hand, the employee level (OHS practitioners’ competence and attitudes) embodies the evidence implementation segment in the same JBI Model. Evidence implementation refers to activities that engage key stakeholders with evidence to inform decision-making and enhance the quality of healthcare services [1, 18]. The evidence implementation segment of the JBI model consists of context analysis, facilitating change and evaluating processes and outcomes. In this phase, evidence-based practice becomes a concrete reality in OHS through decision-making both at the enterprise level and that of individual service users.
The EBP-OHS is a new validated tool for measuring evidence-based practice in OHS. It focuses on EBP concerning physicians and nurses, regardless of whether they are involved in patient care, administration, or both. This new scale expands the range of tools available for OHS as previous ones have focused on primary health care and nurses, such as EBPQ [11] and Quick-EBP-VIK [12], or solely on occupational health physicians [4]. The EBP-OHS offers several avenues for supporting the development of evidence-based practice. It enables comparisons both cross-sectionally (between units or organisations) and prospectively (within the same unit or organisation over time). Within-organisation comparisons facilitate allocating development resources according to needs, whereas over time comparisons enable setting achievable and measurable goals and supporting continuous development. The scale is likely most relevant when used in countries with similarly organised occupational health services, such as the Netherlands and other Nordic countries.
The EBP-OHS is based on a previous tool developed within nursing science, and it builds upon a valid and intuitive theoretical understanding of evidence-based practice [1] whilst acknowledging the requirements of the occupational health arena. Unlike the Fresno scale, which focuses on the skills required to find and understand scientific evidence [9], the EBP-OHS starts with the assumption that respondents are already familiar with the basics of EBM. This way, the respondents are free to explore how they feel about how evidence is transferred and implemented into practice within their organisation. In other words, the EBP-OHS and Fresno scales may be considered mutually complementary.
When compared to similar studies (cf. [8]), one strength of ours is the large sample size used (N=524). So far, the tool has been tested only in Finland. We support efforts to conduct a cross-cultural validation study.
5. Conclusion
Based on its psychometric properties, the EBP-OHS is a valid scale to measure evidence-based practice within OHS. The scale is a practical tool that substantiates relevant theory, specifically the phases of evidence transfer and implementation of the JBI Model of Evidence-Based Healthcare into the OHS context.
Acknowledgements:
The authors thank the Finnish Occupational Physicians’ Association, the Finnish Union of Occupational Health Nurses, and all the Occupational Health Service providers who helped us recruit survey responders.
Supplementary Materials:
None.
Funding:
This research was funded by THE FINNISH WORK ENVIRONMENT FUND.
Institutional Review Board Statement:
The study was conducted according to the guidelines of the Declaration of Helsinki and the ethical principles of research conducted with human participants in Finland. Ethical review and approval were waived for this study due to the chosen methodology in which participants could decide if they chose to participate based on a detailed description of how the study would obtain, analyse and publish results based on the data participants would provide anonymously. According to the guidance provided by the Finnish National Board on Research Integrity, studies of this kind do not need to obtain a priori ethical approval.
Informed Consent Statement:
Responding to the questionnaire was deemed equal to providing informed consent.
Declaration of Interest:
The authors declare no conflict of interest. The funders had no role in the study’s design, data collection, analysis, interpretation, manuscript writing, or decision to publish the results.
Author Contribution Statement:
J.R. and K.P. contributed to the design and implementation of the research, to the analysis of the results, and to the writing of the manuscript. K-P.M. and T.O. contributed to the writing of the manuscript.
Declaration on the use of A.I.:
None.
References
- Jordan Z, Lockwood C, Munn Z, Aromataris E. The updated Joanna Briggs Institute Model of Evidence-Based Healthcare. Int J Evid Based Healthc. 2119;17(1):58–71. doi: 10.1097/XEB.0000000000000155. Doi: 10.1097/XEB.0000000000000155. [DOI] [PubMed] [Google Scholar]
- Pearson A, Wiechula R, Court A, Lockwood C. The JBI model of evidence-based healthcare. Int J Evid Based Healthc. 2005;3(8):207–215. doi: 10.1111/j.1479-6988.2005.00026.x. Doi: 10.1111/j.1479-6988.2005.00026.x. [DOI] [PubMed] [Google Scholar]
- Brämberg EB, Nyman T, Kwak L, et al. Development of evidence-based practice in occupational health services in Sweden: a 3-year follow-up of attitudes, barriers and facilitators. Int Arch Occup Environ Health. 2017;90(4):335–348. doi: 10.1007/s00420-017-1200-8. Doi: 10.1007/s00420-017-1200-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Heselmans A, Donceel P, Aertgeerts B, Van de Velde S, Ramaekers D. The attitude of Flemish occupational health physicians toward evidence-based occupational health and clinical practice guidelines. Int Arch Occup Environ Health. 2010;83(2):201–208. doi: 10.1007/s00420-009-0449-y. Doi: 10.1007/s00420-009-0449-y. [DOI] [PubMed] [Google Scholar]
- Kinnunen-Amoroso M. How occupational health care professionals experience evidence-based guidelines in Finland: a qualitative study. J Eval Clin Pract. 2013;19(4):612–616. doi: 10.1111/j.1365-2753.2012.01850.x. Doi: 10.1111/j.1365-2753.2012.01850.x. [DOI] [PubMed] [Google Scholar]
- Bianchi M, Bagnasco A, Bressan V, et al. A review of the role of nurse leadership in promoting and sustaining evidence-based practice. J Nurs Manag. 2018;26(8):918–932. doi: 10.1111/jonm.12638. Doi: 10.1111/jonm.12638. [DOI] [PubMed] [Google Scholar]
- Shaneyfelt T, Baum KD, Bell D, et al. Instruments for evaluating education in evidence-based practice: a systematic review. JAMA. 2006;296(9):1116–1127. doi: 10.1001/jama.296.9.1116. Doi: 10.1001/jama.296.9.1116. [DOI] [PubMed] [Google Scholar]
- da Silva AM, Valentim DP, Martins AL, Padula RS. Instruments to Assess Evidence-Based Practice Among Health Care Professionals: A Systematic Review. Health Educ Behav. doi: 10.1177/10901981231170154. Published online June 1, 2023:10901981231170154. Doi :10.1177/10901981231170154. [DOI] [PubMed] [Google Scholar]
- Ramos KD, Schafer S, Tracz SM. Validation of the Fresno test of competence in evidence based medicine. BMJ. 2003;326(7384):319–321. doi: 10.1136/bmj.326.7384.319. Doi: 10.1136/bmj.326.7384.319. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sanders K, Vassev A. Fresno test. https://www.sralab.org/rehabilitation-measures/fresno-test. Published November 14, 2018. Accessed November 24, 2023. https://www.sralab.org/rehabilitation-measures/fresno-test. [Google Scholar]
- Upton D, Upton P. Development of an evidence-based practice questionnaire for nurses. J Adv Nurs. 2006;53(4):454–458. doi: 10.1111/j.1365-2648.2006.03739.x. Doi: 10.1111/j.1365-2648.2006.03739.x. [DOI] [PubMed] [Google Scholar]
- Paul F, Connor L, McCabe M, Ziniel S. The development and content validity testing of the Quick-EBP-VIK: A survey instrument measuring nurses’ values, knowledge and implementation of evidence-based practice. J Nurs Educ Pract. 2016;6(5) Doi: 10.5430/jnep.v6n5p118. [Google Scholar]
- Mikkonen K, Tomietto M, Watson R. Instrument development and psychometric testing in nursing education research. Nurse Educ Today. 2022;119:105603. doi: 10.1016/j.nedt.2022.105603. Doi: 10.1016/j.nedt.2022.105603. [DOI] [PubMed] [Google Scholar]
- Korhonen T, Siltanen H, Hahtela N, Holopainen A. Toteutuuko Näyttöön Perustuva Toiminta Suomessa? Raportti Nykytilasta Hoitotyön Edustajien Kuvaamana. 2018 [Google Scholar]
- Polit D, Hungler B. Nursing Research: Principles and Methods. 6th ed. Lippincott Williams & Wilkins; 1999. [Google Scholar]
- Finnish National Board on Research Integrity. The Ethical Principles of Research with Human Participants and Ethical Review in the Human Sciences in Finland.; 2019. https://tenk.fi/sites/default/files/2021-01/Ethical_review_in_human_sciences_2020.pdf. [Google Scholar]
- Nunnally J, Bernstein I. Psychometric Theory. 3rd ed. McGraw-Hill; 1994. [Google Scholar]
- Lockwood C. Applying Theory Informed Global Trends in a Collaborative Model for Organizational Evidence-based Healthcare. Journal of Korean Academy of Nursing Administration. 2017;23(2):111. Doi:10.11111/jkana.2017.23.2.111. [Google Scholar]