Development and psychometric properties rating scale of “clinical competency evaluation in mental health nurses”: Exploratory factor analysis

Abstract

Introduction

Evaluation of clinical competency in nurses has a distinct importance in healthcare due to its significant impact on improving the quality of patient care and creation of opportunities for professional promotion. This is a psychometric study for development of the “Clinical Competency of Mental Health Nursing”(CCMHN) rating scale.

Methods

In this methodological research that was conducted in 2015, in Tehran, Iran, the main items were developed after literature review and the validity and reliability of the tool were identified. The face, content (content validity ratio and content validity index) and construct validities were calculated. For face and content validity, experts’ comments were used. Exploratory factor analysis was used to determine the construct validity. The reliability of scale was determined by the internal consistency and inter-rater correlation. The collected data were analyzed by SPSS version 16, using descriptive statistical analysis.

Results

A scale with 45 items in two parts including Emotional/Moral and Specific Care competencies was developed. Content validity ratio and content validity index were 0.88, 0.97 respectively. Exploratory factor analysis indicated two factors: The first factor with 23.93 eigenvalue and second factor with eigenvalue 2.58. Cronbach’s alpha coefficient for determination of internal consistency was 0.98 and the ICC for confirmation inter-rater correlation was 0.98.

Conclusion

A scale with 45 items and two areas was developed with appropriate validity and reliability. This scale can be used to assess the clinical competency in nursing students and mental health nurses.

1. Introduction

Clinical competency evaluation is a critical challenge in the nursing profession (1). Competency in the health service sector and different medicine, education, clinical, and management fields is an area of debate (2). One definition of clinical competency is efficient utilization of technical and communicational skills, knowledge, clinical reasoning, emotions and values in a clinical environment (3). Simply having the ability to carry out routine tasks cannot provide a reliable path for achieving professional goals of nurses, and modern and efficient nursing entails skills such as critical thinking, problem solving, and ability to make clinical decisions (4), it is essential to possess clinical competency before becoming a successful professional nurse (1). With respect to the current high pace of changes in the health monitoring system, necessity of provision of safe and cost-effective services, increase of public information about health and hygiene, general expectation for quality health services, and tendency in health providers to recruit skillful work forces, it is essential to pay more attention to clinical competency of the professions pertinent to health services (5). In general, with significant effect on quality of health services to patients, competency evaluation is of critical importance as it provides opportunity to develop and improve the nursing profession (6). Measures of evaluation of clinical competency not only lead to more knowledge and awareness among nurses and managers about their competency, but also reveals the skill and cognitive problems in their work (1). The mental health nurse is a key member of a psychiatric team who undertakes the treatment program overnight. The nurse prepares the patients to deal with their situation and regain their previous abilities (7). Dealing with health service takers in different situations is effective in the development of clinical competency of mental health nurses (8). Providing guidelines based on evidence, cooperating with the health service taker and cultural competency are some of the clinical competencies of a mental health nurse (9). Despite the great deal of studies on and utilization of a variety of tools to measure clinical competency, there is evidence regarding a poor quality of clinical evaluation methods. Some say that clinical competency evaluation methods do not have enough comprehensiveness to fulfill the task they are intended to perform. One of the greatest challenges in evaluating clinical competency is instability of measurement techniques and uncertainties as to the evaluating process. Some problems are rooted in subjective nature and stressfulness of the processes, inconsistency of the processes with actual specifications and functions to be measured, and the great challenge of increasing satisfaction of the subjects of assessments (10). It is notable that there is no tool to evaluate all educational purposes such as knowledge, skills, behavior, and attitude; therefore, it is essential to coordinate assessment methods with the competencies that ought to be possessed (11). A 2014 study by Mohtashami designed a valid and reliable checklist with 73 statements for clinical competency assessment (9). Hakimzadeh et al. designed and evaluated psychometric aspects of a tool with 31 statements and evaluated clinical competency of nursing students in three fields of emotional, cognitive, and psychomotor competencies (12). Following Benner’s theory “from the beginner to professional”, a study by Meretoja et al. designed a questionnaire with 73 statements and evaluated clinical competency of general nurses (13). Parsa Yekta et al. in a qualitative study found that six factors are influence on nurses’ clinical competency which are experience, opportunity, environment, personal characteristics, motivation and theoretical knowledge (14). Vanaki and Memarian described a model to achieve clinical competency for undergraduate nurses with six stages, responsibility and professional commitment, self-learning, critical thinking, application of knowledge and expertise, acquisition of effective experience, and clinical competency acquisition (15). According to preliminary surveys and studies, the majority of available tools are “self-assessment” tools. In addition, none of the tools designed in Iran and other countries are specifically designed for measuring clinical competency of mental health nurses. Taking this gap into account, the present study is aimed at designing a tool to evaluate “Clinical Competency of Mental Health Nurses” (CCMHN). This scale can be used as a comprehensive, practical and useful tool to assess psychiatric nurse’s clinical competency. Also, this scale will be utilized to evaluate psychiatric nurses’ function by administrators.

2. Material and Methods

2.1. Design and setting

A methodological study was conducted in 2015 in Tehran (Iran). The development process to design a measurement tool was based on literature review, clinical observations, interviews, and qualitative and combined studies based on the results of literature review, comparing the available tools pertinent to clinical competency of mental health nurses and the task-statements of mental health nurses approved by the Ministry of Health and Medical Services. The study population was different at each stage. For facial and content validity, the samples comprised of 15 teachers’ schools of nursing and midwifery and nursing managers working in psychiatric centers associated with Tehran-based hospitals of Medical Sciences and 223 nurses (for contrast validity) working in psychiatric wards of the hospitals and Tehran Welfare and Rehabilitation University. Inclusion criteria included BSc or MSc and higher degrees and at least six months’ experience in a psychiatric ward. This study was carried out in two sections. Phase 1: Item generation, phase 2: psychometric evaluation. In the first phase, the statements were designed based on reviewing the texts and publications for mental health nurses approved by the Iran Ministry of Health and Medical Services and other available tools. After literature review (included texts, journals and Iranian Health policy), preliminary statements were designed with 49 items. The second phase, psychometric evaluation of CCMHN was conducted through assessing its face, content and construct validity as well as internal consistency and intra-class consistency reliability.

2.2. Face and Content Validity Assessment

Face and content validities (CVR and CVI) were measured simultaneously by consulting with 15 experts. To determine CVR, Lawshe minimum value table and to determine CVI, Walts and Bassel’s method were adopted; so, that if more than 75% of the experts scored “relevance” of a statement equal with 3 or 4 (from 1 to 4), the statement remained in the tool (16).

2.3. Construct Validity Assessment

Explorative factor analysis was used to measure contrast validity of the tool. We recruited a sample of 210 nurses working in psychiatric hospitals (nearly 5 people per item). Samples included nurses who worked at psychiatric wards in Shahid Beheshti Medical Sciences University, Iran Medical Sciences University, Tehran Medical Sciences University and Welfare and Rehabilitation University of Tehran city with at least a bachelor degree and one year experience. All the nurses who wished to participate in this study were selected. At the first stage of factor analysis, and to assess adequacy of correlation model between variables, Bartlett test and KMO measure were used. Based on KMO measure, adequacy of sampling was obtained as 0.96 (17).

2.4. Reliability Assessment

Reliability was determined through internal consistency and obtaining Cronbach’s alpha for the whole tool and the sub-scales. The tools that depend on direct observations of behavior need to be recorded systematically and tested regarding reliability of the observer (18). By obtaining intra-class consistency (ICC), reliability of the observers was obtained through correlation method. The tool was filled out by head nurses of psychiatric wards for the nurses working in the ward. To determine reliability of the tool based on internal consistency and reliability among the observers, the tool was given to two observers (one head nurse and one experienced nurse in the ward). Two observers were identical regarding accuracy, skill, knowledge, and their familiarity with the staff. Each observer filled the tool for 13 nurses in the psychiatric ward. The participants expressed their consent to participate after a brief introduction and being ensured regarding confidentiality of their information. The collected data were analyzed using descriptive statistical tests such as frequency, mean and standard deviation by SPSS version 16 (SPSS Inc., Chicago, Illinois, USA).

3. Results

The preliminary statements (n=49) were written after defining clinical competency. The tool was designed based on a ranking scale in four fields and Likert’s five-point scale (never = 0, barely = 1, sometimes = 2, mostly = 3, always = 4). Some of the statements were modified at face validity stage to improve readability of the statement; statements with similar meaning were mixed to avoid repetition; and some statements were clarified by adding examples.

3.1. Content Validity

Based on CVR of the tool, statements with CVR < 0.49 were removed. CVR values above 0.88 indicated acceptable content validity. At the end of this stage, the number of statements had reached 45 and CVI of the tool was obtained 0.97 (17).

3.2. Construct Validity

For contrast validity of the tool with explorative factor analysis, it was filled out by 210 nurses working in psychiatric hospitals. At the first stage of factor analysis, and to assess adequacy of the correlation model between variables, Bartlett test and KMO measure were used. Based on KMO measure, adequacy of sampling was obtained as 0.96. To check the significant difference between the created correlation matrix and zero before doing factor analysis, Bartlett’s test was obtained as 8.595 (P<0.005). Subsequently, latent factors of the tool were determined by employing scree diagram (Figure 1) and factor load. Six specific value factors above 1, which explained 70.17% of the variance, were determined. Explorative factor analysis results indicated two factors; one factor with 28 statements (specific value of 23.93), and one factor with 17 statements (specific value of 2.85). Factors No. 3, 4, 5, and 6 were removed because their factor load was less than 0.3. All the statements in factors No. 1 and 2 had factor load above 0.3. None of the statements were removed at this stage and given the results of factor analysis; the tool was designed in two fields of Specific Care competencies (factor No. 1) with 33 statements and Emotional/Moral competency (factor No. 2) with 12 statements (Table 1). Consequently, the final version of CCMHN consists of 45 items scored on a five –point Likert scale range, from never (0) to always (4).

Figure 1.

Scree plot for the sample in this study (n = 210).

Table 1.

Summary of principal components (with eigenvalues) contributing the items as a result of explorative factor analysis of the “Clinical Competency Evaluation in Mental Health Nurses”(CCMHN)

Items	Factor Loading
	Factor 1	Factor 2
Demonstrate patience when taking care of the patient		0.82
Show respect to the patient		0.83
Avoid rushing into judgment about the patient		0.77
Keep the secrets of the patient (unless there is risk to the patient or the others)		0.72
Never discuss the patient’s issues in presence of others		0.75
Observe the patient’s rights (e.g. privacy, right to decide about treatment, etc.)		0.73
Show responsiveness and reliance in doing the assigned tasks		0.60
Offer nursing care without discrimination or based on racial, cultural, and religious specifications		0.71
Show respect in dealing with colleagues		0.70
Show honesty in their interactions with the patient		0.74
Have ability to control one’s emotions (e.g. anger, anxiety, and fear)		0.61
Show eagerness and interest in taking care of the patient		0.64
Introduce oneself to the patient	0.30
Observe the principles of interviewing from the beginning to the end	0.57
Gain the patient’s trust to establish communication	0.57
Address the patients by their name	0.57
Motivate the patients to express their feelings		0.56
Observe the limits (physical distance, to address the patient with respectful titles, )		0.61
Follow an active listening method in communication with the patient	0.59
Help the patient to improve their self-confidence through improving capabilities and advantages of the patient	0.63
Explain medical procedures before starting them (e.g. Injection, vital sign checkup, administering medicines)	0.53
Use the medical procedures as a chance to communicate with the patient	0.43
Use a variety of references (e.g. family, medical file, etc.) to collect information	0.62
Use different screening tools (narcotic drug abuse, personality disorder, depression, etc.) to examine hazardous behavior (e.g. aggressiveness, suicide, murder, etc.).	0.71
Record and report any change in the mental health of the patient and other problems	0.64
Record and recognize defensive mechanisms used by the patients (e.g. projection, denial, displacement, etc.).	0.77
Prioritize psycho-nursing diagnosis in providing health care to the patients	0.72
Employ practical skills (e.g. doing injections, administering medicines, checking vital signs, etc.).		0.58
Carry out nursing skills based on reliable references and new researches (evidence based)	0.60
Carry out the measures recorded in the medical file of the patient	0.60
Use stress shooting approaches (e.g. relaxation, deep breath, etc.) when the patient is anxious or disturbed	0.72
Carry out the required health care in emergency and attack cases (e.g. physical restrain, isolation room, etc.).		0.54
Provide a safe environment for the patient (by taking care of oneself, other, escaping, etc.).	0.66
Observe the patient carefully to ensure that the patient has taken their medicine	0.62
Use nursing measures in the fields of using psychedelic drugs and the side-effects	0.71
Pay attention to optimum usage of medicines, materials, and equipment	0.71
Cooperate and consult with other members of the health team	0.56
Provide required training to improve the patient and their family’s health	0.65
To use proper training methods based on age, education level, and nature of disease	0.71
To prepare discharge plan for the patient and their family	0.75
To negotiate with the patient’s family to take part in providing health care service	0.81
To motivate the patient to follow the treatment program	0.74
To help the patient to take care of themselves as much as possible	0.81
To follow psychological rehabilitation principles and standards (e.g. self-care, treatment follow up, etc.) to improve patient’s health and to help them in regaining their abilities	0.82
To use learning opportunities such as continuing studies, and participating in educational workshops to improve personal and professional progress	0.62

All factor loading >0.30 are reported

3.3. Reliability

Cronbach’s alpha through internal consistency for the whole tool, specific care competencies, and emotional-moral competency were 0.98, 0.98, and 0.95 respectively. Subsequently, inter class coefficient (ICC) of observations was obtained (ICC of emotional-moral competency = 0.96, ICC of specific care competencies = 0.98; and ICC of the whole tool = 0.98). The total point of competency of each case was reported as the sum of the scores of all statements. The obtained scores by the tool were classified at excellent (score 136–180), good (score 91–135), average (46–90), and weak (score <46) levels.

4. Discussion

Competency is a complicated concept that encompasses different fields of learning such as knowledge, skills, and attitude. It is a combination of skills and personal attitudes such as motivation (19). As recommended by the surveys on the available tools, all of them are designed to measure the competency of general nurses, and there was no special tool for mental health nurses. In addition, all the available tools were self-assessment (11–13, 20). Thereby, the authors felt compelled to design a valid and reliable tool as a ranking scale with 45 statements to measure clinical competency of nurses in two fields (i.e. emotional-moral competency with 12 statements and specific care competencies with 33 statements). Reliability (internal consistency and correlation between the observers) and validity (face, content and contrast validity) of the tool were proved. Emotional – moral competency field was featured with statements about patience, showing respectful behavior, avoiding prejudgment, observing confidentiality of the patient’s information, observing the patient’s rights, responsiveness, avoiding discrimination, and showing honesty and respect toward colleagues. These statements represent the most important moral and emotional characteristics of nurses. Mohtashami et al. devised emotional and moral competencies as two separate fields in their tool; however, none of the available tools have adopted such an approach (21). Taking into account the field of activities of the mental health nurse in specific care competencies issues, a subscale titled “competency of establishing therapeutic relationship” was devised in the tool. The foundation of the nursing services is the communication between the patient and the nurse (22, 23). In specific care competencies issues, therapeutic communication subscale, active listening and supporting self-confidence of patient through supporting strengths and capabilities of the patient, wining patient’s trust, and motivating the patient to express their feelings were implemented in the tool. Mohtashami et al. devised ability to establish communication as a separate field in their tool; while Hakimzadeh et al. (12), Jabari et al. (11), Bahreini et al. (22), and Liou & Cheng (20) did not. The inconsistency can be explained by the fact that their tool was not a competency evaluation tool specifically designed for mental health nurses, while communicational ability is a key skill for mental health nurses. Among the other statements in specific care competencies issues were: using different sources to collect information about the patient, using screening method to detect hazardous behavior of the patient, recording and reporting changes in patient’s condition in a proper way, detecting defensive mechanisms in patients, and general practical skills (injection, administering medicines, and so on). CVR and CVI of the tool were determined simultaneously by providing the tool to 15 experts. Other researchers (11, 12, 20–22) followed the same method to determine face and content validity of their tools. Lo Biondo-Wood & Haber (18) and Polit & Beck 2010 (24) argued that to examine face validity of a tool, it is provided to experts to examine clarity of the structure. They also argued that in the case of content validity, the extent to which the field under measurement is covered by the statements is determined by the experts. CVR and CVI of the tool designed in this study were 0.88 and 0.97 respectively, both of which indicate acceptable content validity of the tool. Contrast validity results showed that the tool is comprised of two factors including emotional-moral competency (specific value = 2.58), and specific care competencies (specific value = 23.93). No statement was added or removed at the end of this stage. Liou & Cheng (20) used an explorative factor analysis method to determine contrast validity. Polit & Beck (24) stated that factor analysis at validity test stage was a method to determine the cluster of the statements in a tool. The same method was followed here to determine clusters of the designed tool. Having ascertained contrast validity, reliability of the tool was determined through internal consistency and Cronbach’s alpha. The same method was followed by other researchers (11, 12, 20–22). To determine reliability through correlation between the observers, inter class coefficient of the two observers was examined and total ICC was obtained as 0.98 – which is an acceptable level of consistency. Mohtashami et al. (9) used ICC method to survey reliability through the correlation between observers. Lo Biondo-Wood & Haber (18) noted that, for the tools that need direct behavior observation, reliability should be obtained based on correlation between the observers.

5. Conclusions

A ranking scale to measure clinical competency of mental health nurses (CCMHN) was designed with 45 statements in two fields of emotional/moral competency and specific care competencies. The results regarding face, content, and structure validity and reliability based on internal consistency and correlation between the observers, showed acceptable validity and reliability of the tool. Nursing managers may use the tool in their programs to improve health service quality in their wards. In addition, the tool can be helpful to evaluate nurses in psychiatric wards. In addition to clinical competency, the tool can be used to measure educational needs of mental health nurses. With regard to limitations of the study, paucity of similar studies (in Iran and other countries) on clinical competency of mental health nurses and refusal of some of the head nurses to fill out the tool due to their work volume are notable.