Exploring a Common Data Element for International Research in Long-Term Care Homes: A Measure for Evaluating Nursing Supervisor Effectiveness

Abstract

The aim of this study is to recommend a common data element (CDE) to measure supervisory effectiveness of staff working in LTC homes that can be used in international research. Supervisory effectiveness can serve as a CDE in an effort to establish an international, person-centered LTC research infrastructure in accordance with the aims of the WE-THRIVE group (Worldwide Elements to Harmonize Research in Long Term Care Living Environments). A literature review was completed and then a panel of experts independently reviewed and prioritized appropriateness of the measures with mindfulness of their potential applications to international LTC settings. The selection of a recommended CDE measure was guided by the WE-THRIVE group’s focus on capacity rather than deficits, the expected availability of internationally comparable data and the goal to provide a short, ecologically viable measurement, specifically for low- and middle-income countries. Two measures were considered as the CDE for supervisory effectiveness, Benjamin Rose Relationship Scale and the Supervisory Support Scale; however, given that the latter measure has been translated in Spanish and Chinese and has been tested with nursing assistants in both of these countries with good psychometric properties, our group recommends it as the CDE going forward.

Researchers from various international countries founded the WE-THRIVE initiative (Corazzini et al., 2019) whose mission is to develop core measurement domains for LTC environments in order to address gaps in the provision of care and to contribute to person-centered care (Corazzini et al., 2019). The call for the creation of standardized domains stem from the need to account for experiences of staff members as well as the residents living in the environment, linking the interactions between the two, and addressing internationally low- and middle-income LTC homes (LTCHs). Through the creation of what is known as common data elements (CDEs), defined as variables that are operationalized and measured in identical ways across studies (Redeker et al., 2015), the sharing of data and aggregation can be promoted, leading to improved quality of data and to more persuasive comparisons (Corazzini et al., 2019).

To drive this forward, in 2017 the WE-THRIVE Group identified four core measurement domains: (i) Organizational Context, (ii) Workforce and Staffing, (iii) Person-centered Care, and (iv) Care Outcomes (Corazzini et al., 2019). The domain workforce and staffing includes five concepts: staff skills, attitudes, and knowledge; staff collaboration and teamwork; training and self-efficacy; staff retention and turnover; and leadership and supervisory effectiveness (Corazzini et al., 2019) of which the latter is the focus for this paper. In terms of distinguishing between the concepts of leadership and supervisory effectiveness, there is ambiguity between these concepts in terms of profiles, theories, and how these roles often intersect in positions of authority (Curtis et al., 2011; Jeon et al., 2015). Leadership can be viewed as the ability to effectively engage individuals to achieve organizational goals using tasks and oriented behaviors, whereas supervision is viewed more as a role that involves less subtle task focused activities such as communication, team-building, and developing supportive relationships with the supervised staff (Hawkins & Shohet, 2006; Mintzberg, 1998).

The most influential factor in whether workers feel valued and respected at work is their relationship with their nursing supervisors (Hawkins & Shohet, 2006). As the nursing supervisor is often the first workplace contact for educational, instrumental, and moral support of nursing assistants (NAs) in LTCHs (McGilton, 2010) a decision was made to focus on the effectiveness of the supervisory role, as opposed to the leader in the LTCH. An integrative review focused on supervisory effectiveness revealed that nurse supervisors are in pivotal positions to influence the quality of LTCH and resident care (McGilton et al., 2016). Additional evidence suggests that NAs provide better resident-centered care and a higher quality of care when they have supportive supervisors (Ericson-Lidmann et al., 2014). Supervisory nurse performance is related to NAs’ job satisfaction, (Bishop et al., 2009; Choi, 2010), intent to turnover (Bethell et al., 2018), and actual turnover (Chu et al., 2014), which in turn influences quality of resident-centered care (Ericson-Lidman et al., 2014). From a more recent study focused on LTCH that had supervisors who were rated highly supportive (Escrig-Pinol, Corazzini, et al., 2019), results indicate that LTCHs could improve their resident’s quality of life by prioritizing and strengthening the relationships between nurse supervisors and NAs. With such strong evidence for the need of effective supervisors in LTCH, a decision was made by the expert panel to focus on the effectiveness of the supervisory role. Leadership is outside the purview of this paper, as there is WE-THRIVE working group that will focus on the organizational domain related to leadership in LTCHs.

The nurse supervisor in LTCH directly oversees front-line care staff and encourages individual self-reflection and learning from practice; provides on-going education, socioemotional, and instrumental support; and evaluates quality of care and resident outcomes (Orgambídez & Almeida, 2019; Williams & Irvine, 2009; Wong et al., 2013). Accruing evidence suggests that the most effective supervisory behaviors embrace values of empathy and respect (McGilton et al., 2007). Supervisors acknowledging staff personhood has been positively correlated to employee job satisfaction and perception of their work environments, which in turn may decrease average staff turnover rates of 70% to 100% and thus result in improved quality care delivered to residents (Anderson et al., 2004; McGilton, 2010; Noelker et al., 2009). Examples of such activities include active listening to employees’ unique aspirations and interests, as well as establishing policies that assist individuals in balancing their occupational and familial needs with their health and safety (Berta et al., 2018; McGilton et al., 2007; Chu et al., 2016). Although there is consistency in common attributes of supervisors in LTCHs, there is a gap on the best measure of supervisory effectiveness.

To date, several researchers have operationalized effective supervision and created various measures based on these conceptualizations. Researchers have used a variety of different measures to assess the concept of supervisory effectiveness which makes international comparisons difficult to achieve. Supervisors in LTCHs have been identified as having four dimensions to their role: clinical, supervisory, team support, and managerial (Escrig-Piscol, Hempinstall, et al., 2019). To this end, researchers have developed measures that focus on various aspects of these roles. Many measures have focused on the managerial role such as delegation, conflict management, coaching, and intellectual stimulation (Backman et al., 2017; Kiefer et al., 2005; Keisu et al., 1999). Other researchers have focused measurement on the supportive role of the supervisor, being reliable, empathic, with a focus on connecting with their staff (McGilton, 2010; Noelker & Ejaz, 2001).

However, despite the inconsistencies in the measurement of effective supervisors, similar NA outcomes have been realized, most notably, that effective supervisors have influence on NA retention and turnover (Anderson et al., 2003; Kiefer et al., 2005; Tourangeau & McGilton, 2004). These universal results provide evidence that effective supervisors make a difference in LTCHs. However, in terms of selecting a CDE for effective supervision, it would be preferable to have only one measure which has demonstrated feasibility, reliability, and validity, and has the potential to be easily translated into numerous languages to enable the aggregation of data for cross-national comparisons.

The aim of this study is to recommend a CDE to measure supervisory effectiveness of staff working in LTCHs that can be used in international research.

Methods

In terms of selecting the best candidate measure we followed the Best Practices for Identifying Common Data Elements identified by Redeker et al. (2015) which involved: (1) ensuring there is conceptual consistency of a measure, which involved conducting a literature review focused on supervisory effectiveness measures; and the (2) convening a group of WE-THRIVE experts to review the CDE providing them with the measures to review, to rank them and select a measure.

Literature Review on Supervisory Effectiveness Measures

For the first phase of identifying measures of supervisory effectiveness within LTCH, we reviewed the available literature on nursing authority positions and identified scales used to measure them. Concepts identified as appropriate for this review of nursing supervisory effectiveness used search terms “leadership,” “leader,” “lead,” “charge,” “manager,” “nursing,” and “nurse,”. These were further paired with terms “nursing home” and synonyms, “questionnaires,” “psychometry,” and “measurement.” We, conducted the review on June 2019 on Medline, Embase, CINAHL, and HAPI.

Inclusion criteria for selected measures included publications in English, possibility for or previous use in LTC/dementia settings, and surveyors as front-line workers (e.g. registered nurses, registered practical nurses, unregulated NAs) and/or LTC residents and their relatives. Exclusion criteria included use of open-ended questions, community, acute, and sub-acute/rehabilitation care settings, supervisors other than nurses, and nursing students as NAs. The decision to focus on LTCHs as opposed to community was based on the rapidly growing nature of these LTCHs that is projected to eclipse all other health sectors growth (York & MacAlister, 2015).

Ranking Process and Selection of Candidate Measures

Two reviewers independently analyzed the literature review and by using the inclusion and exclusion criteria, eleven measures were identified (n = 11 measures). Two measures were removed retroactively, given they were not financially accessible to LTCHs in countries of low-income and thus did not complement the WE-THRIVE values. The nine identified measures from the literature review were listed into an Excel workbook containing an evaluation grid for each measure. The table containing the measures were then sent out to five WE-THRIVE steering committee members to review and offer reasoning to support their choices. Members were provided with information about each measure and published studies that had utilized them. The members were then asked to individually review the remaining measures (n = 9) and rank according to those being most appropriately suited for international use in LTCH. Several aspects were considered when evaluating the measures, such as conceptual consistency of the measure, the number of items of the scales, the relevance of scale items, accessibility of the measure in different languages, demonstrated relationships of the measure to outcomes, and the psychometric properties of the scales. Lastly, members’ rankings were influenced by their related academic knowledge and/or experience in this area of nursing research, and in analysis of how well the measures operationalize the concept of supervisory effectiveness.

Consulting participants in this review were asked to rank measures they believed to be the best as “1,” and least appropriate as “9” within the table. Once the reviewers had submitted their rankings, measures were evaluated by averaging members’ respective scores. The lowest scores represented the preferred measures for evaluating supervisory effectiveness.

Results

The search generated nine measures that were reviewed by the experts and all of them were self-report measures asking NAs to rate their primary supervisors’ behaviors. For the majority of the measures, supervisory effectiveness was measured with a subscale that was a part of a larger questionnaire that focused on measuring work environment and performance, hence the rating of the supervisors behaviors were not the primary focus. Of the nine measures identified, all had good psychometric properties, and all have published results on the positive influence of supervisors on NA outcomes. In terms of common conceptualizations of the measures, seven of the supervisory scales were primarily focused on the management skills of the supervisor such as delegation and conflict management, while deemed important by the experts, were not deemed as essential as the supportive skills of the supervisor.

The two highest ranking measures were developed exclusively to measure the supportive skills of the supervisory, the Benjamin Rose Relationship with Supervisor Scale (ranked 1.8, the highest) (Noelker & Ejaz, 2001) and the Supportive Supervisory Scale (ranked 2.4, the second highest) (McGilton, 2010). They both had a focus on the relational aspects of their role; listening, being empathic, reliable, respectful and recognizing staff for their contributions which experts perceived as essential qualities for supervisors in LTCHs. Both scales were ranked highest in terms of usability, good psychometric properties, relevant conceptualizations of supervisory behaviors in LTCHs, demonstrated relevancy to outcomes, short time to complete, and easily translatable. The proposed measures are discussed in detail below. All of the other measures considered in this review and their rankings can be found in Appendix 1.

Appendix 1.

Score	Literature review source of measure identification	Brief description of measure
3.2	Learn, Empower, Achieve, Produce Survey—Leadership Behaviors Subscale (Temkin-Greener et al., 2009)	The Leadership subscale of this survey contains 10 items of a total of 34 utilized in the instrument, which includes four other subscales that regard job satisfaction, organizational climate, work empowerment, and work effectiveness (Temkin-Greener et al., 2009)Items of this subscale explore a supervisor’s ability to inform, mentor, motivate, and support colleagues, resolve conflicts, clarify roles of the team, as well as consult and delegate in the respective work environment. This subscale has an acceptable internal consistency of 0.94 and has demonstrated very good reliability and validity elsewhere (Kiefer et al., 2005; Temkin-Greener et al., 2009)
6.6	Measuring work environment and performance in nursing homes (Temkin-Greener et al., 2009)	A three-dimensional measure developed to evaluate LTC setting nature and perceptions regarding work effectiveness (Temkin-Greener et al., 2009). Components of the 56-item total scale include a five-dimension scale regarding work environment and performance; a 12-item scale from the LEAP Leadership Behaviors subscale regarding management style and openness of supervisors to input from staff; and lastly, a questionnaire examining demographics, job details and design, and prior work experience (Temkin-Greener et al., 2009). Themes touched on by this measure are that of leadership, coordination and communication, conflict management, cohesion of the work group, and perceived work effectiveness.
	Direct Care Staff Survey Instrument
7.8	Measuring long-term care work: A guide to selected instruments to examine direct care worker experiences and outcomes. (Kiefer et al., 2005)	A self-administered, 36-item and 11-subscale measure developed in an effort to measure physical and psychological strain in a variety of occupational environments (Marshall et al., 1991). Of the 11 subscales, five are concern factors, including: staff overload, dead-end job, hazard exposure, supervision, and discrimination in the workplace (Kiefer et al., 2005). Six reward factors exist: helping others, decision authority, challenge, supervisor support, recognition, and satisfaction with salary (Kiefer et al., 2005). This instrument is validated through confirmatory factor analysis. Scores are calculated per subscale by averaging 4-point Likert-type scale values. Interpretation of these values depends on the subscale: low values in concern factor subscales represent poorer outcomes from the workplace; high values in reward factor subscales are correlated with poorer outcomes related to stresses outside of the occupational environment (Kiefer et al., 2005)
	Job Role Quality Questionnaire, Supervision Subscales
6.0	Characteristics of highly rated leadership in nursing homes using item response theory (Backman et al., 2017)	A measure designed and tested in 13 countries to identify highly rated leadership features (Ekvall & Arvonen, 1994). With a 6-point Likert-type scale ranging from “completely disagree” to “completely agree,” completion of items can lead to a score range of 24 to 144. Higher scores represent the most effective leadership behaviors to bring about a more efficient and effective staff (Ekvall & Arvonen, 1994). This measure has been demonstrated to reveal the five-most highly-rated supervisory characteristics in a LTC context. They are as follows: willingness to experiment with new ideas, coaching and provision of direct feedback, controlling work closely, ability to rely on subordinates, and handling conflicts in a constructive fashion (Backman et al., 2017)
	Leadership Behavior Questionnaire©
8.6 (omitted due to cost)	Turnover of regulated nurses in long-term care facilities (Chu et al., 2014)	Initially developed for educational use, this scale is a 30-item measure consisting of five leadership practices. Characteristic of the transformational leadership style, it tests behaviors associated with modeling the way, inspiring a shared vision, challenging the process, enabling others to act, and encouraging the heart. Scoring of this measure is completed on a 10-point Likert-type scale with options ranging from “almost never” to “almost always.” An abridging version is available, which has been used in research regarding nursing turnover in LTC settings (Chu et al., 2014; Tourangeau & McGilton, 2004). Further research is required to compile psychometric properties of this scale in nursing contexts (Tourangeau & McGilton, 2004). Must be purchased for use
	Leadership Practices Inventory: (Posner & Kouzes, 1994)
7.2	Nursing homes as complex adaptive systems: Relationship between management practice and resident outcomes (Anderson et al., 2003)	The index developed by Roberts and O’Reilly (1974) consists of 35 items that cover 12 subscales. Such subscales are as follows: trust, influence, mobility, desire for interaction, directionality (upward, downward and lateral), accuracy, summarization, gatekeeping, overload, and satisfaction (Roberts & O’Reilly, 1974). Items are scored on a 7-point scale ranging from “definitely false” to “definitely true” (Anderson et al., 2003). This measure has been used in a variety of occupational settings, including LTC (Anderson et al., 2003)
	Perceptions of management practices(subscales from Roberts & O’Reilly, 1974)
9.0 (omitted due to cost)	Employee effort-reward balance and first-level manager transformational leadership within elderly care (Keisu et al., 1999).	A 45-item measure that explores transformational, transactional and passive avoidant leadership styles (Avolio & Bass, 2002). It is offered in 30+ languages. The six factors examined in this instrument include charisma, individualized consideration of followers, contingent reward, intellectual stimulation, active management, and passive-avoidant leadership (Keisu et al., 1999). A 5-point Likert-style scale is utilized for item rating, with the lowest number representing “not at all,” and the highest number representing “frequently, if not always.” Must be purchased for use (Avolio & Bass, 2002)
	Multifactor Leadership Questionnaire (5x-short)
6.2	Toward a mediation model for nurses’ well-being and psychological distress effects of quality of leadership and social support at work (Van der Heijden et al., 2017)	A cross-sectional survey created to explore the psychological distress of nurses related to the quality of social support and leadership in their occupational workplace, inclusive of LTC settings (Van der Heijden et al., 2017). Nine subscales comprise of a total of 51 items. Five subscales regard the dependent variables of job satisfaction, worker satisfaction with salary, positive affectivity, burnout, and negative affectivity (Kristensen, 2000; Van der Heijden et al., 2009; Van der Heijden et al., 2017; Watson et al., 1988). Three subscales examine the independent variables of the quality of nursing leadership, social support from supervisor, and social support from colleagues (Kristensen, 2000; Van Der Heijden et al., 2009, 2017). One mediator exists as well for the factor of over-commitment (Siegrist, 1996;Van der Heijden et al., 2017). Likert-type scales are used for measuring each item, with the number of scale points varying between 4 and 10, dependent on the subscale.
	Social support from immediate supervisor (Van der Heijden et al., 2009)
7.2	Leadership, staffing and quality of care in nursing homes (Havig et al., 2011)Leadership style was measured by a scale (Yukl Northouse and Bass & Stogdill)	A self-report questionnaire designed to examine the effects of task- and relationship-oriented leadership, staffing numbers and ratio of licensed/unlicensed nursing staff on the structure, process and outcome quality of care of LTC residents (Havig et al., 2011). Leadership style was measured by an index based off of selected items of other leadership resources (Bass & Stogdill, 1990; Northouse, 2001; Yukl, 2006). Item number varied by those surveyed: nine, eight, and seven items were completed by LTC staff, resident relatives, and field observers, respectively (Havig et al., 2011). All items were measured on a 7-point Likert-type scale ranging from “strongly disagree” to “strongly agree” (Havig et al., 2011)

Benjamin Rose Relationship with Supervisor Scale

The Benjamin Rose Relationship with Supervisor Scale is a measure designed by the Margaret Blenkner Research Institute that is utilized to determine the perceptions of NAs relationships with their supervisor (Noelker & Ejaz, 2001). An 11-item scale, it evaluates aspects such as provision of recognition for contributions, quality of communication, active listening, respect of abilities, as well as the practice of empathy via a 3-point Likert-style scale (Noelker & Ejaz, 2001). Options for prompts regarding the participants’ supervisor include “most of the time,” “some of the time,” and “hardly ever/never.” This scale’s score ranges between 0 and 22, with scores on the highest end indicating the most positive reviews of participants’ supervisors (Kiefer et al., 2005; Noelker & Ejaz, 2001). Outcomes with use of this scale have demonstrated that higher supportive relationships between NAs and their direct supervisors result in both improved relations with collegial staff and job satisfaction (Kiefer et al., 2005). Psychometric properties have been established with use in LTC over 10 years, with an acceptable internal consistency and alpha co-efficient of 0.90 (Kiefer et al., 2005). Permission must be granted prior to use. As of the time of publishing, this scale is only offered in English.

Supervisory Support Scale

The Supervisory Support Scale (McGilton, 2010) is based on the definition of a supportive supervisor as an individual who is dependable, empathetic, and nurtures the personal side of the relationship with his or her staff. A 15-item scale was thus refined, with five items operationalizing each domain. A 5-point Likert-style scale was used to measure the supervisors’ behaviors. The response options included “always,” “often,” “occasionally,” “seldom,” and “never.” Content validity of the 15-item instrument was established by a panel of experts. Psychometric properties were established with a sample of 75 staff in three LTCHs, with an internal consistency of 0.94 and test-retest score of 0.70 (McGilton, 2010). The supportive supervisor scale is a summated rating scale out of a possible 75, with higher scores indicating more positively perceived relationships (McGilton, 2010).

The experts felt that either the Benjamin Rose Relationship with Supervisor Scale or the Supervisory Support Scale could be used as a CDE, to measure supervisory effectiveness. Since the ranking of the scales with the expert was conducted, the Supervisory Support Scale has been translated into Spanish (Alconada-Romero et al., 2020) and has been used in 37 LTCH in Spain demonstrating a significant positive relationship with job satisfaction of 394 NAs in Spain (McGilton et al., 2020). The scale has also been recently translated into Chinese (Tian et al., 2020) and validated with 300 NAs in four LTCHs and has established good psychometric properties which makes the Supervisory Support scale more appropriate and feasible in different regions of the world for international comparisons.

Discussion

This article presents the process of recommending a CDE for supervisory effectiveness, from the perspective of a consortium of international researchers. With utilization of a systematic approach, a literature review was completed that reflected the interests of the effective supervisors in LTCHs with a focus on their relational skills. A panel of experts independently reviewed and prioritized appropriateness of nine potential CDEs with mindfulness in their potential application to international LTCHs. Two major conceptualizations of the supervisor role were highlighted in the nine measures found; seven measures focused on the managerial role of the supervisor, while two focused on the relational role of the supervisory. The Benjamin Rose Relationship with Supervisor Scale and the Support Supervisory Scale included the following characteristics of a supervisor: attentive listening; provision of credit, recognition, praise and vocalized appreciation in appropriate contexts; encouragement of individual self-growth and fullest application; and receptiveness to staff remarks, concerns, and ideas. As the WE-THRIVE is focused on person centeredness, the experts were more in favor with these two measures of effective supervision which took into account the relationship between the individual NA and supervisor. In terms of making a final recommendation for the CDE, the Supportive Supervisory Scale which has been cross culturally validated and has demonstrated preliminary construct validation was selected as the candidate CDE measure.

Our results highlighted the lack of any international comparison research conducted on the role of the supervisor in LTCHs. Most of the research comes from North America and high-income countries. The questions of whether supervisors in LTCHs internationally are seen as important or whether low socioeconomic countries have supervisors in their homes are worth pointing out. Another concern with this CDE is the power differentials in the supervisor-supervisee relationship and how this hierarchy may skew the supervisee ratings. Rating their supervisors may be perceived as a risk by staff so ensuring anonymity of the data will be essential when collecting this information.

Strengths of this article include the international and multidisciplinary nature of the WE-THRIVE initiative, and the expert-judge review panel of this article spanning various countries such as Canada, Spain, Sweden, Switzerland, and the United Kingdom. The authors of this article with variant academic and experiential backgrounds were able to consider measures in a number of contexts relatable to low- and middle-income countries and agree on proposed CDEs. Further, the systemic approach in which CDEs were identified and captured should strengthen confidence in, and relevance of, the proposed CDEs for international studies in this field.

While we were able to recommend a candidate CDE measure for supervisory effectiveness, a number of limitations are noted. Firstly, the systematic process of identifying candidate measures was based on a limited number of search terms, and only in the English language. Therefore, there is a risk that all possible measures appropriate for this research may not have been included in the review. Only WE-THRIVE experts ranked the measures but perhaps NAs’ perceptions are required to determine the ranking of measures as they are the target population responding to the survey items. They have the unique ability to assess item relevance and potential realities of completing these measures. Another limitation is that the expert-judge panel of reviewers may not have accurately represented the full lengths to which LTCHs may differ internationally, as the panel did not include representatives from Africa, Asia, Australia, or Southern America even though we made strides to do so. As such, the process of review may not echo all relevant research in the WE-THRIVE concept of supervisor effectiveness, or accurately embrace relevant values of equivalent LTC organizational structures in other contexts. There is a continued need to accentuate research into the nursing supervisor role in low- and middle-income countries, as well as their organizational contexts and professional practice models of LTCH. Our ultimate goal is to test these measures with LTC stakeholders in low-and middle-income countries (Corazzini et al., 2019) to ensure they are applicable to their LTCHs.

Implications for Practice, Policy, and/or Research

This CDE offers the opportunity for capacity building within LTCH, which may in turn promote productive changes to bring about improved perceived outcomes via supervisees. Previous research on the role of the nursing supervisor has articulated the complexity of the role and the lack of preparedness of those in the role (Escrig-Piscol, Hempinstall, et al., 2019). A better understanding of how supervisors effectively work with their NAs across countries will provide a baseline understanding of what is required to inform the development of training programs for supervisors and what factors need to be considered to enhance their role globally. While our experts believe the role of a nursing supervisor exists in most LTCHs internationally the role of supervisor can be held by different health care personnel other than nurses; however, the main qualities espoused by the supervisory scales selected should be universal to all supervisors in LTCH, but further investigation is warranted to support this claim.

In summary, the recommended CDE for supervisory effectiveness is the Supportive Supervisory Scale. With the use of CDEs generated by WE-THRIVE members, comparison between various settings will become possible and persuasive. With the ability to compare and contrast research data and studies, areas with optimal development and LTC work can be identified. Further work is needed to explore the usefulness of the WE-THRIVE CDEs from the perspective of LTC providers and also, other low- and middle-income countries in non-English speaking countries.