NUCare: Advancing Research on Technological Integration for Self-management in the Aging Population

Structured Abstract

Background

The Center for Technology in Support of Self-Management and Health (NUCare) is an exploratory research center funded by the National Institute of Nursing Research’s P20 mechanism positioned to conduct rigorous research on the integration of technology in the self-management of the older adult population.

Purpose

The purpose of this paper is to describe the development and application of an evaluation plan and preliminary evaluation results from the first year of implementation.

Method

This evaluation plan is derived from and consistent with Dorsey et al.’s (2014) logic model. Dorsey’s model provided guidelines for evaluating sustainability, leveraging of resources, and interdisciplinary collaboration within the Center.

Discussion

Preliminary results and strategies for addressing findings from the first year of evaluation are discussed. A secondary aim of this paper is to showcase the relevance of this center to the advancement and maintenance of health in the aging population.

INTRODUCTION

As the population of older adults in the United States continues to grow, so does the need to harness the power of technology to support the development of methods for addressing self-management among ethnically diverse older adults. The integration of technology with self-management among the aging population is critical to the health and independence of this population and nursing leadership is essential to this effort. The older adult population is rapidly growing and diversifying, with the subpopulation of Americans aged 65 and older expected to reach 88.5 million, with a record 18 million aged 85 years and older by the year 2050 [1, 2]. In this time, the Hispanic population is projected to nearly double and estimates are similar for the American Indian and Alaskan Native populations. More specifically, the Hispanic population aged 85 years and older are expected to increase by 15 percent and the non-Hispanic population is projected to triple to approximately 16.2 million by 2050. [2,3] Additionally, many older adults struggle to support themselves with an estimated 2.4 million older adults classified as “nearly poor”, and this number is projected to increase rapidly over the next several years. [4]

Many older adults prefer to age in their home rather than in institutional settings such as nursing homes or assisted living facilities. [5] Achieving this goal requires access to critical supportive resources that are difficult to access for lower income urban older adults, underscoring the urgency to develop accessible assistance. Emerging wearable and other technologies are well positioned to help improve quality of life and allow more people to realize their desire to age in place by supporting self-management and the improvement of health and independence. [7, 8] However, the utility and sustainability of new devices in diverse populations of older adults is an identified gap in the literature. [9]

Given their unique position to provide critical insight and expertise on the needs of specific patient populations, nurses are an essential asset in the development of new technology to support self-management in diverse and under-served older adult populations. The involvement of nurses in this work must move beyond limiting their role to that of key informants or gate keepers and instead work intentionally to support the growth and development of nurse research scientists. This paper describes the development and evaluation of an innovative P20 center developed at Northeastern University’s School of Nursing that offers a training and mentoring model to develop a cadre of nurse science leaders who will make significant contributions to the fields of self-management, technology, and aging. The center supports collaboration between faculty in nursing and computer and information science toward the development of new technologies for addressing issues of aging in diverse urban populations. In order to monitor the progress of this innovative effort, a detailed program theory and evaluation plan were developed. The development of these tools and results from the first year of the evaluation plan implementation are described here. The documentation and dissemination of this process will help other institutions develop and evaluate similar efforts to address urgent issues in health.

Overview of NUCare Center

The Center for Technology in Support of Self-Management and Health (NUCare) is an exploratory research center funded by the National Institute of Nursing Research (NINR) created to advance the science of self-management for older ethnically diverse adults. The overall goal of NUCare is to facilitate quality nursing research in the integration of the state of the art technology with self-management for the aging urban population. Understanding that evaluation is a crucial component in the success of the center, NUCare developed a detailed evaluation plan to measure process and outcome progress. After describing the structure of the NUCare center, the development of the evaluation plan and its first year of implementation are described.

The mission of NINR includes supporting and conducting “clinical and basic research and research training on health and illness across the lifespan to build the scientific foundation for clinical practice, prevent disease and disability, manage and eliminate symptoms caused by illness, and improve palliative and end-of-life care.” [10] The institute’s strategic plan includes harnessing “the strengths of nursing science to power an ambitious research agenda that will meet current and future health care needs and anticipate future health challenges and priorities.”[10] In alignment with NINR’s mission and strategic plan, the NUCare center provides a centralized infrastructure that supports the integration of state-of-the-art technology to facilitate the timeliness, scalability, and sustainability of effective self-management interventions developed by nurse scientists.

NUCare’s infrastructure includes the following three cores: The Administrative Core, the Technology and Analysis Core, and the Pilot Projects Core. Each core includes at least three senior faculty members with relevant expertise. NUCare receives guidance from two external advisory boards: The External Advisory Committee (EAC) comprised of leading scientific experts in aging and technology, and the Community Advisory Board (CAB) comprised of community members and stakeholders with an interest in self-management and a particular focus on ethnically diverse underserved aging populations.

NUCare annually releases a call for pilot study proposals for Northeastern University nursing faculty to integrate technology to address self-management and to promote healthy aging among ethnically diverse aging populations. NUCare funds two projects per year and facilitates just-in-time training for nurse scientists throughout their respective project’s lifecycle. The Center faculty provide mentoring and coordinated services in the areas of experimental design, community engagement, technology selection/development, data collection, storage and security, human subject assurance, interface design/usability testing, and methods of analysis.

The Administrative Core provides overall leadership and is responsible for allocating resources, overseeing budget issues and meetings, coordinating relationships between researchers and the community, and conducting evaluations of the activities to ensure the continued viability of the center. The Technology and Analysis Core focuses on mentoring potential and funded nurse investigators who serve as principal investigators (PI) of pilot projects. Pilot projects focus on the areas of design, data sampling, filtering, storage, privacy assurance, and analysis. This core is also responsible for facilitating and monitoring the integration and use of sensor technology, mobile assessment techniques, coaching software, and communications platforms.

The Pilot Projects Core is responsible for soliciting proposals and fostering the development of junior nurse scientists’ investigation of novel technology-driven self-management interventions focused on ethnically diverse and underserved older adults through mentorship and active engagement of interdisciplinary teams. The Pilot Core is also responsible for evaluating the on-going progress of all funded pilots. NUCare funded three pilot projects in its first two years, these include: 1) Patient-centered Outcomes in Communicating Knowledge and Enhancing Transitions (POCKET), 2) Use of a Wellness Coaching Model to Support Successful Aging, and 3) Promoting Self-Management of Oral Health and Chronic Conditions (please see NUCare website for more information on these pilots).

The purpose of this paper is to describe the development of the NUCare Center evaluation plan and report the preliminary results from the first year of its implementation. The evaluation plan includes an overall process and outcome assessment aimed at identifying the extent to which the center is achieving its intended goals and aims. The process evaluation focuses on whether NUCare is operating as planned, while the outcome evaluation focuses on whether NUCare is achieving its intended impacts [11].

METHODS

Program Theory Development

A program theory, also known as a logic model or action plan, is a “set of assumptions about the relationships between the strategy and tactics the program has adopted and the social benefits it is expected to produce.”[11] The program theory serves as a tool to aid in the identification of evaluation questions, organizing research strategies, and interpreting the results of the evaluation. [11] The NUCare program theory was developed through an iterative process in which the NUCare team worked to adapt aspects of the NINR Logic Model for Center Sustainability [12] to the needs and goals of the center. The NINR Logic Model for Center Sustainability was developed through the collaborative efforts of fifteen directors of NINR-funded Center grants who consolidated important suggestions from the 2013 Centers Directors Meeting on “Sustainability, Leveraging Resources, and Collaboration in NINR Centers,” making it an excellent template from which to build upon for the center. The NUCare program theory excludes items from the NINR Logic Model for Center Sustainability that specifically pertain to larger P30 grants and to other NINR Centers with missions that differ widely from the NUCare center. The NUCare program theory adds more explicit detail about the center, specifically highlighting important areas that drove the development of an action-oriented NUCare plan for ongoing evaluation.

The NUCare program theory is illustrated in Figure 1. The left side of the graphic includes two circles that depict the NUCare center infrastructure (core faculty and advisory boards) and the School of Nursing (SON) faculty. These entities work together through faculty mentoring, coaching, professional development and learning, training, and feedback from CAB and EAC (depicted as inputs) to facilitate the development of the annual pilot projects described in the second box. The boxes on the right side represent expected short- (1–2 year), medium- (2–4 year), and long-term (5 years) outputs that are expected to ensure the sustainability of the center. Short-term outputs include scientific publications and presentations at refereed conferences, the development of new technology or innovative adaptations of existing technology, the production of new grant submissions by nurse scientists, and markers of the development of NUCare’ s infrastructure such as adoption of a business model approach, budget, and cost sharing activities with the university. The medium-term outputs expected during years three and four include continuing scientific publications, presentations, grant submissions and awards, and patenting and licensing of innovative technology. The long-term outputs by year 5 are the submission and awarding of grants from the National Institutes of Health (NIH) beyond NUCare for both individual nurse scientists and the center as a whole.

Figure 1.

The NUCare program theory. CAB, Community Advisory Board; EAC, External Advisory Committee; SON, School of Nursing.

Translating Program Theory into an Evaluation Plan

The NUCare program theory provided the frame of reference for elucidating important areas for ongoing process and outcome evaluation and the development of the detailed NUCare Center Plan for Ongoing Evaluation (Appendix). The evaluation plan explicates how the NUCare Cores’ specific aims are operationalized and evaluated.

The first column of the evaluation plan lists specific aims for each of the NUCare center’s two advisory boards, the center as a whole, and its three core support teams. The second column identifies the methods for evaluating these aims and indicators for evaluating whether each aim has been achieved are listed in the third column. The last two columns include the proposed short, medium, and long-term impacts generated from the achieved aims. Short term impacts are expected to be achieved within the five-year tenure of the grant, while the medium and long-term impacts are expected to be achieved beyond NUCare’s funding period. An important distinction to note is that the NUCare program theory refers to short, medium, and long-term outputs within the life of the grant whereas the evaluation plan refers to short, medium, and long-term impacts that extend beyond the life of NUCare’ s initial five-year project period (2014–2019).

Column two describes methods that are used to determine whether the advisory boards, the center, and the cores are progressing toward meeting their specific aims. The methods include examining meeting minutes, seminar attendance records, feedback from CAB and EAC advisory teams, counting publications, presentations, and grant submissions, examining the number and content of materials disseminated by the center, and measuring faculty perceptions of their experiences with the NUCare center. The process of developing the faculty survey is described in the next section.

Faculty Survey Development

The survey is designed to evaluate NUCare’s broad goals such as increasing department-wide interest in nursing research related to self-management, health disparities among older adult populations, and the incorporation of technology into intervention research; as well as more specific goals such as providing mentorship to individual faculty members with assistance on sensor technology, mobile assessment techniques, coaching software and communications platforms. The survey draws upon guidelines from Dillman, Smyth, and Christian’s text [13] regarding the wording and ordering of survey questions. Skip patterns and branching logic are included to allow for the elicitation of more detailed information from respondents who received pilot funding, such as reporting on their overall experience with NUCare and with mentorship.

The survey begins with broad questions about patterns of participation in NUCare activities including attendance at and feedback on seminars. The survey also includes questions to assess whether or not NUCare related activities have increased faculty interest in developing research that includes underserved older adult populations, self-management approaches and/or the integration of technology. The survey goes on to assess faculty’s current level of comfort with incorporating technology into their research, teaching, and/or clinical practice. These questions allow participants to indicate whether they feel comfortable integrating technology into their research and practice with or without NUCare’s assistance.

To determine their experience seeking or receiving NUCare assistance, faculty were asked to report on whether or not NUCare center core faculty were approachable, and whether they helped address the faculty member’s issues or concerns related to their proposed pilot project. A group of four questions with four-point Likert scale response choices (strongly disagree to strongly agree) were used to assess how nursing faculty perceived NUCare core faculty’s level of responsiveness and quality of mentorship they received.

All nursing faculty were asked to report the number, the funding agency, and focus area of relevant grant proposals submitted and/or funded during NUCare’ s first and second years. Finally, respondents were asked to list three suggestions/areas for improving the activities of the NUCare Center. At the end of NUCare’ s first year, the survey study was approved by the Northeastern University Institutional Review Board, and a live link to the survey was emailed twice over a one-month period to all nursing faculty. The email included an introductory description of the purpose of the survey and invited faculty to participate in the survey anonymously online.

FINDINGS

Preliminary results from the first year of the implementation of the evaluation plan are described here. To facilitate the organization and presentation of these findings the evaluation methods are described in three sections: 1) Preliminary survey results, 2) Meeting minutes and feedback from advisory boards, and 3) Infrastructure development and additional methods of evaluation.

Preliminary Survey Results

Preliminary survey results revealed that NUCare is making progress toward key indicators of success and was able to identify areas where NUCare needs to invest more resources to ensure its goals are achieved. Thirty-seven participants out of 42 total faculty members (88%) completed the survey. Two survey responses were excluded from the analyses because the respondents only answered the first question on the survey.

The survey revealed that NUCare has been successful in improving faculty interest in key areas. The majority of the respondents (26/35) reported that NUCare has increased their interest in incorporating technology into their research, 34% (12/35) reported that NUCare has increased their interest in conducting research among ethnically diverse and underserved aging populations, and nearly half (17/35) of the respondents indicated an increased interest in self-management research. Most of the respondents (28/35) reported that they are likely or very likely to incorporate technology into their research. From these twenty-eight, 61% reported feeling confident that they could use technology in their research with help from NUCare, 7% reported feeling confident without help from NUCare and 32% reported that they were not confident at all about incorporating technology into their research. These results are particularly promising in light of the context of the School of Nursing’s historic focus on education and clinical practice, versus the development of nurse researchers. These results indicate an increased interest among faculty in participating in research that includes technology, and seeking assistance from the center to do so.

The survey also revealed important areas for improvement. NUCare hosts weekly seminars as a key method for increasing interest in technology, self-management, and health disparities in aging research. The seminars feature presentations from researchers both within and outside of Northeastern University. The goal of the seminars is to provide examples of the application of innovative research methods and introduce novel approaches to integrating technology into research. While the seminars were thought to be an excellent way to introduce the utility and accessibility of many forms of technology to a broad audience, the majority of the survey participants (28/35) reported attending three or fewer seminars. Fourteen provided information on why they did not attend the seminars. Nine reported that they had scheduling conflicts which prevented them from attending, two faculty reported that they were not interested in the topics presented and two others stated that they were unaware of the seminars. Qualitative responses revealed that some faculty members felt overwhelmed with their teaching and administrative duties and therefore did not have time to attend the seminars.

Six respondents provided feedback on the mentorship they received from NUCare. This feedback (not presented here to avoid compromising the confidentiality of the few respondents) will be incorporated into the NUCare plans for improving mentoring among nurse scientists at Northeastern University. Finally, faculty were asked to indicate the number of grants that they had submitted and were awarded in the first and second years of the NUCare center and how many of the ones they submitted had been influenced by the NUCare center. There were not notable differences in any of these measures.

Meeting Minutes and Feedback from Advisory Boards

The EAC and CAB are instrumental to NUCare’s progression towards its goals and specific aims. The EAC meets once per year to review progress, advise on current projects, provide feedback on proposed projects and participate in strategic planning for future activities and infrastructure. The CAB meets three times per year to identify “big questions” in healthy aging, assist with engagement and dissemination of research plans and findings, aid in communications to constituencies, and help with the identification of community partners and gathering formative data. Examination of meeting minutes and a formal feedback report from the EAC, and meeting minutes and brief surveys at the conclusion of each CAB meeting are used to measure process and outcomes of the advisory boards.

Review of the meeting minutes from the EAC and CAB meetings show evidence that the specific aims of these advisory boards are being met. NUCare has consistently scheduled meetings with the advisory boards, presented the work of the center and the pilot projects at each meeting, and received important feedback from the boards about the projects. Review of the minutes show that the EAC provided feedback about the scientific integrity of the work and insight on how to sustain the projects and the center. This feedback was then incorporated into discussions at executive committee meetings where next steps were determined. The CAB, in line with its aims, provided feedback on perspectives from the community about the big questions in aging and advises on the relevance of the research for specific sub groups of community members.

Review of the meeting minutes from the CAB show that varying attendance, ranging from as few as two community members/stakeholders to as many as eight is a critical area for improvement. In order to address this concern, the center planned to establish a community chair who, as a member of the community, can help to recruit others, set an agenda that is most relevant for the community, and help find most appropriate methods for disseminating findings to the community.

Infrastructure Development and Additional Methods of Evaluation

Additional methods in the NUCare evaluation plan included the examination of yearly reports, the number of grant submissions and awards and the number of scientific publications and presentations for evidence that the center is building a sustainable infrastructure. The faculty survey described above includes questions about numbers of grants submitted and funded on material relevant to the center. It is important to note that there are a number of grant submissions, scientific presentations and publications authored by the center’s core faculty that are not captured in the survey, as core faculty were not considered as participants. Review of the yearly report confirms that the center has been successful in managing its budget and has achieved its goal of funding two pilot projects per year. In year 2, core faculty had submitted two grant submissions, two publications, and two formal presentations. At least two additional relevant grant submissions were planned for following summer.

Infrastructure development was assessed through an annual review of the additional resources provided by the center including seminars, trainings, and website resources. Over two years, NUCare hosted nearly 40 seminars, with speakers from nursing, health sciences, computer science, engineering, epidemiology, and beyond. The NUCare website was continuously updated and improved to include videos of previous seminars, sample use cases and personas, and guidance on integrating technology into research. Nursing faculty members provided feedback on website materials that identified gaps in training. Additional training materials on data analysis are currently in development. The NUCare health coaching platform continues to be developed, with five modules near completion (oral health, cognitive games, interactive video exercise, socialization, and sleep) and two modules to begin development shortly (hypertension management and tobacco cessation through stress management). Core faculty members plan to demonstrate these modules at regularly scheduled nursing faculty meetings, which is thought to be a more effective means of engagement with the nursing faculty. Core faculty attend these meetings to encourage the development of pilot projects that utilize existing modules. Northeastern University as a whole has enthusiastically supported the NUCare center. The two collaborating colleges, the Bouvé College of Health Science and the College of Computer and Information Science, each contribute funds on an annual basis to a cost sharing account that ensures that the Core faculty can allocate a significant portion of their research efforts to NUCare.

The ability of the center to support the development of research scientists is exemplified in the productivity of the pilot project PIs and extends beyond nursing faculty to the development of doctoral student researchers. Three doctoral students were supported in the first two years of the NUCare Center. These students gained valuable research and development experience by supporting specific pilot projects as well as the core faculty. Specific examples of products of this mentorship include the program theory and evaluation plan, which were developed by a doctoral student under the guidance of the core faculty. In addition to those working directly on center projects, about fifteen students representing the School of Nursing and the Computer and Information Science attended weekly seminars in which speakers presented their experiences in the processes, challenges, and benefits of state-of-the-art technology development.

Impact of pilot Projects

In addition to the training and mentorship activities reported above, the specific pilot projects illustrate the real-world influence of the center on its intended populations. During the evaluation period, two pilot projects were underway: 1. A study to determine the feasibility, usefulness and relevance of a personalized wellness coaching model for promoting physical activity among urban older adult populations and 2. A qualitative investigation of the impact of an integrated care model for patient and family engagement on improving outcomes in transitions between hospital and home.

The first study included a sample of primarily African American and Latino older adults residing in one of two subsidized housing sites. Fifteen participants (5 females and 10 males, mean age 77.4 ± 7.9 years) were assigned to the intervention group and 13 older adults (7 females and 6 males, mean age 75.8 ± 9.1 years) from another site comprised the wait-list control group, crossing over to the intervention after six months. InterRAI’s Health & Social Check-up and the Lifestyle Survey were the primary data collection tools and participants also completed the Patient Activation Measure (PAM). Independent t-tests revealed statistically significant (p < .05) improvement in physical activity and self-reported health status and quality of life for the group receiving the intervention for one year. These urban-dwelling, low-income older adults were able to identify their goals and, through a wellness coaching process, engage in the development of their personal healthy aging plan, facilitating their involvement in health care self-management. The results from this pilot project suggest that capitalizing on social networks and other low or no-cost options to support goal achievement may be a key strategy in resource-restricted environments. Notably, the increase in physical activity level was an important outcome that was supported by improvements in quality of life and self-reported health.

The second pilot included focus groups and key informant interviews of participants, staff and caregivers from an innovative health care model that provides high-quality care to nursing home-eligible seniors with chronic care needs within their own communities. The study explored experiences in care transitions, including facilitators and barriers to effective care, and health care utilization based on rates of hospitalizations, emergency department and skilled nursing facility transfers, and time to nursing home placement. The results informed recommendations related to enhancing care transitions, information exchange, and the use of technology in self-care/self-management of complex, dually eligible older adults.

DISCUSSION AND RECOMMENDATIONS

The development of the NUCare Program Theory and the NUCare Plan for Ongoing Evaluation have allowed for the systematic assessment of process and outcomes related to NUCare’s goals and specific aims. Results from the evaluation are encouraging with much evidence that the center is meeting its goals and moving toward sustainability. The purpose of the ongoing evaluation is also to identify areas for improvement and the evaluation plan proved useful in achieving this goal.

The faculty survey revealed the promising result that many nursing faculty members have an increased interest in self-management, technology, and health disparities among aging populations. Importantly, the evaluation also revealed that faculty members would like to incorporate technology into their research, practice, and teaching, that many were aware of the services offered by NUCare and that they felt these services would be helpful to them in achieving their goals. The survey also identified important areas for improvement, as exemplified by the faculty’s low attendance rates at NUCare seminars.

The results of this evaluation should be interpreted in light of its limitations. The methods were designed to track progress over time in the development of nursing faculty, but did not include a control group from which to compare the successes and challenges of the NUCare center. Additionally, due to the small number of faculty at the School of Nursing, it was not possible to share detailed information on participants or responses on the qualitative portions of the faculty survey without compromising respondents’ confidentiality.

By identifying areas for improvement, the evaluation plan has allowed NUCare to strategize ways to respond. For example, the NUCare core faculty initiated new strategies to improve faculty engagement with the NUCare center. These strategies include increasing NUCare presence at regularly scheduled faculty meetings (presenting useful technology and examples of ways to integrate specific technology into research), inviting the faculty to the technology lab for one of the monthly meetings, and convening a faculty work group with the goal of providing space and support for nursing faculty to brainstorm research ideas and ways to integrate technology into their research. Additionally, in response to fluctuating attendance at CAB meetings, NUCare has taken steps to establish a community chair to help recruit additional members and improve the center’s communication with the community.

CONCLUSIONS

Organizational improvement expert H. James Harrington has stated that evaluation “is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.” [14] Developing and implementing this evaluation plan has been instrumental in generating new strategies and approaches to improve NUCare through a better understanding of the challenges and opportunities of integrating the Center’s goals and objectives into the culture of the School of Nursing in the Bouvé College of Health Sciences and ensuring that the Center achieves its goal of increasing the number of nurse scientists engaged in research aimed at integrating technology and self-management to advance, maintain, and sustain health in the aging population at NU. NUCare’s Plan of Ongoing Evaluation will enable the NUCare core faculty and administration to take steps toward ensuring the effectiveness, growth, productivity, and sustainability of the Center.

NUCare represents an approach to collaboration between nursing, computer, and other health sciences to improve self-management in under-served older adult populations through the use of technology. The work of the NUCare center serves as evidence of the importance of high quality mentorship for the development of junior faculty and doctoral students. The center builds upon the unique expertise of nurses who provide community-based care to vulnerable older adults, recognizing the cost benefit of supporting aging in place. [15] The results of the NUCare evaluation can be a guide for other institutions who plan on developing similar centers. More specifically, these results highlight specific challenges and assets that can be expected and offer a template for program evaluation.

Highlights.

• This paper discusses the successful adaptation and application of an existing logic model (Dorsey et al, 2014) to develop a program theory and evaluation plan for an NINR-funded exploratory research center.

• This paper shares preliminary findings from the first year of implementation and discusses strategies for addressing these findings.

• The intent of the paper is to serve as an exemplar for other P20 centers’ evaluation.

• A secondary aim of this paper is to showcase the relevance of this center to the development of the next generation of nurse scientists.

APPENDIX - attached PDF of Evaluation plan

Appendix Figure 1.