Development and testing of a measure designed to assess the quality of care transitions

Abstract

Background

To improve the quality of care delivered to older persons receiving care across multiple settings, interventions are needed. However, the absence of a patient-centred measure specifically designed to assess this care has constrained innovation.

Objective

To develop a rigorously designed and tested measure, the Care Transition Measure (CTM).

Setting

A large, integrated managed care organisation in Colorado with approximately 55,000 members over the age of 65 years.

Participants

Patients 65 years and older who were recently discharged from hospital and received subsequent skilled nursing care in a facility or in the home.

Methods

Six focus groups of older persons and their caregivers (n=49) were established. Standard qualitative analytic techniques were applied to written transcripts and four key domains were identified: (1) information transfer; (2) patient and caregiver preparation; (3) self-management support; and (4) empowerment to assert preferences. Specific CTM items were developed, pilot tested, and refined. Psychometric testing, conducted in a different population but selected using the same entry criteria (n=60), included content and construct validity, intra-item variation, and floor/ceiling properties.

Results

Older patients and clinicians found the measure to be highly relevant and comprehensive (i.e. content validity). Construct validity was assessed by comparing items from the CTM to selected items from a measure developed by Hendriks and colleagues (Medical Care 2001; 39(3): 270–283). Inter-item Spearman correlations ranged 0.388–0.594. No significant floor or ceiling effects were detected.

Conclusions

The CTM was developed with substantial input from older patients and their caregivers. Psychometric testing suggested that the measure was valid. The CTM may serve to fill an important gap in health system performance evaluation by measuring the quality of care delivered across settings.

Introduction

During an episode of illness, patients may require care from different practitioners in multiple settings, placing them at risk from fragmented care. This is important for elderly patients in particular as they often suffer from chronic illnesses. For example, in a given month, the same individual may receive care from his or her primary care physician or a specialist in the ambulatory setting, a hospital physician and nursing team during an inpatient admission, a different physician and nursing team during a brief stay in a skilled nursing facility, and finally, from a visiting nurse in the home.

During an episode of illness, care may become fragmentated when different components of a health delivery system (e.g. different professionals and different institutions) work independently of one another, or worse, at cross purposes [1, 2]. When multiple providers operate independently with no single plan of care, older patients in particular may be adversely affected. This fragmented care can result in conflicting recommendations regarding chronic disease self-management, confusing medication regimens with a high potential for error and duplication, inadequate follow-up care, and inadequate patient and caregiver preparation to receive care at the next healthcare setting [3, 4]. Families and informal caregivers can become frustrated and overwhelmed. Furthermore, care fragmentation may lead to greater hospital and emergency utilisation, with their associated higher costs of care [5–7].

To reduce care fragmentation across settings of geriatric care, effective interventions are needed. However, this line of inquiry is severely constrained by the absence of a validated measure designed to capture the essential attributes of successful care transitions. The closest approximations to a measure of care transition—measures of care co-ordination [8, 9] and continuity [10–12]—are limited by their exclusive focus on the care delivered by the primary care physician. Such measures do not inquire about the care received in other settings, nor do they emphasise the extent to which care is integrated across settings. Quality assurance and satisfaction measures have traditionally addressed the care delivered at a particular site of care such as the hospital, nursing home, or primary care clinic [13–17]. Other measures have been designed to assess care fragmentation from the perspective of health systems [18]. While system measures may be informative to health planners and administrators, they are much less relevant to the care experienced by older patients and their caregivers.

Nevertheless, older patients and their caregivers do not experience their care in discrete episodes according to the arbitrary divisions of the healthcare system. Rather, they experience their care in a continuous manner irrespective of the particular setting care in which it is delivered. Because they are the only common thread weaving across the health care continuum, the management of care across healthcare settings has, by default, become the responsibility of older patients and their caregivers. To accomplish this task, they need to be adequately prepared for their course of treatment, understand how they can positively influence the management of their illness, know who to contact when they need answers to health-related questions, and be encouraged to assert themselves when their needs are not being properly met. As important as these domains are to the lives of older patients, they are not found in existing measures.

In order to assess the effectiveness of interventions designed to improve care transitions, a validated measure needs to be developed that captures the essential domains of successful care transitions. The origins of such a measure should be based on the actual experiences of older persons with chronic illness and their caregivers who have made a transition. The specific aim of this study was to design and test a new measure, the Care Transition Measure (CTM).

Methods

Settings and participants

Kaiser Permanente, Colorado Region, is a large group-model health maintenance organisation that serves approximately 355,000 members in the Denver metropolitan area. Approximately 55,000 of its members are age 65 or older. Kaiser Colorado receives a monthly per capita payment from the Medicare program for each eligible enrolled member. Kaiser Colorado is a partially integrated health care system in that it owns and manages its outpatient facilities, but it contracts with non-Kaiser providers for hospital, skilled nursing and home health care.

A purposeful sampling strategy was used to identify persons who had recently experienced one or more care transitions. This strategy was designed to ensure that there was adequate representation of minorities, women, and persons of lower socio-economic status. Six focus groups were established. Three of the focus groups were held at clinical practices with a high representation of minorities, including Hispanics and African-Americans. Another focus group was held in a clinic setting attended by persons of lower education and socio-economic status, and a further focus group was held in a clinic serving more highly educated individuals. The sixth and last focus group was held in a more suburban clinical setting.

Participants were identified using administrative data. The initial data abstraction included persons over the age of 65 whose primary care physician was selected from one of the six-targeted clinics. Those patients who had been admitted to the acute care hospital at least once and received subsequent skilled nursing care in either a facility or in the home in the past six months were selected. Patients who had died were not considered. Selected patients were telephoned by one of the researchers and invited to voluntarily attend a focus group. Participants were given refreshments but were not paid. Upon attendance at the focus group, each participant was asked to fill out an anonymous demographic data sheet collected for the purpose of characterising the study population. Colorado Multiple Institutional Review Board approved the protocols employed in this study.

Conduct of focus groups

Each focus group was held on-site at the six respective primary care clinics to facilitate transportation and encourage participation (i.e. most patients had an established transportation system in place for accessing the clinic). Caregivers were encouraged to attend. Participants received a reminder to attend via a telephone call the night before. The focus groups took place in the clinic conference rooms and lasted approximately 90 minutes. Attendance ranged from 7 to 10 participants (n=49), in addition to accompanying caregivers.

The purpose of the focus groups was to elicit patient and caregiver perspectives regarding recently experienced care transitions. Standard focus group techniques were employed [19]. Discussion was directed by a series of open-ended questions administered at each session (see Table 1). All six sessions were moderated by two of the researchers (EAC and JDS). Each participant was encouraged to contribute to the discussion and the moderators encouraged group interaction and sharing of experiences. In addition, notes were taken to capture changes in emotion, enthusiasm, and non-verbal communication associated with responses to each question.

Table 1.

Focus group questions

Think back to when you were in the hospital …

– What was most helpful in getting you back home to your normal routine?

– What aspects of your discharge did you feel were handled particularly well? What aspects were not handled well?

– What did you need to meet your care needs after discharge from the hospital?

– Did you feel confident in knowing the questions you needed to ask about the care you were to receive after leaving the hospital and who to ask them to?

– Did you feel that the reasons that brought you into the hospital in the first place were addressed?

– After leaving the hospital, did you feel fearful or anxious? What would have reduced your fears?

– Did you or your family feel that you were prepared to come home?

– Did you receive care in a nursing facility? Did the nurse understand what had brought you into the hospital and what they did for you?

– Did you receive home care from a nurse? Did the nurse understand what had brought you into the hospital and what they did for you?

– When you returned to your primary care physician, did he or she know about your hospitalisation, nursing facility or your home care experience?

Analysis of focus groups

The six focus groups were audio taped and the tapes were converted to written monographs by a single professional transcriptionist. All possible patient identifiers were removed to protect confidentiality.

Standard qualitative analytical techniques were employed [19]. One strategy required that more than one investigator independently analyse the raw data to evaluate key domains that were then reassembled by the group in order to reach consensus [20]. Four members of the research team (EAC, JDS, TBE, and AMK), each with different professional backgrounds reviewed the six transcripts. Each researcher systematically selected recurring themes accompanied by supporting verbatim quotes that he or she believed captured the essential points of the discussions. The relative frequency, intensity, and consistency of the main points guided the selection of the themes. After theoretical saturation and redundancy was independently reached, the researchers subsequently met together with the aim of agreeing unanimously to the creation of the key domains.

Construction of measure

The CTM was developed to assess the quality of care transitions across healthcare settings. Because, by default, older patients largely manage their transfers across care settings, the measure was designed to be patient-centred rather than provider-centred.

Following the selection of key domains, candidate items were drafted using a similar wording and phrasing as was used in the focus groups. Items were arranged in chronological order reflecting the different phases of a typical care transition. The target reading comprehension level was high, corresponding approximately to 6th grade. Because of the large burden of illness in the population undergoing multiple transitions, the measure was designed for telephonic administration and a script was drafted. The timing of the telephonic administration of the measure was approximately three weeks after the transition. Based on the work of Hendriks and colleagues [21], the response format was designed to be brief and simple. Initially, an identical response format was proposed for all of the respective items.

Because we anticipated that not all patients would have the cognitive capacity to complete the new measure, a protocol was used to determine when a proxy respondent was required. By necessity, the cognitive capacity screen had to be brief, non-intrusive, and verifiable. A four-item screen was constructed, which had been adapted from a previously published mental status instrument [22]. The four items assessed short-term memory, long-term memory, and orientation. These items were believed to be directly relevant to the types of cognitive function required to respond to those contained in the proposed measure. Subjects were asked to state their phone number, year of birth, current age, and the season. Subjects who missed one of these items were asked to name an appropriate proxy to assist in formulating their responses to questions regarding their recent health care experiences.

Pilot testing of measure

Once drafted, the CTM was subjected to a series of pilot tests designed to refine its content, wording, and organisation. First, three focus groups were held at different clinical sites with older patients who had recently undergone care transitions. These patients (n=21) were selected using the same approach as that used for the focus groups. No patient participated in more than one focus group during the course of the development of the entire measure. Patients and their caregivers were asked to review each item, interpret its meaning, and offer an opinion as to whether the item addressed an aspect of the care transition that was important to them. Researchers also asked how they might respond to such a question, how the wording could be improved, and whether any important areas had been overlooked. This critique led to the addition, modification, and deletion of multiple items.

In addition, the CTM draft was shared with local and national experts in geriatric health care delivery. A similar critique was offered and further refinements were incorporated.

One important lesson originating from the first two pilot tests was that a single version of the CTM could not adequately account for the different possible transitions that occurred. Pilot test patients found the chronology of events to be confusing. For example, persons who went from a hospital to a skilled nursing facility to home with a home care nurse were uncertain about the time frame of reference for particular items. This prompted the development of three separate versions of the measure: (1) hospital to home; (2) hospital to home with home skilled nursing care; and (3) hospital to skilled nursing facility to home, with or without home skilled nursing care.

For the next pilot test, the measure was administered over the telephone to a population of 20 patients enrolled in an intervention study who had recently undergone care transitions. Among the lessons learned in this pilot stage was that the same Likert-style response format was not suitable to all items. Further refinement led to a response that was more tailored to reflect the nature of the item's content. The duration of telephone interviews ranged from 8 to 12 minutes for the first version (hospital to home) to 13–20 minutes for the third version (hospital to skilled nursing facility to home).

Psychometric testing of measure

Initial psychometric evaluation of the CTM included content validity, construct validity, floor and ceiling effects, and intra-item variation. Content validity is seldom formally tested [23]. Instead, face validity or clinical credibility is commonly ascertained from experts who review the measure for clarity, relevance, and comprehensiveness [24]. This step was completed during the pilot testing as described in the previous section.

Construct validity typically involves comparing the newly developed measure to a “gold standard”. However, as mentioned earlier, currently there is no validated measure designed to assess the quality of care transitions. In the absence of a gold standard, a reasonable alternative is to compare specific items of the newly developed measure to similar items of an existing measure. We compared related items on the newly developed CTM to items included in a measure developed by Hendriks and colleagues from the University of Amsterdam [21]. The Hendriks measure was designed to assess the quality of hospital discharge from the perspective of the patient and includes items pertaining to the process of discharge transfer out of the hospital. Kappa statistics were used to examine correlations between items. A priori, values between 0.25 and 0.75 were hypothesised to represent reasonable inter-item correlation. Levels above 0.75 were considered as indicating that the two measures were so similar that a new measure was not necessary. Levels below 0.25 were considered as indicating that the inter-item correlations were not similar enough to be considered comparable.

Floor and ceiling effects were examined and items for which greater than 70% of responses were found to be at either extreme of the Likert response format were either modified or discarded. Statistical analyses were performed using SAS (Version 6.12; Cary, NC).

Results

The demographic characteristics of the initial focus group participants are provided in Table 2. The age distribution of participants revealed a relatively wide range for an older population. The majority of participants were women, married, and had achieved at least a high school education. Over 87% of participants were white, 11% were Hispanic, and 2% were categorised as “other”. The income distribution also revealed a wide range with a non-response rate of 12%.

Table 2.

Demographics of Focus Group Participants (n=49)

Demographic characteristic	Percent
Age
65–69	16.3
70–74	26.5
75–79	30.6
80–84	22.4
85–89	4.1
Gender
Male	43.8
Female	66.2
Marital Status
Married	69.4
Not-married	10.2
Widowed	20.4
Education
Grade school	2.0
Some high school	8.2
High school graduate	36.7
Some college	26.5
College graduate	12.2
Graduate school	14.3
Race
White	87.8
Hispanic	10.2
Other	2.0
Income
$0–10,000	2.0
$10,001–20,000	24.6
$20,001–30,000	12.2
$30,001–40,000	22.4
$40,001–50,000	8.2
$50,001 or more	18.4
Non response	12.2

Table 3 illustrates the four primary domains that emerged from the focus groups. The four domains arrived at using qualitative methods, included: (1) Information Transfer; (2) Patient and Caregiver Preparation; (3) Support for Self-Management; and (4) Empowerment to Assert Preferences. Table 3 also provides supporting representative quotes and interpretations for each domain. Under the Information Transfer domain, confusion over the medication regimen in particular was often cited as a central problem. With respect to Patient and Caregiver Preparation, participants often described situations where the care plan was formed around the convenience of the health providers and institutions, as opposed to the older patient and caregiver. Regarding Self-Management, participants frequently voiced concerns that lack of access to health care practitioners impaired their ability to manage their own conditions. Finally, concerning the Empowerment to Assert Preferences domain, many of the participants expressed a strong desire to play an active as opposed to a passive role in their care. These participants uniformly encountered subtle and explicit barriers to asserting their preferences or assuming greater control of their transition care.

Table 3.

Four primary domains with representative quotes and interpretations

Information Transfer

Representative Quotes:

“They overmedicated me like you wouldn't believe [in the NH]. All they had to do was make one call to my primary care doctor”

“It was apparent that the [SNF] nurse had not reviewed my hospital records”

Interpretations:

Sites of care operating independently

Poor inter-professional and inter-institutional communication

Initial reason for hospitalisation often overlooked

Patient and Caregiver Preparation

Representative Quotes:

“The doctor did not know that there was no way my wife could take care of me”

“They came in at 6 PM and informed me that the ambulance was waiting to take me to a nursing home”

Interpretations:

Desire to receive as much information ahead of time as possible—written and verbal

Family and caregiver needs often overlooked or expectations for care provision unrealistic

Lack of specific follow-up reduced confidence

Self-Management Support

Representative Quotes:

“We can't get a hold of anybody—all we have is a quick question”

“A lot of times the questions don't come until you get home”

Interpretations:

Often did not know the questions to ask or the person to direct them to

Not being able to get through on phone to obtain answers needed to manage condition

Medications the area of greatest need

Empowerment to Assert Preferences

Representative Quotes:

“You know, we're responsible for our own healthcare and it's our fault if we fall through the cracks”

“They disregard the patient when he may know full well what is best because he has been through it”

Interpretations:

Patient contribution to care plan not taken seriously

Need for an advocate

SNF staff's lack of empowerment a barrier

The results of formal construct validity testing are provided in Table 3. Selected items from the CTM were compared with selected items from the satisfaction measure developed by Hendriks and colleagues [21]. Spearman inter-item correlation between 0.250 and 0.750 represented the a priori test criteria for a positive construct validity test. For each of the six items compared, the correlation coefficients were within the range pre-designated as acceptable Table 4.

Table 4.

Construct validity testing

Hendriks et al. measure item [21]	Care transition measure item	Spearman inter-item correlation
“What is your opinion of the clarity of information given by the nurses (e.g. about your illness, medication, treatment, laboratory tests, and outcomes)?”	“When you left the hospital or rehab centre/nursing home, did you have a patient discharge form?”	0.594
“What is your opinion of the timing of your discharge from hospital treatment?”	“When you left the hospital, did you think you were discharged…?”	0.554
	– earlier than expected
	– at the perfect time
	– 3. later than you should have been
“What is your opinion of your exit interview by the nurse upon discharge?”	“When you left the hospital, how knowledgeable were you about what to do if your condition got worse?”	0.492
“What is your opinion of the information provided regarding further treatment (e.g. diet, working and resting hours, devices, medications)?”	“When you left the hospital, how knowledgeable were you about managing your condition? For example, the warning signs to watch out for or changing some of your health-related habits?”	0.439
“What is your opinion of the amount of information given (e.g. about your illness, medication, treatment, laboratory tests and outcomes?”	“While you were in the hospital, were you able to get all the information you needed from the doctors and nurses?”	0.534
“What is your opinion of the way information was transferred from one person to another person?”	“During that first home health care visit, did the nurse have the information she/he needed to take care of you (e.g. about your hospital stay and medications)?”	0.388

Discussion

The absence of a patient-centred measure created specifically to assess the quality of care transitions has constrained the advancement of innovative interventions that promote greater care integration. We have designed and tested a new measure to fill this critical void. The CTM was guided by the concerns and insights of older patients and their caregivers who have recently experienced these challenges first hand. The face validity and comprehensiveness of the CTM was well received by both patients and clinicians. To our knowledge, there is no similar transition measure to the CTM to facilitate a direct comparison. We selected items from the satisfaction measure developed by Hendriks and colleagues because it contained a number of parallel items concerning transfer from the acute hospital setting [21]. We had hypothesised that the CTM would have reasonable construct validity if inter-item correlation to the measure of Hendriks et al. was between 0.250 and 0.750. This was indeed confirmed. Furthermore, no significant floor or ceiling effects were exhibited in the population studied. Thus, based on broad patient and clinical input and on acceptable psychometric properties, the CTM may serve to fill an important gap in health system performance evaluation by measuring the quality of care delivered across settings.

Because there have been a paucity of studies in the literature examining the quality of care transitions from the patients' perspective, it is difficult to make many direct comparisons. Levine conducted a series of six focus groups in New York comprised of caregivers to patients who had recently undergone transitions. This analysis identified similar domains, including lack of caregiver preparation, communication barriers, discontinuity, and a lack of empowerment [25]. There are small but important areas of overlap between CTM items and widely used patient satisfaction measures. For example, the Consumer Assessment of Health Plans Study (CAHPS^®) Survey asks patients about whether they have felt involved with their health care decisions, which is similar to the CTM item on empowerment to assert preferences [26]. The Picker Institute Survey primarily focuses on the hospital discharge experience as opposed to care received thereafter. However, the Picker instrument includes items that are relevant to the CTM domains of information transfer, expressing treatment preferences, and self-management [17].

There are multiple strengths of approach used to develop CTM. First and foremost, the measure is based on the actual experiences, both positive and negative, of older patients and their caregivers. This is in contrast to measures developed from the perspective of health care providers or systems of care. At each step in the development of the CTM, the insights and reactions of older patients served as the primary guide. Second, the CTM domains represent more than just a process of care measure. These domains reflect less tangible yet critical components of transitions that include fear and anxiety, empowerment, and caregiver support. It was clear from the focus groups that if the patient's preferences and goals were not accounted for in the formulation of the care plan, adherence was unlikely. Third, the CTM was developed using rigorous methods, including both qualitative and quantitative approaches.

However, our findings need to be interpreted in light of several limitations. This study was conducted within a single health plan. Although the health plan has a large population of enrolled older adults (approximately 55,000), it is possible that those older persons who chose to become members of the health plan may not be representative of the population as a whole. The demographic characteristics of the patients who participated in the focus groups mirrored the enrolled population of the health plan as a whole. Compared to national demographic data for Americans age 65 years and older derived from the 2000 United States Census, the focus group population had a higher proportion of women (66 vs. 56%), a higher proportion educated beyond high school (53 vs. 34%), a lower proportion of married individuals (69 vs. 73%), and a similar proportion of non-white individuals (12 vs. 10%) respectively. Second, we made a deliberate attempt to over-sample persons of diverse racial backgrounds. We selected clinics and practitioners that cared for a disproportionate number of minority members. Despite our efforts, our population was approximately 87% white. Third, the older patients who agreed to participate in this research study may also systematically differ from those who refused. We did not have information available to evaluate this concern. Finally, it is possible that an older person's (or caregiver's) perceptions may not be the same as their experiences. For example, an older person may perceive that the discharge process went well by virtue of the fact that he or she returned home when in fact there were problems with medication errors, lack of follow-up, or poor communication. The converse situation may also be true. Further, there is a high prevalence rate of delirium among older adults recently discharged from a hospital to a post-acute care facility [27]. Thus, although the input from older patients and caregivers was no doubt genuine, it may not have reflected their actual experiences.

In summary, the CTM may serve to fill an important gap in health system performance evaluation by measuring the quality of care delivered across settings. Further testing is needed to evaluate additional psychometric properties and to determine whether scores on the CTM correspond to other indicators of poor care transitions, such as re-hospitalisation rates, inability to return to prior care setting or level of function, or even mortality. The development of a scoring system for the CTM is currently in progress. It is our hope that the development of such a measure may stimulate further innovation into improving care integration.