Validity Testing the Outcomes and Assessment Information Set (OASIS)

Abstract

This study evaluated the criterion validity of the Outcome and Assessment Instrument Set (OASIS) items that measure activities of daily living (ADLs), instrumental activities of daily living (IADLs), cognitive functioning, and depression. A convenience sample of patients (n = 203) from five home care agencies participated in the study. Patient OASIS items were compared to data collected using gold standard instruments. Correlations range from .44 to .69 for ADLs and .20 to .68 for IADLs. A correlation of .62 was found for cognitive functioning while correlations for depressive symptoms are .36 and .26. OASIS ADLs and cognitive status items are sufficiently valid, but the OASIS depression item is not sufficiently sensitive to the prevalence of these conditions.

INTRODUCTION

Nearly a decade ago, the Centers for Medicare and Medicaid Services (CMS) announced that home health agencies (HHAs) were required to collect uniform data for patients receiving skilled services reimbursed by Medicare and/or Medicaid (Health Care Financing Administration [HCFA], 1999). The assessment tool containing standardized data elements is known as the Outcome and Information Set (OASIS). The OASIS addresses six major domains: sociodemographic, environment, support system, health status, functional status, and behavioral status; in addition, selected health service utilization items are included. Registered nurses (RNs), physical therapists (PTs), speech and language pathologists (SLPs), and occupational therapists (OTs) collect OASIS on admission to home health care services, transfer to another site of care (e.g., hospital), resumption of care (following a hospital stay), follow-up (at least 60 days), discharge from home health care, and death (General Accounting Office, 2001).

Home health agencies are required to electronically submit OASIS data to state repositories, which prepare OASIS data for retrieval by a national repository established by CMS (HCFA, 1999). The CMS uses the OASIS data for payment purposes, for public reporting of selected outcomes (Home Health Compare) to be used by consumers and others to select an agency, and for survey and certification purposes for the Medicare and Medicaid programs. Moreover, OASIS items used in the comprehensive assessment of patients serve as a source of data for practitioners when developing a plan of care for the patient. Agencies implement quality improvement activities based on patient and agency-specific outcomes based on OASIS and from reports generated by CMS at the agency level (Shaughnessy et al., 2002). Since initial implementation in 1999, several revisions have been made to the OASIS tool in order to reduce data collection burden on agency clinical staff and to eliminate items determined by CMS to have little benefit in developing a plan of care for the patient. As of August 2008, version B-1 of the OASIS tool is in use, and plans are in place for additional revisions with another version (OASIS C) expected by 2009 (Centers for Medicare & Medicaid Services). In addition to multiple regulatory uses in home health care, OASIS has the possibility of being fertile ground for health services researchers (Fortinsky, Garcia, Sheehan, Madigan, & Tullai-McGuinness, 2003; Keepnews, Capitman, & Rosati, 2004; Madigan, Tullai-McGuinness, & Fortinsky, 2003; Sangl, Saliba, Gifford, & Hittle, 2005). While the reliability of OASIS data has been reported by the developers of the OASIS (Hittle et al., 2003) and others (Kinatukara, Rosati, & Huang, 2005; Madigan & Fortinsky, 2000; Madigan et al., 2003; Madigan & Fortinsky, 2004), there have been few published studies reporting on the validity of any component of the OASIS instrument since its implementation.

The team responsible for developing the originally implemented version of the OASIS instrument conducted a variety of validity analyses on most of the OASIS items during the developmental phase. Items used to measure patient outcomes or for risk adjustment of outcome analysis underwent consensus validation by experts. Criterion or convergent/predictive validity was conducted on items included in the calculation of case mix adjustment for payment. Validity analysis for items used for patient assessment and care planning included consensus validity by experts and validation by practicing clinicians as effective and useful for these purposes (Department of Health and Human Services, 1999). In addition, there is some evidence that OASIS data reflecting depressive symptoms, an important clinical problem of older home health care patients, are not sufficiently valid because they overlook symptoms that are identified with diagnostic interviews (Bruce et al., 2007).

Given the lack of published evidence on the validity of OASIS data, the purpose of the present study was to evaluate the criterion-related validity of selected OASIS items. Criterion-related validity can be defined as an empirical association of an item or scale with a criterion or gold standard (Devellis, 2003). For this study, patient assessment data collected using OASIS items were compared to results yielded by interview-based instruments widely used in clinical and epidemiological research with older adults living in the community. We tested the criterion validity of OASIS items that measure functional disability—depressive symptoms and cognitive status—because functional disability measures constitute the majority of quality indicators used in the Home Health Compare program administered by CMS, and because depressive symptoms and cognitive status are the major mental health-related items available in the OASIS tool.

METHODS

The study reported in this article was part of a larger funded study designed to evaluate risk factors for health outcomes and to improve understanding of how Medicare home health care patient outcomes are associated with home visit patterns. Twenty-eight Ohio, Medicare-certified home health agencies and 1,284 patients were enrolled in the larger study (Fortinsky, Madigan, Sheehan, Tullai-McGuinness, & Fenster, 2006; Madigan & Fortinsky, 2004). The purpose of the present study was to examine criterion validity of specific OASIS items used in the larger study to provide outcome data. We compared OASIS items measuring ADLs, IADLs, cognitive functioning, and depression to patient self-reported data collected on instruments considered to be gold standards in the fields of gerontology and geriatrics: the Short Portable Mental Status Questionnaire (SPMSQ), ADLs and IADLs question-naires of the Older Americans Resource and Services (OARS) Instrument, the Center of Epidemiology Studies Depression Scale (CES-D Scale), and the Brief Symptom Inventory (BSI), respectively.

Sample

The study involved a convenience sample of patients (n = 203) receiving skilled home health services from five Medicare-certified home care agencies participating in the parent study. Each of the five participating Ohio agencies recruited between 36 and 50 patients for the validation component of the study.

Instruments

Functional disability

Functional disability was measured according to an individual’s ability to conduct ADLs and IADLs. In the OASIS tool, each ADL and IADL item is organized conceptually according to whether a person can conduct the activity independently, with the use of an assistive device or human supervision, with the help of another person, or cannot do the activity at all. This is in contrast to some measurement approaches in which a person is asked about the level of difficulty with which they can conduct ADLs or IADLs. The OASIS ADL items are grooming, dressing upper, dressing lower, bathing, toileting, transferring, ambulating, and eating. OASIS IADLs include meal preparation, transportation, laundry, housekeeping, shopping, and telephone. The OASIS Chronicle (Center for Health Services Research, 2002) provides a detailed description of each OASIS item as well as item use, reliability data, and types of validity testing. We focus our discussion here on the OASIS items used in this study and will refer to the number of the item. (Each OASIS item is identified by the letter M and a number between 0001 and 9999).

Response categories for the eight OASIS ADL and six IADL items range from three (0–2) to six (0–5). One of the challenges in using OASIS data for research has been the varying numbers of response categories by item whereby some OASIS items have a 3-point response and others have a 6-point response (Fortinsky et al., 2003). Madigan & Fortinsky’s (2004) evaluation of interrater reliability for ADL and IADL items (except for transportation) found reliability was highly adequate (kappa scores ≥ .60) with all but one item scored higher than .70. Findings of strong interrater reliability are consistent with those reported in the OASIS Chronicle (Center for Health Services Research, 2002). In examining Internal consistency for the present study (n = 203), Cronbach’s alphas for OASIS ADLs (eight items) were .87 and .73 for OASIS IADLs (six items).

The ADLs and IADLs instruments used for comparative purposes in this study were developed at Duke University, as part of the Older Americans Resources and Services (OARS) assessment instrument (Fillenbaum, 1988). Questions on the OARS ADLs instrument relate to the same ADLs as the OASIS, with the exception of dressing; there is only one dressing question on the OARS ADLs instrument, whereas the OASIS uses two questions to assess dressing (upper body and lower body). For each ADL, OARS respondents are ask if (a) they can conduct the activity with or without help, or if they are unable to conduct that activity; (b) if they need help, whether that help is from a person, special equipment, or both; and (c) how much difficulty they have doing the activity. For this study, we did not use the difficulty question because there is no corollary in the OASIS tool.

For each IADL, OARS respondents were asked if they can conduct the activity without help (score = 0), with some help (1), or unable to conduct (2). Six of the OARS IADLs are identical to OASIS items. Cronbach’s alpha for these six items was 0.79.

Cognitive functioning

The OASIS item M0560 assesses cognitive function using a 5-point Likert scale (ranging from 0 = alert/oriented able to focus and shift attention, comprehends and recalls task directions independently to 4 = totally dependent due to disturbances such as constant disorientation, coma, persistent vegetative state, or delirium). We used the Short Portable Mental Status Questionnaire (SPMSQ; Pfeiffer, 1975) as the gold standard to test criterion-related validity of cognitive status. The SPMSQ is a 10-item selfreport that assesses mental functioning for the purpose of identifying organic brain disease in the elderly. The instrument tests orientation, memory function, and capacity to perform serial mental operations. The respondent answers each question with the test administrator identifying if yes, answered correctly or no, answered incorrectly. The numbers of errors are added, with eight to ten errors indicating severe intellectual impairment. Pfeiffer (1975) reported face validity as a measure of organic impairment and test–retest correlations of .82 and .83. Cronbach’s alpha for the SPMSQ for the present study was .82.

Depression

The OASIS item, M0590, assesses for depressive feelings. Clinicians mark all the five depressive feelings (depressed mood, sense of failure or self reproach, hopelessness, recurrent thoughts of suicide, and thoughts of suicide) observed in the patient or reported by the patient. An option is also provided for clinicians to mark “none of the above.” Scores range from 0 to 5, with higher scores representing more depressive symptoms.

Two instruments were used as the gold standards for depressive symptoms. The 20-item Center of Epidemiology Studies Depression Scale (CES-D Scale; Sawyer-Radloff, 1977) was designed to measure symptoms of depression in the general population using a 20-item self-report. Respondents are asked to indicate how often they felt depression symptoms during the last week (e.g., I felt sad.). Questions are scored on a 4-point Likert scale ranging from “rarely or none of the time (less than once/week)” scored as a zero to “most or all of the time (5 to 7 days)” scored as a 3. The possible range of scores is 0 to 60, with higher scores indicating presence of more depressive symptomatology. A cutoff point score of 16 has been found to correlate well with clinical rating scales of depressive symptoms (Kohout, Berkman, Evans, & Cornoni-Huntley, 1993). Internal consistency and test–retest correlationssupport reliability. Validity studies have found that the CES-D is able to discriminate well between psychiatric and the general population but only moderately discriminates among levels of severity within patient groups (Sawyer-Radloff, 1977). For this study, the Cronbach’s alpha of the CES-D was .85.

The Brief Symptom Inventory (BSI; Derogatis & Melisaratos, 1983) was also used as a gold standard to correlate with the OASIS depression item. The BSI, a 53-item self-report, was developed to measure current distress in nine primary symptom dimensions. We used the depression symptom dimension (items 9, 16, 17, 18, 20, 35, 50) for this study. Each item on the BSI is rated on a 5-point Likert scale of distress (0 to 4), ranging from “not-at-all” to “extremely.” Test-retest (0.84) and internal consistency reliability coefficients (.85) have been shown to be very good for symptom dimensions. Convergent validity of BSI dimensions with those of similar instruments and factor analysis support the validity of the depression dimension. Nonpsychiatric patients (n = 719) scored a mean of .28 (SD = .46) on the depression dimension (Derogatis & Melisaratos, 1983). For this study, Cronbach’s alpha for the depression dimension was .83.

PROCEDURE

Home health care clinical staff members at the five participating agencies (n = 188 registered nurses [RNs], 14 physical therapists, and 1 speech and language therapist) making the admission visits obtained written informed consent from patients and completed the OASIS tool per clinical protocol. Staff members then faxed the consent and patient demographic information to the first author, who contacted one of seven research RNs, all of whom had home health care experience, and all who received training on the administration of the “gold standard” instruments by the first author. Research RNs then scheduled home visits, making every effort to interview consented patients using the gold standard measures within five business days after OASIS completion. The SPMSQ was completed first. If the participant missed five or more questions (n = 25, 12.5%), the research RN obtained ADLs and IADLs data from a proxy (caregiver), but data were not collected on the CES-D and BSI. Per study protocol, the research RN read each gold standard question to the participant and held a corresponding laminated 4 × 8 index card with printed responses in a size 18 font, and the subject pointed to the appropriate response, which was then documented by the RN. The RNs spent from 20 to 60 minutes in the subjects’ homes.

The research-specific gold standard data collection occurred independently of the clinical completion of the OASIS; the elapsed time between the clinical staff completion of the admission OASIS and completion of the instruments used for validity analyses averaged five days (M = 5.34, SD = 2.37).

DATA ANALYSIS

Prior to conducting validity analysis using Duke OARS ADL items, scoring issues related to the Duke OARS ADL items were addressed. As mentioned earlier, for every ADL measured by the OARS instrument, two questions were used: need for help and how much help. To configure OARS responses in the format of OASIS ADL items, categories created for each OARS ADL item were the following: independent (score = 0), help with equipment only (1), help from another person or help from another person and equipment (2), and unable to conduct activity (3). After OASIS and OARS ADL and IADL item scores were calculated, composite scores were calculated for OASIS and OARS ADLs and IADLs by adding items together and dividing by the number of items.

To score the SPMSQ, we followed Pfeiffer’s (1975) criteria: 0 to 2 errors, intact intellectual function, scored as 0; 3 to 4 errors, mild intellectual impairment, scored as 1; 5 to 7 errors, moderate intellectual impairment, scored as 2; and 8 to10 errors, severe intellectual impairment, scored as 3. Scores were then correlated with results of M0560.

In examining criterion validity, we correlated data collected on OASIS items related to ADLs, IADLs, cognition, and depression with gold standard instruments as reported by study patients. Pearson’s correlation coefficients were used. The strength of the correlation was evaluated using Cohen’s (1988) recommendations that .50 is large, .30 is moderate, and .10 is small.

RESULTS

Sample Description

The average study participant was 78.6 years of age (SD = 9.8) and 92% were white. Sixty-two percent were females (n = 125) and 38% males (n = 78).

Functional Status

Descriptive data for ADL and IADL items and ADL and IADL composite scores can be found in Table 1, while correlations between OASIS items and OARS items can be found in Table 2. For ADLs, correlations ranged from .44 for transferring to .69 for ambulation. Correlations for IADLs ranged from .20 for shopping to .68 for telephone. In addition, Table 3 shows correlations between the composite scores of ADLs and IADLs as measured by the OASIS and OARS.

TABLE 1.

Descriptive Data for OASIS (n = 203) and Gold Standards: OARS (n = 203), CES-D (n = 165), and BSI-Depression (n = 170)

OASIS items (M0#-Likert-scale)	M(SD)		Gold Standard M(SD)
ADL items			OARS ADL items
Grooming (M0640, 4)	.79 (.93)		.34 (.78)
Dressing upper (M0650, 4)	.78 (.92)
Dressing lower (M0660, 4)	1.10 (1.00)
Dressing (M0650 & M0660)*	.94 (.91)		.93 (1.02)
Bathing (M0670, 6)	2.41 (1.39)		1.36 (0.92)
Toileting (M0680, 5)	.38 (.84)		.74 (.77)
Transferring (M0690, 6)	.71 (.78)		.80 (.77)
Ambulating (M0700, 6)	1.12 (.89)		.94 (.81)
Eating (M0710, 5)	.27 (.63)		.22 (.66)
IADL items			OARS IADL items
Meal preparation (M0720, 3)	.91 (.76)		1.17 (.83)
Transportation (M0730, 3)	1.01 (.21)		1.43 (.68)
Laundry (M0740, 3)	1.65 (.56)		1.54 (.74)
Housekeeping (M0750, 5)	2.80 (.56)		1.60 (.63)
Shopping (M0760, 4)	2.22 (.66)		1.73 (.55)
Telephone (M0770, 6)	.67 (1.47)		.28 (.63)
OASIS ADLs (composite score)	6.63 (4.78)	OARS ADLs  (composite score)	5.33 (4.2)
OASIS IADLs (composite score)	9.25 (3.72)	OARS IADLs  (composite score)	7.74 (2.84)
Depressed feelings (M0590, 6)	.24 (.49)	CES-D* BSI-Depression*	14.10 (10.18) 3.36 (4.2)

Note. OASIS M0590 mean (SD) calculated after summing feelings identified for each patient (possible 0–5). On CES-D scale (0–60), score greater than 16 indicates probable clinical depression. BSI-Depression median = 2.

TABLE 2.

Pearson’s r Coefficients Between OASIS and OARS ADL and IADL Items (n = 203)

Gold Standard OARS items:	Groom	Dress	Bath	Toilet	Ambulate	Transfer	Eat
Oasis Items
Groom	.51 **	.30**	.40**	.44**	.39**	.44**	.43**
Dress	.44**	.47 **	.46**	.51**	.41**	.49**	.43**
Bath	.28**	.33**	.46 **	.38**	42**	.33**	.27**
Toilet	.35**	.24**	.37**	.48 **	.50**	.44**	.36**
Ambulate	.43**	.28**	.44**	.55**	.69 **	.51**	.41**
Transfer	.35**	.29**	.32**	.49**	.55**	.44 **	.32**
Eat	.35**	.22**	.18**	.48**	.35**	.37**	.53 **

Gold Standard OARS items:	Meal	Housekeeping	Laundry	Transport	Shop	Telephone
Meal preparation	.46 **	.20**	.20**	.24**	.26**	.35**
Housekeeping	.28**	.22 **	.17**	.18**	.26**	.35**
Laundry	.25**	.22**	.27 **	.24**	.27**	.26**
Transportation	.18**	.12	.14*	.23 **	.16*	.68**
Shopping	.21**	.21**	.17*	.20**	.20 **	.27**
Telephone	.38**	.18*	.20**	.24**	.15*	.68 **

^*Correlation is significant at the 0.05 level (2-tailed).

^**Correlation is significant at the 0.01 level (2-tailed).

TABLE 3.

Pearson’s r Coefficients Between OASIS and Gold Standard Indices

	OASIS indices
Gold Standard indices	n	ADLs	IADLs	Cognition	Depression
OARS ADLs	196	.71**	.60**	.39**	.01
OARS IADLs	201	.50**	.49**	.34**	.01
SPMSQ	201	.50**	.53*	.62**	−.03
CES-D	165	.15*	.16*	.17*	.36**
BSI (depression)	170	.14	.10	.10	.26**

^**Correlation is significant at the 0.01 level (2-tailed).

^*Correlation is significant at the 0.01 level (2-tailed).

Cognitive Functioning

The OASIS item measuring cognitive functioning (M0560; n = 203) identified 77.8% (n = 158) of the participants as alert/oriented, 11.8% (n = 24) required prompting, 6.4% (n = 13) needed some assistance, and 3.9% (n =8) required considerable or total assistance. The SPMSQ (n = 201) found 69.2% (n = 139) intact, 18.7% (n = 38) mildly impaired, 6.5% moderately impaired, and 5.5% (n = 11) as severely impaired. Table 3 indicates a correlation of .62 between the OASIS cognitive function item (M0590) and the SPMSQ.

Depressive Symptoms

The OASIS item for depressive feelings, M0590, showed that of 203 participants, 159 (78.3%) had no depressive feelings and 44 (21.7%) had at least one depressive feeling. For the CES-D, scores greater than 16 indicate probable clinical depression. Sixty-three of the 165 study patients who were cognitively capable of providing CES-D data (38.2%) had scores greater than 16. Additional descriptive statistics can be found in Table 1. Depressive symptoms for OASIS item M0590 were low to moderately correlated with the BSI depression dimension and the CES-D, Pearson’s r = .36 and .26, respectively, as displayed in Table 3.

CONCLUSIONS

The findings suggest that OASIS is valid for measures of ADLs and cognition, but may not be sufficiently sensitive for depressive symptoms and the IADL items. Our findings may not be surprising in that home health care admission is a clinical encounter, most often with a patient who is recovering from a serious illness or injury. Thus, clinicians prioritize assessments on items that address immediate safety concerns, such as ambulation, toileting, eating, and intact cognition, in which all the correlations exceeded .44. Of note, the highest correlation among these three domains was .69 for ambulation. A high correlation may be related to the clinicians’ ability to easily assess and the importance placed on safe ambulation.

The low correlations for the IADLs were unexpected. There are several explanations for these low correlations. First, most home health care patients are generally very limited in IADLs because of the Medicare requirement that they be homebound (i.e., it takes considerable and taxing effort for them to leave the home) to receive services. Thus, their inability to demonstrate their IADLs performance may be expected. As a result, there may be scoring issues with the comparison based on ceiling and floor effects. The mean scores on the OARS IADL scales and the OASIS IADLs indicate consistent difficulty whereby using the telephone is the least impaired IADL while shopping and housekeeping have the highest levels of impairment for both measures. In past research, the ceiling effect (whereby most patients are severely impaired) has had an impact on the utility of these measures for research (Fortinsky et al., 2003). Second, safety needs indicate that ability to use the telephone and meal preparation are the most important IADLs to remain safely at home. These IADLs also have the highest correlations, suggesting that clinical attention to accuracy of these items may be higher than with other IADLs that are important but not critical to being safe.

Depressive symptoms were not well recognized in home health care patients, despite evidence from the present study that 38.2% of the participants scored 16 or higher on the CES-D, indicating the need for further evaluation. These findings are consistent with those of Bruce and colleagues (2007), who identified similar prevalence and extent of depressive symptoms.

Another plausible explanation is that ADLs and cognitive status assessment are more explicit and observable whereas depressive symptoms are more subtle. As well, the persistent stigma associated with mental health symptomatology may make patients reluctant to report such symptoms. Additionally, home health care clinicians are familiar with measures of ADLs and cognitive status evaluation since these assessments are used in most health care settings. To the contrary, most clinical settings do not include routine assessments of depressive symptoms and anxiety. Thus, some of the low correlations may be explained by less familiarity with how to assess for depressive symptoms and anxiety. Anecdotally, home health care clinicians report reluctance or even resistance to depressive symptom screening because there are generally few resources for them to refer patients to and they are not sure what to do when they find a patient with depressive symptoms.

Consideration should be given to the limitations of this study related to sampling and data collection. We studied only agencies and patients in each agency, and recognize that our results may not be generalizable to all Medicarecertified HHAs. Our sample size was selected based on a reasonable subsample of the sampled patients (10%), and also on the resources available to conduct the gold standard interviews. Sample size was not based on a priori sample size calculation. We also did not check multiple raters for a minimum threshold of interrater reliability in gathering gold standard measures prior to data collection for this study. However, data collectors received a structured orientation and a reference manual.

Based on our findings, we recommend that OASIS items not be used as the only measures for depressive symptom evaluation in research because the OASIS is not sufficiently sensitive to the prevalence of these conditions. This finding is consistent with the proposed revision of the OASIS (OASIS C) that includes a measure for whether a screening for depressive symptoms occurred (Centers for Medicare & Medicaid Services, 2008). Alternatively, using OASIS for the identification of depressive symptoms may indicate moderate or severe depressive symptoms based on the scoring between the CES-D and OASIS. In other words, a score of 1 on OASIS depressive symptoms indicates moderate or greater depressive symptoms.

OASIS is sufficient for ADLs and cognitive status appraisal from a validity perspective. The IADLs represent a persistent challenge because of the scoring effects with many subjects who are severely impacted. Researchers conducting studies in which IADL measures are critical may want to follow the same recommendations as made for depression and use additional measures.