TABLE 3.
OASIS Reliability Studies
| Citation | Type of reliability |
Method/Design | Sample | Items measured | Findings | Limitations/Gaps* |
|---|---|---|---|---|---|---|
| Berg (1999) | Sequential interrater |
Evaluated the reliability of OASIS items for case-mix adjustment purposes; utilized multiple agency staff (RNs, LPNs, therapists) to perform assessments |
144 patients 60 agencies 40 staff Purposive quota sample: all patients scheduled to have second visit within week |
Pt. demographic, health history, symptoms, therapies, prognoses, risk factors, support, sensory, integumentary, cognitive, behavioral & functional status items Using kappa and % agreement Tested original OASIS |
Kappas ranged from poor to moderate on most domains (see table 3) |
Data collected 97–98 Up to 3 days between assessments No control for skill levels of assessors Multiple assessors/ agencies without controls |
| Fortinsky, Garcia, Sheehan, Madigan, & Tullai-McGuinness (2003) | Internal consistency |
Evaluated the use of Rasch modeling to improve disability measures Secondary analysis of data from study on home care resource use & outcomes |
583 patients 1 agency Patients in this cohort were enrolled into a larger ongoing study Sampling method not reported |
I/ADLs via Cronbach’s alpha Tested original OASIS |
Internal consistency among all 15 ADL and IADL items in the study cohort was high (Cronbach’s alpha .91) |
Data from 11/99–9/00 Non-rep. sample Adm. only |
| Hittle et al., (2003) | Sequential interrater |
Evaluated the reliability of most OASIS items & time burden for OASIS completion Sequential assessments by two research clinicians altering who completed the first and second assessments Prospective cross- sectional study |
66 patients (41, Round 1 & 25 Round 2) 5 agencies Random |
Pt. demographic, health history, clinical, support, functional, living arrangements Kappa & % agreement Tested original OASIS |
Mean kappa = .69 Interrater reliability was excellent (kappa >.80) for many OASIS items and substantial/moderate (kappa >.60) for most items (see Table 3) |
Data from 97–98 Not in the usual care context Pt’s assessed at different times but within 24 hrs Nonrep sample No controls for agency factors |
| Kinatukara, Rosati, & Huang (2005) | Sequential (phase I) & simultane- ous (phase II) interrater |
Estimated reliability of various OASIS items via comparison of agency & research clinician assessments Phase I, sequential rater (research clinician) visited same patients within 24–72 hours Phase II simultaneous raters (agency & research clinicians) visit patients jointly Research clinician was a staff nurse with 6 years of home care experience Prospective cross-sectional |
Phase I, 259 patients 1 agency Convenience Phase II, 105 patients 1 agency Convenience |
Various clinical, functional & service items Kappa & % agreement Tested original OASIS |
Reliability for many items was considerably lower than prior studies Phase I, 39 items had poor interrater reliability (<.40), 17 had moderate reliability (>.40–60), 2 items had substantial (>.60–80) reliability, and 2 items had excellent reliability (>.80). Phase II, 19 items had poor reliability, 24 items had moderate reliability, 19 items had substantial reliability, & 4 items had excellent reliability Reliability of 58% of items improved in Phase II |
Phase I from10/ 00–11/01 Phase II, 01/02–9/02 Nonrep sample Only one research clinician used as rater compared to multiple staff clinicians No control for discipline of agency clinician Phase I time lapse |
| Madigan & Fortinsky (2000) | Sequential intrarater Internal consistency |
Evaluated the reliability of agency clinician assessments within a test-retest framework. Same agency staff completed a second OASIS within 48 hours of the first; the staff person was free to use any clinical data for recall, but was not to refer back to the initial OASIS Secondary analysis |
201 patients 10 Ohio agencies Convenience Sampling method not specified |
Functional, clinical, affect, behavioral Cronbach’s alpha & kappa Tested original OASIS |
Internal consistency admission/discharge: Functional—.86/.91 Affect—.36/.56 Behavioral—.24/.53 Intrarater: functional = sufficient; affect & behavioral = good; clinical = low (see Table 3 for detailed kappa results) |
Data from 1996 Prior to OASIS mandatory use Intrarater technique used Time between assessments Nonrep. sample No staff training nor controls for assessor discipline; no controls for agency factors |
| Madigan & Fortinsky (2004) | Simultaneous interrater |
Determine reliability of OASIS items based on independent but simultaneous assessments by agency clinicians Prospective cohort |
88 patients 21 agencies Convenience |
OBQI OASIS items HHRG OASIS items Weighted kappas & % agreement Tested original OASIS |
There were no items with values less than .60, and most items had values higher than .70; these findings suggest that the reliability of these OASIS items is sufficient for use in research, regulatory, & reimbursement purposes |
Data from 11/99–9/00 Nonrep & small sample Several items had insufficient samples; kappa scores not computed for these items No controls for agency factors |
| Madigan, Tullai-McGuinness & Fortinsky (2003) | Simulated interrater (by discipline) |
Compare agreement between nurses & therapists in OASIS completion via video simulation at adm. & DC Exploratory simulation |
436 agency clinicians 337 RNs 68 PTs 21 OTs 10 STs 29 OH agencies Sampling method not specified |
Functional items, dyspnea, & pain Response distributions for RNs vs. therapists, χ2 test of significance Tested original OASIS |
Nurses and therapists agreed in their ratings for a majority of items at adm. (10/16) & DC (14/16); the largest significant differences were for: dressing lower body, transferring, & oral med. management |
Data from 11/99–9/00 Non-rep. sample Not in usual care context No training provided on OASIS No controls for assessor skill or experience No controls for agency factors |
| Neal (2000) | Sequential interrater |
Measured OASIS reliability between RNs, and between RNs and PTs. 2 nurse research clinicians assessed patients 24 hours after the agency clinician Prospective cohort |
23 patients, 14 patients assessed by an RN first; 9 patients by a PT first First 11 patients assessed by Rater 1; last 12 by Rater 2 1 agency Convenience sample |
Functional, affect, behavioral, clinical, demographic items % agreement Tested original OASIS |
23 of the 77 items had IRR of .8 or more Poor reliability defined as < .8 found for: reimbursement, financial situation, location of inpt stay, DC date, inpt dx, change in pt’s regime after inpt stay, dx requiring change in regime in pt’s condition prior to inpt stay, dx & severity of, overall prognosis, high-risk factors, structural barriers in the home, safety hazards in the home, informal support, vision, intractable pain, pt has a skin lesion, pressure ulcer, stasis ulcer or a surgical wound, incontinence, cognitive, affect, behavioral, dyspnea, & functional items >RN experience in HH increased reliability; no difference based on experience for PTs |
Data collection dates not specified Non-rep. & very small sample Prior to use of OASIS as part of the clinical assessment No evaluation of research assessors interrater reliability Inferential stats not appropriate due to sample size & bias Unclear description of reliability coefficient and how calculated |
| Shew, Sanders, Arthur, & Bush (2010) | Sequential interrater |
Evaluated reliability between nurses and PTs for assessments conducted within 24 hours RN completed the initial assessment via laptop computer; PT completed a second OASIS on paper Collected within an operational agency using agency staff Prospective cross- sectional |
52 patients, 18 RNs, 12 PTs 1 agency Convenience sample |
HHRG scores & projected reimbursement rates compared between the two assessments Tested OASIS-B1 Wilcoxon Signed-Rank test |
28 cases (54%) had HHRG scores and reimbursement rates that were equal between RNs and PTs Of the nonequal scores: five reimbursement rate projections were separated by less than 10%; 10 reimbursement rate projections differed between 10% & 20%; 9 rate projections differed between 20% & 30% Overall, the RNs scored higher in 13 of 24 nonequal cases No statistically significant difference between the dollar value of RN & PT ratings |
Data collection dates not specified Non-rep. & small sample size Adm. only Did not evaluate individual OASIS items, just the overall agreement of HHRG and estimated reimbursement rates Relied on existing agency training of assessors Did not control for assessor experience with OASIS |
*
Dates are listed as reported in original articles.