Evaluation of Patient and Proxy Responses on the Activity Measure for Post Acute Care

Abstract

Background and Purpose

Our objective was to examine the agreement between adult patients with stroke and family member or clinician proxies in Activity Measure for Post Acute Care (AM-PAC) summary scores for daily activity, basic mobility, and applied cognitive function.

Methods

This study involved 67 patients with stroke admitted to a hospital within the Kaiser Permanente of Northern California system and were participants in a parent study on stroke outcomes. Each participant and proxy respondent completed the AM-PAC by personal or telephone interview at the point of hospital discharge and/or during one or more transitions to different post-acute care settings.

Results

The results suggest that for patients with a stroke proxy AM-PAC data are robust for family or clinician proxy assessment of basic mobility function, clinician proxy assessment of daily activity function, but less robust for family proxy assessment of daily activity function and for all proxy groups’ assessment of applied cognitive function. The pattern of disagreement between patient and proxy was, on average, relatively small and random. There was little evidence of systematic bias between proxy and patient reports of their functional status. The degree of concordance between patient and proxy was similar for those with moderate to severe strokes compared with mild strokes.

Conclusions

Patient and proxy ratings on the AM-PAC achieved adequate agreement for use in stroke research where using proxy respondents could reduce sample selection bias. The AM-PAC data can be implemented across institutional as well as community care settings while achieving precision and reducing respondent burden.

Introduction

Patient-reported measures of function are important outcomes in stroke research. However, with as many as one quarter of stroke survivors unable to report their status as a result of language or other cognitive deficits, functional assessments from another individual, often a family member or clinician, are typically substituted for the patient’s perspective ^{1; 2, 3, 4, 5, 6}. While the use of proxies may reduce selection bias in stroke research, there are concerns that disagreement between proxy and patient responses may introduce significant measurement error and bias into study results. Patient – proxy disagreement may have an impact on analysis of within-person functional change, may introduce noise in functional outcome assessment, and necessitate larger sample sizes in clinical trials to observe a given treatment effect. The inclusion of unbalanced numbers of proxy responses in different treatment groups may bias the evaluation of an intervention if not appropriately adjusted for in the analysis.

The reliability of proxy respondents has been examined in generic measures of function ^{3, 7, 8, 9, 10}, as well as in stroke-specific instruments.^{11, 12, 13} Previous research in patients with stroke has shown that proxies often systematically rate patient function worse than the patient’s own report^{11, 12, 13, 14} and that the level of agreement between proxies and patients may differ depending on the type of outcome being assessed. Adequate agreement tends to be reported for the assessment of more concrete and observable domains of function such as physical function while lower levels of agreement are more frequently seen in more subjective domains such as psychosocial functioning. ^{12, 15, 16, 17, 18} Agreement between proxy and patient reports has also been reported to be lower among more patients with more severe strokes.^{3, 11, 19, 20} Across different types of proxy respondents, agreement between the patient and proxy rating tends to be better for family members and lower for health care providers ^{14, 21}. Good agreement between proxy and patient is a necessary criterion for a functional outcome measure to be useful in stroke research. The Activity Measure for Post Acute Care (AM-PAC), created to assess patient reported function across post acute care settings, consists of three domains of function: Basic Mobility, Daily Activity, and Applied Cognitive Functioning.²² In previous research, although acceptable proxy-patient agreement has been reported across all three domains of the AM-PAC (ICCs = 0.68 – 0.90)^{23, 24} in patients with major medical, orthopedic, and neurologic conditions who were receiving post acute care, no reports have been published that focus specifically on patients with stroke and that included comparisons across different types of proxy respondents and across different care settings. In this study, therefore, we examined the agreement between family member and clinician proxy reports with those of patient on the three AM-PAC functional domains among survivors of stroke who were receiving post acute care in a variety of institutional and community-based settings.

Methods

AM-PAC Instrument

The AM-PAC instrument was designed to measure functional status in adults across all post-acute care settings. Early content and analytic work with the AM-PAC established three distinct activity content domains upon which three AM-PAC scales were constructed: Basic Mobility, Daily Activities, and Applied Cognitive functioning.²² The AM-PAC instrument, developed on a sample of 1,041 post acute care patients, uses a 4-point difficulty scale rating scale (no difficulty, a little difficulty, a lot of difficulty, cannot/unable to do) across 259 functional tasks and activities. In this study, the computer adaptive test version (CAT) of the AM-PAC was employed. In CAT administration of the AM-PAC, an iterative computer program uses information from a subject’s previous responses to tailor item selection thereby eliminating questions that are too hard or too easy for a particular patient. In the AM-PAC CAT, a small sample of the overall items in an instrument are administered and all scores are generated on the same metric, regardless of the number of items administered, thus facilitating comparisons across time or across groups with different functional levels while reducing respondent burden.^{25, 26} We have conducted a series of simulated validation tests of the AM-PAC CAT software on numerous samples, including patients with stroke,²⁷ complex medical conditions,²⁸ and prospective work on patients with orthopedic conditions ^{26, 29}.

Sample

This sub-study involved 67 patients with a diagnosis of stroke admitted to one of four hospitals of Kaiser Permanente of Northern California and who participated in a parent cohort study on stroke outcomes. Participants were eligible for this proxy reliability sub-study if they were judged by a treating clinician to be able to provide consent without assistance and provide self-report information on their functioning and had an identifiable proxy (clinician or family member) who was willing to participate. Exclusion criteria included: a transient ischemic attack, brain tumor or abscess, significant trauma, and under age 18 years.

Data Collection

Enrollment into the proxy reliability substudy ran from Oct. 2008 through Feb., 2011. The initial assessment was done within 24 hours of discharge from a Kaiser acute hospital following a stroke. Each sub-study participant was administered the AM-PAC CAT by personal interview or by telephone by a trained data collector. Participants and proxy respondents provided responses to questions in all three AM-PAC domains at the point of hospital discharge and/or during transitions to different post acute care settings (inpatient rehabilitation hospital, skilled nursing facility, home care, outpatient therapy). At each assessment period we collected patient reported data from the AM-PAC CAT and proxy reports from a physician or nurse or family member using the AM-PAC CAT. If the patient was receiving care in an institution the first choice of proxy was a clinician while a family member was the first choice if the patient was living at home. These patient and proxy AM-PAC assessments were completed within 1-7 days of each other with a mean interval of 3-days. Self-report data were always collected before obtaining proxy information. In this study we report on 67 complete pairs of family-proxy AM-PAC data and 63 clinician-proxy pairs.

Statistical Analyses

As an overall measure of patient-proxy agreement, we used the linear mixed model to calculate the Intraclass Correlation Coefficient (ICC) which represented the ratio between subject variance to the total variance. We used the linear mixed model to estimate ICC coefficients because we had multiple assessments for some subjects and this model could account the correlation between assessments for each subject. We included the time between stroke onset and assessment, the rater effect, and the interaction between those two variables as fixed effects in the model. The subject effect was treated as the random effect. The standard error of ICC was calculated based on Delta Method³⁰. For family-proxy data, we compared the ICC values in each subgroup, such as first visit vs. second visit or more, in person vs. by telephone and in institution vs. at home. These subgroup comparisons were not done for clinician-proxy – patient pairs due to sample size limitations. For group estimates, reliability was considered high if the ICC is greater than 0.80, substantial if it was between 0.61 and 0.8, moderate between 0.41 and 0.60, and poor to fair if the ICC was less than 0.4.³¹

In addition to ICCs, we constructed a folded cumulative distribution curve for each AM-PAC scale (called ‘mountain plots’) ³² and then generated separate plots for each proxy type (clinician or family). To construct a mountain plot, we calculated the difference scores between proxy respondent and patient respondent (proxy minus self-report) for each AM-PAC scale and then sorted these difference scores in ascending order within each domain. Then the percentile rank for each difference score (the proportion of scores lowers than and equal to that score) was computed. It should be noted that because we “folded” the empiric cumulative distribution plot at the line y=50%, percentile ranks for scores ranked on the second half were calculated using 100 minus the actual percentile rank. Finally, the mountain plot was generated by plotting the percentile rank against the difference score for each AM-PAC scale. For within group (family and clinician proxy) mountain plots, we recalculated the percentile ranks separately for each group.

Results

The 67 participants in this proxy sub-study included a wide age range (30–94 yrs old); 58.2% were female. Twenty two percent of subjects participating in this study had moderate to severe stroke severity according to the modified NIH stroke scale. The vast majority of subjects (92.5 %) had an ischemic stroke (Table 1).

Table 1.

Baseline characteristics of patients in Kaiser’s proxy sub-study

Variable	N = 67
Average age, yrs $\overline{x}$ (SD)	67.3 (13.6)
Age distribution % (n)
<60 yrs	25.3 (17)
60-69	26.9 (18)
70-79	29.8 (20)
80+	18.0 (12)
Gender % (n)
Female	58.2 (28)
Severity of Stroke (NIHSS) % (n)
Mild (0-3)	77.6 (52)
Moderate to Severe (4-31)	22.4 (15)
Stroke Type % (n)
Ischemic	92.5(62)
Hemorrhagic	7.5 (5)
Charlson Index (Deyo Modification) $\overline{x}$ (SD)	1.5 (1.7)
Modified Rankin Scale Score $\overline{x}$ (SD)	.04 (.97)
Days from Stroke to First AM-PAC Assessment $\overline{x}$ (SD)	3.9 (2.9)
AM-PAC Function at Acute Hospital Discharge $\overline{x}$ (SD)
Basic Mobility (Range = 4.2-78.6)	52.9 (14.1)
Daily Activity (Range = 7.3-70.4)	45.6 (13.9)
Applied Cognitive (Range = 28.9-58.8)	43.4 (7.5)

Table 2 presents the overall ICC coefficients by type of proxy. The overall patient-proxy reliability coefficients ranged from 0.50 for the Applied Cognitive scale, 0.63 for the Daily Activity scale, to 0.72 for the Basic Mobility scale. Clinician proxies showed a trend toward less agreement in the Applied Cognitive scale (0.41) than did family member proxies 0.59) while clinicians showed more agreement in the Daily Activity scale (0.78) than did family member proxies (0.57). Agreement was highest and very similar across type of proxy for the Basic Mobility scale. (Table 2 and Figure 1). The mean difference score for patient-reported vs proxy-report data across all three AM-PAC domains is quite modest (0.15-1.39). The magnitude of patient-proxy agreement across all three AM-PAC domains was not different for those with a more severe stroke compared with those with mild or not severe strokes.

Table 2.

Overall ICC correlations and by type of proxy respondent for AM-PAC scale scores

AM-PAC Domain	Family Proxy (N=67) ICC (95% CI)	Clinician Proxy (N=63) ICC (95% CI)	Total Proxies (N=67) ICC (95% CI)
Basic Mobility	0.72(0.06)	0.80(0.05)*	0.72(0.05)
Daily Activity	0.57(0.07)	0.78(0.05)	0.63(0.06)
Applied Cognition	0.59(0.07)	0.41(0.12)	0.5(0.07)

^*: N=62

Figure 1.

ICC correlations by type of proxy respondent for each AM-PAC CAT domain

Figure 1 displays the overall ICC values for patient-family member proxy pairs and by various family member proxy subgroups. There was a trend toward lower ICC values when the interview was conducted by telephone vs face-to-face but no clear trend between first visit assessments vs subsequent assessments or for assessments done in an institution vs when the patient was living at home.

Figure 2 displays the magnitude and direction of divergence between patient and proxy scores across each AM-PAC CAT domain. There does not appear to be a systematic direction in the difference between proxy and patient scores for each AM-PAC scale. Figure 3 illustrates the pattern of disagreement by type of proxy and again shows no systematic bias in the direction of the difference between family member and clinician proxy with the patient scores on each AM-PAC scale. The magnitude and the pattern of the disagreement between patient and proxy is quite similar for each type of proxy ratings; however, the family member proxy rating for the assessment of daily activity function does have a longer tail, suggesting that proxies will rate this domain lower than patient self assessment.

Figure 2.

Mountain Plot for each AM-PAC domain of differences between patient and overall proxy reports

Figure 3.

a: AM-PAC CAT mountain plot for Daily Activity function by type of proxy respondent

b: AM-PAC CAT mountain plot for Basic Mobility function by type of proxy respondent

c: AM-PAC CAT mountain plot for Applied Cognitive function by type of proxy respondent

Discussion

The results of this study suggest that for patients with a stroke proxy data are sufficiently concordant with patient self report for use in research where patient reported outcomes are of interest. The data reveal that concordance is most robust for family or clinician proxy assessment of basic mobility function, clinician proxy assessment of daily activity function, but less concordant for family proxy assessment of daily activity function and all proxies’ assessment of applied cognitive function. In contrast to previous research, proxy respondents did not appear to systematically rate patients as functioning at a lower level in comparison to the patient reported values. It is unclear why this occurred. It could be that there is something inherent in the administration of a CAT instruments that make them less prone to this bias. Most of the papers documenting this phenomenon used standard, paper and pencil instruments. Furthermore, the degree of concordance between patient and proxy was equal to or slightly better for patients with more severe strokes, those who would be most likely to need a proxy due to cognitive or language impairments. In using the AM-PAC among survivors of a stroke, the pattern of disagreement between patient and proxy appears to be, on average, relatively small, random and not clinically meaningful. The overall agreement between patient and proxy reported function was within acceptable limits with little evidence of systematic bias between proxy and patient reports of their functional status. That agreement was lowest in the domain of applied cognitive functioning is not surprising since this domain of functioning is less observable than the areas of mobility and daily activities. Since CAT tools administer different items across patients it was not possible to analyze which subdomains of applied cognitive functioning led to the lower levels of concordance.

The magnitude of agreement between patient and proxy reports of functioning seen in this study was similar to that reported in previous studies of the AM-PAC in different patient groups^{23, 24} and with the literature that compares patient and proxy agreement using other functional outcome instruments.^{14, 15, 16, 17, 18, 21, 33} Similar to existing literature, ICC coefficients for the more observable, more concrete domains of the AM-PAC (ie., Basic Mobility) was substantial to high in contrast to the coefficients for the more subjective, less observable domain (ie., Applied Cognitive) which was only moderate^{12, 15, 16, 17, 18, 23, 24, 33}, although this finding varied across the methods we used to assess concordance. In contrast to other studies, agreement between proxy and patient was the same for those patients with a more severe stroke, those for whom proxy reports are most likely to be needed^{3, 11, 19, 20, 33} and, proxies did not appear to systematically rate patients as functioning at a lower level in comparison to the patient reported values.^{11, 12, 13, 14, 33}

We believe the mountain plot approach used in this study to examine the magnitude and direction of discordance between patient and proxy reports, is an important complement to the interpretation of standard reliability indices such as the ICC.³⁴ The mountain plot approach was developed for an earlier report of the AM-PAC proxy reliability²⁴ and displays the cumulative distribution of the difference scores between proxy and patient raters. The graph is created by computing a percentile rank for ranked difference between respondents’ scores and “folded” at the 50^th percentile rank (percentile rank is defined as the proportion of cases having lower or equal value to the score under consideration). The mountain plot cumulative percentages (y-axis values) against the ranked difference scores (x-axis values). In folding the graph, the percentile ranks for difference scores above the 50^th percentile are obtained by subtracting the actual percentile rank from 100. This mountain plot technique allows the reader to locate the median immediately, clearly illustrates degree of symmetry in the patient-proxy differences, and identifies outliers to determine central or tail percentiles. In this sample, the agreement appears to be quite consistent across AM-PAC domains with no evidence of systematic bias. The absence of a trend in the direction and magnitude of the scoring difference between type of proxy and patient is clearly displayed in Figures 1 and 2 since the plots peak at x=0 and are symmetric around the line of x=0. If proxy reports were systematically different than those of patients the mountain peak would have shifted either to the left or the right of the line x=0.

Although summary indices such as the ICC are the standard in the literature for examining proxy-patient agreement, we have some concerns in using this approach exclusively, because the amount of variability in scores within the sample being evaluated has a strong influence on the magnitude of this statistic.³⁵ For instance, a very low ICC can be obtained when agreement is indeed low or, alternatively, if the variability in the sample is severely restricted. This inconsistency across approaches can be seen in the AM-PAC data we analyzed in this study. For the AM-PAC, the variability of the Applied Cognitive scale (sd=8.5) was substantially smaller than that of the Daily Activity (sd=15.6) and Basic Mobility (sd=13.9) scales. Consequently, it is difficult to interpret whether the lower ICC value for the Applied Cognitive scale was due to lower agreement or to the ratio of the disagreement relative to the underlying variability in the sample. The mountain plot results revealed that the magnitude of agreement between patient and proxy to be similar across all three AM-PAC domains. Another limitation with the ICC approach is that the measure of agreement (the ICC) is not in the metric of the functional instrument which makes practical interpretation more difficult. It is for this reason we presented multiple approaches to examining proxy-patient agreement in this study.

There are several limitations to this study. Because of the limited number of patient-clinician proxy pairs, we were unable to compare agreement and bias within subgroups defined by mode of data collection, location where the assessment was done, or first vs subsequent assessment. Furthermore, the study sample was limited to patients who were able to provide a self-report on the AM-PAC, and therefore the study sample excludes those who would typically require a proxy report of their functioning. Thus, we have to assume the results found in this self selected sub-sample can be generalized to the population needing a proxy. The overall study sample was quite small with a low representation of subjects with a severe stroke. Another concern in this analysis was the lack of independence of the pairs across settings but this was adjusted for in our analyses.

While the Basic Mobility portion of the AM-PAC is quite robust when considering patient proxy comparisons, further work may be necessary to more confidently validate the AM-PAC for all proxy types across other domains, through multiple patient severity rankings and post-stroke time points.

There are suggested patterns of change in ICC coefficients illustrated in Figure 1 that may be important for future investigation. One such pattern is the magnitude and direction of change observed in the daily activity function domain. In most cases, the ICC for daily activity agreement shifts ICC category, i.e. is in the substantial range for first visit but drops to moderate for second visit, similar drop from in person to telephone, and for the in-institution to at-home comparison. The basic mobility and applied cognitive subscales appear to be more consistent across these categories. Further study is also needed on the potential effects of using telephone versus face to face assessment.

In conclusion, patient and proxy ratings on the AM-PAC CAT are sufficiently concordant for use in stroke research where patient reported outcomes are of interest and where the inclusion of proxy reports can reduce selection bias. There was little evidence of systematic bias between proxy and patient reports of their functional status using the AM-PAC CAT instrument. Use of the AM-PAC CAT using patient or proxy reported can be implemented in 2-3 minutes per domain and can be used across institutional as well as community-based and home care settings, thus achieving precise outcome estimates while reducing respondent burden. Although some degree of error is present when using a proxy in place of the patient him/herself, the pattern of disagreement between patient and proxy appears to be, on average, relatively small, random and not clinically meaningful.