Validity of a tool designed to assess the preventability of potentially preventable hospitalizations for chronic conditions

Abstract

Background

Potentially preventable hospitalizations (PPH) are defined as unplanned hospital admissions which could potentially have been prevented with the provision of effective, timely outpatient care. To better understand and ultimately reduce rates of PPH, a means of identifying those which are actually preventable is required. The Preventability Assessment Tool (PAT) was designed for use by hospital clinicians to assess the preventability of unplanned admissions for chronic conditions.

Objective

The present study examined the ability of the PAT to distinguish between those unplanned admissions which are preventable and those which are not, compared to the assessments of an Expert Panel.

Methods

Data were collected between November 2014 and June 2017 at three hospitals in NSW, Australia. Participants were community-dwelling patients with unplanned hospital admissions for congestive heart failure, chronic obstructive pulmonary disease, diabetes complications or angina pectoris. A nurse and a doctor caring for the patient made assessments of the preventability of the admission using the PAT. Expert Panels made assessments of the preventability of each admission based on a comprehensive case report and consensus process.

Results

There was little concordance between the hospital doctors and nurses regarding the preventability of admissions, nor between the assessments of the Expert Panel and the hospital nurse or the Expert Panel and the hospital doctor.

Conclusions

The PAT demonstrated poor concurrent validity and is not a valid tool for assessing the preventability of unplanned hospital admissions. The use of Expert Panels provides a more rigorous approach to assessing the preventability of such admissions.

Key Messages.

• Reducing rates of unplanned admissions is a priority for health systems.

• A way of identifying unplanned admissions which are preventable is required.

• The Preventability Assessment Tool is not a valid tool to assess preventability.

• Expert Panels’ assessments of case reports provide a more rigorous approach.

Background

As rates of unplanned hospital admissions rise (1,2), efforts to understand and reduce such admissions are a priority (3). Potentially preventable hospitalizations (PPH) are broadly defined as unplanned hospital admissions which could potentially have been prevented with effective, timely outpatient care in the period immediately prior to admission (4). Admissions are classified as PPH based on discharge diagnostic codes and are categorized as vaccine-preventable, chronic or acute. Although the specific conditions classified as PPH vary between countries, internationally the majority of PPH are for chronic conditions (5). In Australia nearly three quarters of all chronic PPH are for congestive heart failure (CHF), chronic obstructive pulmonary disease (COPD), diabetes complications or angina pectoris (6).

Rates of PPH have been used as proxy measures of access to timely and effective primary care for several decades. However, the validity of PPH as a marker for actual preventability has not been confirmed empirically (7). Indeed, as classification is based on discharge diagnostic codes rather than any assessment of the actual preventability of the admission, this approach overestimates the number of admissions for these diagnoses that are preventable (7,8). Furthermore, this approach omits preventable admissions for diagnostic codes which are not specified in the classification. A means of identifying which individual PPH are preventable is required to establish the modifiable contributing factors, allowing development of focused interventions and targeting of patient groups. To date, there are no validated approaches or tools for assessing the preventability of individual admissions (8). Efforts to prospectively identify preventable readmissions have been hampered by a lack of consensus between the assessments of patients, their carers and clinicians (9). Oddone et al., however, developed and validated a retrospective medical chart review process to assess the preventability of readmissions (10). Two physicians independently reviewed each medical chart and assessed the preventability of the admission. Disagreements were resolved by the assessment of a third physician, with the majority decision ruling (10). Arozullah et al. subsequently used a slightly modified version of this protocol (11). Our team enhanced the process developed by Oddone et al. (10), with the following modifications: inclusion of a wider range of clinicians in the Expert Panels; the provision of more information to the Expert Panels, including data from the patient’s GP and the patient themselves; and the use of a consensus approach, enabling discussion of differing views and expertise, rather than the majority rule approach (12).

Drawing on an extensive literature review and consultation with clinicians, our group also developed and piloted the Preventability Assessment Tool (PAT) (8). The PAT is intended as a brief tool to be used by hospital clinicians (nurse or doctor caring for the patient) during the patient’s admission, rather than retrospectively, and was specifically designed to assess the preventability of chronic PPH admissions amongst patients aged 45 and older living in the community. Prior to its use in the current study, the PAT had face and content validity assessed and confirmed (8).The feasibility and acceptability of using the tool was assessed in three sites, and demonstrated its utility (8).

The PAT considers individual and social factors more extensively than previous work, and defines the timeframe for preventability as the 3 months prior to admission. When developing a definition of a preventable admission, we were unable to identify any clear timeframes in the literature, although the original work by Billings et al. (4) referred to the ‘period immediately prior to admission’. Based on our understanding of the literature, and in consultation with clinicians, the 3-month time frame was determined as a reasonable time in which to expect any interventions aimed at reducing preventable admissions to be effective (8).

Our group therefore define ‘preventable’ admissions as: Unplanned admissions which could have been prevented if: (i) appropriate, adequate, accessible and good quality support in the community had been available and accessed in the preceding 3 months (support in the community might include primary health care, family/neighbour/friend/social support, health or non-health community services); and (ii) appropriate individual health behaviours e.g. disease self-management, had occurred in the 3 months prior to admission. The definition assumes that all individual health behaviours are modifiable. This pragmatic approach reflects that it is not always possible to make a clear assessment of which health behaviours could have been changed.

Following the initial developmental work with the PAT, the Diagnosing Potentially Preventable Hospitalisations (DaPPHne) study was undertaken. DaPPHne aimed to assess the preventability of chronic PPH in order to (i) determine the proportion of PPH which are actually preventable and (ii) generate an evidence base identifying modifiable factors driving preventable PPH. One objective of the DaPPHne study was to validate the PAT. In this paper, we examine the concurrent validity of the PAT, that is, its ability to distinguish between those unplanned admissions which are preventable and those which are not, compared to the assessments of an Expert Panel (considered the ‘gold standard’).

Methods

Setting and study design

The DaPPHne study was a mixed methods study (12). Patients with unplanned admissions to three hospitals in NSW, Australia, with a primary discharge diagnosis of COPD, CHF, angina pectoris or diabetes complications; aged 45 years or over; community-dwelling; able to give informed consent and not transferred from another hospital, were eligible to participate. Following written consent, data were collected via: (i) patient questionnaire, (ii) the PAT, (iii) GP (family physician) interview and (iv) extraction of hospital clinical data. Patient interviews were conducted as soon as clinically appropriate following admission, and included demographic characteristics, health conditions and self-reported functioning, psychological distress (Kessler 10) (13), engagement with GPs and other health care providers, use of medications and disease self-management (Partners in Health Scale (14)). GPs were interviewed regarding: care provided, including adherence to clinical guidelines; and other chronic conditions or social issues impacting on patients’ self-management. The research nurses extracted clinical data from the hospital records for participants’ current admission and the most recent previous admission. Items extracted included principal diagnoses and comorbidities, medications on admission, and discharge information. The research nurses also copied the clinical notes for the period up to and including the first 24 hours after admission and the hospital discharge summary.

Data were collected for 545 eligible admissions during the recruitment period to the three hospitals (November 2014–June 2017). It was not possible to obtain a complete set of data for all 545 admissions (due to no completed PAT or no GP interview). The 275 admissions included in the current analysis are those for which data were collected from all four sources including at least one PAT.

Assessment of preventability

Assessments of the preventability of each admission were made by: (i) hospital doctors and nurses caring for the patient completing the PAT (based upon their knowledge of the patient), and (ii) an Expert Panel reviewing a detailed case report of each admission (based on data collected from patient questionnaire, GP interview and extraction of hospital clinical data) and completing the Panel-PAT (P-PAT).

The Preventability Assessment Tool

The PAT (see Supplementary Material 1) was completed by senior hospital clinicians caring for the patient (the medical registrar and a senior nurse). These clinicians had direct contact with the patient, as well as clinical expertise and understanding of the hospital setting. The research nurse provided brief explanations of how to use the PAT to assess admissions within the context of the study definition of preventability, and was available to answer questions. The research nurses encouraged completion of the PAT as soon as possible following admission.

When completing the PAT, respondents indicated the primary reason for the unplanned admission (e.g. clinical care, a diagnostic test or need for social support), and rated the extent to which a range of patient (e.g. poor self-management), clinician (e.g. inadequate medical management of existing chronic condition) and system factors (e.g. poor discharge practices) contributed to the admission from one (did not contribute) to four (contributed to a great extent). Other response options were ‘don’t know’ and ‘not applicable’ (if the factor was not relevant to this patient). Finally, whether the admission could have been prevented (yes/no) was recorded, together with suggestions regarding actions that could have been taken to prevent the admission (if assessed as preventable). The PAT took approximately 5 minutes to complete.

Expert Panel assessments of preventability

The ‘Gold Standard’ comparison for the PAT assessments was the judgement of an Expert Panel. Three Expert Panels were established, one for each site, each including a hospital physician, GP and community nurse with expertise in chronic disease management. This membership ensured a diverse range of perspectives on patient care, and local clinical, service and system expertise.

Data from the patient questionnaire, hospital clinical data, the hospital discharge summary and GP interview were consolidated into a non-identifiable, comprehensive ‘case report’ for review by the Expert Panel.

The Expert Panel members received training in using the P-PAT (see Supplementary Material 2) to assess the preventability of admissions. The P-PAT prompted the Expert Panel members to consider the same factors as the PAT (i.e. patient factors, self-care, primary care factors, coordination of care, access to care, hospital admission characteristics). Consensus meetings provided the opportunity to reiterate the protocol and study definition of preventability.

Members of the Expert Panels reviewed each case report, blinded to the assessments of the hospital doctors and nurses (using the PAT) and each other, and provided an assessment of whether they were ‘reasonably confident that this admission was preventable’ (yes/no), and whether they were ‘reasonably confident that this admission was not preventable’ (yes/no). For those classified as ‘preventable’, Expert Panel members identified intervention/s they considered could have prevented the admission. Each case took around 30 minutes to assess.

When discrepancies in the assessment of preventability occurred (127 of the 275 cases (46%)), the Expert Panel members met for a facilitated discussion to reach consensus. At these meetings, the facilitator would summarize the case report, and each Panel member would briefly explain their initial assessment. A facilitated discussion would follow with reiteration of the definition of preventability if necessary. In cases where agreement was not reached, assessments of ‘unclassifiable’ were recorded. Thus, through these two steps, of individual assessments and facilitated meetings, consensus was reached regarding the preventability of each individual admission.

Sample size

Initial sample size calculations indicated that 150 admissions would be required for the validation of the PAT, assuming 20% of PPH admissions be deemed actually preventable. This would give power to estimate a Kappa statistic of >0.6 ± 0.08. However, the final sample size was larger (n = 275), giving power to estimate the Kappa statistic with more precision.

Data analysis

Data analyses were completed using SPSS version 2.2. Descriptive statistics of patient characteristics and their admissions were generated. Concordance between the assessment of preventability made by the hospital clinicians using the PAT, and the assessment of the Expert Panel was calculated (we conducted separate analyses for concordance between the medical registrar and the Panel, and the senior nurse and the Panel) using the Kappa statistic, with 95% confidence intervals (CIs) (15).

Results

Characteristics of patients and their admissions

The characteristics of the 275 admissions included in this analysis are shown in Table 1. Participants tended to be elderly, approximately half were married or cohabiting, while one third lived alone. The most common final principle diagnosis was COPD (35%), and the average number of diagnoses on discharge was 6.4 (median = 6). The average length of stay was 7.1 days (median = 4).

Table 1.

Characteristics of patients with an unplanned admission for a chronic condition to three Australian hospitals between November 2014 and June 2017 (n = 275)

Characteristic	n (%)
Male	153 (56%)
Mean age in years (range)	71.3 (46–93)
Marital status
Married/de facto	139 (51%)
Widowed/divorced/separated/single	136 (49%)
Lives Alone	85 (31%)
Final principal diagnosis
COPD exacerbation/complication	97 (35%)
CHF	69 (25%)
Diabetes with/without complications	80 (29%)
ACS	29 (11%)
Total diagnoses on discharge
1–3	36 (13%)
4–9	199 (72%)
10–20	40 (15%)
Length of stay
1–2 days	73 (27%)
3–10 days	165 (60%)
>10 days	37 (13%)

Agreement between assessments of preventability

There were 247 admissions for which both a hospital nurse and hospital doctor completed a PAT (see Table 2). The PAT was completed by hospital doctors a median of 5 days (range 0–71 days) following admission, and by hospital nurses a median of one day (range 0–33 days).

Table 2.

Agreement between hospital doctors’ and hospital nurses’ assessment of admission preventability of unplanned admissions for a chronic condition to three Australian hospitals between November 2014 and June 2017 (n = 247)

Hospital nurse PAT assessment	Hospital doctor PAT assessment		Kappa statistic	95% CI
	Preventable (n = 95)	Non-preventable (n = 152)b
Preventable (n = 83)	44 (18%)a	39 (16%) a	0.21	0.09–0.34
Not preventable (n = 164)	51 (20%) a	113 (46%) a

^aPercentages shown are % of the total assessments.

^bNon-preventable includes admissions assessed as not preventable and those unclassifiable.

There was low agreement between the assessments of the hospital doctors and nurses regarding which admissions were deemed preventable (Κ = 0.21; 95% CI = 0.09–0.34) (see Table 2). The agreement between hospital nurses and hospital doctors for admissions being preventable was only 18% although agreement for non-preventable admissions (including admissions assessed as not preventable and those unclassifiable) was higher at 46%. Overall disagreement between the hospital nurses and hospital doctors was 36%.

Given there was only low agreement between the hospital doctors and hospital nurses, we compared the nurse and doctor PAT separately with the assessments of the Expert Panels (Table 3).

Table 3.

Agreement between Expert Panel and hospital nurse (n = 268) and hospital doctor (n = 254) assessments of preventability of unplanned admissions for a chronic condition to three Australian hospitals between November 2014 and June 2017

Hospital nurse assessment	Expert Panel assessment		Total in row n (column %)	Kappa statistic	95% CI
	Preventable n (column %)	Non-preventable n (column %)a
Preventable (n = 95)	53 (45%)	42 (28%)	95 (35%)	0.17	0.05–0.28
Non-preventable (n = 173)a	66 (55%)	107 (72%)	173(65%)
Total	119 (100%)	149 (100%)	268 (100%)

Hospital doctor assessment	Preventable n (column %)	Not preventable n (column %)	Total in row n (%)	Kappa statistic	95% CI
Preventable (n = 97)	51 (45%)	46 (33%)	97 (38%)	0.13	0.01–0.25
Not preventable (n = 157)	62 (55%)	95 (67%)	157 (62%)
Total	113 (100%)	141 (100%)	254 (100%)

^aNon-preventable includes admissions assessed as not preventable and those unclassifiable.

There was very low agreement between the Expert Panels and the hospital nurse regarding the assessment of the preventability of individual admissions (K = 0.17; 95% CI = 0.05–0.28) (see Table 3). Of the 119 admissions assessed as preventable by Expert Panel, only 53 (45%) were assessed as preventable by the hospital nurses. Similarly, there was very low agreement between the Expert Panel and the hospital doctor regarding the assessment of the preventability of individual admissions (K = 0.13; 95% CI = 0.01–0.25). Of the 119 admissions assessed as preventable by Expert Panel, only 51 (45%) were assessed as preventable by the hospital doctors.

Conclusions

In order to better understand and ultimately reduce rates of PPH, a means of identifying those PPH which are actually preventable is required. Once preventable admissions are identified, factors associated with these admissions can be determined, enabling the development and implementation of targeted primary care interventions. This study examined the concurrent validity of the PAT, a tool designed to enable hospital clinicians to assess the preventability of an individual PPH during the admission. Despite previously established face and content validity (8), the findings presented here indicate that the PAT is not a valid tool for this use. There was little agreement between the assessments of the hospital doctors and nurses (using the PAT) regarding the preventability of individual admissions, consistent with previous research (9). There was also little agreement between the assessments of the hospital doctors and nurses and the Expert Panels.

The process undertaken in the DaPPHne study to assess the preventability of PPH represents an extension of the previous work done in this area (10,11,16). In our study, more consideration was given to the contribution of patient-, clinician- and system-level factors, and the range of expertise within the Expert Panel membership allowed for diverse perspectives regarding the potential influence of these factors. Panel members also had access to more comprehensive information in the case reports, including from patients themselves and their GPs, allowing in-depth consideration of the complexity of factors which may have contributed to the admission. The local knowledge of Expert Panel members, for example regarding the availability of specialist and other services, ensured that the impact of such factors could be considered. More generally, unlike much of the previous research in this area (16), the assessments were undertaken with a clear definition of preventability including a timeframe. Expert Panel members received training in assessing the preventability of admissions, with consensus meetings providing the opportunity to reiterate the process. Panel members reached consensus on the preventability of each admission via individual assessments and consensus meetings, if required. While there was initial disagreement within the panel on nearly half the cases, the consensus meetings facilitated discussion and agreement, thus enhancing the rigour of the process.

The PAT was developed in the hope that assessments of preventability could be made using a brief tool during the admission, facilitating assessment of modifiable contributing factors so they could be addressed. However, the lack of concordance between the assessments of the hospital doctors and nurses, and between the assessments of the hospital clinicians and the Expert Panels, suggests that completion of the PAT by hospital clinicians is not a valid method for assessing preventability of individual chronic PPH admissions. This may be due to problems with the tool itself (e.g. structure, domains, questions, scale/response options) as well as with hospital clinicians’ engagement. Although designed to be completed during the admission, there was a considerable delay in the completion of the PAT by some, which is likely to have resulted in recall issues. The lack of concordance may also reflect hospital clinicians’ lack of familiarity with patients’ circumstances and the factors contributing to the admission. Anecdotal reflections from the study research nurses would suggest this was the case at times. The lack of concordance is also likely to be partly due to the inherent complexity of assessing preventability. Indeed, even with all the information contained in the case reports, nearly half of the admissions required discussion at an Expert Panel consensus meeting. The lack of concordance in the assessments of preventability give further weight to questions regarding the validity of PPH as an indicator of access to care (7,9).

A strength of this study is that it was conducted in three hospitals geographically distributed across NSW, Australia, with differing catchment populations and service models, thus enhancing the generalizability of the findings. A limitation is that full data could be collected for 275 admissions (50%).

Accurate assessments of the preventability of individual PPH enable identification of the drivers of preventable admissions, and subsequently inform the development and targeting of interventions, including those located in primary care, designed to reduce unplanned admissions. It was hoped that the PAT would enable the swift and timely assessment of the preventability of PPH, with minimal resource implications. However, despite careful design and development, the PAT was found not to be a valid tool for assessing preventability of unplanned admissions for chronic conditions when used by hospital doctors and nurses. At present therefore, although it is a more resource intense and less timely approach, the use of an Expert Panel is the most robust way to assess the preventability of individual PPH.

Ethical approval

Granted by the New South Wales Population and Health Services Research Ethics Committee (ref HREC/14/CIPHS/39).

Funding

The DaPPHne study was funded by: Western Sydney Local Health District, Mid North Coast Local Health District, North Coast Primary Health Network, New South Wales Agency for Clinical Innovation and University Centre for Rural Health, University of Sydney.

Conflict of interest

None.