Factors affecting appropriateness of hospital use in Massachusetts

Abstract

To determine the extent of inappropriate hospital use, and to investigate factors related to variations in appropriateness, 8,031 hospital records of patients discharged from 41 hospitals in 3 Massachusetts professional standards review organization (PSRO) areas were reviewed in 1973 and 1978. The Appropriateness Evaluation Protocol (AEP) was used for the reviews and logistic regression analysis was used to analyze factors associated with inappropriate use.

Based on the results, the authors conclude that utilization review should focus on: longer stays among surgical patients and shorter stays among medical patients; (projected) last third of the stay; and on diagnoses or diagnosis-related groups in which there is less clinical concensus on treatment method. For maximum effectiveness, utilization review must include incentives beyond simple monitoring (e.g., financial incentives).

Introduction

The current consensus in this country is that containing health care costs is not just a desirable goal but an economic necessity. There is widespread recognition that some portion of the utilization of hospital resources is inappropriate (Restuccia and Holloway, 1976; Querido, 1963; Rosenfeld, Goldman, and Kaprio, 1957; Forsythe and Logan, 1960; Van Dyke, Brown, and Thom, 1963; Zimmer, 1974; Berg et al., 1969; Restuccia et al., 1984; Gertman and Bucher, 1969; Morehead, Donaldson et al., 1964; Browning, 1965), in the sense either that patients receive services that provide no significant benefit or that the services could be rendered in a less costly setting. Because hospital costs are the largest single component of total health care expenditures, improving the allocative efficiency of hospital utilization has been a major objective of health planners and policymakers. With the advent of fixed-price, prospective payment by large payers such as Medicare, hospitals must reduce inappropriate use in order to achieve financial viability. Identifying and reducing inappropriate use is now of concern on an institutional as well as a policy level.

A number of programmatic interventions have been implemented to improve hospital efficiency, including professional standards review organizations (PSRO's), health systems agencies (HSA's), health maintenance organizations (HMO's), and various reimbursement schemes. Typically, the evaluations of such programs have been based on changes in total utilization (as measured by changes in admission rates and length of stay). Ideally, reduction in total utilization would be realized by reducing the proportion of use that is inappropriate, not by reducing all use across the board. Based on measurements of changes in total utilization, however, one cannot ascertain whether the observed changes reflect reductions in inappropriate use alone (a desirable means of improving efficiency) or in both appropriate and inappropriate use (an undesirable means). Clearly, to determine what in fact has occurred, and to target cost-cutting efforts effectively, it is important to determine absolute utilization rates, the proportion of use that is appropriate, and the causes of inappropriate use.

This article presents the results of an evaluation of the impact of PSRO's in eastern Massachusetts, the State with the second-highest average cost per hospital discharge in the Nation (United States Department of Commerce, 1984). The findings contribute both to the assessment of the PSRO efforts and to the formulation of the successor to the PSRO's, the peer review organization (PRO).

The Appropriateness Evaluation Protocol (AEP), an instrument specifically developed for such evaluations, was used in the study to determine appropriateness of use. Before the study design and findings are presented, the development and testing of the AEP is briefly described. The reader is referred to Gertman and Restuccia (1981) for more details.

Appropriateness Evaluation Protocol

Although several instruments have been used in the past to identify inappropriate hospital use, fundamental methodological problems with the measurement techniques used, notably poor interreviewer reliability, bias, lack of comprehensiveness, and sampling difficulties (Gertman and Restuccia, 1981; Kurylo, 1976; Holloway et al., 1976), limited the usefulness of those methods. Particularly critical in the poor performance of the instruments was the reliance on subjective, implicit criteria (Donabedian, 1975).

The awareness of the shortcomings of these existing approaches for determining the appropriateness of hospital days of care was the impetus for the development of an instrument for assessing hospital use, the AEP. The design of the AEP was based on a set of a priori objectives. The first was to develop a relatively simple tool that could easily be applied to as many patients as possible, in order to maximize its utility and generalizability and minimize implementation costs. Therefore, the initial decision was to design a diagnosis-independent instrument. It was recognized that this approach would not be applicable to all patients, so the initial goal was to develop one instrument for all adult medical, surgical, and gynecological patients.¹

The second objective was to develop a set of explicit criteria by which, if any single one were met, the day of care would be judged appropriate at an acute hospital level of care. The original criteria set was developed by the AEP physician reviewer and two utilization review nurses, based on previous work on levels-of-care criteria (Gertman and Restuccia, 1981; Holloway et al., 1976; Restuccia and Holloway, 1976). The criteria items were categorized into medical services, nursing/life support services, and patient-condition factors (Figure 1). The first two criteria groups consist of services that would be provided routinely and safely only on an acute hospital level of care. Therefore, a patient receiving any of these services on a given day would justifiably require hospitalization on that day. The third criteria group consists of factors indicating that the patient's condition is so unstable that hospitalization is required, even though no medical or nursing/life support services are provided on a given day.

Figure 1. Criteria of appropriateness of day of care: Massachusetts, 1973 and 1978.

For practical reasons, an upper limit of a total of 30 criteria items was set. This would provide a manageable list that could be readily memorized by reviewers, thereby facilitating increased reliability and efficiency in abstracting information from medical records. As with all utilization review methods based on specifying a priori criteria for determining appropriateness, the AEP should be seen as a screening tool, best applied to aggregates of patients or to flag patients for individualized review by a clinician, and not as the definitive arbiter of appropriateness (Donabedian, 1982).

The final objective was to be able to ascertain the reason for a day being judged inappropriate in order to provide information to target interventions. To accomplish this, a list of reasons adapted from Restuccia and Holloway's “Barriers to Appropriate Utilization” (1976) was incorporated into the instrument. If a day was deemed inappropriate, the reviewer was asked to determine whether physician, hospital, patient, or environmental factors were responsible for the inappropriate day.

An override provision was developed and tested in this study to allow the reviewer to indicate either that the criteria set was not sufficiently comprehensive because some noncriteria service or factor necessitating hospitalization had occurred on that day (i.e., a “false negative”) or, conversely, because a patient meeting one of the criteria nevertheless did not need acute level hospitalization (i.e., a “false positive”), perhaps because the service that met the criterion was not clinically justified. All overrides were reviewed by the AEP nurse and/or physician reviewers. The override provision was subsequently refined and is now available as an AEP option, for reviewers who receive special training and monitoring. The override was used on only 5 percent of the records in this study; it had no net effect on the results, which are therefore reported based solely on the basic criteria, without the use of overrides.

Initial developmental testing of the AEP at Boston University Hospital, in conjunction with feedback from physician committees of the participating PSRO's, resulted in modification and refinement of the original criteria items. The final 27 criteria items are shown in Figure 1. The criteria list was extensively tested for face validity (as determined by review by the PSRO physician committees), construct/concurrent validity (by comparing results from AEP reviews and reviews by a panel of expert physician reviewers), and reliability (interobserver and overall agreement). Results of the AEP evaluations conducted for this and subsequent studies indicated that it marks a significant advance over previous instruments, and that it is a reliable and valid instrument (Gertman and Restuccia, 1981; Restuccia et al., 1984).

Methods

The goal of the study, which was initiated by the executive directors of PSRO's in Massachusetts, was to evaluate the impact of PSRO efforts on improving the appropriateness of hospital utilization in the State. The objectives were threefold:

• To measure the amount of inappropriate hospital use.

• To identify factors associated with variations in inappropriate use among types of patients and hospitals.

• To determine whether there was any change in the amount of inappropriate use after implementation of PSRO-mandated concurrent review.

Of the five PSRO's in the State, three accepted invitations to participate in the study. They were primarily urban, suburban, and mixed urban-suburban-rural. All were in the eastern portion of the State. All the hospitals from each of the two latter PSRO's and a random selection of one-third of the hospitals from the urban PSRO were included in the study.

In each of the 41 surveyed hospitals, a random sample of patients was selected from all adult medical and surgical patients hospitalized on the third Wednesdays of May 1973 and 1978. Thus, the population from which the sample was drawn consists of all days of care received by patients hospitalized on the adult medical and surgical services of the 41 hospitals on these two dates. The first year was chosen as the “preconcurrent utilization review” period, because it pre-dated almost any concurrent utilization review activities in Massachusetts. By 1978, utilization review was well established in each of the PSRO areas.

It should be noted that inference cannot be made to the entire years 1973 and 1978, because only 1 day in each year was sampled and because temporal factors such as day of the week and season of the year may be related to appropriateness of hospital use. However, another study investigating these relationships in four other PSRO areas found them to be statistically insignificant, except in one region where inappropriate days were more likely to occur in fall than in spring and winter (Restuccia et al., 1984).

To determine whether or not the rate of inappropriate use changed markedly since the utilization reviews reported in this article were conducted, the rates of inappropriate admissions and patient days detected in 1973 and 1978 were compared for six of the study hospitals that had had AEP reviews conducted from 1982 to 1985. Although the rates from 1982–85 are not strictly comparable with those from 1973, since the later reviews had the advantage of using objective AEP admissions review criteria (developed and tested in the same manner in which the patient days criteria were developed), the comparison does indicate that, on a gross level, rates of inappropriate utilization have not changed appreciably, at least in these six hospitals, with the passage of time (Table 1).

Table 1. Comparison of rates of inappropriate admissions and patient days in six of the study hospitals: Massachusetts, selected years 1973-85¹.

Hospital	Inappropriate admission rates2		Inappropriate patient days rates
	1973 and 1978	1982, 1984, and 1985	1973 and 1978	1982, 1984, and 1985
A	4.0	9.0	25.0	35.4
B	8.0	10.0	23.7	22.0
C	5.0	7.0	23.4	18.0
D	5.0	36.7	38.1	46.1
E	5.8	10.9	44.0	28.0
F	5.7	5.2	33.3	18.1

¹Hospitals A, B, and C were reviewed in 1982 and hospitals D, E, and F were reviewed in 1984-85.

²Inappropriate admission rates were determined subjectively in 1973 and using objective criteria in the other years.

Data collected for each patient included patient demographic and illness episode characteristics, e.g., age, sex, length of stay (LOS), as well as characteristics of the hospital providing services.

In addition to the day-of-care appropriateness decision, as based on the AEP criteria, the reviewers were asked to make a subjective judgment as to whether the hospital admission was appropriate or inappropriate.

It was recognized that successful implementation of utilization review activities was not equal in all hospitals, and that the marginal effect of the PSRO program was expected to vary across institutions. Therefore, prior to the beginning of the study, the PSRO executive directors were asked to provide numerical ratings of the expected effectiveness of utilization review activities for each hospital in three areas: technical competence of the utilization review coordinator staff, technical competence of the utilization review committee, and willingness of the hospital to police its utilization problems. Based on a summary of the ratings of expected effectiveness, each hospital was designated by the investigators as either “effective” or “ineffective.”

Because individual factors influencing appropriateness may not act independently, considering them together through multivariate statistical analysis can eliminate spurious effects and reveal differences that are otherwise obscured. A log linear model was therefore used to model the joint effect of many factors. In addition, the model estimates the level of inappropriate use free from sampling error.

Results

A total of 8,031 patient days was evaluated. Nine percent of the cases (714) were judged inappropriate admissions based on reviewer subjective assessment of the hospital admission. To avoid overestimating the percentage of inappropriate days, these cases were excluded from subsequent analyses of the days data. Of the days reviewed from cases deemed inappropriate admissions, 75 percent was judged inappropriate, more than 2 times the percent of inappropriate days for patients appropriately admitted.)

To identify particular patient and/or hospital characteristics that significantly affect the level of appropriateness, a list of potential explanatory factors was selected by a combination of subjective judgment and exploratory analysis of the data. The continuous variables—length of stay and age of patient—were converted to multichotomous variables based on a priori judgments and review of the data. Hospital types and services were selected on the basis of a factor analysis. The explanatory factors considered included the following:

• Length of stay: 0–10 days, 11–21 days, 21 or more days.

• Review year: 1973, 1978.

• Part of hospital stay when review occurred: first third, middle third, last third.

• Patient years of age: 1–50(Med)/1–40(Surg), 51–65(Med)/41–65(Surg), 66–75, 76 or over.

• PSRO area.

• Hospital effectiveness: effective, ineffective.

• Type of hospital: teaching (major teaching hospitals with coronary care units (CCU's) and psychiatric units); community hospitals (all others, including secondary teaching and nonteaching). Community hospitals were subdivided into four groups: with CCU and psychiatric units; with CCU only; with psychiatric only; and with neither CCU nor psychiatric units (Table 2).

• Type of insurance: public, private.

Table 2. Ratings of effectiveness of study hospitals, by type of hospital: Massachusetts, 1973 and 1978.

Hospital type	Numbers of hospitals
	Effective	Ineffective	Total
Total	25	16	41
Teaching1	3	4	7
Community
CCU2 and PSYCH3	5	5	10
CCU only	3	2	5
PSYCH only	11	3	14
Neither	3	2	5

¹Each teaching hospital has a CCU and a PSYCH service.

²CCU: Coronary Care Unit. CCU is interpreted to represent high technology service in this sample of hospitals.

³PSYCH: the presence of at least one kind of psychiatric service.

The results indicated that inappropriate hospital use was prevalent in this sample of 41 Massachusetts hospitals. The overall level of inappropriateness was 28.1 percent. For medical patients the level of inappropriate use was 32.2 percent; for surgical patients it was 24.3 percent. Because of this difference between medical and surgical patients, as well as their different clinical and therapeutic needs, each population was subsequently considered separately.

Table 3 summarizes the percent of inappropriate days for the explanatory factors and indicates which explanatory factors were found to be statistically significant, after controlling for interactions through the log linear model.

Table 3. Percent and statistical significance of explanatory factors for inappropriate days, by type of service: Massachusetts, 1973 and 1978.

Factor	Hospital service
	Medical	Surgical
Overall	32.2	24.3
Part of stay reviewed
First third	116.6	114.9
Middle third	131.0	116.6
Last third	148.0	142.2
Length of stay in days
1-10	234.2	117.5
11-21	233.6	127.9
22 or more	228.5	130.2
Patient years of age
15-50 (Med)/15-40 (Surg)	34.4	20.1
51-65 (Med)/41-65 (Surg)	29.3	23.8
66-75	31.1	26.6
76 or more	34.4	30.0
Year of review
1973	134.6	23.3
1978	130.2	25.4
PSRO3
Suburban	29.1	217.5
Urban	32.8	227.2
Mixed (Suburban, urban, and rural)	33.2	223.1
Hospital type
Teaching	128.3	125.7
Community	133.0	123.9
CCU4 and Psychiatric	130.7	121.1
CCU only	133.1	123.9
Psychiatric only	133.3	123.9
Neither	140.6	131.6
Reimbursement
Public	32.1	28.2
Private	32.2	21.9
Interactions
Hospital effectiveness by year of review	1< 0.0001	5N.S.
Hospital type by part of stay reviewed	1< 0.002	5N.S.
Reimbursement by length of stay	1<0.01	5N.S.

¹Significant at 99 percent.

²Significant at 95 percent.

³PSRO: professional standards review organization.

⁴CCU: coronary care unit.

⁵N.S.: not statistically significant.

The strongest effect for both medical and surgical patients is the increase in the level of inappropriate use during a hospital stay. The pattern is slightly different for medical compared with surgical patients (Figure 2). The level of inappropriate use increases constantly during a hospital stay for the former, while for the latter, there is no significant difference between the beginning and middle of the stay, but a significantly higher level at the end of a stay. The absolute length of stay has a highly significant effect only for surgical patients. Their level of inappropriate use is lower for stays of 10 days or shorter compared with longer stays, but there is no significant difference between 11-to-21-day stays and longer ones. The effects of the absolute length of stay and of the third of the stay reviewed are independent.

Figure 2. Percent of inappropriate days, by hospital type and part of stay reviewed: Massachusetts, 1973 and 1978.

Rates of inappropriateness vary greatly by hospital, ranging from 23.2 percent to 54.5 percent for the urban PSRO, from 23.4 percent to 33.3 percent for the suburban PSRO, and from 19.2 percent to 44.6 percent for the mixed PSRO.

For both medical and surgical patients in community hospitals, those hospitals with both CCU and psychiatric services have a significantly lower level of inappropriate use compared with community hospitals with one or none of these services. However, when comparing all community hospitals to teaching hospitals, medical and surgical patients exhibit different patterns of inappropriate use. Medical patients were more likely to have inappropriate days when hospitalized in community hospitals, but the opposite holds true for surgical patients. In further contrast to surgical patients, medical patients exhibit an interaction between hospital type and at what point in the stay the review occurs (Figure 2). Inappropriate use increases at a lower rate during the course of a stay in teaching hospitals compared with community hospitals, among which there are no significant differences. For surgical patients, there are no significant interactions between hospital type and part of stay reviewed.

Univariate analysis of the relationship between payer (Medicare, Medicaid, Blue Cross, commercial insurers, other third party, and self pay) and inappropriateness indicated a significant difference among payers, with the two public payers having slightly higher rates of inappropriateness (36.73 percent and 35.56 percent, respectively) than the others (which ranged from 27.86 percent to 30.96 percent). Because of interactions between payer and other factors, however, these relationships were not significant when multivariate analysis was applied (Table 3).

The year of review is a significant factor for medical patients: inappropriateness is lower in 1978 than in 1973. By year, there is a highly significant difference between effective and ineffective hospitals for medical patients. The level of inappropriate use decreased from 36.6 percent to 27.1 percent for medical patients treated at effective hospitals, while there was a slight, nonsignificant increase at ineffective hospitals (Figure 3). There were significant differences by PSRO for surgical patients, with the urban and the suburban areas having the highest and lowest levels, respectively, of inappropriate utilization.

Figure 3. Percent of inappropriate days, by effectiveness of hospitals' utilization review systems and year of review: 1973 and 1978.

The majority (over 88 percent) of the days deemed inappropriate were attributed to physician responsibility. This pattern was consistent across the two services and among the parts of the stay reviewed (Table 4). Environmental factors, reasons for inappropriate days beyond the immediate control of the physician, hospital, or patient, were responsible for approximately 7 percent of the inappropriate days for each service, and hospital and patient/family factors for approximately 1 percent each.

Table 4. Distribution of patient days failing to meet criteria for appropriateness, by service, responsible party, and third of stay reviewed: Massachusetts, 1973 and 1978.

Responsible party	Type of service
	Medical service				Surgical service
	First third of stay	Second third of stay	Last third of stay	Entire stay	First third of stay	Second third of stay	Last third of stay	Entire stay
Total	113	471	614	1,198	110	205	534	849
Physician	89.38	89.17	88.60	88.90	92.73	86.34	87.45	87.87
Hospital	1.77	.64	3.26	.58	1.82	.49	.75	.82
Patient/family	0.00	.85	1.63	1.17	0.00	.98	.94	.82
Environmental	5.31	7.22	7.49	7.18	1.82	6.83	7.49	7.07
No response given	3.54	2.12	1.95	2.17	3.64	5.37	2.62	3.42

NOTE: Days deemed appropriate are excluded from the table.

Nonphysician factors were more likely to contribute to days being deemed inappropriate during the latter portions of the stays (for both services) than during the first third of the stays. Table 5 lists the specific reasons most frequently cited for days failing to meet the appropriateness criteria, by service. This same general pattern of reasons was present in each third of the stay reviewed.

Table 5. Most frequent reasons days failed to meet criteria for appropriateness, by service: Massachusetts, 1973 and 1978.

Reason	Percent of days deemed inappropriate
	Medical service	Surgical service
Service could have been provided at a lower level of care (including outpatient department or home) (P)1	51.84	51.47
Physician's medical management of patient is overly conservative (P)1	28.96	24.85
Physician delays scheduling of test or procedure (P)1	6.09	8.01
Unavailability of SNF2 bed (E)3	3.76	2.12
Patient is convalescing	1.42	3.77
Miscellaneous reasons and no response given	7.93	9.78

NOTE: Days deemed appropriate are excluded from the table.

¹(P) designates physician responsibility.

²SNF: Skilled nursing facility.

³(E) designates environmental responsibility.

Discussion

Perhaps the most noteworthy result is the substantial level of inappropriate use at the time of the study (as indicated by the percentage of inappropriate patient days among appropriately admitted patients): 32.2 percent of the days reviewed of patients on medical services and 24.3 percent of the days of surgical patients were judged to have been medically unnecessary, as indicated by the AEP review. Given the high average length of stay and the high average cost per stay in Massachusetts, the contribution of inappropriate use to the State's high utilization and cost levels warrants further investigation.

Several of the factors found to explain high rates of inappropriate use point to general strategies for planning and targeting utilization reviews: the fact that longer stays among surgical service patients have higher levels of inappropriate days suggests that utilization review efforts be targeted to especially long stays. Among medical patients, shorter stays have higher levels of inappropriate use, suggesting that such efforts to reduce length of stay be targeted to short stays. In addition, the higher percentages of inappropriate use during the last third of the stay, regardless of the absolute length of the stay or the service, suggests that, as much as is feasible, concurrent review efforts for admitted patients should be conducted prior to the (projected) final third of the stay, to anticipate and prevent inappropriate use during that period.²

The relative lack of impact of PSRO utilization review efforts among the ineffective hospitals in reducing inappropriate days suggests that the final effect of the PSRO's on inappropriateness was dependent on institutional cooperation and not on the efficacy of the monitoring process itself. Thus, the mere presence of a monitoring system cannot necessarily be expected to have an impact. Hospitals (and physicians) must be given incentives, financial or otherwise, to respond to information about their particular inefficiencies by reducing inappropriate use. For example, incentives for hospitals to reduce days of care (but not admissions) are inherent in Medicare's diagnosis-related group (DRG) prospective payment system.

The wide variation in rates of inappropriateness among individual hospitals suggests that efforts to reduce inappropriate use should be specifically targeted for maximum effectiveness. The AEP reasons list provides the sort of detail needed to tailor corrective action to a hospital's or an area's problems. Where physician behavior or hospital habit is responsible, educational efforts focused on specific problems may work, especially if reinforced with appropriate (dis)incentives. Where insufficient equipment or understaffing creates wasteful queues, redistribution of capital resources may be the answer. Where a regional lack of skilled nursing facilities and/or chronic disease hospitals cause a large portion of inappropriate acute hospital days, one may be forced to choose between acceptance of the status quo and development of new nursing or home health care programs, possibly at no net savings for the health care system. Also, use of the AEP to develop physician-specific profiles of inappropriate or nonacute use in one region has been accompanied by the reduction of nonacute patient days of Medicare patients by about one-third (from approximately 25.5 percent to approximately 16.7 percent of the total days reviewed) (Borchardt, 1981). However, a recent AEP study of appropriateness in the Rochester, N.Y. area, which has a well-developed system of sub-hospital-level health care services and facilities and relatively low health care costs, found that less than 10 percent of days were nonacute (Delaney, Restuccia, and Refior, 1984). This level may approach a base below which it is not feasible or optimally efficient to reduce inappropriate use in the shortrun.

The fact that most of the inappropriate use detected was attributed to the physician and/or hospital suggests that the potential payoff of interventions directed at hospital management practices and physician practice patterns can have a high return in reducing inappropriate use.

The complex interactions between hospital type and inappropriateness indicate that further investigation is necessary before definitive conclusions can be drawn regarding the relationship between these two variables. A number of factors not directly included in the study (for example, severity of illness, different levels of expertise in psychiatric illnesses, high technology services) but included by proxies (teaching status, existence of psychiatric units and/or CCU's) could account for the differences among types of hospitals.

The greater impact on inappropriateness of the PSRO utilization review among medical as contrasted with surgical patients is noteworthy. It may reflect that the AEP technique does not question the need for a surgical procedure, and that there is less concensus among clinicians on standards for treating medical patients. The latter is also suggested by the greater variability in costs of physician services observed in medical as compared with surgical DRG's (Mitchell, 1985). This implies that utilization review efforts by PRO's, other medical review organizations, and hospitals should be targeted on those diagnoses in which less consensus does exist.

Finally, the fact that there are differences in inappropriateness across different types of hospitals, among different types of patients, and among different types of hospital stays, reinforces the point that the urgency of cutting hospital costs not be allowed to lead to across-the-board cuts in use. Decreasing total utilization of medical facilities will be a social failure if it is accomplished by cutting appropriate as well as inappropriate use. Considering the magnitude of inappropriate use in our findings (even if some reductions have occurred since the time of the study), there is a great opportunity for savings. Concentration on identifying and correcting the specific problems in each hospital or region is the best and most effective way for the new peer review organizations that have succeeded PSRO's, for individual hospital administrators, and for health insurers and policymakers concerned with reducing health care costs, to proceed.