Factors in the Quality of Patient Evaluations in General Hospital Psychiatric Emergency Services

Abstract

Objectives

The study examined the usefulness of a three-perspective model for determining the quality of evaluations in psychiatric emergency services. The model was used to evaluate the hypothesis that the provision of high-quality care in emergency services is primarily influenced by service objectives related to patients’ clinical characteristics rather than by institutional constraints, such as workload or physical facilities, or by social biases, such as clinicians’ attitudes toward patients or perceptions of community expectations.

Methods

The evaluations of 683 persons assessed in nine California public facilities were independently observed. Multivariate techniques were used to test the relative importance of patients’ clinical characteristics, possible sources of social bias among clinicians, and institutional constraints in influencing three quality-of care dimensions: technical quality, the art of patient care, and optimum investment of time.

Results

The findings generally confirmed the hypothesis that patients’ clinical characteristics have more influence on the quality of care provided than institutional constraints or social biases. However, one institutional constraint–increased workload demands–led to reduced technical quality and to less than optimal use of time. Further, social biases reflected in the clinician’s like for and preconceptions about the patient also influenced the quality of their evaluations.

Conclusion

The model is a useful tool for examining quality of care in the psychiatric emergency service. Increasing workload pressures negatively affect quality of care.

Although most evaluations for psychiatric inpatient care occur in general hospital psychiatric emergency services, studies have failed to adequately measure the quality of this process of care or why it varies between settings (1–6). As cost conservation strategies become the norm, preventing sacrifices in the quality of care will be of increasing importance (7).

This objective can be achieved only by adequately defining quality of care in the psychiatric emergency service and by understanding factors that account for its variability. This study used validated instruments to examine the relationship between quality of care in the psychiatric emergency service and three sets of factors: service objectives, institutional constraints, and social biases of clinicians.

Supporting a multimethod approach to quality improvement, the General Accounting Office (GAO) has emphasized that program goals should be better articulated and that a consistent framework for measuring quality and performance characteristics should be established (8). Managed care organizations stress better specification of patient problems and treatment objectives as well as enhanced measurement and analytic efforts (9–11). Both the GAO report and the approach taken by managed care emphasize the need to be responsive to various stakeholders. Both quality concepts are all-inclusive; they refer to system change, change in patient care, and better measurement of all aspects of process and outcomes related to the care structure.

This study focused on the quality of the care process. Our model follows Donabedian’s classic formulation (12,13) of quality of care, linking the caring process to the structure within which it occurs. Elsewhere we have reported on the relation of structure and process to outcomes in the psychiatric emergency service (14).

Quality of care is difficult to define and measure (15, 16), and no single definition has been universally accepted. We have adopted the approach of Rundall and Gardner (17), who note that defining quality depends on one’s frame of reference (18), and thus definitions vary among stakeholders (10, 11). Quality of care may be defined from the perspective of patients, health care providers, and administrators. Patients judge quality by the nature of the communication with the provider (art of care). Providers judge it by the extent to which professional standards are met (technical quality). Administrators value optimizing the investment of time in the tasks required to complete a high-quality evaluation (efficiency).

Although it is difficult for any evaluation to address all sources of variation, three groups of factors influence these three quality dimensions. The first group hinges on the service objectives of the psychiatric emergency service–providing clinically and legally appropriate patient dispositions. Thus the patient’s status as assessed by admission criteria is likely to be associated with the quality of care provided (19–21).

A second set of factors likely to affect the quality of care–institutional constraints–links the care process to the structure of the general hospital psychiatry emergency service. Structure includes the attributes of the personnel providing care, the specific setting, and the organizational context.

The third set of factors–social biases–links quality of care to the clinician’s perceptions about patients and to his or her humanity. Examples include whether the clinician likes the patient, whether the patient was referred by police, whether the patient has nowhere to go on discharge, and the degree to which the patient has been a nuisance in the community (such as engaging in harmless but socially inappropriate behavior in the presence of public officials).

In sum, the study reported here provides a model for evaluating quality of care in the psychiatric emergency service and for determining the relative importance of service objectives, institutional constraints, and social biases as determinants of quality of care in this setting.

Methods

Sample and data collection

Experienced mental health professionals independently observed patient assessments in nine psychiatric emergency services in California public general hospitals over a five-year period (1981–1986). A total of683 patients assessed during that time were included in the sample. An incoming patient was included if both an independent observer and a staff clinician were available for the patient’s evaluation. The observers gathered data by using structured instruments and by reviewing patients’ charts. The patients included were assessed during time intervals spread across the entire spectrum of the day and week. The characteristics of the sample are described elsewhere (22,23).

Measure of the art of care

The Art of Care Scale operationalizes Brook and Avery’s concepts (24,25) about high-quality care. It addresses the clinician’s attempt to engage in a collaborative interaction, elicit information, include the patient in planning, and attend to and respond empathically to the patient’s feelings at a level appropriate to the patient’s level of functioning. This four-item additive index was converted to reflect a proportion of optimal efforts to engage the patient; possible scores ranged from 0 to 1.

Measure of technical quality of care

The Quality of Care Index (26) reflects the professional technical standard for quality of care (the provider’s perspective). The measure was developed by Johnson and associates (26), who asked two panels of psychiatrists and physicians experienced in emergency psychiatric assessment to identify the components of an evaluation, translate them to item format, and weight them according to relative importance using nominal group process techniques (27). The panels then used the 27item instrument to rate the quality of patient evaluations based on patients’ records; the correlation between the mean ratings of the two panels was .89.

Measure of optimum investment of time

This measure matches the complexity of the patients’ clinical needs and presentation with the amount of time allocated to attend to them. The “optimum time” measure is the difference between the time allocated to clinical tasks and the estimated time to complete a high-quality evaluation (18). A negative score on this measure indicates that time was conserved at the expense of quality; a positive score indicates that more time was provided than was required for a high-quality evaluation.

Measures of the sources of variation in quality-of-care admission criteria

The American Psychiatric Association’s model law on civil commitment describes criteria for involuntary admission that focus the psychiatric emergency service evaluation (28). Four of the most critical aspects of the assessment are determining whether the person is suffering from a severe mental disorder, whether the disorder is treatable, whether the person would benefit from hospitalization, and whether as a result of severe mental disorder the person is in a dangerous state–that is, likely to cause harm to self or others or to suffer substantial mental or physical deterioration.

In this study severe mental disorder, the first assessment concern, was defined by the clinician’s diagnosis of a major mental disorder based on DSM-Ill criteria (29). Although the treatability of acute symptoms of serious mental illness is often thought of as dependent on the availability of effective medications, treatability of the disorder, the second concern, has other dimensions. Mainly, they include the patient’s cooperation and compliance, or potential to become engaged in a collaborative effort. The Treatability Scale codifies these indicators and has been construct-validated in work demonstrating the effects of changes in the psychiatric emergency service’s ideology on decision making.

The Hospital Benefit Scale, a measure of the third assessment criterion, estimates the extent to which the patient is perceived as potentially benefiting from hospitalization. Most of the items in the Hospital Benefit Scale were adapted from the work of Henisz and associates (30). Higher scores indicate that the patient is likely to benefit from hospitalization. The scale has been construct-validated (Segal SP, Watson MA, New hill C, unpublished data, 1992).

The fourth and final admission criterion, dangerousness, was measured by the Three Ratings of Involuntary Admissibility scale (TRIAD), a previously developed instrument that operationalizes the clinical concept of dangerousness (28,31). The scale includes subscales assessing danger to self, danger to others, and grave disability, the three statutory grounds for involuntary commitment under the dangerousness standard. The subscale scores are combined into a global assessment of perceived dangerousness. TRIAD is scored from an independent observer’s checklist of applicable items. Previous studies have produced strong evidence for the reliability and construct validity of TRIAD (32–34).

Six institutional constraints were assessed in this study. They include the clinician’s workload, which was measured by a three-item factor score based on the patient-staff ratio, the clinician’s patient load, and the total number of inpatient beds available (the number of inpatient beds loaded negatively on the factor); difficulties in the physical setting; whether the clinician’s ethnicity matched the patient’s; whether the clinician spoke the same language as the patient; whether the patient had insurance coverage; and the clinician’s experience (in years).

Sources of social bias included demographic descriptors associated with discrimination, including age, gender, and minority status, and factors likely to incite significant community reaction if disposition of the case was later questioned, such as the degree to which the patient was a nuisance in the community, whether the patient had been referred by police, and whether the patient had nowhere to go on discharge from the emergency service. Direct measures of the clinician’s attitude were also made. They were based on the observer’s evaluation of whether the clinician liked or disliked the patient and whether the assessment was affected by the clinician’s preconceptions about the patient’s needs. Attitudinal assessments were made on the basis of direct verbalizations and other actions taken by the clinician. For example, although in one case the clinician seemed to be doing a wonderful job with the patient, he privately described the individual as a “real dirt bag.”

Analysis

Three multivariate regression models, one for each quality criterion, indicated how assessment criteria were associated with quality when controlling for institutional constraints and social bias sources. We entered independent variables in three steps: institutional constraints, social bias sources, and assessment criteria.

Two additional confirmatory analyses were run. First, we regressed quality-of-care variables directly on the assessment variables, which produced results similar to those reported below for the hierarchical analysis. Second, we added a variable to the complete list of factors considered. This additional variable, an institutional constraint, indicated whether a less-restrictive alternative was available at the time of the evaluation. We then reran each of the three complete models. The variable was not significant in any model and downgraded the sample size by 32 percent. We therefore eliminated this variable from the final analyses to preserve statistical power.

Results

The art-of-care model

As shown in Table 1, institutional constraints, social bias sources, and admission criteria, as groups of indicators, contributed equally to explaining scores on the Art of Care Scale. The admission criteria of treatability, dangerousness, and major mental disorder were significant variables in this model. As expected, treatability, which was the strongest variable in the equation, was positively related to the scale scores, while dangerousness and major mental disorder were negatively associated. The only social bias indicator that positively affected the art-of-care scores was whether the clinician liked the patient, which was the second-strongest variable. Length of clinician experience was negatively related.

Table 1.

Hierarchical regression analyses of factors affecting three aspects of quality of care of evaluations in psychiatric emergency services of nine California general hospitals¹

	Art of care (N = 663)			Technical quality (N = 649)			Optimum time (N = 680)
Independent variable	β	p	R2	β	p	R2	β	p	R2
Institutional constraints (step 1)			.07			.02			.04
Clinician’s experience	−.19	.00		.08	.03		.09	.02
Clinician-patient language match		ns			ns			ns
Clinician-patient ethnic match		ns			ns			ns
No insurance		ns			ns			ns
Clinician’s workload		ns		−.09	.02		−.13	.00
Difficult setting		ns			ns		.10	.01
Social bias indicators (step 2)			.06			.04			.02
Clinician’s liking of the patient	.21	.00			ns			ns
Clinician’s preconceptions about the patient		ns		−.12	.00			ns
Female patient		ns			ns		.10	.00
Patient age		ns			ns			ns
Patient ethnicity		ns			ns			ns
Police referral		ns		.18	.00		.11	.00
Patient’s nuisance score		ns			ns		−.09	.01
Homeless and resourceless		ns			ns			ns
Previous hospitalizations		ns			ns			ns
Admission criteria (step 3)			.06			.09			.03
Benefit from hospitalization		ns		.16	.00		.11	.00
Treatability	.22	.00		.21	.00		.09	.02
Dangerousness	−.09	.02		.16	.00		.08	.05
Major mental disorder	−.10	.01			ns		.08	.05
Model adjusted R2†			.19			.15			.09

¹Only β coefficients with p values less than .05 are reported because others were deemed not significantly different from zero. Adjusted R² values are reported.

^†p < .001 for all model R² values

Technical quality of care

Scores on the Quality of Care Index, which measured technical quality, were positively associated with dangerousness, treatability, and potential benefit from hospitalization. As shown in Table 1, the patient’s status as assessed by admission criteria contributed more to explaining the variance in technical quality scores than did either institutional constraints or social bias indicators 11.97, 19,629, p<.01).

Among social bias sources, police referral and the clinician’s preconceptions about the patient’s needs were significantly associated with technical quality of care. Patients referred by police received better evaluations. Clinicians’ preconceptions were associated with poorer technical quality.

Among institutional constraints, workload and experience were significantly associated with technical quality of care. Workload was negatively associated, while greater clinician experience was positively associated.

Optimum time model

As Table 1 shows, all three groups of indicators contributed about equally to predicting deviations from time allotted to complete a high-quality evaluation.

Scores on all four admission criteria were significantly associated with less deviation from the estimated time to complete a high-quality interview. The more a patient conformed to each criterion, the smaller the deviation–that is, clinicians were not forced to save time at the expense of the performance of quality-related tasks when patients more clearly matched one or more of the criteria.

Three institutional constraints–clinician experience, workload, and difficulties in the physical setting–were also associated with deviations from time required for high-quality care. Workload was the strongest variable in the equation. As workload increased, clinicians tended to save time at the expense of performing tasks that contributed to the quality of the evaluations. More experienced clinicians were less likely to save time at the expense ofthe performance of these tasks. Difficulties in the setting were associated with spending more time than the average allotted to complete a high-quality evaluation.

In this study, quality of care in the emergency service was most dependent on the patient’s clinical characteristics rather than on biases of the clinician or on institutional constraints.

Three bias indicators were also significant predictors of quality. Evaluations of males and of patients considered a community nuisance were more likely to suffer from time saving than other sample members. Evaluation of police referrals was less likely to be done too hastily than was evaluation of others in the sample.

Discussion

Although caution must be exercised in generalizing the results reported here beyond the nine California general hospitals in which the observations were completed, this study offers the first empirical model for evaluating factors associated with quality of care in the psychiatric emergency service.

Scores on admission criteria specified in the American Psychiatric Association’s model law on civil commitment (28) were associated with artful care, technical quality, and optimum time use, even when many potentially confounding factors were controlled. This finding runs contrary to those of studies indicating that evaluations in the psychiatric emergency service are cursory and biased (1–6), primarily influenced by social and demographic criteria, and lacking an appropriate quality-of care model.

However, clinicians do have human frailties. On the one hand, the technical quality of care provided in the emergency service was found to be positively associated with the criteria of dangerousness and severe mental disorder, which suggests that clinicians were more careful with patients who met these criteria. On the other, the negative association between the art of care and major mental disorder and dangerousness indicates that clinicians were less likely to involve sicker and more dangerous clients in the treatment process. “Good” patients–those who were treatable and likable–and “difficult” high-risk patients–those who were dangerous or psychotic–were more likely than other patients to receive specialized, technically correct evaluations in the psychiatric emergency service. Good and likable patients received more artful care than difficult patients.

Clinicians’ preconceptions about patients and their liking for patients were also associated with quality of care. Liking was positively associated with artful care, and preconceptions were negatively associated with technical quality. If we think of liking as positive affect and preconceptions as a closed cognitive state, this correspondence seems natural. Liking positively influences the affective or process side of quality–the art of care. Preconceptions negatively influence the cognitive side of quality–technical quality.

Patients referred by police drew the attention that we would expect. They received better technical evaluations than the average patient, and clinicians invested more time in tasks leading to high-quality evaluations.

Clinicians’ experience, an institutional constraint, was positively associated with technical quality and optimal use of time but negatively associated with the art of care. We tentatively interpreted this pattern as consistent with burnout.

Finally, workload was associated with two of the three quality measures. Increased workloads apparently caused clinicians to save time by sacrificing technically correct approaches, negatively affecting technical quality-of-care scores.

Conclusions

This study examined quality of care from three perspectives: the patient’s, the provider’s, and the administrator’s. From all perspectives, we found that quality of care in the emergency service was most dependent on the patient’s clinical characteristics rather than on biases or institutional constraints. Clinicians were responsive to community concerns, handling the evaluations of police referrals more carefully and paying special attention to technical quality in these evaluations.

Administrators and policy makers must be attentive to the effects of increasing workload demands in the psychiatric emergency service. Such increases did not lead to efficiency but to reductions in the technical quality of care and to less than optimal use of time. Cost conservation efforts in mental health should be carefully reviewed in light of these findings.