Quality of care indicators for schizophrenia: determinants of observed variations among Italian Departments of Mental Health. Results from the ETAS DSM study

Abstract

Aims.

The primary aim of this study is to analyse the conformance of usual care patterns for persons with schizophrenia to treatment guidelines in three Italian Departments of Mental Health (DMHs). The secondary aim is to examine possible organisational and structural reasons accounting for variations among DMHs.

Methods.

Within the framework of the Evaluation of Treatment Appropriateness in Schizophrenia (ETAS) project, 20 consensus quality of care indicators were developed. Ten concerned pharmacological treatment and ten encompassed general care and psychosocial rehabilitation interventions. Indicators were calculated using data from a stratified random sample of 458 patients treated at three DMHs located in North-Eastern, North-Western and Southern Italy. Patients’ data were collected by combining information from medical charts and from a survey carried out by the health care professionals in charge of the patients. Data on the structural and organisational characteristics of the DMHs were retrieved from administrative databases. For each indicator, the number and percentage of appropriate interventions with and without moderators were calculated. Appropriateness was defined as the percentage of eligible patients receiving an intervention conformant with guidelines. Moderators, i.e., reasons justifying a discrepancy between the interventions actually provided and that recommended by guidelines were recorded. Indicators based on a sufficient number of eligible patients were further explored in a statistical analysis to compare the performance of the DMHs.

Results.

In the overall sample, the percentage of inappropriate interventions ranged from 11.1 to 59.3% for non-pharmacological interventions and from 5.9 to 66.8% for pharmacological interventions. Comparisons among DMHs revealed significant variability in appropriateness for the indicators ‘prevention and monitoring of metabolic effects’, ‘psychiatric visits’, ‘psychosocial rehabilitation’, ‘family involvement’ and ‘work’. After adjusting the patient's gender, age and functioning, only the indicators ‘Prevention and monitoring of metabolic effects’, ‘psychiatric visits’ and ‘work’ continued to differ significantly among DMHs. The percentage of patients receiving appropriate integrated care (at least one appropriate non-pharmacological intervention and one pharmacological intervention) was significantly different among the three DMHs and lower than expected.

Conclusions.

Our results underscore discrepancies among Italian DMHs in indicators that explore key aspects of care of patients with schizophrenia. The use of quality indicators and improved guideline adherence can address suboptimal clinical outcomes, and has the potential to reduce practice variations and narrow the gap between optimal and routine care.

Introduction

Evidence from literature indicates that the current usual treatment practices fall short of what would be recommended based on the best evidence on treatment efficacy for schizophrenia. This underscores the need for greater efforts to ensure that the treatment research results are translated into practice (Lehman et al. 1998).

The past decade has witnessed an increasing interest in quality of care measures in mental health, with a focus on the dimensions of accessibility, continuity, appropriateness, efficacy and safety in schizophrenia and in general in mental health disorders. However, there is lack of consensus between clinician and stakeholders as to which measurement should be used and how conformance measurement results should be applied and acted upon (Weinmann et al. 2010).

In Italy and in other European countries, the process of deinstitutionalisation and the implementation of a community-based model have not been monitored and evaluated through systematic data collection (Munizza et al. 2011).

In Italy, although a national information system for mental health has been enacted with a Ministry Decree in 15 October 2010, to date the development and calculation of mental health indicators have been carried out mainly at the regional level, except for the SIEP Direct's Project (Semisa et al. 2008) and the inquiry ‘Comparative analysis of the efficiency, quality and appropriateness of the Italian local health trusts’ promoted by the Parliamentary Commission of Inquiry of the Senate on the effectiveness and efficiency of the national health service (Nuti et al. 2014). The SIEP Direct's Project emphasised a marked variability among Departments of Mental Health (DMHs) in the pharmacological treatment of first-episode psychosis and a low frequency of psychotherapeutic, psychosocial and rehabilitative approaches in people with schizophrenia and the final report of Commission of inquiry underlined large intra- and inter-regional differences in hospitalisation rates of individuals with schizophrenia and psychotic disorders.

This suggests that more research efforts should be addressed at benchmarking the performance of mental health services located in different regions, to identify discrepancies in practice patterns, health outcomes and regional usages of resources that cannot be justified by differences in patient needs.

In a previous paper, our research group (Bollini et al. 2008a, b) examined the structural and organisational determinants of quality of care in patients with schizophrenia in two mental health departments in Italy, with a focus on dosage of antipsychotic drugs. We found that higher doses than recommended could be accounted for by the high patient caseload per psychiatrist, which leads to limited contacts with patients and their families and to an overreliance upon drug treatment.

Building upon this previous work (Bollini et al. 2008a, b), we hypothesised that the department's culture, work climate, motivation, structural characteristics and resources would favour the implementation of a measurement-based quality improvement (MBQI) process with a plan-do-study-act cycle. Specifically, our assumption was that the presence of a multi-professional staff, a good working climate, the staff participation in training programmes, well-defined responsibilities and adequate financial resources is the prerequisite to improve the quality of delivered care. Moreover, we assumed that the MBQI can be successful only when health care professionals are willing to be measured, to select meaningful measures and to be proactive in promoting changes (Weinmann et al. 2007; Chou et al. 2011).

The MBQI process was implemented in two steps. First, recommendations were extracted from international guidelines on schizophrenia, revised and operationalised in 15 indicators (nine concerning drug treatment and six psychosocial treatment) that were calculated for 807 patients with schizophrenia treated in some DMHs of the Piedmont Region, Italy (Bollini et al. 2008a, b). A characteristic element of this experience was the application of eligibility criteria to treatment (the criteria that the patient must have, to be included in the recommendation), compliance (the criteria that must be present to say that the recommendation has been satisfied) and the inclusion of moderators’ factors (factors that may explain the lack of application of a given recommendation). These include, for instance, the presence of severe physical comorbidity, or side effects that would contraindicate the use of recommended drug treatment or explicit refusal to take the drug.

The Evaluation of Treatment Appropriateness in Schizophrenia (ETAS) study was then developed in continuity with this latter project to promote the quality of care in patients with psychotic spectrum disorders (ICD10 F20-F29) in Italy through the involvement of mental health staff in the critical appraisal of existing guidelines and the evaluation of care appropriateness in routine mental health care. This was done through the selection of a number of additional consensus indicators identified starting from guidelines recommendations, the calculation of the indicators using data from a sample of patients with psychotic spectrum disorders attending three Italian DMHs, a review of the results with the operators and the identification of possible measures to address the criticalities emerged.

The primary aim of this paper is to analyse the conformance of usual care patterns for persons with schizophrenia to treatment guidelines. The secondary aim is to examine the organisational and structural reasons underlying the observed variations in care provided to patients with schizophrenia in order to assess their potential role as determinants of the observed appropriateness.

Methods

Setting

The study was carried out at three DMHs located in different areas of Italy. One Department is located in North-Eastern Italy (NE), in an autonomous province with a catchment area of 513 357 inhabitants that comprises a town and the surrounding mountain area. The province is one of the wealthiest in Italy and is a renowned destination for summer and winter tourism in the Dolomites. However, the geographical characteristics of the area cause some problems of accessibility to health care services. Another (NW) is located in the suburban area of a metropolitan town of North-Western Italy, with a catchment area of 202 400 inhabitants and the third (SO) is located in a small town of Southern Italy and has a catchment area of 115 232 inhabitants (20% of the population of the region). The three DMHs can be considered the representative of their geographical area and were chosen because they provided the full involvement of the DMHs staff and management that was required for this study.

Study design

This is an observational study based on primary and secondary data collected on patients and DMHs.

Participants

In each participating DMH, during an index period of 13 days in the years 2009 (NE) or 2010 (NW and SO) a list of patients being treated was prepared using data from the mental health information system or from paper archives. Inclusion criteria were: an ICD-10 F20-F29 diagnosis, age ≤65 years and having at least one contact with the DMH facilities in the previous year. Patients who died or were transferred to another DMH during the previous year were excluded. Patients being in an acute episode were identified during 2 index days among those hospitalised for the first psychotic episode.

For each DMH, a statistician extracted a random sample stratified by the administrative district with a sampling ratio that was approximately 20% in NW and NE and 50% in SO, to obtain a comparable number of patients. This was done using a predefined list of random numbers matched to patients’ alphabetical list. Because patients with psychosis in the three DMHs could seek treatment at any site according to their degree of disability, level of symptoms and family ties, we sampled all of the facilities available in the catchment area for acute inpatient care, long-term residential care, outpatient clinics, rehabilitation centres and day hospitals.

Quality of care indicators

The selection of indicators was conducted by 30 health care professionals (psychiatrists, psychologists, nurses, educators, social workers) of the participating institutions during a 2-day meeting held in Pergine on 20–21 May 2009. Specifically, the health care professionals reviewed the 15 original indicators reported by Bollini et al. (2008a, b) and nine new indicators from the most recent PORT, NICE, Canadian Psychiatric Association, Royal Australian and New Zealand College of Psychiatrists guidelines identified as relevant by two independent researchers. The health care professionals assigned each indicator a colour according to foreseen problems in their implementation: green (no problem), red (not applicable in the Italian context or having a low level of support from guidelines) and yellow (problematic, in need of adaptation or definition of moderators specific to the local context). Three red indicators were excluded, green indicators were retained and regarding yellow indicators, participants were asked to provide written comments by e-mail after the meeting. After collecting comments, excluding one of the yellow indicators and refining the definition of indicators and their moderators, the final consensual set of 20 quality of care indicators was defined. Ten concerned pharmacological treatment and ten encompassed general care, psychosocial rehabilitation interventions, patient's work potential and collaborative decision-making with the patient's family.

The source guidelines and the criteria making up the indicators are listed in Table 1. Pharmacological indicators covered essential principles of treatment with antipsychotic drugs, namely adequate dosage, length of treatment, monotherapy, use of depot antipsychotics and management of extrapyramidal adverse reactions. Prevention and monitoring of metabolic effects was included among pharmacological treatment indicators based on evidence that second-generation antipsychotics can induce serious metabolic dysregulations, especially in drug-naive, first-episode populations, with olanzapine and clozapine having the highest likelihood to cause these abnormalities. Non-pharmacological indicators encompassed psychiatric interviews, psychosocial interventions, family involvement and vocational interventions. Appropriateness was defined as the percentage of eligible patients receiving an intervention conformant with guidelines. Moderators, i.e., reasons justifying a discrepancy between the interventions actually provided and that recommended by guidelines were recorded.

Table 1.

Indicators of conformance to treatment guidelines for patients with schizophrenia and schizoaffective disorders

Pharmacological indicators
	Indicator	Eligibility	Appropriateness	Moderators	Source
P1	Prescription of an antipsychotic drug for the treatment of acute symptoms	The patient is having an acute episode	On the day of the survey the patient should be prescribed atypical (SGA) or typical (FGA) antipsychotic medication	The patient has a serious comorbidity, which may interfere with antipsychotic medication or has experienced a severe adverse reaction to at least two antipsychotics over the past 6 months, or the patient or the family refuses treatment with antipsychotic drugs	McEvoy rec. 10A PORT rec.1 NICE rec. 1.2.3 PORT update rec.1 NICE 2009 6.2.1 Canadian Psychiatric Association (e.l. A)
P2	Dose of an antipsychotic drug at the first acute episode	The patient is having the first acute schizophrenic episode	On the day of the survey, the patient should be taking an atypical (SGA) or typical (FGA) antipsychotic medication at a daily dose between 300 mg and 500 mg equivalents of chlorpromazine	Specific comorbidities; first trimester gestation; the patient or family refuse the treatment with antipsychotic drug	McEvoy rec. 17A PORT rec. 3 PORT update rec. 3
P3	Daily dose of antipsychotic drugs at the second and subsequent acute episodes	The patient is having a second or subsequent acute schizophrenic episode or schizoaffective disorder	On the day of the survey, the patient should be taking an atypical (SGA) or typical (FGA) antipsychotic medication at a daily dose between 300 mg and 1000 mg equivalents of chlorpromazine	Comorbidities; presence of side effects in the last 6 months to at least two antipsychotic medications (hypersensitivity, malignant neuroleptic syndrome, etc.), or the patient or family refuse the treatment with antipsychotic medication	McEvoy rec. 7A PORT rec. 2 NICE rec. 1.3.2.3 PORT update rec. 2 Royal Australian and New Zealand College of Psychiatrists Canadian Psychiatric Association (e.l. A)
P4	Length of antipsychotic treatment in the maintenance phase, ≤12 months after the resolution of the last acute episode	The patient has had an acute episode within the past 12 months, which was resolved before the day of the survey	On the day of the survey the patient should be taking an atypical (SGA) or typical (FGA) antipsychotic medication	Comorbidities; presence of side effects in the last 6 months to at least two antipsychotic medications (hypersensitivity, malignant neuroleptic syndrome, etc.), or the patient or family refuse the treatment with antipsychotic medication	McEvoy rec. 6 PORT rec. 8 NICE rec. 1.3.3.7 PORT update rec. 4. Canadian Psychiatric Association (e.l. B) NICE 2009
P5	Dose of antipsychotic drugs in the maintenance phase	The patient has had an acute episode within the past 12 months, which was resolved before the day of the survey	On the day of the survey the patient should be taking an atypical (SGA) or typical (FGA) antipsychotic medication at a daily dose between 300 mg and 600 mg equivalents of chlorpromazine	Comorbidities; presence of side effects in the last 6 months to at least two antipsychotic medications (hypersensitivity, malignant neuroleptic syndrome, etc.) or the patient or family refuse the treatment with antipsychotic medication	McEvoy rec. 7A PORT rec. 9 PORT update rec. 5 Canadian Psychiatric Association (e.l. B)
P6	Prescription of a depot antipsychotic to patients with inadequate compliance in the maintenance phase	The patient has had an acute schizophrenic episode within the past 12 months, which was resolved before the day of the survey, and had an inadequate compliance to treatment documented in the medical chart at least twice over the past 5 years	Prescription of depot medication	Severe side effects (hypersensitivity, malignant syndrome neuroleptic, severe extrapyramidal effects, metabolic syndrome, diabetes familiarity, etc.) over the last 6 months after the resolution of the last acute episode; comorbidities that may interfere with the administration of antypsychotic medication; explicit refusal of the patient and/or family	McEvoy rec. 3B PORT rec. 12 NICE rec. 1.4.5.8 PORT update rec. 6 Royal Australian and New Zealand College of Psychiatrists Canadian Psychiatric Association (e.l.B) NICE 2009
P7	Prescription of clozapine	Prescription of clozapine on the day of the survey	The patient has a schizophrenia or schizoaffective disorder diagnosis; has good compliance to treatment (<2 episodes of non-compliance over the past 5 years); the patient has received two antipsychotics, one of which is second generation, for at least 6 weeks at therapeutic doses with no effect; and no history of haematological disorders	Not applicable	McEvoy rec. 2 PORT rec. 13 NICE rec 1.4.5.14 and 1.4.5.15 PORT update rec. 8 Royal Australian and New Zealand College of Psychiatrists Canadian Psychiatric Association (e.l. A) NICE 2009
P8Aa	Monitoring, management of extrapyramidal side effects, level A	The patient had a diagnosis of schizophrenia or schizoaffective disorder and had at least one extrapyramidal side effect recorded in the medical chart	Decrease of the dose of the typical antipsychotic, prescription of an antiparkinson drug, or switch to a second generation antipsychotic (excluding clozapine, unless the patient was treated with an atypical antipsychotic before)	Inadequate compliance with treatment documented in the medical chart at least twice over the past 5 years. Comorbidities which exclude the switch to an oral or depot atypical medication	PORT rec. 16 NICE rec. 1.3.2.5 and 1.3.2.6 PORT update
P8Ba	Prevention and monitoring of metabolic effects, level B	The patient has a diagnosis of schizophrenia or schizoaffective disorder; has started, for the first time, to take an atypical drug in the last 12 months and is continuing treatment	Measurement of the initial weight, at the beginning of the treatment with an atypical neuroleptic. Blood work performed for estimating glucose and lipid metabolism. A family history of diabetes in the medical chart	Patient's refusal	Marder et al. (2004) ADA/APA, 2004
P9	Monotherapy with antipsychotic drugs	The patient has a diagnosis of schizophrenia or a schizoaffective disorder. The patient is not hospitalised at the time of the survey	The patient should be taking one antipsychotic medication	Two antipsychotic drugs were prescribed in less than 6 weeks (or 8 weeks if one agent is clozapine) during the switch phase	NICE rec. 1.3.2.8 and 1.4.5 Royal Australian and New Zealand College of Psychiatrists NICE 2009
Non-pharmacological indicators
	Indicator	Eligibility	Appropriateness	Moderators	Source
NP1	Psychiatric interviews	The patient has a diagnosis of schizophrenia or schizoaffective disorder. The patient is in day hospital treatment, has had no acute crisis and has not been hospitalised before the survey	At least one visit of 30 minutes or more by the psychiatrist in charge has taken place in 2 months before the survey	The patient has severe comorbidity; or has missed the scheduled appointment	McEvoy rec. 10b PORT rec. 10c NICE rec. 1.4.2.1
NP2	Psycho-social rehabilitation	The patient has a diagnosis of schizophrenia or schizoaffective disorder. The patient is not hospitalised at the moment of the survey	The patient has attended at least ten rehabilitation sessions (social, educational, occupational) in the month before the survey	Severe comorbidities that prevent the patient from attending rehabilitation sessions. The patient has severe comorbidity or has good or fair social functioning (SOFAS score ≥ 55). Refusal of the rehabilitation proposals. Travel time over 30 minutes to reach the usual rehabilitation sessions sites (CSM or CD)	McEvoy rec. 10 PORT rec. 23 Royal Australian and New Zealand College of Psychiatrists
NP3Ad	Collaborative decision making with the patient's family Level A.	The patient has a diagnosis of schizophrenia or schizoaffective disorder The patient is in regular contact with the family or lives with them	The family participates in a cycle of psycho-educational encounters conducted by operators with specific training in the last 5 years	Explicit refusal by patient to involve the family	Canadian Psychiatric Association (e.l. B) Royal Australian and New Zealand College of Psychiatrists
NP3Bd	Collaborative decision making with the patient's family Level B.	The patient has a diagnosis of schizophrenia or schizoaffective disorder. The patient is in regular contact with the family or lives with them	At least one meeting, planned in advance, between the patient's family and a mental health professional in charge of the patient has taken place in 12 months before the survey	Explicit refusal by patient to involve the family	McEvoy rec. 10 Canadian Psychiatric Association Royal Australian and New Zealand College of Psychiatrists
NP4	Assessment of the patient's work potential	The patient has a history of employment or has expressed the wish to find a job and is not working or is in a vocational training programme on the day of the survey. The patient is <65 years old	At least one interview has been conducted to assess the patient's work potential in 2 years before the survey	The patient has poor social functioning (SOFAS score ≤ 30)e or severe comorbidities	McEvoy rec. 10 PORT rec. 27 NICE rec. 1.4.6.1
NP5	Work	Schizophrenia or schizoaffective disorder. The patient is <65 years old	The patient works or is enrolled in vocational rehabilitation training on the day of the survey	The patient does not wish to work, has poor social functioning (SOFAS score ≤ 30)e, has a serious comorbidity, or is 60–65	McEvoy rec. 10f PORT rec. 27g NICE rec. 1.4.6.2 and 1.4.6.3 NICE 2009 rec. 9.6.7.1 Canadian Psychiatric Association (e.l. B) Royal Australian and New Zealand College of Psychiatrists PORT update rec. 16
NP6.1h	Assertive community treatment level A	The patient lives on his own, with the family or with someone in charge, the patient is not hospitalised at the moment of the survey and was not hospitalised in the previous month, has poor social functioning (<55). The patient had repeated hospitalisations in the past 2 years, inadequate compliance with treatment or important residential problems	At least four sessions has been made in the previous month, by more than a mental health professional in charge	The patient accepts a rehabilitation psycho-social programme and has a good social functioning	McEvoy rec. 10 PORT rec. 29 NICE rec. 1.4.3 NICE 2009 rec. 9.4.7.1 Royal Australian and New Zealand College of Psychiatrists PORT update rec. 17
NP6.2	Cognitive Behavioural Therapy Level A	The patient is in an acute phase with positive or negative symptoms, also during recovery. The patient is in subsequent phases, with enduring positive or negative symptoms	At least ten consecutive sessions of Behaviour Cognitive Therapy have been offered by the mental health professional in charge with specific training during the 2 (or 5) years before the survey	SOFAS score <30 Explicit refusal by patient to participate in the sessions	NICE 2009 rec. 8.4.10.1 Canadian Psychiatric Association (e.l. B) Royal Australian and New Zealand College of Psychiatrists PORT update rec. 19
NP6.3	Art therapy Level B	Schizophrenia or schizoaffective disorder. The patient is in an acute, also during recovery. Patients in subsequent phases of the disorder	At least ten consecutive sessions of art therapy (art therapy, music therapy, drama therapy, dance therapy) have been offered by the mental health professional in charge with specific training during the 5 years before the survey	Refusal to an art therapy proposal	NICE 2009 rec. 8.3.8.1 and 8.3.8.3
NP6.4	Support and self-mutual help groups, level B	Schizophrenia or schizoaffective disorder. The patient is not hospitalised at the moment of the survey	Regular attendance to support groups and self-mutual help in the 2 years before the survey	Explicit refusal by patient to participate. Distance from the meeting site (≥30 min of travel) Severe physical comorbidities	Canadian Psychiatric Association

^aThe indicators monitoring and management of side effect have two levels of evidence. Level A corresponds to interventions supported by the majority of guidelines for which good scientific evidence suggests that the benefits of the clinical intervention/procedure substantially outweigh the potential risks. Level B corresponds to interventions with weaker levels of evidence and not supported by the majority of guidelines. Level B indicators have been included because they were deemed pertinent and significant for the clinical practice of Italian mental health services. Level A and B interventions are not mutually exclusive.

^bMcEvoy and colleagues recommended physician appointments in conjunction with non-physicians.

^cPORT recommended ongoing assessment of dosage level or the need for maintenance therapy.

^dThe indicators related to family involvement have two levels of evidence A and B, as defined in (a).

^eSOFAS (Social and Occupational Functioning Assessment Scale). Scores range from 0 to 100, with scores ≥55 indicating good social functioning and scores ≤30 indicating poor social functioning.

^fMcEvoy and colleagues recommended vocational rehabilitation services.

^gPORT recommended the availability of a range of vocational rehabilitation services, particularly supported employment.

^hInterventions considered for assertive community treatment include the following: help at home for activities of daily living, family interventions, help with finding a job or supported employment, contact or interaction with other agencies, help with prevention and health care, check of compliance with antipsychotic medication, assistance with neighbours and property owner, help with budgeting and counselling.

McEvoy: (McEvoy et al. 1999).

PORT: (Lehman et al. 1998).

NICE: (National Collaborating Centre for Mental Health Commissioned by the National Institute for Health and Clinical Excellence, 2002).

PORT update: (Lehman et al. 2004).

NICE 2009: (National Collaborating Centre for Mental Health Commissioned by the National Institute for Health and Clinical Excellence, 2009).

Canadian Psychiatric Association: (Canadian Psychiatric Association, 2005).

Royal Australian and New Zealand College of Psychiatrists (McGorry, 2005).

Marder et al. 2004: (Marder et al. 2004).

ADA/APA, 2004: (American Diabetes Association et al. 2004).

Data sources

Patients’ data were obtained by combining secondary data (patients’ medical records) with primary data on interventions and moderators through an ad hoc survey. The survey was conducted by contact persons including all the DMH professionals who were in charge of the selected patients. They were asked to fill out an ad hoc form including patients’ demographic information, psychiatric history, past and current drug treatments, side effects of drugs, the presence of comorbid physical illnesses, psychosocial interventions, family involvement, the breadth of patient's social network, participation in community life, functioning (measured on a 0–100 scale using the Social and Occupational Functioning Assessment Scale (Morosini et al. 2000) and severity of psychotic symptoms (suspiciousness, hallucinations, unusual thought content, conceptual disorganisation, rated on a 1–7 scale).

The contact person was supervised by a senior DMH professional who was in charge of the quality of the data collected. The senior DMH professional conducted the descriptive statistical analyses on patients’ characteristics and provided anonymised aggregated data to the main research team for the analysis of indicators.

Data on the structural and organisational characteristics of the DMHs and on budget allocation were retrieved from the administrative databases of the DMHs or of the Local Health Authority and were updated to the last available year.

Patients’ consent to the use of the collected data was not required because data were analysed in anonymised form, in compliance with the Italian ‘Code of conduct and professional practice applying to processing of personal data for statistical and scientific purposes’ enforced by this Authority http://www.garanteprivacy.it/web/guest/home/docweb/-/docweb-display/docweb/1115480 (Published in the Official Journal no. 190 of August 14, 2004).

The study protocol was approved by the institutional review boards of each of the three local authorities whose DMHs participated in the study.

Statistical analysis

The characteristics of patients were compared among the DMHs using analysis of variance (ANOVA) F or χ² test, where appropriate. Following significant tests, post hoc pairwise comparisons with Bonferroni corrections were conducted at p < 0.016 to adjust for multiple comparisons.

For each indicator, the number and percentage of appropriate interventions and of inappropriate interventions with and without moderators were calculated. Percentages of appropriate care, inappropriate care with moderators and inappropriate care without moderators were compared among DMH using the χ² test. Multinomial logistic regression was used to adjust these comparisons for patients’ gender, age and functioning.

Results

Characteristics of the DMHs

Structural and organisational characteristics of the DMHs are provided in Table 2. The three DMHs differed in many respects: NE had the lowest mean number of community mental health centre (CMHC) opening hours (60.9) and of patients with schizophrenia per full-time equivalent operator (3.6), the largest treated prevalence (240.2) and the highest number of general hospital psychiatric ward (GHPW) beds per 100 000 inhabitants (9.2). NW had fewer beds in GHPW and residential facilities but longer hospitalisations. The number of patients with schizophrenia per operator was similar between NW and SO, but the treated prevalence was lower in NW.

Table 2.

Characteristics of the three departments of mental health

Department	NW	NE	SO
Catchment area	202 400	513 357	115 232
Mean weekly opening hours per CMHC	61.3	60.9	78.0
Number of beds in GHPW × 100 000 inhabitants	4.5	9.2	6.9
Number of beds in residential facilities × 100 000 inhabitants	34.1	40.7	53.8
Average hospital stay (days)	24.3	15.3	14.0
Total no. of patients with schizophrenia aged ≤65 years	430	1233	264
Treated prevalence of schizophrenia × 100 000 inhabitants	212.5	240.2	229.1
Patient caseload per operator (FTE)	6.6	3.6	6.3
Physicians (FTE)	13.6	48.0	9.0
Nurses (FTE)	36.0	143.3	19.0
Rehabilitation operators (FTE)	NA	59.2	NA
Other staff (FTE)	10.0	4.8	7.0
Costs of personnel (€)	4 540 643	13 720 714	2 506 028
Total budget (€)	14 353 581	30 564  402	6 040 353

CMHC, community mental health centre; GHPW, general hospital psychiatric ward; FTE, full-time equivalent; NA, not applicable.

Patients’ characteristics

The study sample included 458 patients with schizophrenia recruited from three DMHs. Patients’ characteristics broken down by DMH are provided in Table 3. Patients were predominantly male, living with their original family, with a secondary school diploma, not working, with a limited social network and with an established and long-term relationship with mental health services (mean time in contact with mental health services: 14.0 years). Significant differences were found in the case mix of patients in the three DMHs on living arrangement (p = 0.003), working status (p = 0.001) and functioning (p < 0.001).

Table 3.

Characteristics of the study samples in the three departments of mental health

	NW (N = 93)	NE (N = 264)	SO (N = 101)	Missing data, n (%)	ANOVA or χ2 test, p-value
Gender (% M)	66.7	56.7	51.5	1 (0.2%)	0.092
Age (mean ± s.d.)	43.1 ± 10.3	46.1 ± 10.8	45.8 ± 10.4	7 (1.5%)	0.065
Education (%)				26 (5.7%)	0.042
Primary school	17.6	11.6	21.2
Secondary school	57.1	49.6	41.4
High school	23.1	30.6	32.3
University	2.2	8.3	5.1
Living arrangement (%)				4 (0.9%)	0.003
Alone	8.6	26.0	23.2
Original family	41.9	35.5	45.5
Own's family	24.7	23.7	14.1
Community	23.7	12.6	17.2
Other	1.1	2.3	0.0
Working status (%)				5 (1.1%)	0.001
Professional/executive	0.0	1.2	0.0
Employee (high-level)	2.2	4.2	0.0
Self-employed	3.2	6.1	1.0
Employee (low-level)	12.9	13.4	4.0
Temporary worker	8.6	17.2	18.2
Unemployed	26.9	11.1	19.2
Not working	46.2	46.7	57.6
Illness duration (mean ± s.d.)	14.5 ± 8.8	14.6 ± 10.1	16.7 ± 10.3	42 (9.2%)	0.186
Time in contact with mental health services (mean ± s.d.)	13.6 ± 8.8	13.6 ± 9.5	15.6 ± 10.2	36 (7.9%)	0.203
Alcohol use (%)	7.7	6.3	10.3	15 (3.3%)	0.455
Substance use (%)	4.4	3.2	8.2	15 (3.3%)	0.127
Social network (%)				8 (1.7%)	0.039
No relations other than family	8.6	2.0	1.0
Minimum	49.5	52.3	57.4
Intermediate	39.8	41.4	37.6
Wide	2.2	4.3	4.0
Participation in community life (%)				11 (2.4%)	0.606
No contacts	26.9	20.2	15.8
Few	45.2	51.4	50.5
Several	26.9	26.5	31.7
Active participation	1.1	2.0	2.0
Functioning (mean ± s.d.)	47.3 ± 14.1	55.9 ± 16.6	53.7 ± 13.3	−	<0.001

NE, North-East; SO, South; NW, North-West.

ANOVA was used for continuous variables (age, illness duration, time in contact with MHS, functioning); χ² test for categorical variables (gender, education, living arrangement, working status, alcohol use, substance use, social network, participation in community life).

Appropriateness indicators

Table 4 provides, for each indicator and DMH, the number of eligible patients, the percentage of patients with appropriate care and with inappropriate care, and the percentage of patients with inappropriate care who had moderators to explain the inappropriateness. Four indicators (NP3A, NP6.2, NP6.3 and NP6.4) could not be computed because no patient met criteria to be included in the numerator of the indicator. This means that, although guidelines recommend family psychotherapy, cognitive behavioural therapy and art therapy, and regular attendance to support groups and self-mutual help, the use of these treatments was virtually absent.

Table 4.

Number and percentages of patients receiving appropriate or inappropriate care in the three departments of mental health

Indicator	% Appropriateness	NW (N = 93)	NE (N = 264)	SO (N = 101)	χ2 test, p-value
P1, Prescription of an antipsychotic drug for the treatment of acute symptoms	Appropriate	0	2 (100.0%)	0	NA
	Inappropriate with moderators	0	0	0
	Inappropriate without moderators	0	0	0
P2, Dose of an antipsychotic drug at the first acute episode	Appropriate	0	8 (33.3%)	0	NA
	Inappropriate with moderators	0	6 (25.0%)	0
	Inappropriate without moderators	0	10 (41.7%)	0
P3, Daily dose of antipsychotic drugs at the second and subsequent acute episodes	Appropriate	2 (50.0%)	7 (50.0%)	4 (50.0%)	NA
	Inappropriate with moderators	2 (50.0%)	5 (35.7%)	3 (37.5%)
	Inappropriate without moderators	0	2 (14.3%)	1 (12.5%)
P4, Antipsychotic dosage during the maintenance phase ≤12 months after the resolution of the last acute episode	Appropriate	11 (100.0%)	37 (88.1%)	13 (100.0%)	0.213
	Inappropriate with moderators	0	5 (11.9%)	0
	Inappropriate without moderators	0	0	0
P5, Antipsychotic dosage during the maintenance phase (at least one acute episode in the last 12 months)	Appropriate	5 (45.5%)	16 (38.1%)	4 (30.8%)	0.651
	Inappropriate with moderators	2 (18.2%)	17 (40.5%)	5 (38.5%)
	Inappropriate without moderators	4 (36.4%)	9 (21.4%)	4 (30.8%)
P6, Prescription of depot antipsychotics to patients with a history of non-compliance (no acute episode in the last 12 months)	Appropriate	1 (8.3%)	9 (30.0%)	9 (34.6%)	0.089
	Inappropriate with moderators	3 (25.0%)	11 (36.7%)	3 (11.5%)
	Inappropriate without moderators	8 (66.7%)	10 (33.3%)	14 (53.8%)
P7, Prescription of clozapine	Appropriate	6 (54.5%)	14 (77.8%)	2 (28.6%)	0.066
	Inappropriate with moderators	0	0	0
	Inappropriate without moderators	5 (45.5%)	4 (22.2%)	5 (71.4%)
P8A, Monitoring, management and treatment of extrapyramidal side effects (Level A)	Appropriate	6 (60.0%)	30 (71.4%)	7 (77.8%)	0.700
	Inappropriate with moderators	2 (20.0%)	6 (14.3%)	0
	Inappropriate without moderators	2 (20.0%)	6 (14.3%)	2 (22.2%)
P8B, Prevention and monitoring of metabolic side effects (Level B)	Appropriate	37 (55.1%)	11 (6.5%)	42 (70.0%)	<0.001
	Inappropriate with moderators	2 (3.0%)	3 (1.8%)	3 (5.0%)
	Inappropriate without moderators	28 (41.8%)	154 (91.7%)	15 (25.0%)
P9, Monotherapy with antipsychotic drugs	Appropriate	64 (71.9%)	174 (74.7%)	73 (74.5%)	0.874
	Inappropriate with moderators	0	0	0
	Inappropriate without moderators	25 (28.1%)	59 (25.3%)	25 (25.5%)
NP1, Psychiatric interviews	Appropriate	81 (89.0%)	154 (61.6%)	61 (62.9%)	<0.001
	Inappropriate with moderators	2 (2.2%)	27 (10.8%)	17 (17.5%)
	Inappropriate without moderators	8 (8.8%)	69 (27.6%)	19 (19.6%)
NP2, Psychosocial rehabilitation	Appropriate	19 (22.4%)	25 (10.6%)	17 (17.9%)	0.001
	Inappropriate with moderators	43 (50.6%)	177 (75.3%)	64 (67.4%)
	Inappropriate without moderators	23 (27.1%)	33 (14.0%)	14 (14.7%)
NP3B, Family involvement	Appropriate	60 (75.6%)	131 (59.3%)	68 (74.7%)	0.007
	Inappropriate with moderators	1 (1.3%)	18 (8.1%)	2 (2.2%)
	Inappropriate without moderators	18 (22.8%)	72 (32.6%)	21 (23.1%)
NP4, Assessment of patient's work potential	Appropriate	7 (41.2%)	8 (29.6%)	2 (13.2%)	0.537
	Inappropriate with moderators	2 (11.8%)	3 (11.1%)	2 (13.2%)
	Inappropriate without moderators	8 (47.1%)	16 (59.3%)	11 (73.3%)
NP5, Work	Appropriate	31 (33.3%)	118 (44.7%)	29 (28.7%)	<0.001
	Inappropriate with moderators	40 (43.0%)	117 (44.3%)	72 (71.3%)
	Inappropriate without moderators	22 (23.7%)	29 (11.0%)	0
NP6.1, Other psychosocial interventions – assertive community treatment	Appropriate	0	0	0	NA
	Inappropriate with moderators	0	3 (50.0%)	1 (25.0%)
	Inappropriate without moderators	0	3 (50.0%)	3 (75.0%)

P, pharmacological; NP, non-pharmacological; NE, North-East; SO, South; NW, North-West; NA, not applicable.

The χ² test compares all the percentages among DMHs. Denominators for the calculation of indicators are eligible patients.

In the overall sample, the level of conformance of pharmacological treatment to treatment guidelines exhibited a wide variability. During the maintenance phase, the antipsychotic dosage appeared conformant to guidelines in the large majority of patients (92%), virtually all, if patients with moderators are included. The prescription of depot to patients with non-compliance was appropriate in 27.6% of patients, with a range from 8.3% in NW to 34.6% in SO. Monitoring of extrapyramidal effects was appropriate in more than 60% of patients, while prevention and monitoring of metabolic effects was appropriate only in 30.5%. Regarding monotherapy, inappropriateness without moderators (i.e., unnecessary polytherapy) was found in 26.0%.

As to non-pharmacological indicators, conformance to guidelines for psychiatric interviews and family involvement was generally high. However, psychosocial rehabilitation was inappropriate in a large number of patients, although most of these cases were accounted for by the presence of moderators. Similarly, involvement in vocational rehabilitation training was appropriate in 38%, but an additional 50% had moderators preventing from work. Notably, assessment of patients’ work potential was poor (59.3% inappropriate without moderators).

Comparison of indicators among DMHs

Given the constraints related to the inclusion criteria and the lack of eligible subjects, only 16 indicators out of the 20 initially identified could be compared among the three DMHs to identify possible heterogeneity in the quality of care provided. For the other four indicators (P1, P2, P3 and NP6.1), the limited number of eligible subjects for one or more DMHs precluded a meaningful statistical analysis. Significant differences were found for indicators ‘Prevention and monitoring of metabolic effects’, ‘Psychiatric interviews’, ‘Psychosocial rehabilitation’, ‘Family involvement’ and ‘Work’ (Table 4). Specifically, appropriateness for monitoring metabolic side effects ranged from 6.5 to 70.0%, for psychosocial rehabilitation ranged from 10.6 to 22.4%, for psychiatric interviews ranged from 61.6 to 89.0% and for vocational training programmes from 28.7 to 44.7%. Compared with the other DMHs, higher level of inappropriateness were found at NE for indicators P8B (Prevention and monitoring of metabolic side effects), NP1 (Psychiatric interviews), NP3B (Collaborative decision making with the patient's family) and at NW for indicators NP2 (Psycho-social rehabilitation) and NP5 (Work). For three of these indicators (‘Prevention and monitoring of metabolic effects’, ‘Psychiatric interviews’ and ‘Work’) differences persisted after adjustment for patients’ gender, age and functioning in multinomial logistic regression.

Lastly, we calculated the percentage of patients receiving appropriate integrated care (at least one appropriate non-pharmacological intervention and one pharmacological intervention). The three DMHs exhibited different levels of integrated care, with a predominance of drug treatment alone at NE and SO and a higher proportion of patients receiving both pharmacological and psychosocial treatment at NW. Specifically, at NE 56.1% received drug treatment alone, 38.3% the combination of drug treatment and psychosocial interventions, and 5.7% other interventions. At SO the percentages were 69.3, 27.7 and 3 and at NW 47.3, 49.5 and 3.2%, respectively (χ² test = 11.9, p = 0.018).

Discussion

This study examined the results of the implementation of a set of consensual quality of care indicators based on guidelines for treatment and management of schizophrenia. Our approach implied an active involvement of the staff in a quality improvement process, consistent with existing MBQI models in Italy focused on care processes (Nuti et al. 2016) and with institutional accreditation system processes that have been implemented since 1990 when the Italian national health system changed to a corporate model (Rossi et al. 2014).

Our findings concerning pharmacological indicators showed that drug treatment during the maintenance phase and monitoring of extrapyramidal side effects were appropriate in the large majority of patients and across all DMHs. In the previous study conducted by our group using the same indicators (Bollini et al. 2008a, b), dosage appropriateness during the maintenance was lower, ranging from 25 to 34%.

Regarding monotherapy, the percentage of patient receiving inappropriate treatment (i.e., polytherapy) in the present study (26%) was higher than a recommended cut-off of 10% (Weinmann et al. 2010), but lower than the percentage of polytherapy (40.6%) reported in an Italian study conducted in 2007 (Santone et al. 2011).

Prescription of a depot antipsychotic in patients with non-compliance was appropriate in 8 to 35% of patients. Results concerning the SIEP Direct's Project (Semisa et al. 2008) indicate that a depot antipsychotic was administered to more than 75% of patients with poor treatment compliance in 14/19 centres surveyed.

The indicator ‘monitoring of metabolic side effects’ showed a high variability among DMHs. One possible explanation is that when the study was conducted there was no concern about metabolic effects of second-generation antipsychotics. In fact, until the publication of the Lancet editorial ‘No mental health without physical health’ in 2011 (Tiihonen et al. 2011), monitoring of physical health was not deemed relevant and was not included among the qualifying goals of the DMHs. So, variability among DMHs was related to the local sensitivity to this problem. Our findings are consistent with the results of SIEP Direct's Project (Semisa et al. 2008), indicating a low attention on physical health across Italian DMHs. Still, because health care setting for persons with mental disorders encompasses other settings outside of mental health (such as primary care), we argue that both primary care and mental health providers might be involved in the management of physical conditions co-occurring with mental disorders.

The appropriateness in the use of clozapine varied from 28.6 to 77.8%, but the small sample size of eligible patients prevents from drawing conclusions.

Non-pharmacological indicators showed a large discrepancy in appropriateness among DMHs, with a higher focus on psychosocial rehabilitation, family involvement and assessment of work potential in NW, while the higher appropriateness achieved at NE for the Work indicator was related to the availability of financial resources provided by the autonomous province.

Concerning organisational determinants, we noted that inappropriateness in the monitoring of side effects, psychiatric interviews and family involvement was high in NE. However, in this DMH patients’ work or enrolment in vocational rehabilitation training was appropriate in 44.7% (reflecting a high reliance on community resources for patient's employment) and attendance in training courses was wider than in NW or SO. On the contrary, results concerning treatment appropriateness in NW indicate a high attention on psychosocial treatments and metabolic complications of antipsychotic treatment. This is consistent with the large percentage of patients receiving psychosocial interventions in addition to pharmacotherapy.

In SO, the small number of beds coupled with the high number of CMHC opening hours and the high level of appropriateness in psychiatric interviews indicates that the model in place relies mostly on community resources for treatment, although vocational rehabilitation and assessment of work potential is appropriate only for a minority of patients.

The predominance of pharmacological treatment alone in our sample (where the mean time in contact with mental health services is 14 years) raises concerns about the distance between the current available care in the Italian Mental Health Departments and the suggested ‘best practice’ of integrated care both for the short- and long-term patients (Lenroot et al. 2003; Van Os & Kapur, 2009).

Our findings should be interpreted keeping in mind strengths and limitations. One strength is the availability of data from different sources, not limited to patients’ clinical charts and the involvement of staff members in the survey. Another strength is that indicators are compared among DMHs after adjusting for demographic characteristics, which reduces a possible bias related to case mix imbalances. Limitations include the small number of patients eligible for the calculation of some indicators, that did not allow to obtain reliable evidence regarding specific aspects of care, the heterogeneous sample, that did not allow us to aggregate our quality indicators into one single adherence index and the cross-sectional study design, that does not allow us to determine whether appropriate care is associated with better outcomes. However,Weinmann et al. (2007) suggested that there is insufficient high-quality evidence to draw firm conclusions about the effects on outcomes of the implementation of specific psychiatric guidelines. To address this last limitation, we designed a second (ongoing) study on a sample of first-onset schizophrenia patients to assess the impact of appropriateness in the early stages of treatment on medium to long-term outcomes. Moreover, our methodology relies strongly on information provided by key clinical informant(s); therefore, it is not generalisable to contexts in which such ad hoc surveys are not feasible. We were in fact aware from our previous experience with medical chart review (Bollini et al. 2008a, b) that some information is reliably recorded (including diagnosis, current pharmacological treatment or episodes of non-compliance), while other information (i.e., psychosocial interventions at the community level, family meetings, reasons for changing medication, presence of side effects) is usually limited to brief and haphazard notes, making it impossible to define conformance and eligibility criteria on the basis of medical records alone. We tried to overcome this limitation by entrusting the collection of data to the professionals in charge of the patients, who complemented data extraction from medical records with his/her first-hand knowledge of patients’ history or by interviewing other staff members involved in the care of the patient. In order to be used in daily routine, this data collection methodology would require the adoption of electronic medical records in which information on functioning, side effects of drugs, refusal to take drugs and family involvement is collected.

In summary, the variability shown by indicators measured in DMHs located in three distant Italian regions confirms the notion that each region has its own treatment model (Ferrannini et al. 2014), in which adherence to guidelines for pharmacological treatment and psychosocial treatments may vary depending on available resources and clinical attitude for integrated care.

Thus, benchmarking among DMHs in the same region and across regions becomes more and more important to ensure horizontal equity of treatment in patients with severe mental illness. It should be noted that in Italy, where the National Health System is based on a corporate model, an institutional accreditation system is in force and no professional accreditation system has been implemented based on performance. Kilbourne et al. (2010) argued that performance incentives such as pay-for-performance should be given at the group and not individual provider level, and should be rewarded based on incremental changes rather than attaining absolute benchmarks. This would reduce costs of performance incentives and maximise the potential for addressing the system-level deficiencies in care.

In the present study, feedback on local results was provided to the staff of each DMH and the Heads of the three DMHs had full autonomy with respect to the implementation of any corrective strategies or incentives.

In conclusion, we argue that MBQI cannot rely on routinely collected data only, but requires the active involvement of health care professionals for the identification of areas for improvement and promoting subsequent actions (Chou et al. 2011). The use of consensus indicators adapted to the local context, in a bottom-up perspective, is a key ingredient to undertake a quality improvement initiative and favour the harmonisation of practices.

Financial Support

This research received no specific grant from any funding agency, commercial or not-for-profit sectors.

Conflict of Interest

None.