Patient experiences with care across various types of mental health care: Questionnaire development, measurement invariance, and patients' reports

Abstract

To describe the development, validation, and findings of a patient experience questionnaire across 7 types of residential and ambulatory mental health care services. Thirty‐five items were hypothesized to cover information, participation, therapeutic relationship, personalized care, organization and collaboration, safety, patient rights, outcomes of care, and discharge preparation and after‐care. Also included were 2 overall rating items (scoring and recommending the organization). This Dutch questionnaire was applied in 79 organizations in Belgium (N patients = 5,168). Exploratory structural equation modelling was conducted on a random split‐half sample to examine dimensionality. Confirmatory factor analysis and multiple group confirmatory factor analyses were conducted on the holdout sample to confirm dimensionality and assess measurement invariance across type of service and patient characteristics. Multilevel logistic regression models linking subscale top box scores to overall rating items were used to assess criterion validity. The hypothesized dimensionality was partly confirmed, and configural and scalar invariance were demonstrated across types of organizations and patient characteristics. Subscale scores were significantly associated with overall ratings. Process evaluation showed that participating organizations strongly support continued use of this questionnaire. This validated patient experience questionnaire supports comparison across organizations from different types of services to improve the quality of mental health care.

1. INTRODUCTION

Evaluating overall and specific patient experiences with care was put high on the agenda after Donabedian's (1988) transformative ideas on how to assess the quality of care. By now, measures of patient experiences with health care are widely used for ranking, public reporting, and value‐based purchasing in general hospitals (Delnoij, 2009; Giordano, Elliott, Goldstein, Lehrman, & Spencer, 2009). Several studies in this setting have linked patient experiences to other dimensions of quality (Aiken et al., 2012; Bruyneel et al., 2015; Jha, Orav, Zheng, & Epstein, 2008). Also in mental health care settings associations between patient experiences and objective quality measures have been shown. In the United States, better experiences were linked with higher follow‐up rates, promptness of follow‐up, and continuity of outpatient care (Druss, Rosenheck, & Stolar, 1999), as well as with appropriate technical quality of care (Edlund, Young, Kung, Sherbourne, & Wells, 2003). Patients staying at psychiatric wards in general hospitals in France who had higher satisfaction rates about their treatment reported higher levels of quality of life and scored higher on treatment adherence (Zendjidjian et al., 2014). In Belgium, higher perceived patient involvement was found to be related to better patient experiences (Tambuyzer & Van Audenhove, 2015). The National Institute of Health and Care quality standard on service user experience in adult mental health services includes 15 quality statements. All explicitly refer to generating evidence from experience questionnaires to measure progress (National Institute for Health and Care Excellence, 2011).

Though the studies mentioned above show the importance of measuring patient experiences in mental health care, no studies have so far constructed a measure of patient satisfaction mapping both global patient satisfaction and specific aspects of patient experiences (as suggested by Donabedian) in a methodologically sound way that involved patients and other stakeholders. Additionally, existing questionnaires are often specifically designed for specific types of mental health services. In the United Kingdom for example, the National Health Service (NHS) Community Mental Health Survey is one of the most widely used questionnaires but focuses on one specific type of mental health service solely (Picker Insititute, 2016). In France, a recently developed questionnaire focuses on hospitalized patients (Zendjidjian et al., 2015). In this study, we describe the development, validation, and detailed findings of a cross‐sectoral patient experience questionnaire in Flanders, Belgium. These efforts are embedded in a region‐wide quality improvement initiative initiated in 2016. This initiative aims to implement a core set of quality indicators to create transparency to the public and enhance quality of mental health care. Professionals, patients, and mental health care organizations have initiated the development of five indicators that are specific and relevant to both residential and ambulant care: presence of peer workers, a suicide prevention policy, a complete medication order, timely ambulant contact after discharge, and, as described in this study, patient experiences with care (Flemish Government, 2016).

2. METHODS

2.1. Questionnaire development

Thirty‐six validated patient‐experience questionnaires were identified through a scoping literature review. Patient representatives and health care professionals from all types of mental health services in Flanders were invited to separate focus group‐type brainstorm sessions about the quality of mental health care. Six patient representatives and 18 professionals participated in 2‐hr sessions, which were moderated by two staff members with an academic background working at the Flemish Patient Platform. Patient and stakeholder participation has previously been shown to be an excellent approach for guiding quality indicator selection (Baribeau, Wong, Monga, Pignatiello, & Ickowicz, 2016). Our approach resulted in a master list of 1,074 items. The same two staff members independently excluded items lacking clarity, items not applicable to the Flemish context, and items not applicable to all types of mental health services.

Next, health care professionals from all types of mental health services selected in two Delphi‐rounds (n in round 1 = 60; n in round 2 = 52) 63 of the remaining items as relevant. Forty‐eight patient representatives subsequently selected 37 of 63 items as both relevant and important. Last, health care professionals and patient representatives finalized the questionnaire. Special attention was given to patient representatives' suggestions on clear wording of the items. The first version of the Flemish Patient Survey of Mental Healthcare includes 8 demographic items, 2 items reflecting global rating, and 35 core questions hypothesized to measure nine domains: information about mental health problems and treatment, participation, therapeutic relationship, personalized care, organization of care and collaboration between professionals, safe care, patient rights, result and evaluation of care, and discharge management and aftercare. Response categories for all items were “never,” “sometimes,” “mostly,” and “always,” except for the two items of discharge management which were scored as “yes” or “no.” All items for the nine domains are shown in Table 4 in the results section. The two global rating items are overall rating of the organization (scored from “0” to “10”), and willingness to recommend the organization (“definitely yes,” “probably yes,” “probably no,” “definitely no”). Demographic items included gender, living situation, age, treatment duration, work situation, health status, education and language. Questionnaire language was Dutch.

Table 4.

Standardized loading pattern for exploratory structural equation modelling with nine specific factors

Item	F1	F2	F3	F4	F5	F6	F7	F8	F9
Information about mental health problems and treatment
1. I received information about my mental health problems.	.795	−.062	.042	.019	.026	−.053	.031	.151	.053
2. I received information about treatment options for my mental health problems.	.986	−.007	−.036	.064	−.054	.042	.021	−.036	.023
3. I received information about my treatment.	.743	.089	.101	−.063	.008	.037	.007	−.019	.027
4. I received information about my medication and possible side effects.	.416	−.109	.292	−.173	.230	.045	.115	.060	−.063
5. I received information about the financial cost of my treatment.	.243	.239	.285	−.216	.133	.103	−.007	.078	−.153
Participation
6. My caregivers encourage me to decide on my treatment, examinations and/or tests.	.250	.234	.441	−.012	.063	.151	.059	−.067	.025
7. I can decide on my treatment.	.135	.185	.580	.087	.014	.178	−.023	−.033	.073
Therapeutic relationship
8. My caregivers explain things in a way I could understand.	.254	.218	.262	.106	.205	−.111	.120	.104	.082
9. My caregivers treat me with courtesy and respect.	.107	.214	.168	.378	.091	.063	.156	.124	.012
10. I have confidence in my caregivers.	.094	.158	.103	.590	.139	.024	.198	.043	.009
11. I feel free to ask my caregivers questions about my treatment and my medication.	.122	.081	.193	.242	.283	−.031	.146	.069	.061
Personalized care
12. My caregivers understand my mental health problems.	.210	.389	.048	.204	.104	.087	−.013	.133	.036
13. My caregivers ask me which problems or complaints are most important to treat.	.210	.467	.067	−.134	.120	.025	−.022	.141	.126
14. My caregivers act on my possibilities rather than my limitations.	.059	.687	.063	.067	.065	−.031	−.017	.090	.074
15. My caregivers look at my whole person rather than at my problems only.	−.116	.819	.028	.098	.000	.007	.084	.069	.003
16. My caregivers encourage me to take responsibility for my care.	.092	.656	.082	−.058	.014	.123	.085	−.060	.075
17. My caregivers pay attention to my physical health.	.039	.319	.038	.075	.201	.117	.210	−.083	.094
Organization of care and collaboration between professionals
18. My caregivers provide similar information.	.094	.189	.023	.056	.110	.023	.422	.184	.004
19. Within this center, my caregivers have good working relationships.	.043	.035	−.058	.143	.087	.114	.564	.120	.061
20. My caregivers collaborate well with external caregivers or services like my general practitioner, independent psychotherapist, …	.067	−.025	−.062	−.019	.167	.278	.394	−.067	.115
21. My caregivers take enough time to care for me.	−.001	.094	.013	.002	.742	.073	.118	−.039	.038
22. I can easily get an appointment with a caregiver from this center.	−.075	−.056	.065	.001	.730	−.011	.133	.096	.082
Safe care
23. During the first contact, my caregivers introduce themselves by name and function.	.088	.207	−.008	−.143	.205	.132	.057	.334	−.109
24. I feel safe during my treatment.	.139	.072	−.072	.260	.202	.122	−.094	.301	.008
25. I think my caregivers respond well when I say that I am not doing well.	113	.157	−.102	.267	.372	.202	−.110	.334	−.013
26. I am satisfied with the arrangements made in case I am not doing well.	.089	.076	−.016	.238	.494	.189	.127	.111	.035
27. I think my caregivers are good at their job.	.099	.074	−.032	.041	.082	.096	.154	.580	.085
Patient rights
28. My caregivers respect my privacy.	.119	.124	−.045	.341	.205	.121	.223	.129	.147
29. My caregivers ask for my permission to share information about me with external caregivers.	.019	.231	.073	−.080	−.126	.217	−.015	.403	.078
30. I have the opportunity to choose a different caregiver.	−.073	−.026	.120	.097	.027	.726	.066	.133	−.023
31. I received information about my right to ask for a second opinion.	.045	−.005	−.019	−.071	.004	.838	.033	−.038	.070
Result and evaluation of care
32. My caregivers and I assess on a regular basis the progress of treatment.	.162	.174	−.078	−.204	.242	.050	−.039	.020	.551
33. I feel like my treatment has helped me.	.212	.131	−.098	.261	.197	.180	−.062	−.003	.291
Discharge management and aftercare
34. I have a say about when my treatment ends.	−.086	−.023	.428	.118	−.034	.062	.105	.241	.474
35. My caregivers and I made arrangements about where I go to after my treatment or in case my problems return or worsen.	.067	.009	.036	.046	−.033	.102	.151	−.025	.643

Notes. F1–F9 = specific factors 1 to 9. Wording of the questionnaire items slightly varied across types of services. Questionnaire items referred to the specific setting where the patient was treated. In the version for sheltered housing and psychiatric nursing homes, “caregivers” was replaced by “mentors” and “treatment” was replaced by “mentoring program.” Bold items: factor loading larger than .300 and no cross loadings larger than .300. Items are arranged according to the hypothesized dimensions.

2.2. Setting and participants

A call for participation was launched by the Flemish Patient Platform and a convenience sample of 79 organizations was obtained. Adult patients admitted to one of the types of mental health care services (Table 1) were invited to participate to the study. Patients could only complete the questionnaire after at least 4 days of admission (psychiatric hospital, general hospital psychiatric ward, psychosocial rehabilitation, and psychiatric nursing home) or at least four sessions or contacts (sheltered housing, ambulatory public funded mental health services, and assertive community teams). Non‐Dutch speaking patients were excluded. Also, since our focus was on organizations, patients treated by a self‐employed caregiver who did not link to a mental health care service were excluded. Last, patients were excluded if they were cognitively unable to complete the questionnaire. Data collection took place in April 2016. Participation was voluntary, without financial incentives, and respondents were guaranteed anonymity.

Table 1.

Types of services available to patients with a psychiatric condition

Residential care ▪ Psychiatric hospitals provide specialized long‐term (>3 weeks) residential psychiatric care. ▪ General hospital psychiatric wards provide short‐term (<3 weeks) residential psychiatric care to adults (>16 years) having a psychiatric crisis. ▪ Sheltered housing offers support for patients with a long‐term psychiatric condition who live independently in the organization's property. ▪ Psychiatric nursing home offers 24‐hr support for patients with a long‐term but stabilized psychiatric condition. Ambulatory care ▪ Ambulatory public‐funded mental health services offer various type of care, all through consultation and coaching only by appointment. ▪ Psychosocial rehabilitation offers support for patients with the objective of resocialization. ▪ Assertive community teams (mobile teams) offer intensive and highly integrated psychiatric support to outpatients in their home environment.

Ethical approval was provided by the Medical Ethics Committee KU Leuven and all local ethics committees of participating organizations.

2.3. Process evaluation

Following administration of the questionnaire, a process evaluation was organized among all participating organizations. A web questionnaire was used to focus on organizations' endorsement of the questionnaire, the full process from distributing the questionnaire among patients to uploading data to the data warehouse, and implementation of the questionnaire in the long term.

2.4. Statistical analysis

We determined the construct‐relevant dimensionality of our questionnaire items and assessed whether this dimensionality fits our population and allows comparison of subgroups of our population. This was done in three steps.

In step 1, we assessed multidimensionality from various angles through exploratory structural equation modelling (ESEM) and bifactor ESEM (B ESEM). ESEM is a more recently introduced technique that allows exploratory factor analysis to combine with aspects of structural equation models (Asparouhov & Muthén, 2009). We opted for an ESEM approach instead of regular exploratory factor analysis as we anticipated that we would need to overcome restrictions from our subsequent confirmatory factor analysis (Marsh, Morin, Parker, & Kaur, 2014). This would not turn out to be the case. Bifactor analysis (Jennrich & Bentler, 2011; Reise, 2012) was added within the ESEM approach. This allows observed items to have two sources of common variation, that is, a general factor that explains item intercorrelations and a number of specific factors that capture the item covariation that is not explained by the general factor. ESEM and B ESEM were conducted on a random split‐half sample, stratified by type of service.

In step 2, an independent cluster confirmatory factor analysis (ICM‐CFA) was used to investigate whether the established ESEM dimensionality and factor‐loading pattern fit the holdout sample (i.e., the other half of the sample).

In step 3, also in the holdout sample, we assessed measurement invariance across type of service and patient characteristics. This was to evaluate whether the same model fit the data across groups of our population. We did not study invariance across Dutch and non‐Dutch speaking patients because of the low proportion of non‐Dutch speaking patients. For the other groups, we first assessed model fit in each group separately. We then conducted multiple group ICM‐CFA to assess various types of invariance (Steenkamp & Baumgartner, 1998). Configural invariance pertains to showing the same pattern of association between items and factors, and the same number of factors. In this model, factor loadings and thresholds are free across groups. Metric invariance refers to equality of factor loadings. Scalar invariance refers to equality of factor thresholds and is a requirement for drawing meaningful comparisons across groups. Since several groups did not contain all values for multiple items, all items were dichotomized. To be consistent in our approach and interpretation, this was done throughout all three steps described above. For ESEM and B ESEM, we used weighted least squares estimation using delta parameterization and oblique Crawford‐Furguson‐quartimax rotation and oblique bifactor Crawford‐Furguson‐quartimax rotation, respectively. For multiple‐group ICM‐CFA, we used weighted least squares estimation using delta parameterization. In this case, only the configural and scalar models are considered since the metric model is not identified. For identification purposes, in multiple‐group analyses factor variances and latent means were fixed to be 1 and 0, respectively (Muthén & Muthén, 2012). Model fit evaluation was based on Hu and Bentler's (1999) cutoff criteria and Chen's (2007) allowed changes in these fit indices when studying invariance for the comparative fit index (Bentler, 1990; ranges between 0 and 1; reasonable if >.90 and very good if >.95; change of ≥−.01 indicates noninvariance, and comparative fit index is the main criterion for assessing change), the Tucker–Lewis index (Tucker & Lewis, 1973; ranges between 0 and 1; reasonable if >.90 and very good if >.95), and the root mean square error of approximation (Steiger, 1990; ranges between 0 and 1; reasonable if <.08 and very good if <.05; change of ≥.015 indicates noninvariance).

Criterion validity was assessed using multilevel logistic regression analysis to estimate the association between factors and global rating of the hospital. Patient clustering within organizations was taken into account through random intercepts for organizations. Subscales were calculated on the basis of top‐box scores (for the items that in ESEM showed to have a factor loading larger than .300; as a rule of thumb, interpretation was also theory driven) and which showed no high cross loadings—also note that this dimensionality was confirmed in ICM‐CFA and multiple‐group ICM‐CFA. In the logistic model, these subscales were operationalized as the sum of top‐box scores. For example, for a subscale with four items, if a patient responded “always” to two of four items, 2 was included as a value for the covariate. Only patients who completed all items for a given subscale were included in the analysis. For descriptive analyses, for a subscale with four items, top‐box scores were calculated as the average percentage of patients that responded “always” to the four items. The top‐box percentage would be 30% if 20% of patients rated “always” to item 1, 25% rated “always” to item 2, 35% rated “always” to item 3, and 40% rated “always” to item 4. Global rating was operationalized as overall rating (dichotomized “9” and “10” as 1 versus “0” to “8” as 0) and willingness to recommend the organization (dichotomized “definitely yes” as 1 versus “probably yes”; “probably no” and “definitely no” as 0). In a hospital setting, these cut points for overall ratings have been found to relate well to structures and process of care (Aiken et al., 2012; Bruyneel et al., 2015; Jha et al., 2008). These top‐box scores and cut points are also used for public reporting in the United States (Centers for Medicare & Medicaid Services, 2011; Giordano et al., 2009) as well as in Belgium (Vlaamse Overheid, 2016). Also in the NHS Community Mental Health Survey, global ratings are dichotomized (Picker Institute, 2016).

Mplus version 7.1 was used to estimate structural equation models (Muthén & Muthén, 2012). The descriptive analysis and multilevel regression analysis for this paper were generated using SAS software, Version 9.4 of the SAS System for Windows. Copyright ©2016 SAS Institute Inc. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc., Cary, NC, USA.

3. RESULTS

3.1. Process evaluation

Fifty‐seven organizations participated. All types of sectors were represented. Fifty‐six of fifty‐seven respondents indicated their role in the organization: board of directors (n = 7), staff members (n = 38), direct patient care (n = 4), nurse managers (n = 3), quality coordinators (n = 2), medical secretary (n = 1), and psychologist (n = 1).

Respondents appreciated clear communication about the objectives and contents of the questionnaire, the clear inclusion and exclusion criteria, and questionnaire length. Many reported a strong engagement of everyone involved within the organization for aiding in data collection. All organizations confirmed that only patients meeting the inclusion and exclusion criteria completed the questionnaire. However, one organization reported that it had provided a questionnaire to non‐Dutch speaking patients, without indicating in which other languages it provided the questionnaire, and another indicated that also minors (adolescents) completed the questionnaire. These records were excluded from the analysis above. Informed consent procedures were viewed as too complex. Contrary to its objective, this led to suspicion about anonymity, which was reported as one of the main reasons not to complete the questionnaire. Across all sectors, organizations indicated that various types of patients received help to complete the questionnaire, and that only a Dutch version of the questionnaire is too restrictive. Post data collection, participants said that manual input was too time‐consuming. This related to the main negative point, mentioned by a number of organizations, which was the short time span in which the data collection and uploading needed to be completed. Whereas several organizations indicated that they would prefer continuous measurement of each discharged patient, a large majority (n = 35, 64%) indicated that it would not be possible to do two rounds a year. This would be too expensive and time‐consuming. Regarding the contents of the questionnaire, multiple residential care organizations indicated that questions on the hotel functions of the organization would be a welcome addition. It was also mentioned several times, across various types of sectors, that more differentiation in the word hulpverlener (caregiver) was requested by patients. That is, they lacked specific questions about nurses, psychologists, psychiatrists, and other types of mental health care professionals.

3.2. Sample

Five thousand one‐hundred sixty‐eight participants from 79 organizations completed the questionnaire. Table 2 shows the number of organizations and number of participants across types of services. It is also shown that, while there are more female respondents in psychiatric wards in a general hospital, ambulatory public‐funded mental health services and assertive community teams, the inverse is observed for psychiatric nursing homes, sheltered housing, and psychosocial rehabilitation. Responses to living situation, age, treatment duration, and work situation are as expected for the type of service. Compared to other types of services, a far greater proportion of participants in long‐term care rated their health status as excellent. Large differences were also seen for education levels. Participants treated in a community mental health service were far more likely to report having obtained a university degree. Almost all participants were native Dutch speakers.

Table 2.

Characteristics of survey participants across various types of services

	Overall	General hospital psychiatric ward	Psychiatric hospital	Ambulatory public‐funded mental health services	Psychosocial rehabilitation	Sheltered housing	Psychiatric nursing home	Assertive community teams
Patients, n	5,168	310	2,844	1,014	260	379	233	128
Organizations, n	79	16	24	8	7	11	9	4
Patients/organization, n (range)	65.4 (7–280)	19.4 (7–47)	118.5 (33–280)	126.8 (12–226)	37.1 (14–110)	34.5 (7–50)	25.9 (13–43)	32.0 (21–40)
Gender, n (%)
Female	2,509 (50.3)	172 (57.7)	1,312 (48.0)	646 (64.5)	118 (45.9)	104 (28.7)	81 (38.9)	76 (59.8)
Male	2,476 (49.7)	126 (42.3)	1,420 (52.0)	355 (35.5)	139 (54.1)	258 (71.3)	127 (61.1)	51 (40.2)
Living situation, n (%)
Living alone	1,905 (39.8)	127 (42.8)	1,091 (42.0)	416 (42.2)	108 (43.4)	50 (14.3)	35 (20.2)	78 (62.4)
With partner, family, or friends	2,135 (44.7)	165 (55.6)	1,227 (47.2)	542 (55.0)	127 (51.0)	13 (3.7)	16 (9.3)	45 (36.0)
Assisted living, service apartment, care home	742 (15.5)	5 (1.7)	283 (10.9)	28 (2.8)	14 (5.6)	288 (82.1)	122 (70.5)	2 (1.6)
Age, n (%)
18–24	419 (8.4)	22 (7.3)	271 (9.9)	87 (8.7)	26 (10.0)	12 (3.3)	0 (0.0)	1 (0.8)
25–34	1,010 (20.3)	57 (18.9)	544 (20.0)	219 (22.0)	101 (39.0)	65 (18.0)	9 (4.3)	15 (11.7)
35–44	1,052 (21.1)	50 (16.6)	600 (22.0)	218 (21.9)	52 (20.1)	73 (20.2)	31 (14.7)	28 (21.9)
45–54	1,161 (23.3)	93 (30.3)	564 (20.7)	250 (25.1)	58 (22.4)	91 (25.2)	58 (27.5)	47 (36.7)
55–64	909 (18.3)	58 (19.3)	476 (17.5)	170 (17.1)	18 (7.0)	90 (24.9)	62 (29.4)	35 (27.3)
65–74	335 (6.7)	16 (5.3)	199 (7.3)	43 (4.3)	4 (1.5)	28 (7.8)	43 (20.4)	2 (1.6)
75–84	87 (1.8)	5 (1.7)	62 (2.3)	10 (1.0)	0 (0.0)	2 (0.6)	8 (3.8)	0 (0.0)
85+	9 (0.2)	0 (0.0)	9 (0.3)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)
Treatment duration, n (%)
<1 week	175 (3.6)	64 (22.3)	105 (3.9)	2 (0.2)	2 (0.8)	1 (0.3)	0 (0.0)	1 (0.8)
1–2 weeks	262 (5.3)	77 (26.8)	170 (6.3)	0 (0.0)	11 (4.3)	4 (1.1)	0 (0.0)	0 (0.0)
2–4 weeks	453 (9.2)	91 (31.7)	329 (12.2)	5 (0.5)	17 (6.6)	3 (0.9)	3 (1.5)	5 (3.9)
4 weeks–2 months	655 (13.3)	40 (13.9)	508 (18.8)	54 (5.5)	(15.2)	9 (2.6)	3 (1.5)	2 (1.6)
2–6 months	960 (19.6)	10 (3.5)	714 (26.4)	139 (14.1)	(19.8)	28 (7.9)	10 (5.0)	8 (6.3)
6–12 months	648 (13.2)	3 (1.1)	373 (13.8)	146 (14.8)	(27.2)	29 (8.2)	18 (9.0)	9 (7.1)
>12 months	1,756 (35.8)	2 (0.7)	501 (18.6)	638 (64.8)	(26.1)	279 (79.0)	167 (83.1)	102 (80.3)
Work situation, n (%)
Payed work (part‐time or full‐time)	1,126 (22.8)	100 (33.4)	566 (20.9)	354 (35.6)	51 (20.3)	37 (10.4)	4 (2.0)	14 (11.4)
Unemployed	985 (20.0)	49 (16.4)	621 (23.0)	137 (13.8)	71 (28.3)	51 (14.3)	44 (21.6)	12 (9.8)
Student	233 (4.7)	12 (4.0)	138 (5.1)	65 (6.5)	7 (2.8)	5 (1.4)	3 (1.5)	3 (2.4)
Retired	651 (13.2)	38 (12.7)	389 (14.4)	93 (9.4)	12 (4.8)	50 (14.0)	56 (27.5)	13 (10.6)
Other	1,937 (39.3)	100 (33.4)	989 (36.6)	346 (34.8)	110 (43.8)	214 (59.9)	97 (47.6)	81 (65.9)
Health status, n (%)
Poor	655 (13.1)	67 (22.2)	349 (12.8)	137 (13.8)	43 (16.9)	25 (6.9)	22 (10.4)	12 (9.8)
Fair	2,348 (47.1)	162 (53.6)	1,267 (46.3)	498 (50.1)	129 (50.6)	155 (42.5)	77 (36.3)	60 (49.2)
Good	1,665 (33.4)	66 (21.9)	923 (33.7)	323 (32.5)	72 (28.2)	153 (41.9)	82 (38.7)	46 (37.7)
Excellent	320 (6.4)	7 (2.3)	199 (7.3)	36 (3.6)	11 (4.3)	32 (8.8)	31 (14.6)	4 (3.3)
Education, n (%)
Elementary school	777 (15.7)	44 (14.7)	431 (15.9)	107 (10.8)	49 (19.4)	76 (21.4)	53 (25.7)	17 (13.5)
High school	2,548 (51.5)	171 (57.0)	1,391 (51.2)	475 (48.0)	136 (54.0)	212 (59.5)	97 (47.1)	66 (52.4)
Higher education, non‐university	1,235 (25.0)	72 (24.0)	697 (25.7)	277 (28.0)	60 (23.8)	56 (15.7)	42 (20.4)	31 (24.6)
University	386 (7.8)	13 (4.3)	197 (7.3)	131 (13.2)	7 (2.8)	12 (3.4)	14 (6.8)	12 (9.5)
Language, n (%)
Dutch	4,647 (95.7)	292 (96.1)	2,554 (96.6)	906 (91.4)	243 (98.8)	343 (98.0)	200 (96.2)	109 (96.5)
Other	210 (4.3)	12 (3.9)	91 (3.4)	85 (8.6)	3 (1.2)	7 (2.0)	8 (3.8)	4 (3.5)

3.3. Construct validity

Table 3 presents the goodness‐of‐fit indices associated with the factor analyses.

Table 3.

Model fit for the factor analytic models exploring dimensionality, confirming dimensionality, and assessing measurement invariance

	χ2	p	df	CFI	TLI	RMSEA (90% CI)
Step 1: Exploration of dimensionality
ESEM with 9 factors (as hypothesized) for the random split‐half sample	488.848	<.0001	316	.997	.994	.015 (.012–.017)
B ESEM with 9 factors (as hypothesized) for the random split‐half sample	429.403	<.0001	290	.997	.995	.014 (.011–.016)
Step 2: Confirmation of dimensionality
ICM‐CFA with 9 factors (ESEM‐informed) for the holdout sample	996.848	<.0001	341	.986	.983	.027 (.025–.029)
Step 3: Assessment of measurement invariance (all for the holdout sample)
Type of services
ICM‐CFA with 9 factors (ESEM‐informed): Psychiatric ward in a general hospital	366.900	.1603	341	.990	.988	.022 (.000–.038)
ICM‐CFA with 9 factors (ESEM‐informed): Psychiatric hospital	770.496	<.0001	341	.985	.983	.030 (.027–.033)
ICM‐CFA with 9 factors (ESEM‐informed): Ambulatory public funded mental health service	411.740	.0051	341	.971	.966	.020 (.012–.027)
ICM‐CFA with 9 factors (ESEM‐informed): Psychosocial rehabilitation	359.956	.2302	341	.991	.981	.021 (.000–.039)
ICM‐CFA with 9 factors (ESEM‐informed): Sheltered housing	351.937	.3300	341	.994	.993	.013 (.000–.030)
ICM‐CFA with 9 factors (ESEM‐informed): Psychiatric nursing homes	360.534	.2237	341	.991	.989	.022 (.000–.042)
ICM‐CFA with 9 factors (ESEM‐informed): Assertive community teams	341.377	.4841	341	.999	.999	.004 (.000–.047)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Configural invariance	2,700.694	.0000	2,387	.990	.988	.019 (.015–.022)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Scalar invariance	2,888.251	.0000	2,507	.989	.987	.024 (.020–.027)
Gender
ICM‐CFA with 9 factors (ESEM‐informed): Female	638.960	.0000	341	.988	.985	.026 (.023–.029)
ICM‐CFA with 9 factors (ESEM‐informed): Male	628.019	.0000	341	.986	.984	.026 (.023–.029)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Configural invariance	1,266.969	.0000	682	.987	.984	.026 (.024–.028)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Scalar invariance	1,324.203	.0000	702	.986	.984	.027 (.024–.029)
Living situation
ICM‐CFA with 9 factors (ESEM‐informed): Living alone	548.894	.0000	341	.989	.987	.025 (.021–.029)
ICM‐CFA with 9 factors (ESEM‐informed): With partner, family or friends	644.741	.0000	341	.982	.979	.029 (.026–.032)
ICM‐CFA with 9 factors (ESEM‐informed): Assisted living, service apartment, care home	382.917	.0583	341	.993	.991	.018 (.000–.027)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Configural invariance	1,535.544	.0000	1,023	.988	.986	.025 (.022–.028)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Scalar invariance	1,576.891	.0000	1,063	.988	.986	.025 (.022–.027)
Age
ICM‐CFA with 9 factors (ESEM‐informed): 18–24	390.958	.0320	341	.975	.970	.027 (.009–.039)
ICM‐CFA with 9 factors (ESEM‐informed): 25–34	450.681	.0001	341	.987	.985	.025 (.018–.031)
ICM‐CFA with 9 factors (ESEM‐informed): 35–44	431.550	.0006	341	.991	.989	.022 (.015–.028)
ICM‐CFA with 9 factors (ESEM‐informed): 45–54	486.581	.0000	341	.987	.984	.027 (.022–.033)
ICM‐CFA with 9 factors (ESEM‐informed): 55–64	391.232	.0313	341	.995	.994	.018 (.006–.026)
ICM‐CFA with 9 factors (ESEM‐informed): 65–74	376.948	.0875	341	.987	.984	.025 (.000–.039)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Configural invariance	2,470.143	.0000	2,046	.990	.988	.023 (.019–.026)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Scalar invariance	2,638.072	.0000	2,146	.988	.987	.024 (.020–.027)
Treatment duration
ICM‐CFA with 9 factors (ESEM‐informed): < 1 week	334.197	.0593	341	1.000	1.000	.000 (.000–.038)
ICM‐CFA with 9 factors (ESEM‐informed): 1–2 weeks	381.079	.0663	341	.989	.986	.028 (.000–.043)
ICM‐CFA with 9 factors (ESEM‐informed): 2–4 weeks	387.863	.0406	341	.988	.986	.025 (.006–.037)
ICM‐CFA with 9 factors (ESEM‐informed): 4 weeks ‐ 2 months	411.105	.0055	341	.986	.984	.025 (.014–.034)
ICM‐CFA with 9 factors (ESEM‐informed): 2–6 months	408.922	.0067	341	.991	.989	.021 (.012–.028)
ICM‐CFA with 9 factors (ESEM‐informed): 6–12 months	389.252	.0365	341	.991	.989	.021 (.006–.030)
ICM‐CFA with 9 factors (ESEM‐informed): > 12 months	513.402	.0000	341	.989	.987	.024 (.019–.028)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Configural invariance	2,721.103	.0000	2,387	.992	.990	.020 (.016–.024)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Scalar invariance	2,966.613	.0000	2,507	.990	.989	.022 (.019–.025)
Work situation
ICM‐CFA with 9 factors (ESEM‐informed): Payed work (part‐time or full‐time)	486.138	.0000	341	.981	.977	.027 (.021–.032)
ICM‐CFA with 9 factors (ESEM‐informed): Unemployed	402.221	.0125	341	.994	.993	.019 (.009–.026)
ICM‐CFA with 9 factors (ESEM‐informed): Student	384.314	.0528	341	.953	.943	.035 (.000–.052)
ICM‐CFA with 9 factors (ESEM‐informed): Retired	379.415	.0743	341	.994	.993	.018 (.000–.028)
ICM‐CFA with 9 factors (ESEM‐informed): Other	.579.991	.0000	341	.987	.985	.027 (.023–.031)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Configural invariance	2,138.881	.0000	1,705	.989	.987	.023 (.019–.026)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Scalar invariance	2,274.069	.0000	1,785	.988	.986	.024 (.020–.026)
Health status
ICM‐CFA with 9 factors (ESEM‐informed): Poor	425.098	.0013	341	.989	.987	.028 (.018–.036)
ICM‐CFA with 9 factors (ESEM‐informed): Fair	605.014	.0000	341	.987	.985	.025 (.022–.029)
ICM‐CFA with 9 factors (ESEM‐informed): Good	434.242	.0000	341	.989	.987	.021 (.016.026)
ICM‐CFA with 9 factors (ESEM‐informed): Excellent	353.441	.3099	341	.996	.995	.015 (.000–.034)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Configural invariance	1,756.303	.0000	1,364	.990	.988	.021 (.018–.024)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Scalar invariance	1,835.723	.0000	1,424	.990	.988	.022 (.019–.024)
Education
ICM‐CFA with 9 factors (ESEM‐informed): Elementary school	384.046	.0538	341	.994	.993	.018 (.000–.026)
ICM‐CFA with 9 factors (ESEM‐informed): High school	635.991	.0000	341	.986	.984	.026 (.023–.029)
ICM‐CFA with 9 factors (ESEM‐informed): Higher education, non‐university	468.719	.0000	341	.990	.988	.024 (.019–.030)
ICM‐CFA with 9 factors (ESEM‐informed): University	396.844	.0198	341	.987	.984	.029 (.013–.041)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Configural invariance	1,806.175	.0000	1,364	.990	.988	.023 (.020–.026)
Multiple group ICM‐CFA with 9 factors (ESEM‐informed): Scalar invariance	1,877.988	.0000	1,424	.989	.988	.023 (.020–.025)

Notes. B ESEM = bifactor exploratory structural equation model; CFI = comparative fit index; ESEM = exploratory structural equation model; ICM‐CFA = independent cluster model confirmatory factor analysis; RMSEA = root mean square error of approximation; TLI = Tucker–Lewis index.

In our exploration of dimensionality (step 1), both ESEM and B ESEM showed an excellent fit for a solution with nine factors. Inclusion of less or more factors did not improve model fit nor did it improve interpretability of the factors. B ESEM displayed the best representation of the data and all items had a strong factor loading on the general factor. However, model fit was only marginally better compared to the ESEM solution, and the inclusion of the general factor slightly distorted the loadings of the specific factors. Table 4 displays the standardized factor loading pattern from ESEM analysis with nine factors. Our psychometric multidimensionality consists of nine specific factors that partly overlap with the nine hypothesized factors and cover 29 of 35 items. Most notable about the factor structure was that (a) two factors originated from the items measuring organization of care and collaboration between professionals. Organization of care in addition included item 26, which dealt with the same topic but was hypothesized to measure safe care; (b) item 34 was hypothesized to measure discharge management but also strongly loaded on the factor participation, which, in hindsight, was not surprising given the wording of the item; (c) item 28 was hypothesized to measure patient rights, but again the wording was closely related to that used in another factor, namely, therapeutic relation; (d) there was no clear distinction between the hypothesized factors of results and evaluation of care, and discharge management and aftercare. Items 5, 8, 11, and 33 were excluded from the dimensionality because of low loadings, and items 25 and 34 were excluded because of high cross loadings.

In step 2, ICM‐CFA confirmed the ESEM‐based dimensionality presented in Table 4 for the holdout sample.

In step 3, ICM‐CFA and multiple group ICM‐CFA, also for the holdout sample, showed an excellent fit to the data and supported both configural and scalar invariance of the ESEM factor structure across type of service and patient characteristics previously presented in Table 2.

3.4. Criterion validity

Table 5 displays further information on these factors.

Table 5.

Description and association between composite measures, single items, and global ratings

	Items, n (items)	Descriptive analysis									Regression analysis
		All factor items/single item completed	Top box %, overall	Top box %, types of services (min‐max)							Top box responses, n	Overall rating 9/10	Definitely recommend service
				General hospital psychiatric ward	Psychiatric hospital	Ambulatory public funded mental health services	Psychosocial rehabilitation	Sheltered housing	Psychiatric nursing home	Assertive community teams		Odds ratio*	Odds ratio*
ESEM‐informed factors (from step 1)
Information about mental health problems and treatment	4 (1,2,3,4)	3,977/5,168	32.6	32.5 (17.9–53.6)	27.1 (17.3–39.8)	47.7 (37.5–54.0)	31.1 (21.8–53.6)	37.3 (23.1–50.0)	28.8 (16.2–44.1)	34.7 (15.3–51.5)	0/4	1	1
											1/4	2.27	2.12
											2/4	3.39	3.22
											3/4	5.47	5.26
											4/4	10.07	8.50
Personalized care	6 (12,13,14,15,16,17)	4,705/5,168	45.5	46.6 (26.9–100)	37.5 (23.1–58.3)	63.5 (47.2–77.8)	49.8 (32.1–68.3)	49.2 (32.6–66.1)	39.7 (11.1–65.0)	48.8 (34.3–67.1)	0/6	1	1
											1/6	1.95	2.01
											2/6	2.66	2.92
											3/6	4.26	4.44
											4/6	6.30	7.10
											5/6	8.17	8.17
											6/6	22.65	18.73
Participation	2 (6,7)	4,673/5,168	34.3	30.4 (0.0–64.3)	28.2 (12.8–48.3)	57.9 (45.8–69.1)	39.2 (15.4–60.0)	37.9 (14.3–53.2)	22.8 (11.5–37.5)	45.6 (41.4–50.0)	0/2	1	1
											1/2	2.41	2.51
											2/2	5.42	6.23
Therapeutic relationship	3 (9,10,28)	4,735/5,168	60.9	63.7 (40.0–100)	51.7 (37.6–71.3)	84.3 (77.8–88.8)	58.4 (23.8–80.3)	62.6 (46.0–79.6)	55.5 (35.7–73.3)	69.9 (50.0–82.3)	0/3	1	1
											1/3	1.84	2.51
											2/3	4.14	4.44
											3/3	12.94	13.20
Organization of care	3 (18,19,20)	4,535/5,168	47.3	48.7 (23.3–100)	37.4 (25.4–51.0)	64.9 (60.7–71.6)	43.1 (23.8–66.7)	56.1 (30.8–76.3)	43.7 (23.8–63.3)	56.5 (33.3–71.1)	0/3	1	1
											1/3	3.25	2.34
											2/3	5.75	4.10
											3/3	14.59	10.28
Patient rights	2 (30,31)	4,127/5,168	34.8	36.5 (8.3–100)	26.6 (16.0–45.0)	54.9 (48.1–63.8)	29.1 (17.6–50.0)	38.1 (8.3–51.6)	29.2 (20.0–40.0)	49.7 (46.2–57.4)	0/2	1	1
											1/2	2.83	2.86
											2/2	7.03	5.81
Collaboration between professionals	3 (21,22,26)	4,065/5,168	39.0	39.3 (21.2–100)	30.7 (19.3–42.3)	48.1 (39.1–63.0)	37.6 (28.6–52.9)	47.3 (32.5–56.0)	39.6 (17.3–55.6)	47.4 (29.8–63.1)	0/3	1	1
											1/3	2.72	2.56
											2/3	5.87	4.66
											3/3	12.43	8.00
Safe care	4 (23,24,27,29)	4,436/5,168	55.5	57.2 (40.9–100)	46.2 (29.7–73.2)	76.2 (69.2–81.2)	56.1 (38.5–71.0)	60.7 (46.3–75.0)	48.3 (33.3–75.0)	64.6 (43.1–79.8)	0/4	1	1
											1/4	2.59	2.41
											2/4	5.47	4.35
											3/4	10.70	8.41
											4/4	27.66	20.70
Evaluation and aftercare	2 (32,35)	3,435/5,168	60.5	63.0 (37.5–100)	57.8 (39.7–73.1)	63.2 (55.0–75.0)	65.6 (44.7–84.4)	62.8 (50.0–75.9)	51.1 (37.0–66.7)	66.9 (60.0–78.6)	0/2	1	1
											1/2	2.61	2.10
											2/2	9.68	7.85
Single items with loadings < .300 or cross‐loadings > .300 in ESEM solution
Information about costs	1 (5)	4,784/5,168	33.2	12.6 (0.0–28.6)	29.8 (16.1–55.4)	75.0 (63.5–90.9)	47.9 (28.6–60.0)	36.5 (14.3–64.5)	24.0 (13.3–38.9)	38.8 (30.0–48.5)	0/1	1	1
											1/1	2.83	2.80
Clear explanation	1 (8)	5,060/5,168	50.4	49.6 (23.1–100)	40.6 (28.9–57.6)	73.4 (60.0–81.6)	52.7 (33.3–78.1)	56.5 (34.9–81.3)	43.7 (26.7–66.7)	60.6 (38.1–74.3)	0/1	1	1
											1/1	4.90	4.57
Free to ask	1 (11)	5,014/5,168	59.0	60.8 (30.8–100)	52.1 (40.2–71.9)	74.5 (64.3–83.3)	64.2 (53.8–78.1)	59.9 (28.6–76.7)	52.2 (31.0–64.7)	66.5 (61.3–73.5)	0/1	1	1
											1/1	4.57	3.82
Caregivers respond well	1 (25)	4,819/5,168	49.9	51.6 (26.7–100)	40.7 (27.5–58.1)	74.4 (66.7–80.0)	47.5 (38.1–67.7)	50.0 (34.1–65.5)	49.3 (29.6–72.2)	54.5 (29.6–73.5)	0/1	1	1
											1/1	5.87	4.90
Treatment helped	1 (32)	4,800/5,168	42.7	43.1 (18.2–100)	31.7 (19.1–51.5)	49.5 (33.3–57.9)	46.0 (28.4–61.3)	53.2 (31.8–78.1)	46.9 (25.9–65.0)	49.1 (42.9–59.0)	0/1	1	1
											1/1	5.93	5.42
Say about ending	1 (34)	3,698/5,168	81.5	87.9 (50.0–100)	79.0 (54.8–93.9)	95.7 (88.8–100)	85.6 (63.6–100)	83.1 (58.3–100)	55.9 (32.0–90.9)	89.1 (78.6–96.0)	0/1	1	1
											1/1	3.53	4.06
Global rating
Overall rating 9 or 10	1	4,859/5,168	36.9	33.5 (6.3–100)	28.8 (13.2–53.4)	52.2 (33.3–62.6)	31.4 (0–50.0)	43.6 (32.5–51.9)	40.9 (10.7–64.7)	50.1 (40.0–58.3)	‐‐		‐‐
Definitely recommend hospital	1	4,941/5,168	53.3	49.4 (16.7–100)	47.4 (23.3–76.4)	73.9 (66.7–77.5)	59.9 (30.8–81.8)	54.7 (38.6–85.7)	43.6 (13.3–78.9)	69.0 (61.3–77.5)	‐‐		‐‐

Note. Descriptive findings are the average of the average per organization. ESEM = exploratory structural equation modelling.

^* p < .001.

Items are organized into factors suggested by ESEM and reproduced by (multiple group) confirmatory factor analysis (column 1).

The respective number of items is shown as well as the corresponding items from Table 4 (column 2).

The third column shows the number of respondents who completed all items for the specific factor, with participation and evaluation and aftercare showing a clearly higher number of missing values.

The fourth column displays the overall top box percentage. Less than 4 in 10 patients (36.9%) gave the organization a 9 or 10 but about half of the patients (53.3) would definitely recommend the organization. Variation across and within the various types of services is shown in columns 5 to 11.

The three columns on the right list the odds ratios (all p < .001) for the association between the top box responses and the two global ratings. By way of example, compared to patients with no top box ratings, if patients have one or two top box ratings for the subscale reflecting participation, the odds of rating the organization 9 or 10 are 2.41 and 5.42, respectively. In most cases, the odds for rating the organization are higher compared to the odds for recommending the organization. A sensitivity ICM‐CFA analysis was conducted in which the two global rating items were regressed on the nine hypothesized factors treated as latent variables, confirming the above findings.

The middle part of Table 5 presents findings for the six items that were excluded from the dimensionality. This illustrates that also for these items there is large variation across and within type of service. Also, these items are related to global ratings.

4. DISCUSSION

We developed a questionnaire with overall ratings and multiple factors capturing specific aspects of patient experiences with mental health care, identified by patients as important quality aspects, and allowing comparison across various types of services and patient characteristics.

The latter was assessed through a series of factor analytic models assessing multidimensionality and measurement invariance. In many studies, the multiple group ICM‐CFA approach applied here would lead to poor model fit as the assumption of ICM‐CFA is that cross loadings between items and nontarget factors are exactly 0. In that regard, the less restrictive ESEM model is gaining popularity (Marsh et al., 2014). Cross‐cultural researchers on the other hand focus on attempting to explain measurement invariance in multilevel structural equation models (Davidov et al., 2016; Davidov, Dulmer, Schluter, Schmidt, & Meuleman, 2012). Here, we showed a well‐fitting model within each group that we studied (type of service, gender, living situation, age, treatment duration, work situation, health status, and education). In multiple group analyses, we then found that the questionnaire has the same meaning across groups. This provides evidence that the requirement for valid group comparisons has been satisfied. The questionnaire can be restructured according to the proposed dimensionality. Also of importance is to discuss with stakeholders what the six items not included in the multidimensionality tell us, that is, whether these questions should be dropped from future versions or should be included at the end of the questionnaire as separate questions, given their strong association with overall ratings. When more data become available, longitudinal measurement invariance must be studied. In addition, with several organizations aiming to administer the questionnaire electronically, measurement invariance across administration mode must be examined.

Overall scores (9 or 10 on a scale of 0 to 10) of patients (36.9%) were low compared to acute‐care patients in general hospitals in the United States (70.4% in 2014; Papanicolas, Figueroa, Orav, & Jha, 2017) but also compared to the 48 Flemish acute hospitals for whom data are publicly reported (54.9% in 2016; Vlaamse Overheid, 2016). In the Community Mental Health Survey, a cut point of 7 instead of 9 is used. In 2016, 65% evaluated their overall experiences with a score of 7 or above (National Health Service Care Quality Commission, 2016). A sensitivity analysis for our data showed that 86% evaluated their overall experiences with a score of 7 or above.

For these overall ratings and the specific subscales, ambulatory public‐funded mental health services scored consistently highest and had lowest variation between organizations. Psychiatric hospitals and psychiatric nursing homes scored lowest but always included organizations that scored among the best in the total sample. Room for improvement is high for all subscales but particularly for information about mental health problems and treatment, for patient rights, and for collaboration between professionals. Our process evaluation and experience during feedback to participating organizations highlighted that participating organizations are eager to continue measuring patients' experiences with mental health care and initiate a learning community. Several organizations would appreciate the availability of the questionnaire in languages other than Dutch to include a higher number of patients. It would be desirable to collaborate with French‐speaking organizations in using one standardized instrument for the Belgian population. A core indicator set on the national level would also be required if it comes to pay‐for‐performance initiatives, which are currently being introduced in general hospitals and where the same issues on cross‐regional availability of valid and reliable quality indicators present an issue yet to be resolved. Pay‐for‐performance was recently shown not to have positively impacted patient experiences in general acute hospitals (Figueroa, Tsugawa, Zheng, Orav, & Jha, 2016). This is not surprising, since designing specific interventions to improve patient experience in daily practice is a yet to be accomplished mission. Two systematic reviews recently concluded that actual interventions to improve patient experiences are rare (Davidson et al., 2016; Gleeson et al., 2016). Also in Belgium, patient experiences, although frequently measured, have not been part of any formal quality improvement method. With this validated questionnaire, we aim for patient experiences to play a major role throughout quality improvement initiatives in mental health care.

A first limitation of this study is that no response rates were reported. Future data collection efforts will include such information and predict hypothetical experience of nonrespondents to estimate the impact of response rates. A study in English NHS hospitals found that hospitals with higher response rates have better patient experience scores (Saunders, Elliott, Lyratzopoulos, & Abel, 2016). Second, case‐mix adjustment is important to allow for fair comparisons across organizations, also in mental health care services (Köhler, Unger, Hoffmann, Steinacher, & Fydrich, 2015). Particularly in this cross‐sectoral study that showed large differences in patient characteristics across types of services, more evidence on the impact of case‐mix needs to be generated. Third, further testing of the psychometric properties, for example, an assessment of test–retest reliability of the questionnaire for evaluation of short‐ and long‐term reliability should be assessed, again in each type of service.

In conclusion, our systematic approach including patient involvement from the very beginning resulted in a questionnaire instrument that supports comparisons across various types of mental health care organizations and patient groups. In general, patient experiences with Belgian mental health care seem poor. Although the measurement of patient experiences is in itself an accomplishment and learning collaboratives might lead to quality improvement, there is an urgent need for making the right choices in designing robust interventions to improve patient experiences.

CONFLICT OF INTEREST STATEMENT

The authors have no conflicts of interest to declare.

Bruyneel L, Van Houdt S, Coeckelberghs E, et al. Patient experiences with care across various types of mental health care: Questionnaire development, measurement invariance, and patients' reports. Int J Methods Psychiatr Res. 2018;27:e1592 10.1002/mpr.1592