Does implementing a development plan for user participation in a mental hospital change patients' experience? A non‐randomized controlled study

Abstract

Background

Governments in several countries attempt to strengthen user participation through instructing health‐care organizations to implement user participation initiatives. There is, however, little knowledge on the effect on patients' experience from comprehensive plans for enhancing user participation in whole health service organizations.

Objective

To investigate whether implementing a development plan intending to enhance user participation in a mental hospital had any effect on the patients' experience of user participation.

Design, setting and participants

A non‐randomized controlled study including patients in three mental hospitals in Central Norway, one intervention hospital and two control hospitals.

Interventions

A development plan intended to enhance user participation was implemented in the intervention hospital as a part of a larger reorganizational process. The plan included establishment of a patient education centre and a user office, purchase of user expertise, appointment of contact professionals for next of kin and improvement of the centre's information and the professional culture.

Main outcome measures

Perceptions of Care, Inpatient Treatment Alliance Scale and questions made for this study.

Results

A total of 1651 patients participated. Implementing a development plan in a mental hospital intending to enhance user participation had no significant effect on the patients' experience of user participation.

Discussions and conclusions

The lack of effect can be due to inappropriate initiatives or challenges in implementation processes. Further research should ensure that initiatives and implementation processes are appropriate to impact the patients' experience.

Introduction

User participation is highly emphasized in the health services, and governments in several countries are encouraging and instructing health service organizations to enhance user participation.1, 2 Such initiatives should take place in individual treatment (e.g. in shared decision making in consultations) and in organizational settings (e.g. involving service users in committees and boards).

There are many proposed benefits from user participation in health care: reducing medical errors and increasing patient safety,3 making services more accessible and improving information,4 improving the coordination of care in chronic diseases,5 improving clinical decision making6 and aligning the goals and agendas of patients and providers.7 The knowledge base on the effect from user participation in health services is, however, weak due to a limited number of studies.8, 9, 10

According to reviews, there are some organizational user participation initiatives that have been shown to be effective.8, 11 Involving users in the development of information material led to more relevant, readable and understandable information, which, in turn, led to better patient knowledge.12, 13 Involving users in deliberate health policy discussions led to stronger engagement in users.14 Employing users as case managers in mental health care influenced the delivery of health services and improved several aspects related to those who were involved.11 Involving users in training of professionals (in a course in assertive community treatment services) and in the education of mental health nursing students improved the professionals' attitudes towards mental health users and mental illness.11 Research on participation in individual treatment (e.g., shared decision making in consultations) has, so far, shown improvement of patient satisfaction,15 increase in well‐being,16, 17, 18 improvement of treatment adherence17, 18 and increase in patient knowledge.19, 20

As governments are emphasizing more user participation, health service organizations are increasingly working to develop and implement user participation initiatives. Some organizations have implemented smaller user participation initiatives in some areas of the organizations,8, 9 but few have aimed at implementing more complex and comprehensive plans to enhance user participation.21, 22 One of the main aims for enhancing user participation in health care is improved quality of care and better health outcomes for service users.23 Thus, the implementation of user participation initiatives in health service organizations should first and foremost be beneficial for the service users. Although there is little knowledge about the effects of the implementation of isolated user participation initiatives, there is even less knowledge of the effects of more comprehensive plans on whole health service organizations. And, very few have investigated the effects of such initiatives on the general population of patients who are using the services.

Aim

The aim of this study was therefore to investigate whether implementing a development plan intending to enhance user participation in a large health service organization had any effect on the patients' experience of user participation.

Methods

Study design and ethics

This was a non‐randomized, controlled study including patients from three mental hospitals in Central Norway. One of the hospitals (intervention hospital) implemented a development plan to enhance user participation, and two hospitals participated as the control group. A non‐matched sample of patients from the three hospitals filled out the same questionnaire two times, 16 months apart. The effect was measured by comparing the change in the intervention hospital with the control hospitals. The study took place from November 2008 to December 2010 and was approved by the regional committee for medical and health research ethics in Central Norway, the Norwegian Data Inspectorate and each hospital's management. The effects of the development plan on the professionals' knowledge, practice and attitudes towards user participation have been previously published.22

Setting

The three hospitals are part of the same hospital trust and offer general mental health treatment in one of the trust's main hospitals. The intervention hospital covers a catchment area of 96 000 persons, with urban and semi‐rural areas including parts of a large Norwegian city. The two control hospitals cover catchment areas of 74 000 and 47 000 persons respectively, with urban, semi‐rural and rural areas including parts of the above‐mentioned city. The three hospitals provide the same types of service: inpatient treatment (5.4 beds per 10 000 inhabitants), outpatient treatment and ambulatory services. The intervention hospital was relocated and reorganized in January 2009. Several units were merged and colocalized, and an ambulatory acute treatment team was established. The reorganization was based on an overarching plan focusing on professional development and improvement of collaboration and patient flow. The reorganizational plan also included the intervention in this study, a development plan for user participation.

Intervention

As part of the structural reorganization and relocation, the intervention hospital formulated and approved a development plan for user participation. The development plan was formulated by a project group, constituting administrators, health professionals and user representatives (recruited from mental health user organizations), working from the fall of 2007 to June 2008. Thus, the plan was based on clinical experience and knowledge and the hospital administration's aim for the health services they provide. The final plan was approved in the health‐care trust in June 2008.

The development plan included several initiatives to enhance user participation both on a organizational level and an individual level. All initiatives are described in Table 1. The initiatives in the plan were chosen by the project group based on their knowledge about user participation and the services they wanted to provide after the reorganization. In April 2010, 15 months after the implementation started, some of the initiatives were completed, while some were not. Information on the implementation status at this time was collected through interviews with key professionals and the documents produced by the hospital. The implementation status in April 2010 described in this article was confirmed by the hospital's management.

Table 1.

Initiatives in the development plan

Planned initiatives in development plan sanctioned in June 2008 and planned implemented from January 2009	Status for implementation in April 2010
Establishing a patient education centre	A patient education centre was established in November 2009 and employed 2 persons. A user representative participated in the planning and starting of the centre, and representatives partake in the daily work
Establishing an office run by users where various user representatives shall be available to the users of the centre	An office and information centre for users was established in January 2010. The office provides information material, telephone and Internet for patients and next of kin. Two user organizations and representatives from the regional labour and welfare administration use the office weekly
Purchasing user expertise up to 17.5 h/week	The centre's budget allows for buying up to 17.5 h of user expertise per week, but normally buys 10–12 hours per month. A user representative is employed 20% for the research project on self‐administered places/beds
Establishing a strategy for education of user representatives	Not implemented. Education of user representatives has been assigned to the user organizations
Appointing contact personnel for next of kin in each section	In March 2009, one personnel from each unit has been appointed contact person for next of kin
Allowing money in the budget for patient education	Money for patient education has since January 2009 been a part of the patient education centre's working budget
Tentative proceedings with places/beds administered by the patients themselves	A randomized controlled trial on places/beds administered by patients was started in May 2010. One user representative is participating in the steering committee and two in the research group. User expertise equivalent to 20% employment is bought during this study
Improving the centre's communication and information materials	A group was established before relocation to evaluate and suggest measures to improve the centre's communication and information materials. The work in this group stopped after a few meetings. Outwards communication has been discussed at several staff meetings during 2009 and 2010
Formulate and implement a strategy for quality assurance of attitudes and culture among the personnel	Tentative plans were discussed with user representatives in spring 2009. A philosopher was temporarily employed during the fall 2009. He conducted group sessions with health personnel to discuss attitudes towards user participation. The work stopped in 2009. The implementation group (administrators, health personnel and user representatives) discussed attitudes and culture at 6–8 meetings during the implementation process
Implementing a Web‐based system (Sampro) for collaborating and coordinating individual plans and individual education plans for patients	An educational course led by an external course supervisor was held for 4 patients and their therapists in April 2010. In one of the inpatient units, therapist has received training in using the system, and patients are continuously offered to use this system
Informing patients; in general about the centre, about their right to change therapist and about setting treatment goals	Information has been discussed at several meetings in the executive group, but no concrete initiatives have been planned or implemented
Tentative proceedings with using client‐directed outcome informed therapy in outpatient sessions	A research trial on client‐directed outcome informed therapy in outpatient sessions started in February 2010 and is currently running
(Not in development plan)	The patient education centre reviewed each unit's work with patient education from January 2010 and decided to appoint one contact person for patient education per unit. Per April 2010 6 out of 8 units had contact persons
(Not in development plan)	To ensure identification of and care for inpatients' children, a group in charge was appointed in January 2010
(Not in development plan)	All inpatient units conduct regular ‘house meetings’ where patients are encouraged to raise issues which are subsequently discussed in management meetings
(Not in development plan)	Patients and users are represented in the panel overseeing the quality of the services and are participating in the processes of introducing new service initiatives

Implementation

The intervention was implemented from January 2009. The hospital manager was in charge of the implementation process, and the everyday executive responsibility was ensured by one of three unit managers at the hospital. An implementation group consisting of the unit manager, several administrators, health professionals and user representatives (recruited from mental health user organizations) was established in August 2009 to supervise and follow up the implementation. The group had six meetings until January 2010 when the group was dissolved.

Participants

All patients registered as users of the hospitals' services the last month before the questionnaires were mailed received a questionnaire. This included patients receiving inpatient, outpatient or ambulatory treatment or services from the hospitals. There were no exclusion criteria. Patients were identified from updated lists provided by the hospital trusts' office. Lists were kept for 3 weeks to allow for one reminder before being destroyed. The group of patients answering the questionnaire was therefore not the same at the first and the second time of measurement. The participants concordantly constituted four independent samples; sample 1 (baseline) and sample 2 (follow‐up) constituted the intervention group, and sample 3 (baseline) and sample 4 (follow‐up) the control group.

Data collection

All patients at the intervention hospital were invited to fill out a questionnaire in December 2008 (before the implementation of the development plan started) and in April 2010 (16 months later). During the same period, all patients at the two control hospitals were invited to fill out a questionnaire two times within a 16‐month period. The questionnaires were sent by post to the patients' private address with pre‐paid return envelopes that were returned directly to the researchers. One reminder was sent.

Outcome measures

To measure the patients' experience of user participation, two questionnaires and self‐made questions were used.

The Perceptions of Care (PoC)

The PoC24 was used to measure the patients’ perception of the treatment and care they received from the hospital. The version of PoC used in this study consists of 16 questions.24 The PoC questions are presented as questions 1–16 in Tables 3 and 4. For questions 1–8, the answer categories are on a 4‐point scale, recoded to ‘never/sometimes’ and ‘usually/always’ for number 1–7, and to ‘not at all/somewhat’ and ‘quite a bit/a great deal’ for number 8. Questions 9–16 have ‘yes’ and ‘no’ as possible answers. The PoC questionnaire has no total score.

Table 3.

Patients' experiences at baseline and follow‐up for intervention and control group with tests of difference within each group. Numbers are percentages of total N for each sample unless otherwise stated

	Intervention			Control
	Baseline N = 324a	Follow‐up N = 419a	P‐valuec	Baseline N = 477a	Follow‐up N = 431a	P‐valuec
1. Were you involved as much as you wanted in decisions about your treatment? (% always/usually)	76.9	78.3	0.655	75.3	80.3	0.075*
2. Did the staff treat you with respect and dignity? (% always/usually)	91.7	94.0	0.213	89.9	93.0	0.102
3. Did the staff explain things in a way you could understand? (% always/usually)	90.4	91.6	0.569	87.8	89.0	0.566
4. Did the staff listen carefully to you? (% always/usually)	86.6	85.1	0.541	87.1	86.2	0.675
5. Did the staff who treated you work well together as a team? (% always/usually)	88.1	83.8	0.107	85.6	86.8	0.609
6. Did the staff spend enough time with you? (% always/usually)	75.2	77.2	0.540	79.7	81.8	0.419
7. Did the staff give you reassurance and support? (% always/usually)	78.2	81.3	0.297	80.3	80.9	0.806
8. How much were you helped by the care you received? (% pretty much/a lot)	67.4	61.0	0.072*	62.7	65.6	0.351
9. Did the staff give you information about the rules and policies of the program? (% yes)	56.9	56.6	0.928	59.6	63.0	0.295
10. Did the staff give you information about your rights as a patient? (% yes)	42.7	39.4	0.362	43.3	50.2	0.039**
11. Did the staff tell you about the risks and benefits of the medication(s) you are taking? (% yes)	38.9	32.4	0.066*	43.1	39.2	0.248
12. Did the staff review with you the plans for your continued treatment? (% yes)	59.4	55.6	0.293	59.1	62.8	0.264
13. Were you told who to contact in case you have a problem or emergency? (% yes)	56.3	57.8	0.675	61.0	58.0	0.352
14. Did the staff tell you about self‐help or support groups? (% yes)	29.8	32.7	0.399	43.3	40.5	0.410
15. Did the staff give you information about how to reduce the chances of a relapse? (% yes)	44.6	46.1	0.686	47.6	49.3	0.620
16. Would you recommend this facility to someone who needed mental health treatment? (% yes)	86.3	86.3	0.996	87.0	86.7	0.906
#17. Do you know if the unit has a user's committee? (% yes)	3.8	3.4	0.758	13.0	12.5	0.827
#18. Do you know if the unit has representatives or spokespersons on behalf of the users? (% yes)	3.1	3.4	0.854	9.5	12.1	0.224
#19. Have you received information about how to make a complaint about your treatment? (% yes)	17.7	16.8	0.767	18.6	23.9	0.053*
#20. Have you received information about the confidentiality of your records? (% yes)	70.3	73.6	0.323	74.4	76.7	0.425
#21. Have you been offered an Individual plan? (% yes)	37.5	33.0	0.212	39.6	39.7	0.986
#22. Have you received information about your rights to read and make corrections in your records? (% yes)	26.4	24.4	0.535	28.9	26.7	0.474
#23. How has your therapist(s) cooperated with your next of kin? (% good/very good)	53.3	45.8	0.199	48.7	46.8	0.746
#24. How would you evaluate the information you have received about the treatment options for your problems? (% good/very good)	45.7	38.1	0.041**	45.1	44.1	0.767
#25. How would you evaluate the information you have received about your mental health problems? (% good/very good)	51.1	48.0	0.414	54.8	52.5	0.485
#26. Overall, how would you evaluate the help you have received for your mental health problems? (% good/very good)	64.2	60.4	0.296	66.2	69.1	0.361
Inpatient Treatment Alliance Score (I‐TAS) [mean of total score between 0 and 6 (SD)]b	4.55 (1.36)	4.51 (1.33)	0.679d	4.57 (1.41)	4.63 (1.36)	0.545d
27. I work well with my therapist 28. I feel that my therapist has a good understanding of my problems 29. I feel that the therapist listens to my problems 30. I think my therapist will be available if I need him/her 31. I feel that my therapist wants me to partake fully in my treatment 32. I feel that my therapist wants to help me 33. I feel like an active participant in my treatment 34. I feel respected by my therapist 35. My therapist and I agree on what has to change in order to conclude my treatment 36. I think my treatment will be successful (answered from 0 to 6 where 0 is ‘False’ and 6 is ‘Completely true’)

Questions marked with # (no. 17–26) were made by the researchers.

^aN is the no of participants who returned completed questionnaires. The N in the four samples varied for each question due to missing answers on the variables (0–8%).

^bFor total score of I‐TAS a higher score indicates a stronger treatment alliance.

*P < 0.1, **P < 0.05.

^c P‐value calculated using Pearson's chi‐square test.

^d P‐value calculated using independent samples t‐test.

Table 4.

Multivariate analyses of changes within and between the groups. For the within‐group analysis, adjusted odds ratio (AdjOR) >1.0 favours increase from baseline to follow‐up. For the between‐group analysis, ratio odds ratio (ROR) >1.0 favours intervention

Variable	Within				Between
	Intervention N = 743a		Control N = 908a		Intervention vs. control
	AdjOR (95% CI)	P‐valuec	AdjOR (95% CI)	P‐valuec	Ratio OR (95% CI)	P‐valuec
1. Were you involved as much as you wanted in decisions about your treatment? (% always/usually)	1.12 (0.78–1.62)	0.540	1.29 (0.93–1.80)	0.131	0.9 (0.5–1.4)	0.574
2. Did the staff treat you with respect and dignity? (% always/usually)	1.86 (1.01–3.45)**	0.048	1.47 (0.89–2.43)	0.137	1.3 (0.6–2.8)	0.561
3. Did the staff explain things in a way you could understand? (% always/usually)	1.11 (0.65–1.92)	0.695	1.07 (0.70–1.65)	0.750	1.0 (0.5–2.1)	0.917
4. Did the staff listen carefully to you? (% always/usually)	1.00 (0.64–1.55)	0.987	0.88 (0.58–1.31)	0.516	1.1 (0.6–2.1)	0.677
5. Did the staff who treated you work well together as a team? (% always/usually)	0.72 (0.46–1.14)	0.161	1.13 (0.74–1.70)	0.577	0.6 (0.3–1.2)	0.151
6. Did the staff spend enough time with you? (% always/usually)	1.10 (0.76–1.57)	0.620	1.13 (0.80–1.61)	0.486	1.0 (0.6–1.6)	0.917
7. Did the staff give you reassurance and support? (% always/usually)	1.23 (0.84–1.81)	0.280	1.00 (0.70–1.42)	0.996	1.2 (0.7–2.1)	0.437
8. How much were you helped by the care you received? (% pretty much/a lot)	0.74 (0.54–1.02)*	0.064	1.17 (0.88–1.54)	0.288	0.6 (0.4–1.0)**	0.034
9. Did the staff give you information about the rules and policies of the program? (% yes)	1.08 (0.80–1.47)	0.608	1.14 (0.87–1.51)	0.345	0.9 (0.6–1.4)	0.796
10. Did the staff give you information about your rights as a patient? (% yes)	0.93 (0.68–1.26)	0.625	1.32 (1.01–1.74)**	0.044	0.7 (0.5–1.1)*	0.095
11. Did the staff tell you about the risks and benefits of the medication(s) you are taking? (% yes)	0.81 (0.59–1–10)	0.178	0.86 (0.66–1.13)	0.287	0.9 (0.6–1.4)	0.775
12. Did the staff review with you the plans for your continued treatment? (% yes)	0.95 (0.70–1.29)	0.749	1.21 (0.92–1.60)	0.178	0.8 (0.5–1.2)	0.250
13. Were you told who to contact in case you have a problem or emergency? (% yes)	1.17(0.86–1.58)	0.329	0.91 (0.69–1.19)	0.484	1.3 (0.9–1.9)	0.228
14. Did the staff tell you about self‐help or support groups? (% yes)	1.19(0.86–1.66)	0.293	0.87 (0.67–1.15)	0.336	1.4 (0.9–2.1)	0.149
15. Did the staff give you information about how to reduce the chances of a relapse? (% yes)	1.07 (0.79–1.45)	0.654	1.09 (0.83–1.43)	0.555	1.0 (0.7–1.5)	0.929
16. Would you recommend this facility to someone who needed mental health treatment? (% yes)	1.11(0.70–1.75)	0.667	1.00 (0.66–1.53)	0.985	1.1 (0.6–2.1)	0.742
#17. Do you know if the unit has a user's committee? (% yes)	0.84 (0.38–1.86)	0.678	1.00 (0.67–1.49)	0.995	0.8 (0.3–2.0)	0.701
#18. Do you know if the unit has representatives or spokespersons on behalf of the users? (% yes)	1.10 (0.47–2.57)	0.821	1.35 (0.87–2.09)	0.181	0.8 (0.3–2.1)	0.675
#19. Have you received information about how to make a complaint about your treatment? (% yes)	1.02 (0.86–1.53)	0.910	1.44 (1.03–2.04)**	0.032	0.7 (0.5–1.1)	0.130
#20. Have you received information about the confidentiality of your records? (% yes)	1.29 (0.92–1.81)	0.136	1.11 (0.81–1.52)	0.518	1.2 (0.7–1.8)	0.524
#21. Have you been offered an Individual plan? (% yes)	0.83 (0.60–1.15)	0.274	0.98 (0.74–1.29)	0.857	0.8 (0.6–1.3)	0.447
#22. Have you received information about your rights to read and make corrections in your records? (% yes)	0.91 (0.65–1.29)	0.604	0.95 (0.71–1.29)	0.759	1.0 (0.6–1.5)	0.853
#23. How has your therapist(s) cooperated with your next of kin? (% good/very good)	0.74 (0.46–1.19)	0.213	0.97 (0.59–1.58)	0.892	0.8 (0.4–1.5)	0.438
#24. How would you evaluate the information you have received about the treatment options for your problems? (% good/very good)	0.80 (0.59–1.09)	0.151	0.98 (0.75–1.29)	0.900	0.8 (0.5–1.2)	0.331
#25. How would you evaluate the information you have received about your mental health problems? (% good/very good)	0.91 (0.67–1.22)	0.515	0.97 (0.74–1.27)	0.823	0.9 (0.6–1.4)	0.756
#26. Overall, how would you evaluate the help you have received for your mental health problems? (% good/very good)	0.90 (0.66–1.23)	0.502	1.16 (0.87–1.55)	0.305	0.8 (0.5–1.2)	0.241

	Diff (95% CI)	P‐valued	Diff (95% CI)	P‐valued	Diff (95% CI)	P‐valued
Inpatient Treatment Alliance Score (I‐TAS) (questions no 27–36) (mean of total score between 0 and 6)b	−0.039 (−0.232 to 0.154)	0.693	0.083 (−0.096 to 0.263)	0.362	−0.028 (−0.162 to 0.106)	0.682

Questions marked with # were added by the researchers.

^aN is the no of participants who returned completed questionnaires. The N in the four samples varied for each question due‐missing answers on the variables (0–8%).

^bFor total score of I‐TAS, a higher score indicates a stronger treatment alliance.

*P‐value < 0.1, **P‐value < 0.05.

^c P‐value calculated using logistic regression and test of proportions.

^d P‐value calculated using analysis of covariance (ancova).

Inpatient Treatment Alliance Scale (I‐TAS)

The I‐TAS25 was used to measure the patients' perception of treatment alliance with therapists. Inpatient Treatment Alliance Scale scores early in treatment have been shown to be significantly associated with early symptom improvement, patient satisfaction, length of stay in hospital and patients' initial reduction in symptom severity.25 I‐TAS has 10 items where the participants tick one box from 0 (False) to 6 (Completely true). Inpatient Treatment Alliance Scale questions are presented as questions 27–36 in Tables 3 and 4. The total I‐TAS score is the mean of all answers. Only participants answering 5 or more of the 10 items were included in the analyses of I‐TAS. Inpatient Treatment Alliance Scale has strong psychometric properties: adequate internal consistency, good item‐to‐item correlation and good test–retest correlation.25 Studies have found it to be unifactorial, with one primary factor counting for 66.4% of the variance.25

Self‐made questions

Ten self‐made questions are presented as questions 17–26 in Tables 3 and 4. These questions were added by the researchers to ensure that all aspects of the experience of user participation were included. The questions were developed based on the Norwegian health service context in discussions among the authors and other researchers. Questions 17–18 were recoded to ‘yes’ and ‘no/don't know’. Questions 19–22 had ‘yes’ and ‘no’ as possible answers. Questions 23–32 were recoded to ‘very poor/quite poor/neither good or poor’ and ‘good/very good’.

Statistical analysis

The results from the two control hospitals were combined. The data thereby included results from four different samples of patients – sample 1 (baseline) and sample 2 (follow‐up) for the intervention group, and sample 3 (baseline) and sample 4 (follow‐up) for the control group. Changes from baseline to 16 months were analysed for the intervention group and the control group, respectively. The effect was measured by comparing the change in the intervention group (differences between results in sample 1 and sample 2) with the change in the control group (differences between results in sample 3 and sample 4).

Pearson's chi‐square tests were used to identify any differences in demographic variables between the baseline and follow‐up sample in the intervention and control group, respectively. Demographic variables with trends for difference (P < 0.1) within each group (intervention group; gender and number of people in treatment team – control group; disability benefit, unemployment, physical health and age) were included in the multivariate analysis models.

Multivariate analyses

To analyse changes in PoC questions (1–16) and added questions (17–26), binary logistic regression was used. Results from the bivariate analyses of the demographic variables were added to the regression model. A test of proportion26 was used to calculate ratio odds ratio (ROR) to compare the effect in the intervention group with the effect in the control group. To analyse changes in the I‐TAS score from baseline to 16 months and to compare the change in the intervention group with the change in the control group, analysis of variance (ancova) was used. Results from the bivariate analyses of the demographic variables were added as covariates.

A significance level of 5% (P < 0.05) was chosen for all analyses. Analyses were carried out with SPSS 17.0 for Windows (SPSS Inc., Chicago, IL, USA).

Results

A total of 1651 patients participated. At the intervention centre, 1202 questionnaires were mailed to patients at the first time point and 324 (27%) responded. Sixteen months afterwards, 1350 questionnaires were mailed and 419 (31%) responded. At the control centres, 1387 questionnaires were mailed at the first time point and 477 (34%) responded. Sixteen months later, 1346 questionnaires were mailed and 431 (32%) responded. The average response rate was just below 30%, as in similar studies in this patient group.27, 28, 29

The total sample (N = 1651) included 65.5% women, and the mean age was 41.4 years (SD 13.0). Of these patients, 35.4% had more than one person in their treatment team, 19.6% had been hospitalized due to mental health problems during the last year, and 78% reported that they had someone they could talk confidentially with. 37.6% had education on a university level, 27.5% were working, while 12.1% were students, 23.1% received disability benefits, and 29.3% received rehabilitation benefits. Of these patients, 22.4% rated their physical health as very good/excellent, while 8.7% rated their psychological health as very good/excellent.

Differences in demographic variables

The demographic variables at baseline and follow‐up for the intervention group and control group, respectively, are presented in Table 2. In the intervention group, there were significantly more women at follow‐up (P = 0.03) and a trend for fewer patients having more than one person in their treatment team at follow‐up (P = 0.085). In the control group, the patients were significantly younger at follow‐up (P = 0.035). In addition, there were significantly fewer patients receiving disability benefit at follow‐up (0.048) and a trend for more patients being unemployed at follow‐up (P = 0.063) There was also a trend for fewer patients reporting very good or excellent physical health at follow‐up (P = 0.1). These variables were added to the binary logistic regression models (for categorical variables) and the analysis of covariance (for continuous variables) and were thus controlled for when comparing the changes within and between the groups.

Table 2.

Demographic variables – comparison of proportions at baseline and follow‐up (total N = 1651). Numbers are percentages of total N for each sample unless otherwise stated

	Intervention			Control
	Baseline N = 324a	Follow‐up N = 419a	P‐valueb	Baseline N = 477a	Follow‐up N = 431a	P‐valueb
Female	61.0%	68.7	0.030	64.9	66.4	0.628
Do you have somebody you can talk to? (% yes)	79.1	80.4	0.645	77.7	75.2	0.398
Education
Elementary school	15.3	15.4	0.964	16.7	17.9	0.641
High school	47.0	41.1	0.107	47.5	48.5	0.781
College graduate	23.7	28.6	0.132	23.3	21.6	0.545
Post graduate	14.0	14.9	0.735	12.4	12.0	0.848
Work/benefit status
Working	23.8	23.9	0.974	29.6	31.6	0.514
Sick benefit	10.8	10.0	0.730	10.9	13.2	0.282
Disability benefit	21.9	17.7	0.147	28.7	23.0	0.048
Rehabilitation benefit	29.0	28.6	0.911	29.6	39.9	0.903
Education	16.7	19.3	0.350	6.5	7.9	0.417
Unemployed	5.2	5.7	0.776	1.9	3.9	0.063
Other	4.9	6.0	0.543	6.5	7.2	0.679
Hospitalized the last 12 months	21.9	20.5	0.645	19.6	17.0	0.320
Self‐reported physical health
Excellent/very good	23.6	25.6	0.528	22.6	18.1	0.100
Self‐reported mental health
Excellent/very good	7.9	9.3	0.513	8.3	9.2	0.627
More than one person in treatment team	39.8	33.6	0.085	35.9	33.2	0.404
Age (mean, SD, range)	39.6 (12.6) (21–87)	38.2 (13.3) (19–87)	0.144c	44.4 (12.3) (20–83)	42.6 (12.9) (19–83)	0.035c

^aThe N in the four samples varied for each question due to missing answers on the variables (1–3%).

^b P‐value calculated using Pearson's chi‐square test.

^c P‐value calculated using independent samples t‐test.

Changes in experience of user participation from baseline to follow‐up within groups

The results on all questions on user participation on baseline and follow‐up, respectively, are presented in Table 3. In the intervention group, user participation had decreased in three questions (number 8, P = 0.072; number 11, P = 0.066; and number 24, P = 0.041). In the control group, user participation had increased in three questions (number 1, P = 0.075; number 10, P = 0.039; and number 19, P = 0.053).

The results from multivariate analysis of changes from baseline to follow‐up for each group are presented in Table 4. Within the intervention group, there was significant change from baseline to follow‐up in the question ‘Did the staff treat you with respect and dignity?’ (number 2, AdjOR = 1.86, P = 0.048) where a larger proportion answered ‘always/usually’ at follow‐up. In addition, there was a trend for change in the question ‘How much were you helped by the care you received?’ (number 8, AdjOR = 0.74, P = 0.064) where a smaller proportion of patients answered ‘pretty much/a lot’ at follow‐up. There was more experience of user participation at follow‐up for one question and less user participation for one question. There was no change in treatment alliance (I‐TAS score) from baseline to follow‐up in the intervention group.

Within the control group, there was significant change in two questions; ‘Did the staff give you information about your rights as a patient?' (number 10, AdjOR = 1.32, P = 0.044) and ‘Have you received information about how to make a complaint about your treatment?’ (number 19, AdjOR = 1.44, P = 0.032). On both questions, a larger proportion of patients answered ‘yes’ at follow‐up, reflecting more experience of user participation at follow‐up for both questions. There was no change in treatment alliance (I‐TAS score) from baseline to follow‐up in the control group.

There were thus only very small changes (changes on only a few questions) from baseline to follow‐up in the patients' experience of user participation within both the intervention hospital and the control hospitals.

Comparison between the groups

Results from the multivariate between‐group analyses are presented as ROR and estimated difference in Table 4. Comparison of changes between the intervention group and the control group showed significant difference in change in only one question; ‘How much were you helped by the care you received?’ (number 8) with a ROR of 0.6 (P = 0.034) indicating more improvement in user participation in the control group than the intervention group. In addition, in question number 10 ‘Did the staff give you information about your rights as a patient?’ there was a trend for more improvement in user participation in the control group than the intervention group (ROR = 0.7, P = 0.095). There was no difference in changes in treatment alliance between the intervention group and the control group (P = 0.682).

Discussion

The main result from this study was that implementing a development plan in a mental hospital intending to enhance user participation had no significant effect on the patients' experience of user participation. Some of the results indicated, in fact, more improvement in user participation in the control hospitals than in the hospital with the development plan.

Strengths and limitations

The main strength of this study is that it is one of the first large controlled studies investigating the effect from a large organizational initiative on patients' experience of user participation. The sample was varied and included both patients who had received outpatient and inpatient treatment.

As the development plan was initiated in one hospital by the management, the hospitals in this study could not be randomized. To maintain the participants' anonymity, the samples in this study were not matched. This could have rendered the samples considerably different and thereby influenced the findings. Thus, this constitutes a limitation. The present study included all three mental hospitals at this treatment level in one hospital trust and, thereby, provided a good representation of patients in this area. Studying patients in a different part of mental health services or in another area or country could have provided different results. Although the response rate (30%) was in line with similar studies on patients in mental health services27, 28, 29 and was similar in the two groups, it is still low and constitutes a limitation.

Both the Perception of Care and the I‐TAS questions were translated to Norwegian for this study. Although translation was thorough, no transcultural validation30 was conducted. Such validation would be helpful to ensure the appropriateness for the Norwegian culture and health‐care system. As studies using the PoC questionnaire after 2002 have used the extended version with 20 questions,31 there is little knowledge of the psychometric properties of the version used in this study. The I‐TAS questionnaire has previously been used for inpatients only,25 and the psychometric properties for other health service user groups are not known. The measures used in this study therefore constitute a limitation. Before conducting this study, we searched the literature for appropriate questionnaires to measure patients' experience of user participation. We found that the PoC and I‐TAS were appropriate for this task despite the weaknesses. It is also reasonable to ask whether the PoC and the I‐TAS questionnaires fully cover the phenomenon of user participation. User participation is related to several other concepts (e.g. patient‐centred medicine,32 empowerment33 and patient education).34 Several authors have highlighted the lack of robust measurement of user participation and related phenomena, especially regarding change and impact.35, 36, 37, 38, 39 To remedy this, we added several self‐developed questions on user participation.

Discussion of results

This is the second publication on a study investigating the effect from implementing the development plan intended to enhance user participation in this mental health hospital. The investigation of effect from the development plan on the professionals' knowledge, practice and attitudes towards user participation did not show any effect.22

This is not the first study that fails to show any measurable effect in‐patients from user participation initiatives.8, 9, 10 Similar results were also found in another Norwegian study by Storm et al., where there was no improvement on the patient‐reported outcomes on satisfaction with treatment and care from several user participation initiatives.21 Qualitative studies on initiatives at the organizational level have shown that the intended effect from user participation initiatives sometimes fails to appear and that some aspects are hard to impact.1, 40, 41 Although it was found to be a good and friendly collaboration between users and health professionals, the treatment ideology, the status of users and other important factors in health services were hard to influence.

User participation initiatives

The lack of effect found in this study might be due to inappropriate initiatives in the development plan. Although several of the initiatives were accomplished, this does not necessarily mean that patients' experience of user participation will be changed. The development plan included initiatives both at an organizational level (e.g. improving patient information and establishing a patient education centre) and at an individual level (e.g. introducing systems for coordination of care or increased participation for individual patients). These initiatives should ideally increase users' participation in the hospital as a whole and lead to changes in the patients' experience. Authors have argued that studies of change pay little attention to which characteristics of the interventions that should facilitate or hinder any change.42 Although the initiatives in the plan are perceived as appropriate to enhance user participation, authors have argued that the active component of participation is still unknown.40, 43 This complicates the endeavour to conduct studies with a rigid design such as controlled studies.40

As described in the introduction, most effects from user participation have been found when investigating smaller and rather isolated initiatives in limited parts of health service. It would be reasonable that isolated single initiatives are easier to implement and evaluate, and more often show measurable effect, than larger and more comprehensive programmes. Endeavours to change the organization's culture and attitude towards user participation would be harder to plan, implement and measure than more isolated and limited initiatives.

Previous research has also shown that patients and health professionals have dissimilar perceptions of the meaning and implications of user participation.22, 40 In Storm and colleagues' study, the professionals reported a stronger focus on user participation, more patient collaboration and more involvement of family members.21 Initiatives perceived as appropriate by the professionals might therefore not be adequate to have impact on the patients' experience of user participation in their treatment.

Implementation processes

Another potential reason for the lack of effect in this study is that the process of implementing the development plan in the hospital was inadequate. Abundant literature shows that results from implementation processes are mixed,44, 45, 46 and there are many potential barriers to the implementation of interventions.47, 48 Many user participation implementations have been hampered by lack of clear directions and clarification of responsibility.49

Organizations have been described as complex social systems where it is difficult to make changes44, 46 and established practices and institutional beliefs have been described as very persistent.1 Implementation processes in general include several steps where many complex systems interact and where each step has to be conducted thoroughly to ensure the quality of the whole process.44, 45 To achieve measureable effects on the patients' perception of user participation, the organizational change must permeate every aspect of the organization's services. Some have argued that it is flawed to assume that changes at a high organizational level, although perceived as happening by management, will necessarily trickle down to other levels in health service organizations.50 Although many of the initiatives in the development plan were completed as planned, this might not have changed the organization enough for the service users to experience their treatment differently.

Due to the long history of stigmas and myths, changes in culture and attitudes towards user participation might be even more difficult and slow in mental health care, where the present study took place. User participation has been described as widespread1, 51 and particularly valuable in mental health care.52, 53 Participation from patients suffering from mental health problems has, at the same time, been viewed as difficult. Although mental health professionals embrace the ideal that the user is an equal and competent partner, users' concerns and views are often attributed to their mental health problems.1 Mental health professionals have been shown to hang on to their control over the decision‐making process, to use language that patronizes and devaluates users during participation processes, and to prefer to mainly consult users instead of acknowledging them as partners.54 Research on professionals' and patients' perceptions of limits for user participation shows that mental health problems were perceived by both parties as a challenge and obstacle towards participation.55 It might be that mental health is a particularly challenging area to make organizational changes intended to enhance user participation.

Conclusion

This is one of the first controlled studies investigating the effect of implementing a development plan in a mental hospital to enhance user participation on patients' experiences. The plan had no significant effect on the patients' experience of user participation. This can be due to the quality of the initiatives in the development plan or the implementation process. Further research with controlled studies is needed to build knowledge on the effects from user participation initiatives on patients' experience of participation. Further research should ensure that user participation interventions and implementation processes are appropriate to impact the patients' experience.

Authors' contributions

MBR designed the study, collected, analysed and interpreted data, and wrote and completed the manuscript. AS supervised and designed the study, interpreted data and contributed to the writing and completion of the manuscript. Both authors approved of the final version.

Funding

This project has been financially supported by the Norwegian ExtraFoundation for Health and Rehabilitation through EXTRA funds.

Conflict of interests

No competing interests have been declared.