Abstract
Background.
Increasing therapeutic inpatient activities may improve the quality and outcomes of care. Evaluation of these interventions is necessary including assessment of cost-effectiveness. The aim of this paper is to describe the development and reliability of a tool to collect information on care contacts and therapeutic activities of patients on inpatient wards.
Method.
The development of the tool consisted of: 1) literature review, 2) interviews with staff, 3) expert consultation, 4) feasibility study, 5) focus groups with staff members, and 6) reliability tests. Service use data were collected with the tool and costs calculated.
Results.
Service users' reported more use of activities than that contained in case notes during a 7-day period. This resulted in a cost difference of £10 per person. Case notes had more one-to-one nursing contacts, with a cost difference of £4 per person. One-day data showed less nurse contact time reported by participants compared to observational data (p < 0.001) but similar use of activities. Costs were £46 for the tool and £67 for the observational data.
Conclusions.
This tool is a good source of information on the number of activities attended by service users and contacts with psychiatrists. There is some disagreement with other sources of information on interactions between service users and nurses, possibly reflecting different definitions of a ‘meaningful contact’. This does not have a major impact on cost given that for much of the care received there is reasonable agreement.
Key words: costs and cost analysis, inpatient care, economic evaluation, resource use, hospital care
Spending on mental health services varies substantially across countries; in Europe ranging from around 2.5 to 14% of total health expenditure (McDaid et al. 2007). Psychiatric bed numbers have declined in most European countries in recent years (Knapp et al. 2008), but remain an important component of the mental health care system. The move towards community services has had some effects on the characteristics of inpatient patients; they have a higher level of severity of symptoms and behavioural problems, and the percentage of detained patients has increased. Consequently, it has been argued that the inpatient services can sometimes appear more custodial than therapeutic (Bowers, 2005).
Activities on wards have been key in improving the quality and outcomes of inpatient care. Lower quality-of-life has been found to be associated with inactivity (Kelly et al. 2001), activity scheduling on inpatient wards has produced reductions in depression compared with psychotherapy (Hopko et al. 2003), and activity as a whole may reduce challenging behaviour (Fialko et al. 2005). This evidence has not been translated to clinical settings as shown in a review by Sharac et al. (2010). Low activity and social engagement for patients have remained stable in recent years and limited nursing time is spent in direct contact with patients (Sanson-Fisher et al. 1979; Sandford et al. 1990) and even less providing therapeutic activities (Ryrie et al. 1998; Whittington & McLaughlin, 2000; Bee et al. 2006). Furthermore, it has been shown that higher-grade nurses spend less of their time directly with patients (Cormack, 1976; Higgins et al. 1999; Bee et al. 2006).
A systematic and comprehensive evaluation of inpatient activities could support greater implementation, and assessments of cost-effectiveness are important. Economic evaluations need to collect information on the use of services associated with different models of care but with inpatient services this use is usually only recorded as the number of days spent in hospital. For evaluations of an inpatient intervention, it would be more helpful to identify all the care inputs received while on a ward. This distinction is important because, even if the length of stay is the same, the use of resources might be different between patients depending on activities attended and the care received – both of which may be associated with patient clinical and demographic characteristics.
The aim of this study was to describe the development and assess the reliability of a tool (the CITRINE) to achieve the objectives of collecting data on the care contacts and therapeutic activities of patients while on a psychiatric ward. This would potentially enable a more precise costing of inpatient psychiatric services.
Methods
Initial draft
In developing the tool, we conducted interviews with inpatient staff to discuss its content and structure. The professionals interviewed included two ward managers, three occupational therapists, one nurse and one ‘therapy and benefits coordinator’. All were working at psychiatric inpatient units in the South London and Maudsley (SLAM) NHS Foundation Trust. The interviews lasted for approximately 30 min and the main objective was to identify the group activities that take place on the wards and which professionals were most likely to have contacts with inpatients. Staff sometimes provided detailed schedules of patient activity on their wards. The first draft of the tool was then circulated for comments and suggestions to a group of experts including psychiatrists, psychologists and nurses.
Feasibility
Twenty-five service users from inpatient psychiatric wards were then interviewed using the questionnaire. The objective was to determine the acceptability of the tool, service users' opinion about difficulties in answering the questions and to evaluate how practical it was to collect data in this way. Two focus groups were conducted in parallel with this feasibility component. The first included eight occupational therapists, with seven nurses from an inpatient ward comprising the second. Participants were asked for their opinions about the tool and for any suggestions for improvements.
The process of consultation with staff resulted in the development of a tool that allows activities specific to an inpatient ward to be measured. The list-of-group activities were amended to include only the activities that had taken place in the previous week. Interviewers list the availability of activities (based on the ward calendar for the relevant week) and therefore each version of the tool is unique to that particular ward and time period. Initial questions on contacts with members of staff were shortened. The staff contacts section was also revised so that service users could indicate the name of professionals they have met even if they did not know or remember their job title. The tool is shown in the Appendix.
Reliability
This assessed the level of congruence between the information provided by service users using the final version of the tool and information that could be obtained from other sources. This took place in two stages. Initially, information on the activities attended by 41 service users over the preceding 7 days was obtained from the occupational therapist of each ward and the number of contacts with nursing staff members was collected from the patient's records for the same period. This information was also collected from the same patients and period using the tool.
Subsequent to this, information on the number and duration of staff contacts and activities attended by 22 other service users was collected over a 14-h (08:00–22:00 h) direct observation period. An adapted 1-day version of the CITRINE questionnaire was used to collect the relevant information from the same service users over this period. (This stage was added later and so the patients are different to the 41 previously included.)
Contacts with staff members were combined with unit costs measured in UK pounds for the financial year 2007/2008 from an established source (Curtis, 2008). In the first phase (n = 41), we assumed an average nurse contact of 15 min. Activity costs were calculated specifically for the study based on session duration, preparation required, staff involved and materials for each activity provided on each ward. These data were combined with unit costs of staff time (Curtis, 2008) and with information on the average number of service users attending each activity to obtain an estimated individual cost for each group activity. Activities were subsequently classified in categories according to their type and cost. (A list of all unit costs is available from the authors.) An average of these activity costs was used in the analyses presented. The significance of differences in reported and observed/recorded mean resource use was assessed using a paired t test and the agreement was further assessed using the concordance correlation coefficient.
Approval for the PERCEIVE study was provided by an appropriate NHS ethics committee and participants were asked to provide written consent.
Results
The CITRINE tool typically took 5–10 min to complete and most of the service users who participated in the feasibility study found it easy or very easy to answer the questions. Table 1 reports 7-day data from the CITRINE and from information covering the same period from occupational therapists and case notes for 41 service users. The mean age was 37 years and 19 of them were women. Most (68%) had a primary diagnostic of psychosis or bipolar disorder and the average length of stay at assessment was 43 days. However, there was great variation in this represented by an s.d. of 66 days.
Table 1.
Mean number and cost (£) of activities and nursing contacts reported by service users and from OTs/case notes
| Mean (s.d.) contacts/duration | Mean (s.d.) cost (2007/2008 £s) | |||||
|---|---|---|---|---|---|---|
| One-week stage (n = 41) | Data provided by service user | OT/case note data | Data provided by service user | OT/case note data | T valuea | Significancea |
| Activities (number) | 5.2 (5.6) | 3.8 (3.0) | 39.5 (43.2) | 29.2 (23.2) | 2.159 | 0.037 |
| One-to-one contacts with nursing staff (number) | 2.8 (2.7) | 3.2 (3.9) | 30.4 (29.4) | 34.3 (47.3) | −0.501 | 0.619 |
| One-day stage (n = 22) | Data provided by service user | Observation data | Data provided by service user | Observation data | T value | Significance |
| Activities (number) | 1.0 (0.8) | 1.1 (1.1) | 7.7 (6.3) | 8.4 (8.2) | −0.624 | 0.539 |
| One-to-one contacts with nursing staff (duration in minutes) | 7.1 (13.8) | 29.8 (23.0) | 5.1 (9.9) | 21.3 (16.5) | −5.394 | <0.001 |
| One-to-one contacts with psychiatrists (duration in minutes) | 5.7 (16.3) | 5.7 (11.1) | 30.2 (86.0) | 29.9 (58.2) | 0.027 | 0.979 |
| One-to-one contacts with other staff (duration in minutes) | 3.2 (12.9) | 9.1 (13.6) | 2.7 (10.7) | 7.6 (11.3) | −1.982 | 0.061 |
| Total cost | 45.6 (92.9) | 67.2 (72.7) | −1.990 | 0.060 | ||
aTests of significance apply to both contacts/duration and cost.
Service users' report attending more activities than is contained in case notes (Table 1). Attaching an average unit cost of activities suggest a cost difference of £10 per person. Case notes also report more one-to-one nursing contacts, resulting in a cost difference of £4 per person. Although neither difference is large nor the discrepancy in the number of activities reported is statistically significant.
The comparison of data obtained from the 1-day observational study and the 1-day version of the tool show good congruence in terms of activities attended and psychiatrist contacts. However, the congruence in terms of contact with nurses and other staff is less good. The differences in these latter services were significant or of borderline significance. The concordance correlation coefficient for total costs from this part of the reliability study was 0.79. The cost implications for the difference in nursing contact is important given that the observational data suggest that this accounts for one-third of the total cost.
Discussion
The process of developing the CITRINE tool has involved input from a wide range of staff involved in providing care to those receiving psychiatric inpatient services (nursing staff, OTs, ward managers and psychologists). Furthermore, key input to the process has been provided by the recipients of this care. The tool collects data directly from service users. From a pragmatic perspective this is appropriate as it is the most practical way of obtaining the breadth of activity data that we require for economic studies. In addition, it emphasizes the service user's perspective in reporting activity. Of course, this could mean that activities that are entirely unmemorable will not be recalled or costed (e.g. conversations with nurses that could be very similar everyday such as ‘you have a telephone call’, ‘do you want to go for a walk?’, etc.). However, this type of very brief contact that is not remembered by service users is not the focus of the tool.
The result is a tool that has its objective in collecting information on the therapeutically relevant activities and staff contacts that take place on psychiatric wards and at the same time is acceptable to service users and relies on them as the main source of the relevant information. There have been other attempts to describe interventions and services provided in inpatient mental health settings, such as the International Classification of Mental Health Care (ICMHC) (de Jong, 2000). However, this instrument is a tool to classify services according to the type of care, rather than one that provides information on the specific services that patients receive.
The main function of the tool will be its use in economic studies, combined with information on relevant unit cost of the services and staff time. Therefore, its level of accuracy in reflecting what is actually taking place within psychiatric wards is paramount. The reliability study carried out offers some reflections on this issue. In particular, the questionnaire: (i) is a good source of information on the number of activities attended by service users and on contacts with psychiatrists, (ii) may provide acceptable information on interactions between service users and non-nursing care staff (OTs, psychologists, social workers, etc.) and (iii) may be more limited in recording time spent in service user–nursing staff interaction.
In relation to nursing staff contacts, there are some aspects that should be considered. First, this type of contact is the one that is common and consequently creates difficulties for accurate recall. Second, service users might report only contacts that they think are significant or meaningful for them. This classification is subjective and can result in some contacts being labelled as ‘non-contacts’ by service users. For example, a member of staff may have spent some time asking how a patient was, and would regard this as a contact, but it may not have been recognized as such by a patient if it was very brief or unwanted. This may have been the key reason for the discrepancy between the observed nurse–patient contacts and the patient-reported contacts. The mental health status of some of the service users may affect their ability to provide accurate information, although data on this are lacking. Although time spent with nursing staff needs to be measured correctly, the difficulty seems only to apply to one-to-one contacts. Contacts as part of organised activities are more readily measured and therefore the disagreement over total cost is limited.
The alternative sources of information on inpatient psychiatric activities and staff contacts are not free of problems. Registers and electronic databases are designed to support clinical care and not to record activities within wards. Furthermore, there might be intra- and inter-ward variability on the level of completion of these and on the accuracy of the information recorded. Observational data are an alternative but require too many resources to be a realistic when studies involve large number of individuals/wards with data collected at several time points.
There are limitations with this work. First, we did not conduct the full range of psychometric tests, although we did assess validity and reliability. It should be stressed that we do not regard this as a clinical instrument but rather a simple recording schedule to ascertain more accurately the use of care on inpatient wards. We anticipate the tool being adapted for different circumstances and extensive tests on one particular version would have limited usefulness. Second, and related, we did not assess inter-rater reliability. The content of the tool though is such that ambiguity of responses should be limited and hence inter-rate reliability is not as important as with a tool where interviewer interpretation is required. Third, the activities related to wards in one particular hospital. It may be that in another setting a more complex array of activities would present more of a challenge. The tool does, however, require the listing of specific activities at the start and so this problem is reduced. Fourth, the relatively poor concordance on contacts with nurses is a course for some concern. This has been discussed above, but the wording of the question may not be clear. Greater definition of what constitutes a contact may be required. Fifth, service users may not always have been clear about the profession of the person they had contact with. This is a problem for all research involving self-report data and one for which data on the extent to which this is a problem is required. However, definitions in the wording of the tool could be refined if necessary. Sixth, to rely on self-report data might be a limitation of the tool although a number of studies have suggested that patient recall of service use is acceptable (Calsyn et al. 1993; Goldberg et al. 2002; Patel et al. 2005). Asking for the duration of contacts may be difficult but these durations are meant to be approximations.
In conclusion, the CITRINE is a tool that, despite some limitations, provides adequate information on the activities that take place within psychiatric wards. Therefore, its use is recommended, alone or in combination with other sources, in economic analyses of inpatient care.
Appendix. Survey of inpatient activities and services receipt
Ward: ___________________Patient ID number: ___________________
We are interested in finding out what activities patients have been to in the last 7 days. We would also like to find out what staff members patients have seen. Please use a copy of the ward's activity timetable if it would help remind you about activities you have participated in.
Section 1: Group activities on the ward
Please indicate what activities you have taken part in since last___________ by filling out the table below.
| Please list the activities you have taken part in: | How many times since last | Extra notes (researcher use only) |
|---|---|---|
| Activities that take place at the ward | ||
| □ Community meeting | ||
| □ | ||
| □ | ||
| □ | ||
| □ | ||
| □ | ||
| □ | ||
| □ | ||
| Activities that take place at the OT resource centre/gym/swimming pool/community/… | ||
| □ | ||
| □ | ||
| □ | ||
| □ | ||
| □ | ||
Section 2: Meeting with staff members
Please note any individual meetings you have had with any of the staff members listed below. You do not need to include times where other patients have been involved e.g. in a therapeutic group.
One-on-one time with nurses:
Please indicate if you have spent one-on-one time with a nurse since last_______ by filling out the table below:
| How many times? | On average, for about how long? | |
|---|---|---|
| One-on-one time with nurses, other nursing staff or health care assistants | _________minutes |
Other professionals:
Please indicate if you have met with the following staff members since last______ by filling out the table below:
| Position | How many times? | On average, for about how long ? |
|---|---|---|
| Psychiatrist or consultant | _________minutes | |
| Other doctor | _________minutes | |
| Occupational therapist | _________minutes | |
| Care coordinator (includes telephone conversations) | _________minutes |
Please indicate if you have met with any other staff (not nurses) since last__________
e.g. pharmacist, chaplain, psychologist, hairdresser, dentist, physiotherapist, optician, chiropodist, advocate, solicitor
| Professional | How many times? | On average, for about how long? | Where did you meet? |
|---|---|---|---|
| ________minutes | □ On the ward | ||
| □ Off the ward | |||
| _________minutes | □ On the ward | ||
| □ Off the ward |
Thank you for taking part in our survey!
Declaration of Interest
This study presents independent research commissioned by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research scheme (RP-PG-0606-1050). The views expressed in this publication are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. We also acknowledge the financial support of the NIHR Biomedical Research Centre for Mental Health, South London and Maudsley NHS Foundation Trust/Institute of Psychiatry (King's College London).
References
- Bee PE, Richards DA, Loftus SJ, Baker JA, Bailey L, Lovell K, Woods P, Cox D (2006). Mapping nursing activity in acute inpatient mental healthcare settings. Journal of Mental Health 15, 217–226. [Google Scholar]
- Bowers L (2005). Reasons for admissions and their implications for the nature of acute inpatient psychiatric nursing. Journal of Psychiatric and Mental Health Nursing 12, 231–236. [DOI] [PubMed] [Google Scholar]
- Calsyn R, Allen G, Morse G, Smith R, Templehoff B (1993). Can you trust self-report data provided by homeless mentally ill individuals? Evaluation Review 17, 353–366. [Google Scholar]
- Cormack D (1976). Psychiatric Nursing Observed. Royal College of Nursing of the United Kingdom: London. [Google Scholar]
- Curtis L (2008). Unit Costs of Health and Social Care 2008. PSSRU: Canterbury. [Google Scholar]
- de Jong A (2000). Development of the international classification of mental health care (ICMHC). Acta Psychiatrica Scandinavica, Supplement 405, 8–13. [PubMed] [Google Scholar]
- Fialko L, Jabati S, Gregory R, Grey S (2005). Improving provision and uptake of therapy activities on an in-patient ward: procedures for regular monitoring and review. Proceedings of the British Psychological Society 13, 170. [Google Scholar]
- Goldberg RW, Seybolt DC, Lehman A (2002). Reliable self-report of health service use by individuals with serious mental illness. Psychiatric Services 53, 879–881. [DOI] [PubMed] [Google Scholar]
- Higgins R, Hurst K, Wistow G (1999). Nursing acute psychiatric patients: a quantitative and qualitative study. Journal of Advanced Nursing 29, 52–63. [DOI] [PubMed] [Google Scholar]
- Hopko DR, Lejuez CW, LePage JP, Hopko SD, McNeil DW (2003). A brief behavioral activation treatment for depression. A randomized pilot trial within an inpatient psychiatric hospital. Behavior Modification 27, 458–469. [DOI] [PubMed] [Google Scholar]
- Kelly S, McKenna H, Parahoo K, Dusoir A (2001). The relationship between involvement in activities and quality of life for people with severe and enduring mental illness. Journal of Psychiatric and Mental Health Nursing 8, 139–146. [DOI] [PubMed] [Google Scholar]
- Knapp M, McDaid D, Medeiros H, Becker T, Johnson S, Kilian R, Salvador-Carulla L, Simon J, Tatar M (2008). Economics, Mental Health and Policy: An Overview. Mental Health Economics European Network: London. [Google Scholar]
- McDaid D, Knapp M, Medeiros H, the MHEEN group (2007). Mental health and economics in Europe: findings from the MHEEN group. Eurohealth 13, 1–6. [Google Scholar]
- Patel A, Rendu A, Moran P, Leese M, Mann A, Knapp M (2005). A comparison of two methods of collecting economic data in primary care. Family Practice 22, 323–327. [DOI] [PubMed] [Google Scholar]
- Ryrie I, Agunbiade D, Brannock L, Maris-Shaw A (1998). A survey of psychiatric nursing practice in two inner city acute admission wards. Journal of Advanced Nursing 27, 848–854. [DOI] [PubMed] [Google Scholar]
- Sandford DA, Elzinga RH, Iversen RA (1990). A quantitative study of nursing staff interactions in psychiatric wards. Acta Psychiatrica Scandinavica 81, 46–51. [DOI] [PubMed] [Google Scholar]
- Sanson-Fisher RW, Desmond Poole A, Thompson V (1979). Behaviour patterns within a general hospital psychiatric unit: An observational study. Behaviour Research and Therapy 17, 317–332. [DOI] [PubMed] [Google Scholar]
- Sharac J, McCrone P, Sabes-Figuera R, Csipke E, Wood A, Wykes T (2010). Nurse and patient activities and interaction on psychiatric inpatients wards: a literature review. Journal of Nursing Studies 47, 909–917. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Whittington D, McLaughlin C (2000). Finding time for patients: an exploration of nurses' time allocation in an acute psychiatric setting. Journal of Psychiatric and Mental Health Nursing 7, 259–268. [DOI] [PubMed] [Google Scholar]