Examining Outcomes of Acute Psychiatric Hospitalization among Children

Abstract

Within the past two decades, few studies have examined outcomes of acute psychiatric hospitalization among children, demonstrating change in emotional and behavioral functioning. A secondary analysis of pre-test/post-test data collected on 36 children was conducted, using the Target Symptom Rating (TSR). The TSR is a 13-item measure with two subscales – Emotional Problems and Behavioral Problems and was designed for evaluation of outcome among children and adolescents in acute inpatient psychiatric settings. Results of this study, its limitations, and the barriers encountered in the implementation of the TSR scale as part of routine clinical practice are discussed.

Mental healthcare has undergone significant changes over the past several decades in the United States. Most notable are reduced reliance on inpatient psychiatric care, and more recently, a greater demand for the delivery of treatments that are evidence-based (Hoagwood, Jensen, Petti & Burns, 1996). Several developments in mental health policy and practice have prompted these changes: (1) A conceptual shift toward community-based care rather than institutionalized care (Anthony, 1993; Stroul & Friedman, 1986); (2) the emergence of managed care with demands for cost effective treatments and accountability (Anthony, 1993; Stroul & Friedman, 1986); and (3) the consumer movement, led by advocates of mentally ill Americans, who have not only fought to attain parity for mental health coverage but also for the delivery of mental health care in the least restrictive treatment environment possible (Goldman, McCulloch & Sturm, 1998). Inpatient psychiatric care has become ‘acute’ care, to be used primarily for symptom stabilization in cases of mental health crisis (Goldman et al., 1998).

A number of studies have investigated utilization rates and cost of child and adolescent mental health care, producing different results depending on sample and time period studied (Case, Olfson, Marcus & Siegel, 2007; Glied & Cuellar, 2003; Martin & Leslie, 2003; Ringel & Sturm, 2001; Pottick, McAlpine, & Andelman, 2000). Some studies have reported significant declines in child inpatient utilization rates (e.g., Martin & Leslie, 2003; Pottick et al., 2000) while a more recent nationally representative study showed no significant changes in discharges or daily cost over a 10-year period (Case et al., 2007). However, findings across several studies report utilization rates of about 2 out of 1000 children aged 0 to 17, with rates being highest for adolescents. Cost of inpatient psychiatric care remains disproportionally high, but has decreased vis-à-vis outpatient cost due to shortened stays in care and greater utilization of community-based treatments. Utilization rates among children aged 6 to 13 is lower compared to adolescents but has been reported to have increased significantly (Case et al., 2007). While lengths of stay in inpatient psychiatric settings have declined by as much as 63% in the decade between 1990 and 2000 (Case et al., 2007; - Glied & Cuellar, 2003; Pottick et al., 2000), rehospitalization rates have almost doubled, suggesting to some that the decline in long-term hospitalization has led to ineffective psychiatric treatment (Heggestad, 2001; Lien, 2002; Wickizer, Lessler & Boyd-Wickizer, 1999).

These developments along with calls for greater accountability have resulted in increasing emphasis on the assessment of outcomes within psychiatric settings (Hoagwood et al., 1996). The current demand for evidence-based practices further underscores the need to identify valid and reliable measures that could capture changes in functioning and determine effectiveness within the context of short-term interventions. However, within the field of child and adolescent acute psychiatric care, there are few measures conducive to this task, particularly for younger children.

Explanatory models of mental illness have identified a range of psychosocial and biological factors that place children at risk for emotional and behavioral problems. Central to evidence based programs based on these models is risk reduction and the enhancement of protective factors (Department of Health and Human Services, 1999). The purpose of the current study, using key risk factors identified in prior studies, is to report on a pilot project conducted in a large child inpatient unit, aimed at implementing an outcome measure specifically designed for acute inpatient psychiatric care and evaluating the outcomes of children.

Background

Outcome Studies of Inpatient Psychiatric Care

Few studies within the past two decades have examined outcomes of acute psychiatric inpatient care among children and adolescents (Pottick, Hansell, Gaboda, & Gutterman, 1993). Even fewer studies have focused solely on children, and excluded adolescents. A review of peer-reviewed outcome studies in this area found a total of 34 studies between 1970 and 1988; six which specifically investigated children, 17 studies which focused on adolescents only, and 11 studies which combined the two populations (for a review of these studies see Pfeiffer & Strzelecki, 1990). Between 1989 and 1992 no outcome studies were reported to be published (Pottick et al., 1993). More recently, Bettmann and Jasperson (2009) conducted a review of the outcome literature concerning adolescents in inpatient settings. However, their review was limited to studies involving adolescents between 12 to 18 years old and included studies of both inpatient psychiatric care and residential treatment.

Since Pfeiffer’s and Strzelecki’s (1990) review, a handful of studies, which included children, were conducted. Healy and Fitzgerald (2000) conducted a longitudinal follow-up study of a sample of 50 children who were admitted to a psychiatric inpatient unit 16 years prior and found that a majority of participants experienced poor long-term outcomes which were defined as death, imprisonment, unemployment or psychiatric illness. Rice, Woolston, Stewart, Kerker, and Horwitz (2002) compared younger, middle and older children who were admitted to a psychiatric inpatient unit and found that younger children tended to be hospitalized for more serious emotional, behavioral and family problems. Using a retrospective design, Cornsweet Barber and colleagues (Cornsweet Barber, Neese, Coyne, Fultz, & Fonagy, 2002) examined treatment outcomes among a sample of child and adolescent inpatients with a median length of stay of 13 days and found that significant improvements occurred between admission and discharge.

A review of the combined literature indicates that most outcome studies of child and adolescent inpatient psychiatric care assess outcomes in terms of symptom reduction, quality of care indicators, such as rehospitalization, and service effectiveness outcomes, such as cost of services (Chung, Edgar-Smith, Palmer, Bartholomew & Delambo, 2008; Fontanella, 2008; Hoagwood et al., 1996; Pottick, Hansell, Miller & Davis, 1999). These studies generally report significant improvements in child behavior and emotional symptoms (Gavidia-Payne, Littlefield, Hallgren, Jenkins, & Coventry, 2003; Swadi & Bobier, 2005) as well as overall functioning (Gold et al., 2009). Green et al. (2007) examined inpatient treatment in child and adolescent psychiatry using a prospective pre-treatment, intra-treatment, and post-treatment cohort study of multiple units in the U.K. along with a 1-year follow-up post discharge. Consistent with other studies, the results indicated significant health gains since baseline. Also reported, high levels of aggression were associated with shorter lengths of stay and less therapeutic change.

A number of studies have investigated child, family and service factors in relation to inpatient psychiatric outcome (e.g., Chung et al., 2008; Fontanella, 2008; Harnett, Loxton, Sadler, Hides, & Baldwin 2005; James et al., 2010) either as predictor or mediator variables. Findings of these studies have been inconsistent depending on methodology used, but factors such as clinical diagnosis and symptom severity, length of stay in care and prior hospitalizations have been relatively consistent predictors across studies (see Chung et al., 2008 and Fontanella, 2008 for a review of this literature). In most studies, greater symptom severity has been associated with poorer outcome (e.g., Fontanella, 2008); however, Mayes et al. (2001) determined that greater symptom severity was associated with a higher rate of change. With regard to length of stay in care, some studies have found shorter lengths of stays to be associated with an increased risk for poor outcome, such as rehospitalization (Case et al., 2007; Wickizer, Lessler & Boyd-Wickizer, 1999) whereas others have reported a significant relationship between longer stays and poor outcome (James et al., 2010).

While findings of the limited outcome literature are encouraging, definitive conclusions cannot be drawn due to small sample sizes, missing data, lack of control groups and standardized measures (Gavidia-Payne et al., 2003; Green et al., 2007; Rice et al., 2002; Sourander & Piha, 1998; Sourander & Leijala, 2002). Our knowledge is further limited due to few studies reflecting current short stays in care. Length of stay was found to have decreased by 90% between 1988 and 1994. Average length of stay was 9 months in 1988, but only 5 weeks in 1994 (Pottick, McApline & Andelman, 2000). Since then, it has further decreased to a median 4.5 days in care (Case et al., 2007). The few studies conducted with children in inpatient psychiatric settings that reflect current short stays (Cornsweet Barber et al., 2002; Mayes et al., 2001; Swadi & Bobier, 2005) show that children can experience improvement even during short time periods.

Measures Utilized in Child and Adolescent Inpatient Settings

Assessing changes for children within a short-term inpatient setting is challenging, especially since a child’s response to inpatient treatment is believed to be mediated by multiple factors (Sourander et al., 1996). Few measures exist which are suitable in examining the effectiveness and quality of mental health services in acute psychiatric settings for children and adolescents (Cornsweet Barber et al., 2002). Measures designed to assess outcomes in inpatient psychiatric settings, impose numerous challenges when implemented in settings with brief lengths of stay. Table 1 presents a comprehensive overview of measures that have been utilized in studies of child and adolescent inpatient psychiatric care. Some of the most common measures utilized will be highlighted next.

Table 1.

Overview of Outcome Measures for Children and/or Adolescents in Inpatient Psychiatric Settings

Measure	Measure Properties and Use	Psychometric Properties	Strengths	Limitations
Achenbach (1991) & (1992) Child Behavior Checklist/4 –18 (CBCL/4–18) Child Behavior Checklist/2 – 3 (CBCL/2–3)	Behavior problem checklist which consists of 113 items. Measures emotional and behavioral problems in children and adolescents. Responders are given three options: 0 = not true of their child; 1 = somewhat or sometimes true; and 2 = very true or often true.	Reliability: Internal consistencies ranged from .92 to .96. For the internalizing scale the reliability ranged from .88 to .92 Validity: Strong discriminant validity. High concurrent correlations with related instruments and the CBCL.	Used in large national studies. Translated into 50 different languages	Multiple items make it a challenge to administer such an instrument in a short term inpatient setting.
Bracken, & Keith (2004) Clinical Assessment of Behavior (CAB)	Purpose is to aid in the assessing, diagnosing, screening and treating of children between the ages of 2 and 18 years old. Consists of 3 rating forms: the Parent Rating Form (CAB-P) consisting of 70 questions, the Parent Extended Rating Form (CAB-PX) consisting of 170 questions, and the Teacher Rating Form (CAB-T) consisting of 70 questions. Items on all scales utilize a 5-point likert scale.	Reliability: Test-retest reliability ranged between .75 and .93. Inter-rater reliability for the scales and subscales ranged between .40 and .58. Validity: Correlation coefficients were found to range between .57 and .77. Construct validity ranged between .71 and .95.	Can be utilized to track the progress of clients during treatment. Designed to closely reflect the content of current behavioral disorder literature and child/adolescent psychopathology. Possesses strong psychometric properties.	It cannot be used among individuals lacking a proficient knowledge of the English language.
Burlingame, Jasper, Peterson, Wells, Reisinger, & Brown (n.d) Youth Outcome Measure (YOQ 30.1)	This measure is a shortened version of the Youth Outcome Questionnaire. It consists of 30 likert item questions, which may be completed by parents, clinicians, adolescents or teachers. Designed to be used both at intake and at the end of the treatment. A parent and a self-report version of the scale exist.	Reliability: Scale has high internal consistency, with a coefficient alpha ranging between .92 and .94. The Self-report version has a test-retest reliability of .91 and the parent-report version has a test-retest reliability of .80. Inter-rater reliability was in the low to moderate range, .32 to .77.	Enables clinicians to track the progress of children and adolescents. It allows clinicians to track the overall distress of a patient.	Supporting psychometric evidence regarding whether the YOQ 30.1 provides valid information is limited and hence does not provide clinicians with the informed clarity needed to make decisions based on this measure.
Eisen, Dill, & Grob (1994) Behavior and Symptom Identification Scale (BASIS-32)	This scale consists of 32 items and items are assessed on a 5-point scale. Utilized to measure outcomes in inpatient settings	Reliability: good test-retest reliability and internal consistency was established (Klinkenberg, Cho, & Vieweg, 1998) Vailidity: Good concurrent and discriminant validity.	The use of the scale has been examined in both inpatient and outpatient settings. It obtains a patient-report and can be easily administered and scored (Hoffmann, Capelli, & Mastrianni, 1997)	Takes approximately 20–30 minutes to complete. Questionable utility in measuring outcomes among adolescents (Hoffmann et al, 1997; Klinkenberg et al., 1998).
Eyberg, & Pincus (1999) Eyberg Child Behavior Inventory (ECBI) Sutter-Eyberg Student Behavior Inventory Revised (SESBI-R)	Examines conduct problems in children between 2– 16 years old. Consists of a total of 38 items; 13 of the items are unique to the SESBI-R. Behaviors are rated on two scales, a Yes-No Problem scale, which identifies problematic behaviors, and a 7-point intensity scale regarding the frequency of the child’s behavior.	Reliability: internal consistency ranged from .98 to .95. Test-retest reliability ranged between the .80s and .70s. Validity: Discriminant validity and convergent validity was established.	Takes into account the perspective of both parents and teachers in the assessment of conduct problems among children. Can be considered as initial screening devices.	Does not adequately take into account the changes in conduct-related behaviors among children between the ages of 2 and 16, due to development. Lacks age-specific norms, limiting the utility of these scales. Scarce validation studies in support of this scale exist.
Gadow & Sprafkin (2002) Child Symptom Inventory-4	Measure screens for 13 major childhood psychiatric disorders in the DSM-IV using a 4-point scale. Each item is rated: never, sometimes, often, and very often.	Reliability: Test-retest reliability ranged between .70 to .87. Validity: Convergent validity was established through multiple studies and criterion validity was established in one study.	User-friendly scale, which is easy to interpret and administer. Closely linked to the DSM making it suitable for practical and research use.	Scale's validity depends on the DSM-IV.
Gowers et al (1999). Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA)	Comprised of 15 scales of which the first 13 are used to compute the total score. The additional two items relate to lack of knowledge within the family about the nature of the child’s disorder and of information about the services. The scale is scored on a 0–4 point rating ranging from ‘no problems’ to ‘severe problems’.	Reliability: Inter-rater reliability was established for 20 cases in one study. Validity: Face validity existed and satisfactory sensitivity to change was demonstrated in one study.	Found to be feasible in clinical settings and was acceptable for use by a range of clinicians.	Requires significant amount of training in order to be used effectively as an outcome measure (Bebbington et al., 1999). Also, for pre-school aged children, certain subscales were found to be unsuitable.
Hodges, Wong & Latessa (1998) The Child & Adolescent Functional Assessment Scale (CAFAS)	Measures the effect of emotional, behavioral, or psychiatric problems and the degree of impact on the functioning of youth. Consists of 315 items, with 11 subscales. Severity measured by four levels: Severe, Moderate, mild, minimal or no impairment.	Reliability: Internal consistency ranged between .73 to .78. A score of .78 was found for test-retest reliability and a score of .92 was found for inter-rater reliability. Validity: Acceptable criterion validity existed.	Attempts to reduce rater bias are in place, by requiring raters to provide justification through behavioral descriptions of the youth.	Burden for clinicians because required to provide justifications as part of the rating process, causing them to see this measure as added paperwork
Kronenberger, Carter & Thomas (1997) Pediatric Inpatient Behavior Scale	47-item scale completed by nurses. Behaviors observed during hospitalization are rated on a 0–3 frequency level. Consists of 10 Subscales: Positive-Sociable, Distress, Elimination Problem, Withdrawal, Oppositional-Noncompliant, Conduct Problem, Anxiety, Self-Harm, Self-Stimulation, & Overactive.	Reliability: Strong inter-rater reliability, with correlations greater than 0.70, was found for all subscales except for withdrawal. For 8 of the 10 subscales, strong internal consistency was found. Validity: Strong construct validity was established among a sample of child inpatients.	Good psychometric properties for 8 out of 10 subscales. Designed specifically for children with primary childhood diagnoses.	Due to the total number of items and numerous subscales, may require extensive time to complete. Also, 2 of the 10 subscales have poor psychometric properties.
LeBuffe, & Naglieri, (2003). Devereux Early Childhood Assessment-Clinical Form.	Assesses the behavior of children between the age of 2 to 5 years old, with a focus on emotional, and social resilience, including other concerns relating to behavior. Comprised of 62 items with 7 subscales. Each item consists of a 5 point frequency rating ranging from never to very frequently.	Reliability: Internal reliability for the two total scores, Protective Factors and Behavioral Concerns, ranged from .88 to .94. Validity: Since the items are formulated based on the DECA, the DSMD, and the DSM-IV, the scales content validity is said to be demonstrated.	This scale is useful in not only assessing behavioral concerns but also is capable of assessing both risk and resiliency factors of the child which is beneficial in developing individual treatment plans and outcomes research.	This instrument is suitable only for children who range in age between 2 to 5 years old, and hence not equipped to assess children who are above 5 years of age.
Lyons (1998) Severity and Acuity of Psychiatric Illness—Child and Adolescent Version (CAPI)	Comprised of 20 anchored ratings, ranging from 0 to 3, with 0 representing an extremely healthy pole and 3 representing an extremely unhealthy pole (Lyons, McCulloch & Romansky, 2006; Lyons, Terry, Martinovich, Peterson, & Bouska, 2001). Four domains: Symptoms, High Risk Behaviors, Functioning, and System Support (Lyons et al., 2001).	Reliability: Reliability for all subscales was .70 or higher. Inter-rater reliability also existed with the average reliability being .76 (Kappa) (Lyons et al., 2001). Validity: The CAPI had strongly correlated with the CAFAS and the Child Behavior Checklist (Lyons et al., 2001).	Valid and reliable (Lyons et al., 2006). Brief and takes relatively little time to complete, of approximately 5 to 10 minutes.	Limited in that it does not measure educational attainment, healthy development or functional skills (Lyons et al., 2006).
Shaffer et al. (1983) Children’s Global Assessment Scale (CGAS)	Psychiatric Severity is measured. It is used among children ranging in age from 4 to 16 years old. It ranges from 1 to 100, where for each decile, descriptions of behavioral functioning across various life situations are provided.	Reliability: Three studies established test-retest reliability. Several past studies also established inter-rater reliability for this scale (Schorre & Vandvik, 2004). Validity: One study examined concurrent validity and it was supported.	CGAS is the most studied scale. Easy to administer and can be used on normal populations as well. Also utilized to measure psychosocial functioning in somatic patients (Schorre & Vandvik., 2004).	For children under 4 years old, psychometric properties were not established (Schorre & Vandvik., 2004). The scale relies exclusively on clinical report, thus the possibility of bias when completing the scale exists (Gold et al., 2009).
Williams & Bloomer (1978–1987) Bay Area Functional Performance Evaluation	Consists of two components which are the Social Interaction Scale (SIS) and the Task-Oriented Assessment (TOA). Seven aspects of social behavior are rated through the SIS based on observations in five different social settings. Performance, affective and cognitive functioning is assessed by the TOA through the administration of five tasks.	Reliability: Inter-item reliability ranged from .73 to .89. Test-retest reliability was said to exist, but psychometric information was not provided. Validity: Concurrent validity was established between the CGAS and the Functional Life scale. Initial construct validity was also established in some studies of the scale (Hemphill-Pearson., 2007).	BaFPE can be quickly administered and scored. BaFPE can be utilized to complement other life skills assessments and support clinical observations. It can be used to enhance patients’ self esteem due to its ability to be used as a therapeutic medium (Hemphill-Pearson., 2007).	TOA may be measuring constructs not intended by the authors. Also, TOA constructs were not clearly described or formulated.

The Children’s Global Assessment Scale (CGAS) relies on an observational process and standard clinical interview to assess overall functioning of children inpatient treatment (Cornsweet Barber, 1990; Shaffer, Gould, Brasic & Ambrosini, 1983; Sourander & Leijala, 2002). Achenbach’s Child Behavior Checklists (CBCL/4–18 and CBCL/2–3) (Achenbach, 1991; 1992) consist of a total of 113 items comprised of eight syndrome scales that measure social problems, thought problems, attention problems, aggressive behavior, withdrawn, delinquent behavior and somatic complains; three subscales examine social competence as well as internalizing and externalizing behavior problems. Although this measure has good psychometric properties and is widely used in child mental health research (e.g., Blader, 2004; Brinkmeyer, Eyberg, Nguyen, & Adams, 2004; Parmelee et al., 1995), the number of items make it challenging to implement in acute inpatient settings. A few outcome studies have also utilized the Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) (Gowers et al., 1999), a 15 item scale to assess changes in functioning over the course of psychiatric hospitalization (Harnett et al., 2005; Swadi & Bobier, 2005).

The Devereux Early Childhood Assessment-Clinical Form (DECA) is a recent instrument which examines behavior and emotional resilience using a 62-item, 5-point rating scale, specifically among preschool children. However, this measure is primarily an assessment tool to be used in conjunction with other outcome measures. It also examines the frequency of behaviors over a 4-week period making it difficult to implement in acute settings (LeBuffe & Naglieri, 2003). The Children’s Depression Inventory, Reynolds Child Depression Scale, and Weinberg Depression Scale for Children and Adolescents (Kovacs, 1997; Reynolds, 1989; Weinberg, Harper & Emslie, 1987) are examples of scales that focus on particular dysfunctions, in this case depression among children. However, these scales are not versatile in capturing outcomes among children who display other symptoms.

Unlike instruments used in past studies, the Target Symptom Rating (TSR) Scale is a relatively new measure, which consists of an emotional problems subscale and a behavioral problems subscale. It was developed specifically for use in acute psychiatric inpatient settings (Cornsweet Barber et al., 2002). The scale was used as a measure in a follow-up study of 1,723 patients in residential, day treatment and an inpatient setting. Scores on both subscales significantly decreased, suggesting adequate sensitivity of the measure, making it a promising outcome instrument. However, this scale has only been utilized in the retrospective study described, and the study protocol was plagued by high staff turnover and inability to conduct structured diagnostic interviews.

Research Questions

This analysis addresses two research questions: (1) Do children hospitalized on an inpatient psychiatric unit show improvement in behavioral and emotional functioning between admission and discharge as measured by the TSR Scale? (2) Can change in behavioral and emotional functioning be predicted by diagnosis, previous hospitalization and length of stay?

Methods

Setting and Target Population

The pilot project was conducted at a large private nonprofit psychiatric facility, located in the Southwest of the United States, providing a full range of mental health services across all age groups, including inpatient and intensive outpatient treatment. This 89-bed facility is one of two remaining comprehensive psychiatric facilities in the largest county in the United States, and the only one in a smaller region in the county offering care to children below 13 years of age. The facility has 29 beds for its child and adolescent inpatient programs. The majority of the patients hold private insurance; approximately 20% are admitted through public insurance. More than 50% of the patient population is White, followed by Hispanic, African-American and Asian. Services for children include intensive outpatient programs, partial hospitalization, and for more severe cases, inpatient hospitalization. Staff include child psychiatrists, pediatricians, rotating residents and medical student interns, social workers, clinical counselors, behavioral health specialists, and a school teacher. Typical length of stay is approximately one week. Upon being admitted to the inpatient unit, child patients are immersed in a structured treatment milieu where they participate in psychiatric assessments, occupational therapy, psychoeducation groups, academic classes, and family discharge planning meetings.

The Behavior and Symptom Identification Scale (BASIS 32) is currently utilized to examine outcomes among adolescent inpatients (ages 13 –17), but as a self-report scale it is not appropriate for use with younger patients. As such, the need for a valid and user-friendly measure suitable for the child unit existed. In 2008, the facility piloted the TSR-Scale after a careful review of the existing outcome literature. During the 5-month pilot phase, the TSR was supposed to be filled out for all child patients admitted to the inpatient program as part of standard intake and discharge procedures. The current study presents a secondary analysis of the data and was given exempt status by the Institutional Review Board. In accordance with HIPAA regulations, any identifying information regarding a patient was removed and was not used for analysis.

Data Collection Procedures

Data on the TSR-Scale was collected by clinical staff at admission and discharge. Clinical staff included three Masters in Social Work therapists, two clinical counselors, and one Marriage and Family Therapist. To obtain post-discharge data, the school teacher was also involved. Staff was trained to complete the TSR-Scale using a training manual developed for this project. The initial biopsychosocial assessment was usually conducted within 48 hours following admission into the hospital, during which time the TSR-Scale was expected to be filled out. The scale was supposed to be completed again at the time of discharge by the discharging clinician. Responses to the TSR scale were based on assessment information obtained from parents or caregivers and observation of the patient. Complete pre-test and post-test data were obtained for 36 children, ranging in age from 6 years old to 13 years old. Among the 36 children examined, 13 were female and 21 were male. In regards to their racial background, 24 identified as being Caucasian, seven identified as being Hispanic, and three identified as being African American. Demographic data was missing for two of the 36 children reviewed.

Measures and Variables

The Target Symptom Rating (TSR) Scale was used to measure behavior changes in patients between admission and discharge. The scale takes approximately ten minutes to complete and is scored as follows: The 13 items are divided into two subscales, Emotional Problems and Behavior Problems. The Emotional Problems Subscale is comprised of five scale items, which are depression, anxiety, psychosomatic, suicidality and psychotic symptoms items. The Behavioral Problems Subscale consists of eight scale items, which are family conflict, peer relationship, school difficulties, self destructive, aggression, substance abuse, runaway/out of control/legal problems and impulsivity. Each item is rated along a 5-point scale from 1 (no signs or symptom) to 5 (severe symptoms). Scores for these subscales are derived by taking the mean of the involved items, creating a continuous variable. In a survey of 22 clinicians, using a 5-point rating scale, ranging from 1 (very poor) to 5 (excellent), content validity was established on four dimensions: Relevance to Inpatient Settings, Relevance to Outpatient Settings, Relevance to Residential Settings, and Clarity of Anchor Definitions. For Relevance to Inpatient Settings, the mean of all item ratings was 4.27, and for Clarity, the mean was 4.08. Moderate inter-rater reliability and significant correlations, ranging from .51 to .73, existed between the TSR Emotional Problems and Behavioral Problems Subscales, and the Child Behavioral Checklist’s (CBCL) and YSR’s Externalizing and Internalizing Subscales (Cornsweet Barber et al, 2002). As part of the pilot project, staff also recorded patients’ diagnosis, previous psychiatric hospitalizations and the lengths of stay of the current episode.

Clinical diagnosis was determined using the DSM IV-TR, which uses a multi-axial system of diagnosis (American Psychiatric Association, 2000). The child patient’s primary Axis I diagnosis at discharge was used in the analysis. Discharge diagnosis was utilized because unit staff identified it to be the most accurate diagnosis, in comparison to the diagnosis provided on admission, which represents an initial ‘working’ diagnosis. Since patients presented with a range of different diagnoses, a variable was created that collapsed all diagnoses into two broad categories. One category captured those patients who had been diagnosed with an internalizing disorder only, which included all mood and anxiety disorders. A second category captured patients who either had externalizing disorders only (Attention Deficit Disorder, Intermittent Explosive Disorder, Oppositional Defiant Disorder) or who had been diagnosed with both externalizing problems as well as other disorders, with 0 = internalizing disorders only; and 1 = externalizing disorders or externalizing disorders plus other disorders.

Prior psychiatric hospitalization was operationalized as any previous inpatient psychiatric hospitalizations that occurred for the patient, prior to their admission at the described inpatient program. This variable was dichotomous (1=yes; 0=no).

Length of stay in this study refers to the number of days spent overnight at the child inpatient program from the time of admission to time of discharge.

Analysis

The statistical software program SPSS 15.0 was used for management and analysis of data. All data were entered into a SPSS database and were screened for missing data and outliers. Frequencies for all individual TSR items were run. Two dependent variables were computed and used for all analyses: the Emotional Problems Subscale score and the Behavioral Problems Subscale score. Frequencies were generated for both. The next step involved running descriptive statistics for the three predictor variables.

To calculate whether change occurred between pre-test and post-test, we used a Paired-Samples t test to compare the pre-test and post-test means for each of the TSR scale items, as well as for both the emotional and behavioral subscales. Linear regression analysis was used to examine if outcomes on the emotional and behavioral TSR subscales could be predicted by each predictor variable, length of stay, diagnosis, and re-hospitalization, independently. This technique examines the nature of the relationship between the dependent variable by regressing one or several independent variables onto the dependent variable (Tabachnick & Fidell, 2001). Diagnostic screening confirmed that assumptions of normality, linearity, homoscedasticity of residuals, and independence of errors between variables were met. Subsequently, each predictor variable was regressed on each of the two TSR subscales.

Multiple regression analyses were then conducted to assess if together, the three predictor variables were associated with change in the TSR subscales. Two separate regression analyses were conducted. The first multiple regression analysis examined the relationship of the three predictor variables on the change score of the Behavioral Problem Subscale. The second multiple regression analysis examined the relationship of the three predictor variables on the change score of the Emotional Problem Subscale.

Results

Data collection had occurred at two time points – at admission and discharge. During the 5-month pilot phase, a mean of 53.4 child patients were admitted to the unit per month (sd = 3.29). Despite the original goal of collecting data on at least 82 children to provide sufficient power for multivariate analysis, admission or discharge data were collected for 56 patients. However, results presented here are restricted to the matched sets of pre-test and post-test data for 36 patients. For the remaining 20 patients, either pre-tests or post-tests were obtained, but not both. Beyond problems in the implementation of the protocol (see Discussion), it is important to note that the mean lapse of time between patient admission and the time the form was completed at baseline was 2.38 days (sd = 1.62). The mean lapse in time between patient discharge and the time the form was completed upon discharge was 4.75 days (sd = 7.23).

With regard to the predictor variables, results revealed that 14 children had internalizing disorders only, while 22 had any externalizing disorders. Among the 36 patients examined, 17 had previous hospitalizations, while 19 patients had no prior hospitalizations. The mean length of stay during this episode at the facility was 6.25 days (sd = 2.38).

Changes in Outcome

The results of the Paired Sample t test for each of the thirteen TSR scale items and their respective subscales are presented in Table 2. Scores reflect the average score on a 5-point Likert scale with 1 = No significant problem on a particular item, 2 = minor problems posed, 3 = moderate problem, 4 = serious problem and 5 = severe problems. For each of the two TSR subscales, statistically significant decreases in emotional and behavioral problems between pre-test and post-test were found. The mean score on the Emotional Problem Subscale decreased from 2.60 (sd = .49) at pre-test to 1.50 (sd = .31) at post-test (p<.001; t =11.56, df = 34). This was also the case for the behavioral subscale where the mean pre-test score decreased from 3.22 (sd = .62) to 2.31 (sd = .55) (p<.001; t = 9.02, df = 34). Statistically significant improvements between pre-test and post-test were found across all individual items with the exception of the substance abuse item. Changes ranged from a score reduction of 0.03 to 1.72 with an average reduction of almost one point (mean=0.98; sd =0.47).

Table 2.

TSR Items - Pre-Posttest Results and Change Scores

TSR-Items	Pre-Test	Post-Test	Change Score	t
Emotional Subscale	2.61(0.49)	1.50 (0.31)	1.10 (0.56)	11.56***
Depression	3.56 (0.73)	2.00 (0.59)	1.56 (0.88)	6.74***
Anxiety	2.56 (0.81)	1.56 (0.61)	1.00 (1.01)	5.92***
Psychosomatic	2.47 (0.77)	1.50 (0.65)	0.97 (0.94)	6.20***
Suicidality	2.64 (1.07)	1.25 (0.44)	1.39 (1.08)	7.74***
Psychotic Symptoms	1.71 (0.89)	1.26 (0.61)	0.46 (1.01)	2.68*
Behavioral Subscale	3.22 (0.62)	2.31 (0.55)	0.91 (0.58)	9.02***
Family Conflict	3.92 (0.87)	2.79 (0.69)	1.12 (0.95)	6.89***
Peer Relationships	3.50 (0.86)	2.70 (0.84)	0.80 (0.69)	6.74***
School Difficulties	3.39 (0.93)	2.94 (0.83)	0.44 (0.88)	3.04**
Self-Destruction	3.22 (0.87)	2.00 (1.04)	1.22 (0.96)	7.64***
Aggression	3.69 (1.26)	1.97 (0.84)	1.72 (1.19)	8.71***
Substance Abuse	1.33 (0.67)	1.30 (0.67)	0.03 (0.45)	0.37
Runaway	3.14 (1.03)	1.97 (0.79)	1.17 (1.12)	6.17***
Impulsivity	3.53 (0.94)	2.61 (0.77)	0.92 (1.05)	5.22***

^*p<.05

^**p<.01

^***p<.001

Bivariate Findings

Linear regression was employed to examine the relationship between each individual predictor variable - diagnosis on Axis I, previous hospitalization, length of stay, and each of the TSR subscales. Results are presented in Table 3. None of the predictor variables were significantly related to the Emotional Problems Subscale change score. With regard to change in the behavioral domain, previous hospitalization was found to be a significant predictor of Behavioral Problems Subscale change scores (β = −.42, p = .02), accounting for 17% of the variance. Findings indicate that children with previous hospitalizations had on average, a change score that was .42 lower compared to the change score of children with no previous hospitalizations. Neither diagnosis nor lengths of stay were statistically significant.

Table 3.

Bivariate Linear Regression Analysis Predicting Outcomes on the TSR Emotional and Behavioral Subscales

Predictor Variable	β	SE	p
Emo Change Scores (CONSTANT)
Length of Stay	.06	.06	.74
Diagnosis on Axis	−.21	.19	.24
Previous Hospitalization	.02	.19	.93
Beh Change Scores (CONSTANT)
Length of Stay	.04	.05	.84
Diagnosis on Axis I	.12	.21	.52
Previous Hospitalization	−.42	.19	.02*

^*p<.05

^**p<.01

^***p<.001

Multivariate Findings

Table 4 presents results of both multiple regression models. Mirroring bivariate findings, the three predictor variables failed to predict the Emotional Problem Subscale change score, and accounted for only 6% of the variance. Previous hospitalization, however, was a significant predictor of behavioral changes after accounting for the effects of diagnosis and length of stay (β = −.47, p = .01), with children with prior hospitalizations showing less change. Altogether, length of stay, diagnosis on Axis I, and previous hospitalizations accounted for 21.7% of the variance in Behavioral Problems Subscale scores.

Table 4.

Multivariate Regression Analysis Predicting Outcomes on TSR Emotional Subscale and Behavioral Subscales

Predictor Variable	β	SE	p
Emo Change Scores (CONSTANT)
Length of Stay	.66	1.11	.51
Diagnosis on Axis I	−1.34	1.10	.19
Previous Hospitalization	.20	1.05	.84
Beh Change Scores (CONSTANT)
Length of Stay	.10	.04	.57
Diagnosis on Axis I	.17	.20	.32
Previous Hospitalization	−.47	.19	.01**

^*p<.05

^**p<.01

^***p<.001

Discussion

In a secondary analysis, we examined pre-test/post-test changes in functioning for a small sample of children in one inpatient psychiatric setting. Mirroring current trends in inpatient psychiatric care, the average length of stay was six days for this sample. Results of the study showed that acute psychiatric inpatient hospitalization for children produced favorable outcomes as measured by the TSR scale in most areas of functioning. At pre-test, problems ranged from a minor to a serious problem level. At post-test, children had experienced a ‘one-unit’ improvement on average. This means that a child with an initial report of ‘major problems’ in one area presented with ‘minor problems’ at post-test. These improvements are consistent with the findings of previous studies (e.g., Harnett et al., 2005; Mayes et al., 2001; Sourander & Leijala, 2002). Problems decreased at a statistically significant level in 12 of the 13 domains captured by the TSR scale. The greatest degree of change occurred in the area of depression, aggression, suicidality, self-destruction, family conflict, peer relationships, impulsivity, psychosomatic symptoms, and anxiety. School difficulties and psychotic symptoms were found to decrease to a lesser degree in comparison to other scale items. This is probably due to the fact that progress in academics could not be accurately assessed, as clinical staff’s exposure to a child’s academic performance over the course of their hospitalization is limited to the few hours spent in a classroom with the inpatient program’s contract teacher, lesser time when compared hours spent in a community-based school. Significant changes in this area within a short period of time are also relatively unlikely. The minor reduction in psychotic symptoms is likely due to the fact that only a small portion of the child patients studied, presented with symptoms of psychosis. The only area in which improvements in outcome did not occur at a statistically significant level was substance use. Findings showed that substance use was rated to occur at a very minor level among this young sample. Furthermore, if substance use problems are present, visible changes are not likely to occur within the context of a brief inpatient stay.

Significant improvements were also reported for the two subscales (Emotional and Behavioral Problems subscales). These findings were expected since the goal of psychiatric hospitalization is emotional and behavioral stabilization. Children are not expected to be discharged unless significant improvements in their emotional state and behavioral patterns are made. Thus, these findings provide some evidence that positive change does occur as a result of psychiatric hospitalization between admission and discharge. Whether these changes were sustained upon discharge is unknown and would require follow-up data collection.

Findings may also carry clinical significance. Clinical significance, in a strict sense, means that participants who were once considered as “troubled” or “disordered,” experience reliable change after treatment, and are no longer distinguishable from a meaningful and representative non-disturbed reference group as well at the level of the individual participants (Becker, 1997). However, rather than moving to a range that is considered normal, it is argued that clinically significant improvements are considered to occur even when finer gradations on the outcome scale are made, for instance, such as a shift from severe to moderate (Becker, 1997). This latter argument suggests that this study’s average improvement of one scale point (i.e., from serious to moderate; from moderate to mild) within an acute psychiatric setting might indeed be clinically meaningful.

This study also tested the effect of three predictor variables - clinical diagnosis, length of stay and previous hospitalization – on outcome. Results showed no significant relationship between length of stay and either subscale of the TSR scale. Previous studies have been equivocal about the relationship between length of stay and outcome, with some studies showing that longer stays lead to better outcomes (Harnett et al., 2005; Sheerin, Maguire & Robinson,, 1999) while a number of studies did not report such an association (Healy & Fitzgerald, 2000; Sourander & Piha, 1998). Contradictory to previous findings, however (e.g., Gavidia-Payne et al., 2003; Harnett et al., 2005; Rice et al., 2002), a significant relationship was not found between diagnosis on Axis I and outcome. Our classification into two diagnostic categories may have been too gross. However, a more detailed and perhaps accurate classification schema was not possible due to small numbers in each category. Severity of clinical functioning using GAF scores and its relationship with each subscale was examined, and found not to be significant either. Besides small sample size, the finding could also be related to the fact that this study used the discharge diagnosis as it was considered to be most accurate. Future analysis with a larger sample size may allow for a more fine-grained examination of clinical diagnosis and outcome.

A majority of the children in this sample had experienced previous hospitalizations. However, mixed results were obtained regarding previous hospitalization. The moderate, negative relationship between rehospitalization and the Behavioral Problem Subscale score indicates that children with previous hospitalizations show improvement in behavioral functioning to a lesser degree than children with no prior hospitalizations. Prior hospitalizations are sometimes interpreted as an indicator or a proxy for greater behavioral severity (James et al., 2010). The relationship between previous hospitalization and behavioral functioning persisted in the multivariate context even after controlling for the effects of clinical diagnosis and length of stay. Not surprisingly, rehospitalization accounted for the largest portion of the variance. Virtually negligible relationships existed between Emotional Problems Subscale scores and all three predictor variables. Interestingly, previous hospitalization was not predictive of changes in emotional functioning. This may be due to the fact that internalizing symptoms tend to be more difficult to assess.

Strengths and Limitations

Although the sample size was small, the significant changes between pre-test and post-test are promising and suggest that inpatient hospitalization may contribute to symptom reduction. Furthermore, this study is among the few studies, which examined the outcomes of acute psychiatric hospitalization solely for children and excluded adolescents from the sample. Also, this project was conducted in a setting with short stays, reflecting current trends in acute inpatient psychiatric care.

However, the study had a number of limitations, which need to be considered when interpreting the findings. First, the analysis is based on a small sample. Second, as many other studies before, this study did not include a comparison group, thus limiting the internal validity of the study. A major difficulty in conducting outcome research in inpatient psychiatric settings is the identification of an equivalent comparison group, such as children and adolescents with severe emotional and behavioral disorders who do not receive inpatient treatment. This is largely due to ethical issues. However, pre-test/post-test studies are considered an important first step in the development of research (Burns & Hoagwood, 2002). Third, the sample was limited to children in one particular psychiatric setting. This limits the generalizability of the findings. At best, findings can be generalized to similar inpatient psychiatric settings. Fourth, inter-rater reliability between clinicians was not assessed; thus it is unknown if the clinical judgment of different clinicians was consistent. Finally, difficulties in implementing the research protocol with fidelity lead to a cautious interpretation of the findings. Intervention fidelity has been identified as an important factor in explaining and maintaining improved outcomes (Moncher & Prinz, 1991). This study had put into place several mechanisms to ensure fidelity. Multiple trainings on the use of the TSR scale were conducted to ensure understanding of the TSR scale and its administration as well as to accommodate the needs of all clinicians who participated in the data collection process. The study also used a training manual in all training sessions to ensure standardization of training. However, no measures were used to assess consistency in training. Though attempts were made to standardize data collection procedures, the fast-paced nature and unpredictability of the unit presented a challenge.

Challenges in Implementing the TSR Scale

Special attention was paid to the barriers that existed to implementing an outcome measure in an acute inpatient setting. There is a growing body of literature that discusses barriers to the implementation of more outcome-focused and evidence-based practices in mental health care (e.g., Garland, Kruse & Aarons, 2003). Numerous barriers were encountered during the implementation process of this study’s research protocol that might have implications for future efforts to implement standardized assessment procedures in clinical settings. The first barrier involved establishing a valid baseline finding. Ideally, the TSR scale should have been completed immediately upon a child’s admission. However, from a practical standpoint, this was not always feasible. A limited number of available clinicians, and the fact that clinicians at the psychiatric facility are given a 48-hour period within which to complete psychosocial assessments, required that the TSR scale would be completed as part of the psychosocial assessment based on clinical impressions obtained from the assessment. Cornsweet Barber et al. (2002) recommended that parents, patients and clinical staff should provide information on the assessment. While this would be ideal, engaging parents in short-term clinical settings can be challenging. In many instances, children admitted to the inpatient unit were not escorted by their parents, or may have come from a group home or residential treatment facility. As such, reliable caretaker information is not always available.

Another barrier was the lack of consistency in who completed the scale at both time points. The same clinician who completed the scale upon admission was frequently unable to complete the scale upon discharge. Again, this was due to the quick turnover rates of child patients and varying shift schedules, which do not coincide with admission or discharge dates of patients.

Staff perceptions also played a role. Although clinicians, on a conceptual level expressed an appreciation for the TSR scale as being able to capture salient information regarding a patient, and potentially contributing to the completion of thorough assessments, from a practical standpoint it was viewed as time-consuming and burdensome. One clinician was primarily responsible for all the child patients admitted to the unit, and thus was mainly responsible for completing the TSR scale both upon admission and discharge. Psychosocial assessments typically ranged between 60 to 90 minutes and due to the fast-paced nature of the unit, clinicians described having limited one-to-one contact with patients, restricting their capacity to rely solely on their own observations. Clinicians also questioned the reliability of using a standardized measure, since items, such as runaway behavior and substance abuse, seemed to have less relevance for this population and setting. This suggests that some items may need to be dropped. Overall, some dissatisfaction was reported by clinicians regarding the requirement of implementing standardized outcome measures by licensing and accrediting bodies. Since this study was conducted as part of routine clinical practice, factors such as staff changes due to leave of absences and vacation time, which had occurred during the course of this study, further impacted the study’s ability to deliver the research protocol with fidelity.

Given the factors outlined above, the data collection stage proved difficult. It was challenging to obtain matched sets of data. Post-tests were particularly difficult to obtain, since this was completed at the time of discharge. When the clinician primarily responsible for discharging child patients was unavailable, the burden of completing the TSR fell upon other clinicians or nursing staff who did not have the opportunity to observe the progress of child patients, and lacked the requisite training in completing the TSR scale. With the help of the clinical supervisor this issue was resolved with the identification and training of a contract teacher who had the opportunity to closely observe patients through classroom interaction. Once the teacher was identified, the number of matched sets obtained and the rate at which they were completed was positively impacted.

In summation, numerous logistical details needed to be addressed throughout the course of the study. Strict adherence to the proposed protocol was not possible due to the realities of day-to-day acute psychiatric care. Slight deviations were required in order for the TSR scale to be implemented.

Implications for Research and Clinical Practice

Outcome evaluation has become a critical aspect of mental health treatment, and an essential component of evidence-based practice (Koch, Lewis & McCall, 1998). Major accreditation organizations, like the Joint Commission on Accreditation of Healthcare Organizations (JCAHO), incorporate outcome evaluations as part of their requirement (Cesare-Murphy, McMahill & Schyve, 1997). However, the process of effectively implementing standardized measures as part of clinical practice remains a challenge, as was evident throughout this pilot project. In a qualitative study by Garland, Kruse, and Aarons, 2003, 50 mental health providers in San Diego County who participated in state mandated assessments were interviewed and participated in focus groups in order to obtain information regarding the clinical utility of outcome measurements. The findings revealed that a large majority of clinicians were ambivalent about standardized outcome assessments, and expressed similar concerns as those expressed by this inpatient program’s clinicians (Garland et al., 2003). Nevertheless, implementing standardized measures to rate level of functioning and severity at admission and discharge was found to be feasible across a range of inpatient and outpatient facilities for mental health care for children (Gold et al., 2009).

In order to address some of the challenges posed, future research should be conducted in the area of implementation to understand how the utility of a standardized outcome measure can be increased. These studies should closely examine the barriers encountered, clinicians’ attitudes towards the proposed outcome measurement, and the clinicians’ utilization and understanding of data once it is obtained.

The challenges encountered in this study also suggest the need for further education and training on the importance of outcome evaluation to clinical practice. Staff should be educated on the importance of tracking patient progress in order to guide the goals of treatment and direct interventions. Outcome evaluations also provide insight for clinicians regarding what techniques are effective and those which are not.

The TSR scale was useful instrument to capture behavioral and emotional changes of children admitted to an acute psychiatric inpatient setting in this study, and continues to be utilized at the psychiatric facility described, as part of routine clinical practice. Its relative comprehensiveness make it a promising instrument for acute psychiatric settings. However, future research is warranted to further assess the psychometric properties of this scale as well as its versatility across a range of informants. Furthermore, the inclusion of comparison groups is needed, in order to establish more conclusive results regarding the effectiveness of acute psychiatric inpatient care among children.