Assessing Program Needs and Planning Change

Abstract

Assessments of treatment staff training needs, preferences, and barriers can help guide and improve training activities and transfer of evidence-based technologies into clinical practice. The TCU Program Training Needs (PTN) assessment consists of 54 items organized into seven domains: Program Facilities and Climate, Program Computer Resources, Staff Training Needs, Preferences for Training Content, Preferences for Training Strategy, Training Barriers, and Satisfaction with Training. Data collected from 589 counselors representing 194 treatment programs showed the PTN was psychometrically sound and predictably associated with results from a more comprehensive assessment of organizational functioning. Importantly, fewer barriers to training and greater staff satisfaction with training were reported in programs with higher levels of organizational functioning. In addition to representing an efficient source of staff perceptions about organizational operations and needs, the PTN empowers staff with a “voice” they can contribute to strategic planning and priority setting for organizational actions.

1. Introduction

The role of needs assessment in the design of training programs has long been recognized in the organizational literature (Goldstein, 1991; Goldstein & Ford, 2002; McGehee & Thayer, 1961; Peterson, 1998; Wexley & Latham, 1981). This appraisal affects nearly all phases of the training process including determining specific training needs of individuals in the organization, selecting the most appropriate training content and delivery methods, and evaluating the effectiveness of the training procedures. In addition, it can play an important role in assessing the organizational context regarding resources, management support, and other organizational climate factors that either hinder or facilitate the successful transfer of a training initiative. As pointed out by Amodeo, Ellis, and Samet (2006), this is an important first step to help organizations gauge their own “developmental” state and determine whether they need to alter any internal structure or procedural deficiencies.

Backer (1993) suggests the first condition necessary for effective technology transfer is that appropriate innovations must be brought to the attention of the organization and made accessible for dissemination. Staff assessments of program training needs and preferences are useful in identifying topics and techniques that offer the greatest appeal in training settings. Training plans should be developed using a strategic sequence that builds on staff inputs concerning needs, readiness, and capacity. This represents a strategic planning process and systematic assessment with regard to organizational needs, facilities, resources, preferences, and barriers. Use of this approach has been supported in the prevention literature where the selection of evidence-based prevention strategies tailored to community profiles of risk and protection have increased the likelihood that planned interventions can be fully implemented (Hawkins, Catalano & Arthur, 2002; Arthur & Blitz, 2000).

This paper describes a strategic planning assessment tool for health and social service agencies. As reflected in the TCU conceptual models for treatment process and program change (Simpson, 2002, 2004; Simpson & Flynn, this issue), such an assessment is an important step in studying the process of technology transfer where evidence-based treatment interventions are incorporated into every-day counseling practices.

1.1. Organizational characteristics and technology transfer

Organizational attributes of substance abuse treatment programs are important elements in the adoption and implementation process for treatment innovations (Simpson, 2002, 2004; Simpson & Dansereau, in press). Some real-world examples of efforts to transfer innovative treatments demonstrate the types of challenges faced with regard to adopting new medications (Roman & Johnson, 2002; Thomas, Wallack, Lee, McCarty, & Swift, 2003), comprehensive services for adolescents (Liddle et al., 2002), and cognitive-based counseling tools (Dansereau & Dees, 2002). There is increasing evidence that organizational factors (e.g., stress, communication, financial pressures) are more important in transferring research to practice than the methods used to distribute the materials (Backer, David, & Soucy, 1995; Simpson, 2002). Thus, in order to transfer new technology more effectively (i.e., drug treatment interventions), it may be important to first determine an organization’s readiness and capacity for implementing innovations.

Simpson and Flynn (this issue) describe the process of program change involved when new technologies or knowledge are introduced. The key action steps include training, adoption, and implementation based on planning and preparation. The systematic assessment of needs and staff readiness for innovations as part of strategic planning should precede these steps. The adoption stage represents an explicit intention to try an innovation. While this might be a “formal decision” made by program leadership, it also includes subtle levels of commitments made by individual staff members at a more personal level about whether an innovation is appropriate and should be tried. In this stage, assessment of the resources and climate can be helpful in addressing barriers. Implementation comes next, implying that there is a period of trial usage of the new innovation to allow testing of its feasibility and potential. Addressing potential barriers continues to be important in incorporating an innovation into regular use and sustaining the practice. However, these stages of change are impacted by institutional and personal readiness (e.g., motivation and resources), and organizational dynamics, including climate for change and staff attributes. Therefore, organizations with insufficient traits are less likely to attain successful technology transfer. Selection of appropriate scales, reliability and validity of their measurement, choosing individuals to properly represent the organization, and methodological alternatives for aggregating data are issues that require careful attention (Hermann & Provost, 2003).

The TCU Organizational Readiness for Change (ORC) assessment was developed to accommodate these applications and focuses on organizational traits that predict program change (Lehman, Greener, & Simpson, 2002). It includes 18 scales covering four major domains – needs and pressures, resources, staff attributes, and climate. Needs and pressures (motivation for treatment) includes program needs, training needs, and pressures for change, while program resources are evaluated in regard to office facilities, staffing, training, equipment, and Internet. Organizational dynamics include scales on staff attributes (growth, efficacy, influence, adaptability, and clinical orientation) and program climate (mission, cohesion, autonomy, communication, stress, and flexibility for change). The ORC scales are useful indicators of the measured constructs regarding global strengths and weaknesses and are useful for identifying potential trouble spots for program consideration.

1.2. Program Training Needs (PTN)

The TCU Program Training Needs (PTN) assessment was developed as a complement to the ORC scales to help identify and prioritize treatment issues that program staff believe deserve attention at a more detailed level than addressed in the ORC. Additionally, it identifies training content and preferences, barriers to training, and satisfaction with previous training, because these would be useful to informing issues of technology transfer.

The PTN provides a basis for identifying treatment innovations that are priorities for program staff. It evolved from a series of surveys conducted to assess staff perceptions about needs for improvements and how to achieve them. The information collected became progressively more structured and useful for engaging staff in the planning and action process. As discussed by Backer, Liberman, and Kuehnel (1986), counselors deserve more attention as advocates and potential barriers to adoption and maintenance of treatment interventions because they carry the front-line responsibility of interpreting and converting research findings into practice. The TCU Program Change Model (Simpson & Flynn, this issue), illustrates how staff within treatment organizations are initially linked to the model. In particular, the strategic planning function involves surveying staff needs and functioning, and providing an integrative review of these elements. The PTN content domains focused on Program Facilities and Climate, Program Computer Resources, Staff Training Needs, Preferences for Training Content, Preferences for Training Strategy, Training Barriers, and Satisfaction with Training. Collectively, this type of information is intended to help guide overall training efforts as well as predict the types of innovations that participating programs are most likely to seek out and adopt.

1.3. Research questions

There were three goals of this paper. The first was to confirm the factor structure and establish basic psychometric properties of the seven PTN scales while focusing on internal structure (reliability and dimensionality) and exploring relationships among the scales. A second goal was to determine similarities and differences in how clinical supervisors and staff characterize their training issues. Clinical supervisors and staff have different levels of responsibility in organizational functioning and thus bring different perspectives. Differences (and similarities) in how supervisors and staff view their training requirements are an important factor as they address agreement among staff and management, as well as their views of program needs. Finally, program training needs as measured by the PTN need to be explored in relation to more global organizational processes. More specifically, relationships between the PTN and the ORC, the companion assessment of organizational functioning described above (Lehman et al., 2002), were examined.

Using the TCU Program Change Model as a framework, it was expected that the more specific constructs measured in the PTN would be related to the more global measures in the ORC. For instance, the brief PTN scales focused on program facilities and climate and computer resources should be positively related to the ORC resources scale. The PTN staff training needs scale should be positively related to the broader ORC needs and pressures scales. Additionally, The ORC climate scale should be positively related to PTN satisfaction with training and negatively related to training barriers.

2. Method

2.1. Procedures for data collection

Data for this study were obtained in collaboration with treatment programs located in 2 states within the Gulf Coast Addiction Technology Transfer Center (GCATTC) region. Data collection occurred over a period of 2 years and also was coordinated with the state drug and alcohol agencies in the member states to accommodate their training initiatives. Many participating agencies had several treatment units – that is, different treatment sites under the oversight of the same parent organization. Each treatment unit has its own unique organizational attributes and climate, so these units were asked to voluntarily administer a package of forms to be completed by their local clinical supervisors and staff. Methods and procedures for collecting these forms were carried out in accordance with protocols approved by the Institutional Review Board at Texas Christian University.

2.1.1. Data collection from State A

In February 2004, 330 PTN assessment forms were sent to 59 treatment units and a total of 192 forms were returned (59%). Participation by programs/units in the assessment was voluntary and a passive consent procedure was used in which completing the assessment and returning it to TCU in the attached self-addressed and stamped envelope implied consent to participate. Using the same data collection procedure, the ORC assessment had been administered 4 months earlier (in October 2003), with 330 forms mailed to the same treatment units and 174 ORC assessments returned (52%). This return rate for the ORC assessment forms was slightly lower than the 56% to 64% rates for employees surveyed by mail as generally reported in the organizational literature (Schneider, Parkington, & Buxton, 1980; Schneider, White, & Paul, 1998).

2.1.2. Data collection from State B

In the second state, the data for the PTN assessment was collected during November and December 2003 by GCATTC staff (working collaboratively with the TCU research staff) using online services provided by the PsychData Corporation. A total of 397 PTN assessments was completed, representing 135 separate programs. In an earlier wave of data collection (June 2003), the ORC data was also collected via the Internet with a total of 405 forms collected from a larger sample of approximately 173 treatment units. It was not possible to calculate participation (response) rates for these PTN and ORC assessment forms which were completed online.

2.2. Participants

The primary sample of 589 participants consisted of PTN assessments that were collected from 194 treatment units from both states. This sample included 192 PTNs (33%) from State A (59 treatment units) and 397 PTNs (67%) from State B (135 treatment units). A total of 579 ORCs was collected from staff representing 231 treatment programs. There were 174 ORCs (30%) collected from State A and 405 ORCs (70%) from State B.

Participating staff included 63% who were Caucasian, 24% African American, and 14% Hispanic. Overall 69% were female. Just under half (45%) had at least 5 years of experience in drug treatment counseling, and 21% had been on their present job for at least 5 years (with about 43% in their present position for less than a year).

Individual staff members were matched with their respective treatment units when data were analyzed at the treatment unit level. This resulted in an average of approximately 2.7 PTN forms per treatment unit (range 1-13) and 2.4 ORC forms (range 1-8) per treatment unit. No minimum number of respondents was required for aggregation of treatment unit scores because this allowed all the data available to be used for establishing treatment unit scores based on a simple additive composition model (Chan, 1998).

Clinical supervisors completed 81 (14%) of the PTN assessments collected and the treatment counseling staff completed 502 (86%). The final analysis sample used in this study was limited to programs that had linked-data available for both PTN and ORC. This requirement reduced the samples to about one-third, and resulted in 195 PTN assessments and 174 ORC assessments from 97 matched treatment units.

2.3. Assessments

2.3.1. Program Training Needs (PTN) assessment

The PTN includes 54 items organized into seven scale domains, including Program Facilities and Climate, Program Computer Resources, Staff Training Needs, Preferences for Training Content, Preferences for Training Strategy, Training Barriers, and Satisfaction with Training. Responses to each item were scored as follows: “strongly disagree” = 10; “disagree” = 20; “undecided” = 30; “agree” = 40; and “strongly agree” = 50. Mean scores for each scale range from 10 to 50, with scores above 30 indicating overall agreement (and increasingly higher scores indicating stronger agreement). Table 1 lists the individual items contained in each domain and a mean summary score and standard deviation for each item. The Program Computer Resources domain was not developed as a traditional scale, but instead was intended to provide brief checklists of critical resource categories that could be summed together as an index. This is also the case for the Preferences for Training Content and Staff Training Needs scales.

Table 1.

Staff Survey Responses to PTN Items (N = 589)¹

	Summary Scores Mean (SD)	CFA Loadings
Program Facilities and Climate (Alpha = .79)	34.9 (7.23)
1. Offices, equipment, and supplies are adequate at your program.	34.1 (11.9)	.500
2. Your program has enough counselors and staff to meet current client needs.	29.3 (12.5)	.561
3. Your program has adequate resources for meeting most medical and psychiatric client needs.	30.1 (12.2)	.479
4. Most program staff feel positive and confident about the quality of services at your program.	37.8 (9.4)	.751
5. Your program has a secure future ahead.	37.6 (9.2)	.593
6. Program staff here get along very well.	39.5 (9.3)	.595
7. Program staff morale is very good.	36.1 (10.6)	.772
Program Computer Resources (Alpha = .44)	33.0 (5.9)
1. Most client records for this program are computerized.	37.3 (10.7)	.333
2. Program staff here feel comfortable using computers.	36.2 (9.4)	.517
3. More computer resources are needed here.	35.8 (10.8)	.332
4. Program staff here have easy access for using e-mail and the Internet at work.	38.2 (10.3)	.476
5. This program has policies that limit program staff access to the Internet and use of e-mail.	31.1 (11.9)	.219
Staff Training Needs (Alpha = .92)	35.0 (7.84)
1. assessing client problems and needs.	32.8 (10.7)	.766
2. increasing client participation in treatment.	36.2 (9.9)	.744
3. monitoring client progress.	33.7 (10.5)	.806
4. improving rapport with clients	32.1 (11.2)	.753
5. improving client thinking skills.	36.7 (9.5)	.855
6. improving client problem-solving skills.	37.0 (9.5)	.864
7. improving behavioral management of clients.	36.7 (9.8)	.785
8. improving cognitive focus of clients during group counseling.	37.0 (9.4)	.795
9. using computerized client assessments.	32.9 (10.9)	.508
10. working with staff in other units/agencies.	34.4 (9.9)	.533
Preferences for Training Content (Alpha =.79)	37.7 (5.91)
1. You want more scientific information on the neurobiology of addiction.	38.9 (8.1)	.354
2. More pharmacotherapy information and training are needed on new medications.	40.5 (7.7)	.434
3. Program staff need sensitivity training for dealing with special populations.	36.6 (10.4)	.630
4. Program staff training is needed on ethics and confidentiality of information.	34.0 (10.9)	.574
5. Specialized training is needed for improving family involvement and related issues.	39.1 (8.5)	.694
6. Program staff training is needed on dual diagnoses and appropriate treatment.	40.3 (8.7)	.712
7. Training to use brief diagnostic screening tools would be helpful to program staff.	38.1 (8.9)	.627
8. Program staff need to be trained to understand other staff functions.	34.5 (10.4)	.607
Preferences for Training Strategy (Alpha = .71)	37.0 (4.4)
1. General introductory sessions on multiple topics is an effective workshop format.	32.9 (10.0)	.275
2. Intensive full-day training on special topics is an effective workshop format.	38.3 (8.5)	.430
3. A conceptual treatment process model documenting how treatment activities contribute to “recovery” would be helpful.	38.6 (7.2)	.551
4. Training workshops should be based on evidence-based interventions.	38.4 (7.1)	.532
5. Training workshops should be based on manual-guided interventions.	31.8 (8.5)	.399
6. Training workshops should include role playing and group activities.	37.0 (9.7)	.424
7. Telephone consultations following specialized Training would be useful.	34.4 (9.1)	.506
8. Specialized training made available over the Internet would be useful.	37.9 (9.1)	.347
9. Exchanging ideas with other programs that have Interests similar to yours would be helpful.	41.2 (6.4)	.593
10. On-site consultation following training would be helpful.	39.1 (7.3)	.602
Training Barriers (Alpha = .83)	30.4 (6.86)
1. The workload and pressures at this program keep motivation for new training low.	32.8 (11.6)	.643
2. The budget does not allow most program staff to attend professional conferences annually.	36.3 (11.9)	.578
3. Topics presented at recent training workshops and conferences have been too limited.	30.5 (10.3)	.588
4. The quality of trainers at recent workshops and conferences has been poor.	24.5 (9.3)	.458
5. Training activities take too much time away from Delivery of program services.	26.8 (10.1)	.448
6. Training interests of program staff are mostly due to licensure or certification requirements.	32.2 (10.8)	.457
7. It is often too difficult to adapt things learned at workshops so they will work in this program.	27.5 (10.4)	.615
8. Limited resources (e.g., office space or budget) make it difficult to adopt new treatment ideas.	33.3 (11.5)	.606
9. The background and training of program staff limits the kind of treatment changes possible here.	28.2 (11.1)	.645
10. There are too few rewards for trying to change treatment or other procedures here.	31.6 (11.7)	.721
Satisfaction with Training (Alpha = .75)	35.5 (7.74)
1. Good in-house (inservice) training is provided to program staff.	36.6 (10.8)	.597
2. You found good outside training events to attend last year.	35.6 (10.6)	.591
3. Your state-funded drug or alcohol agency provided good training in the past year.	36.5 (9.9)	.810
4. Regional authorities or groups (e.g., ATTC, ACA) provided good training in the past year.	33.3 (9.8)	.685

Seven areas are assessed by the PTN.

• Program Facilities and Climate (7 items) refers to the adequacy of overall office and physical space available. Inadequacy of these resources reduces the ability of staff to incorporate new treatment approaches and is likely to be related to an overall lack of financial resources. This scale also assesses climate issues such as staff morale.

• Program Computer Resources (5 items) deals with adequacy and use of computer equipment in general. This scale also contains items related to use and access of Internet and e-mail.

• Staff Training Needs (10 items) focuses on the types of skills the counselors think might help them do their jobs better. These include behavioral and cognitive interventions in addition to assessment and monitoring techniques.

• Preferences for Training Content (8 items) assess perceptions of need for specific types of training in several areas. This section may be modified to reflect specific training areas of interest to the specific population. Areas of interest relevant to training in the present study included dual diagnosis, special populations, diagnostic screening tools, and the neurobiology of addiction.

• Preferences for Training Strategy (10 items) focuses on the types of training the staff prefer to attend. This includes logistical aspects of the training experience such as intensive full day training or on-site consultation, and use of a conceptual recovery model framework. Other areas include questions about training workshop components such as the use of role-playing.

• Training Barriers (10 items) deals with staff ideas about barriers to training. This includes staff perceptions about lack of budget for attending training, difficulties in adapting ideas from workshops, or workload pressures that might interfere with attendance at training events and implementation activities.

• Satisfaction with Training (4 items) concerns how satisfied the staff report being with the sources, quality, and types of training they have been offered in the last year.

2.3.2. Organizational Readiness for Change (ORC) assessment

The rationale, scale descriptions, and psychometric properties of the ORC were reported by Lehman et al. (2002). The ORC includes 115 items representing 18 scales covering four major areas: Needs/Pressures, Resources, Staff Attributes, and Organizational Climate. Items use 5-point response categories (disagree strongly, disagree, uncertain, agree, agree strongly), and scale scores are calculated by reflecting items that need to be reverse scored and computing the mean and multiplying by 10. Thus, 30 represents a neutral score and scores over 30 indicate stronger levels of agreement (similarly, scores below 30 indicate stronger levels of disagreement). Scale scores were calculated only if the respondent completed at least half of the items in a scale.

ORC composite scores were computed as a summary index for each of the four domains using all of the respective domain scales. The Needs and Pressures Index (NPI) (motivation for treatment) consisted of the mean of the scores on the three needs and pressures scale (training needs, program needs, and pressures for change). The Organizational Climate Index (OCI) was a composite measure of organizational climate and is the mean of the scores on the six scales (mission, staff cohesiveness, staff autonomy, communication, stress, and openness to change) measuring climate in the ORC (the stress scale was reverse scored). The Institutional Resources Index (IRI) consisted of the mean of scores of the five institutional resources scales (offices, staffing, training resources, equipment, and Internet). Finally, the Staff Attributes Index (SAI) consisted of the mean of the scores on the four staff attribute scales in the ORC (growth, efficacy, influence, and adaptability). Greener, Joe, Simpson, Rowan-Szal, and Lehman (this issue), used an expanded sample to calculate reliability scores for these ORC composite measures. The motivation composite index representing program needs and pressures for change (NPI) had an alpha of .69, the organizational climate index (OCI) had an alpha of .88, the institutional resources index (IRI) had an alpha of .71, and the staff attributes index (SAI) had an alpha of .70.

2.4. Analysis

2.4.1. Reliability, dimensionality, and correlations among PTN scales

Confirmatory factor analysis using Proc CALIS (SAS, 2003) – assessed the fit of the 54 items to the hypothesized scales. Coefficient alpha reliabilities were computed for each scale and the relationships among the seven PTN scale composites were examined using Pearson’s correlations.

2.4.2. Comparisons between clinical supervisor and staff responses

Clinical supervisor and staff responses on the PTN were compared by using ANOVA to examine mean score differences between the two groups (calculated at the treatment unit level). That is, staff responses within a treatment unit were averaged and compared to the clinical supervisor’s responses from the same treatment unit.

2.4.3. PTN correlations with ORC domain indexes

Relationships of PTN scales with organizational functioning scales (as measured by the ORC) were examined based on analyses conducted at the treatment unit level. For the organizational functioning analyses, the four ORC domain indices (described above) served as overall summaries that measured distinct aspects of organizational functioning (see Courtney, Joe, Rowan-Szal, & Simpson; Greener, Joe, Simpson, Rowan-Szal, & Lehman, this issue). The relationships between PTN and ORC index domains were examined using Pearson’s correlations.

3. Results

3.1. Psychometric properties (based on confirmatory factor analysis, internal reliability, and correlations among PTN scales)

The 54 PTN items were subjected to a seven-factor confirmatory factor analysis (CFA) using the CALIS software program (SAS 9.1, 2003). The RMSEA estimate for the model was .06 and the Root Mean Square Residual (RMR) was .065; these fit indices suggest that the seven factors defined a priori is an appropriate solution in describing the PTN, although not a “good fit.” As noted by Browne and Cudeck (1993), a RMSEA of .05 or less indicate a close fit in relation to the degrees of freedom, and values up to .08 indicate the fit as having reasonable errors of approximation in the population. The work by Hu and Bentler (1999) indicate that a RMSEA of about .06 and a SRMR of close to .08 are among the criteria that need to be met for a relatively good fit between the hypothesized model and the observed data. As shown in Table 1, loadings from the CFA suggest that all of the items in the Program Facilities and Climate, Satisfaction with Training, Staff Training Needs, and Training Barriers scales would be marker variables for the scales, as all are above .40. Understandably, some of the items for the Preferences for Training Content scale (1 of 5 items), Preferences for Training Strategy (3 of 10 items), and Program Computer Resources (3 of 5 items) did not meet this cutoff. This result accounts for why the fit was not better. There is some disagreement in the literature, however, as to whether the minimum factor loading in exploratory factor analysis should be .30 or .40 before becoming sufficiently trivial as not to be expected to generalize to other samples (Guion, 1998, p. 263; Hatcher, 1994, p. 89). It is notable that in the CFA results of this study, even the lowest loading of .219 still had a significant t value of 4.14.

Additionally, there were substantial correlations among the seven confirmatory factors. That is, Program Facilities and Climate was positively correlated with Satisfaction with Training (.47) and Program Computer Resources (.61), and negatively correlated with Training Barriers to (-.69). Likewise, Training Barriers was also negatively correlated with Satisfaction with Training (-.54) and Program Computer Resources (-.51). Other substantial correlations involved Preferences for Training Strategy with Preferences for Training Content (.61) and Staff Training Needs (.54). Preferences for Training Content and Staff Training Needs were correlated .65.

Reliability of each of the PTN scales was computed using Cronbach’s alpha. Results are presented in Table 1 for 575 staff. Most of the scales had reliabilities of .70 or higher (except computer resources) ranging from .71 to .92.

Table 2 reports correlations among the PTN scale composites, based on a total of 194 program treatment units. Results showed that Program Facilities and Climate had its highest positive correlation with Satisfaction with Training (.38) and a strong negative correlation with Training Barriers (-.56). Staff Training Needs had high positive correlations with both Preferences for Training Content (.56) and Preferences for Training Strategy (.56). Preferences for Training Content and Preferences for Training Strategy were highly related (.47), while Training Barriers and Satisfaction with Training were negatively related (-.42). These are in accord with the estimated correlations among the factors from the confirmatory factor analyses.

Table 2.

Correlations among Program Training Need scales (N = 195 treatment units)

	Program Facilities and Climate	Program Computer Resources	Staff Training Needs	Preferences for Training Content	Preferences for Training Strategy	Training Barriers	Satisfaction with Training
Program Facilities and Climate	_ _	.28***	-.26***	-.26***	-.22***	-.56***	.38***
Program Computer Resources		_ _	-.24***	n.s.	n.s.	-.28***	n.s.
Staff Training Needs			_ _	.56***	.56***	.27***	-.23***
Preferences for Training Content				_ _	.47***	.28***	n.s.
Preferences for Training Strategy					_ _	.26***	-.14***
Training Barriers						_ _	-.42***
Satisfaction with Training							_ _

*p < .05, **p < .01, ***p < .001

3.2. Clinical supervisor and staff agreement

Analysis of clinical supervisors and staff from respective treatment units revealed significant differences in only one domain. Comparisons between the perceptions of these two groups showed clinical supervisors rated Satisfaction with Training events significantly higher (38.1) than did staff (34.8; F (1,51) = 5.22, p < .05). Inspection of individual items within this scale revealed clinical supervisors gave higher ratings on two items (i.e., “You found good outside training events to attend last year” and “Your state-funded drug or alcohol agency provided good training in the last year”) than staff. There were no significant group differences on any other PTN scales.

3.3. Relationship of PTN to ORC domain indices

Based on the establishment of favorable psychometric properties for the PTN scales, cross relations of the PTN and ORC assessments were addressed. Organizational functioning as measured by the ORC is an important indicator of how successful treatment agencies are likely to be in implementing new treatment and training innovations. The four ORC domain indices were used, with aggregated staff scores computed for each treatment unit. These were then correlated with the seven PTN scales based on the 97 treatment units that had both PTN and ORC scores.

Table 3 indicates the ORC Needs and Pressures Index (NPI) was positively related to two PTN scales – i.e., Staff Training Needs (.37) and Preferences for Training Content (.39). This suggests that agencies wherein staff reported higher training needs and higher preferences for specific training content also received high needs and pressures (reflecting motivation) for organizational change. As expected, results also show that treatment units with higher PTN scores for program facilities and climate, also have higher ORC indices for climate and institutional resources (representing more comprehensive assessments). More importantly, treatment units characterized as higher in resources and climate reported higher satisfaction with training opportunities and fewer barriers to overcome.

Table 3.

Correlations of Program Training Need scales with ORC Indexes (N = 97 treatment units)

	Needs and Pressures (NPI)	Climate Index (OCI)	Institutional Resources (IRI)	Staff Attributes (SAI)
Program Facilities and Climate	n.s.	.35***	.45***	.22*
Program Computer Resources	n.s.	n.s.	n.s.	n.s.
Staff Training Needs	.37***	n.s.	n.s.	n.s.
Preferences for Training Content	.39***	-.19*	-.28**	n.s.
Preferences for Training Strategy	n.s.	n.s.	n.s.	n.s.
Training Barriers	n.s.	-.41***	-.43***	n.s.
Satisfaction with Training	n.s.	.33***	.36***	.21*

^*p < .05

^**p < .01

^***p < .001

4. Discussion

The TCU Program Training Needs (PTN) assessment was developed to be a brief tool for helping programs identify staff training needs, adequacy of facilities and resources, and likely barriers to adopting innovations. In short, it has evolved as a strategic planning aid by addressing parts of the Simpson (2002) model of program change. Namely, staff perceptions about training needs, program operations, and barriers to innovation can help guide efforts to transfer evidence-based technologies into clinical practice. The PTN is an important companion assessment to the TCU Organizational Readiness for Change (ORC). While the ORC addresses important organizational functioning indicators more comprehensively, the PTN addresses training needs in specific areas.

The psychometric analyses indicate the PTN adequately fits the pre-conceptualized seven-factor structure, and its scales have acceptable reliabilities. The reliabilities computed at the program level showed that all scales (except program computer resources) had a coefficient alpha above .70. Regarding relationships among the PTN scales, one of the strongest associations was between Staff Training Needs and Preferences for Training Content which reflects complimentary purposes describing training needs from the client and counselor points of view. The Staff Training Needs scale focuses on the types of skills that counselors think can help them reach their particular client population more effectively. Conversely, the Preferences for Training Content scale assesses staff perceptions of needs for specific types of training, especially for gaining better insights into client needs and problems. It should be pointed out, however, that these scales were not designed as permanent or rigid measures. Instead, they should allow flexibility so that as new training interventions are introduced to the substance abuse field they may be incorporated into the PTN assessment format.

Another interesting relationship with the PTN was the strong negative correlation of Program Facilities and Climate with Training Barriers, indicating that programs with more resources generally face fewer barriers to training. This is consistent with several studies reporting that institutional resources are important for technology transfer (Backer, 1988). Lehman et al. (2002) likewise reported that adequacy of agency resources and climate indicators were positively correlated with budget growth and organizational stability. That is, agency resources and climate play an integral role in supporting training efforts, and as expected, budget limitations can be a major barrier to training efforts. The positive relationship between the Program Facilities and Climate scale and Satisfaction with Training scale also indicates that programs with higher resources are more satisfied with the training they received. Finally, Training Barriers was negatively correlated with Satisfaction with Training. This implies that programs reporting more barriers were less satisfied with training because they actually received fewer training opportunities during the past year.

The comparisons between training perceptions of clinical supervisors and staff revealed that these groups tended to agree on most training issues. Although both groups have important and differing perspectives of the organization, they tend to agree on critical new training areas to help improve clinical work with their clientele. Findings from Courtney et al. (this issue), however, suggest that uniformity (or concordance) in staff viewpoints also is an important consideration. The one area of disagreement involved Satisfaction with Training, and in this case, the clinical supervisors were more satisfied (compared to staff) with the training they had received in the last year. This was not altogether surprising because clinical supervisors were likely involved in the selection of training opportunities and might have actually received more training opportunities than general staff.

The utility of the brief PTN assessment was further supported by comparisons with the more comprehensive ORC assessment of organizational functioning. First, the most significant relationships involved PTN Program Facilities and Climate, Training Barriers, and Satisfaction with Training with ORC Institutional Resources (IRI) and Organizational Climate (OCI) indices. Treatment units with higher staff ratings of institutional resources and climate scores on the ORC also reported better facilities, fewer barriers to training, and greater satisfaction with training. These patterns are consistent with research indicating the important role institutional resources exert in overall organizational functioning (Backer, 1988; Lehman et al., 2002). Adequate institutional resources are a fundamental part of inviting and activating change in organizations because without resources an agency often is unable to secure appropriate training. This also is consistent with the finding that programs with higher ORC staff attributes scores had higher PTN ratings for Program Facilities and Climate as well as Satisfaction with Training.

The Needs and Pressures index (NPI) from the ORC was positively related to both the Staff Training Needs and the Preferences for Training Content scales from the PTN. This is consistent with the motivation for change literature showing that unless ample motivation to change exists, innovative changes are unlikely to occur (Backer, 1995, 2000; Backer et al., 1995). Agencies that report higher motivation for change (NPI index) also were found to have staff who reported more training needs, which may reflect motivation for training. Finally, the Preferences for Training Content scale was negatively related to both the OCI and IRI indices, indicating that programs with less resources and lower climate scores have staff perceiving higher needs for training in a variety of areas. Examination of how the training scales are related to the ORC scales can further help in determining the overall functioning of an organization and in turn the potential success in implementing new interventions.

4.1. Concluding comments

A structured instrument for assessing program training needs has been developed to better understand and address staff needs. It represents a strategic tool for guiding program planning. This paper establishes that the seven domains included in the PTN have acceptable psychometric properties and that they have important relationships with other relevant measures of organizational functioning (from the ORC).

Additional research is needed to verify and extend these findings. In particular, the PTN needs testing in larger, more diverse samples. To date, it has been completed in a variety of drug treatment programs that were located in one geographical region. In addition, modifications and customized subscales of the PTN may be needed in settings or agencies that treat special populations. For example, a version appropriate for staff working with criminal justice treatment populations is currently being tested. Further research will be needed to test the generalizability of the constructs included in the PTN for use in these and other types of organizations.

Finally, assessments such as the PTN and ORC should not be viewed as static tools. Strategic readministrations over time (see Simpson, Joe, Rowan-Szal, this issue) can provide dynamic insights about change, both intentional and opportunistic. For instance, unplanned changes in budget allocations, staffing, leadership, and so forth can be monitored through periodic reassessments. More significant, however, is the value of repeated assessments and their potential use in strategic and intentional decisions and actions to effect change.