The Development and Application of the RAND Program Classification Tool

Short abstract

The RAND Program Classification Tool is an instrument containing questions and response options across eight core domains that allows managers of a portfolio of programs to quickly, easily, and consistently describe and compare their programs.

Abstract

As a result of extended military engagements in Iraq and Afghanistan during the past decade, the U.S. Department of Defense (DoD) has implemented numerous programs to support servicemembers and family members who experience difficulty handling stress, face mental health challenges, or are affected by a traumatic brain injury (TBI). As these efforts have proliferated, it has become more challenging to monitor these programs and to avoid duplication. To support DoD in this area, RAND compiled a comprehensive catalog of DoD-funded programs that address psychological health and TBI. In creating the catalog of programs, RAND recognized the need to consistently describe and compare multiple programs according to a set of core program characteristics, driven largely by the lack of a single, clear, widely accepted operational definition of what constitutes a program. To do this, RAND developed the RAND Program Classification Tool (R-PCT) to allow users to describe and compare programs, particularly those related to psychological health and TBI, along eight key dimensions that that define a program. The tool consists of a set of questions and responses for consistently describing various aspects of programs, along with detailed guidance regarding how to select the appropriate responses. The purpose of this article is to describe the R-PCT, to help potential users understand how it was developed, and to explain how the tool can be used.

Despite the recent drawdown of troops in Iraq, the increased pace of deployments over the past decade, longer deployments, and frequent redeployments have resulted in significant mental health problems among service members. Among those who had been deployed to Iraq and Afghanistan as of October 2007, approximately one-fifth reported symptoms consistent with current posttraumatic stress disorder (PTSD) or major depression, and about the same number reported having experienced a probable TBI while deployed (Tanielian and Jaycox, 2008). In the wake of the 2007 report of the DoD Task Force on Mental Health (Department of Defense Task Force on Mental Health, 2007), DoD has implemented numerous programs, interventions, and policies to address the increased prevalence of these conditions and their effects on service members and their families.

As these efforts have proliferated, it has become more challenging for DoD to monitor these programs and to avoid potential duplication of effort. To support DoD efforts in this area, RAND compiled a comprehensive catalog of relevant programs and created a taxonomy for them (see Weinick et al., 2011). In developing the catalog of DoD programs related to psychological health and TBI, RAND encountered a fundamental problem: the lack of a single, clear, universally accepted definition of the term program.

To help in creating the catalog, RAND developed a conceptual framework that distinguishes programs from other types of services that may be used by service members and their families. While this framework distinguishes programs from routine services and systems of care, it does not provide operational guidance about how to bound the set of activities that constitute a program or how to identify variation among programs. The lack of an operational definition of a program creates some practical challenges for individuals who manage portfolios of programs.

To address this gap, we developed the RAND Program Classification Tool (R-PCT) to allow users to understand and compare programs, particularly those related to psychological health and TBI, along key dimensions. The tool consists of a set of questions and responses for consistently describing various aspects of programs, along with detailed guidance regarding how to select the appropriate responses. This articles describes how the R-PCT was developed and explains how the tool can be used.

The R-PCT is one tool created as part of a larger effort to catalog and evaluate programs related to psychological health and TBI. Information on this and other tools is available at the “Innovative Practices for Psychological Health and Traumatic Brain Injury” web page.*

How the R-PCT Was Developed

We first conducted a literature review to identify how the term program is defined across various fields of study and to identify defining characteristics of programs. After this review, we consulted a variety of experts, including those based at DCoE, regarding which characteristics are of greatest importance in the context of military mental health, and then used this feedback to help focus the review findings into eight essential program characteristics:

• Program goals

• Program barriers

• Evaluation experience

• Evaluation readiness

• Participant interaction

• Scale

• Scope

• Transferability

We then developed a specific set of questions corresponding to these key characteristics to enable descriptions of DoD-funded psychological health and TBI programs.

Once the characteristics and questions were finalized, we developed corresponding response options that would target responses to the questions for each of the specified characteristics and make scoring simple and consistent when the R-PCT was applied by different individuals and to different programs. To develop both relevant questions and response options, we examined the existing literature in the field of program evaluation. We subsequently engaged RAND researchers in a pilot test of the R-PCT to assess how well the characteristics, questions, and response options captured the variability in programs as well as their usefulness for comparing multiple programs.

Characteristics and Questions Included in the R-PCT

The R-PCT includes eight characteristics for describing and characterizing programs. When applied to a portfolio of programs, the R-PCT could be used to describe and compare a wide variety of programs.

Suggested Uses of the R-PCT

Individuals who manage portfolios of programs (e.g., those who work for government agencies, foundations, and intermediary and grant-making organizations) could use the R-PCT to describe and compare the programs they manage. For example, examining the frequency of responses to the question on program barriers could be a way to identify common barriers faced by multiple programs. R-PCT information about program evaluations (e.g., whether a program has conducted an outcome evaluation in the past year) can provide useful data about the extent to which programs within a portfolio are being evaluated. If used more than once during a program's implementation, the R-PCT would allow users to describe changes in program characteristics over time. If a program portfolio manager is trying to compare programs to identify the best candidates for scaling up, he may want to use R-PCT data on the transferability and scale of current programs. Finally, if a program manager is trying to decide whether a new program should be developed or funded, she may want to use the R-PCT to help identify whether the goals of the new program fill a gap in their portfolio.

Conclusions

The R-PCT is an instrument containing questions and response options across eight core domains that allows managers of a portfolio of programs to quickly, easily, and consistently describe and compare their programs. The program characteristics included in the R-PCT are integral to understanding the goals and objectives of programs and how they function, and offer a set of characteristics along which programs may be delineated. Having consistent metrics is crucial for enabling comparisons of the characteristics of multiple programs in the same content area, and is a first step to developing a more robust operational definition of a program.

The tool (a short set of questions and response options) is not meant to replace more formal program evaluation efforts but to provide a simple, user-friendly way to systematically aggregate data across multiple programs so that individuals managing multiple programs can quickly and easily describe and compare the programs in their portfolio. This information can also inform decisions about what types of technical assistance (e.g., help designing or implementing an outcome evaluation) are needed and for which programs.

We recommend that users continue to adapt and expand the R-PCT, with the ultimate goal of using the R-PCT to classify programs into a typology, which would allow users to better target technical assistance to specific types of programs. In addition to enhancing the R-PCT itself, we also recommend that the R-PCT continue to be tailored for use across a variety of program types. Although developed for use in understanding psychological health and TBI programs, the R-PCT is not specific to a single content area and can be used, with modification, to describe characteristics of programs across a variety of subject areas.