Evaluating and Training Substance Abuse Counselors: A Pilot Study Assessing Standardized Patients as Authentic Clients

Abstract

Clinician training and supervision are needed to transfer evidence-based practices to community based treatment organizations. Standardized patients (SPs) are used for clinician training and evaluating. However, to be effective for substance abuse counselors, SPs must realistically portray substance abuse treatment clients. The current study assessed authenticity of SPs as substance abuse treatment clients. Twenty-one substance abuse counselors interviewed SP(s) with differing profiles. Counselors provided quantitative and qualitative ratings of SP authenticity. Counselor responses to the study procedures were analyzed as well. Quantitative results include high authenticity ratings for the SPs but counselors’ subjective responses varied. Counselor’s rated the experience of participating in the study positively and provided constructive comments for future applications of this methodology. Results support future work on SPs as teaching and evaluation tools in substance abuse counseling. Findings also illustrate the need to refine definitions of authenticity for SPs as substance abuse clients.

INTRODUCTION

Although effective treatment for people with chemical dependency is available when programs incorporate evidence-based therapeutic practices, (1–12) recent research suggests that quality of care for substance abuse clients may be declining (13–15). The Institute of Medicine reported that evidence-based practices appear to be used infrequently by clinicians working in community drug or alcohol abuse treatment programs (16). The challenge, then, is to transfer efficacious technologies to community based treatment organizations (17, 18). Clinician training and supervision, while not sufficient (19), are necessary components of this process.

Accurate clinician assessment is of paramount importance considering recent requirements that addictions treatment programs implement evidence based practices including detection of co-occurring disorders (20). However, measuring the extent to which clinicians apply evidence based practice is difficult considering issues of confidentiality and self-report bias. Implementing relevant, cost-effective training becomes additionally difficult without accurate knowledge of what occurs in clinical interactions.

Measures of clinician behaviors (such as diagnostic and treatment decision-making) are often based on self-assessments, questionnaires, or chart notes which may have little to no relationship to counselor performance (21). In addition, difficulties can arise in obtaining tape recordings of sessions between clinicians and clients (22). For example, in a project designed to evaluate training in motivational interviewing (22), only three of 22 clinicians who had volunteered for the study provided complete audio tapes of sessions. Nonetheless, the Institute of Medicine recommended that “clinicians and organizations providing mental health and substance abuse services should use measures of the processes and outcomes of care to continuously improve the quality of the care they provide” (23). Standardized patient methodology could ameliorate problems associated with access to clinical interactions, measuring clinician behaviors and subsequent training. However, for this methodology to be useful, SPs must be able to perform convincingly for substance abuse counselors.

Standardized patients (SPs) also known as “simulated patients” (24) are actors trained to portray a prescribed patient role similarly across clinical encounters (25, 26). SPs are a methodology recognized and applied internationally as a valid interpersonal mechanism for training and evaluating clinical capabilities. SPs are given information about how to portray physical and emotional symptoms as well as “typical” responses to common questions. Utilizing SPs, each clinician sees exactly the same “patient.” This approach allows for comparison of clinician performances. Conversely, in real clinical contexts, patient-to-patient variation hinders comparison of providers with one another (27).

Research questions appropriate to respective clinical fields guide development of checklists that SPs complete following interaction with a practitioner. Typically, one SP assesses multiple clinicians on specific and consistent dimensions. During a clinical interview, the SP is aware that he/she will be completing the checklist, allowing a level of vigilance toward particular clinician behaviors that “real” clients may miss. Videotaped footage allows for a complete view of the clinician/client interaction, providing a data source for tracking relevant interpersonal constructs (e.g., motivational interviewing). Using real patients considerably restricts such direct access into otherwise confidential encounters.

At least 80 percent of medical schools in the United States utilize SPs for training and evaluation purposes (28, 29). SPs assist in evaluating and teaching dentists (30), nurses (31), physician assistants, pharmacists, and others (32). As of 1993, Canada implemented SPs in the Canadian National Medical Licensing Examination (25). Beginning with the class of 2005, the National Board of Medical Examiners (NBME) requires an SP exam as part of the United States Medical Licensing Examination for physician licensure in the United States. Medical students have rated SPs as more helpful than real patients or videotapes (33).

Standardized Patients and Behavioral Health

Standardized patients have been used to portray a variety of patients including people with mental health problems. For example, studies have found SP examinations appropriate for assessing psychiatric clinical skills in third year medical students (34–36) and psychiatry residents (37). SPs have been used to study primary care providers’ ability to detect major depressive disorder (38). In a study in which unannounced SPs were sent to the offices of volunteer primary care clinicians, investigators found SPs could effectively portray people suffering from mental health disorders such as depression (39, 40). Impressively, the majority of these covert SPs went undetected by providers.

A small but growing literature exists on the use of SPs to portray people with substance abuse. Substance abuse and mental health clients are often characterized by notable physical and emotional symptoms. As such, portraying substance abusing patients with mental health or social co-occurring problems poses more challenges than enacting other, less emotionally or physically demanding, roles. SPs have been used to portray alcohol as well as marijuana abusing individuals in order to evaluate clinicians on motivational interviewing (22, 21). Several authors have reported on medical student education programs that involve SPs trained to portray people with alcohol problems (33, 41–46). SPs portraying individuals with a history of alcohol and/or drug use have been used to assess nursing (47) and medical (48, 49) students’ ability to obtain substance abuse histories. Importantly, in a survey of community based clinicians, Saitz et al. (50) found that trainings for screening and brief intervention for substance abuse problems that included role play with standardized patients were associated with high percentages of reported impacts on practice, use of formal screening questionnaires, and assessments for readiness to change.

While these studies indicate that SPs have been effectively used to portray a variety of substance abuse related cases, roles that include complex emotionality and case histories have met with criticism (51). Additionally, the majority of studies did not report any measure of SP authenticity. Without assessments of SP case or portrayal authenticity, it is unclear which substance abuse client life stories, including the depiction of complex mental and physical symptoms, may be most feasible for SP replication and implementation with treatment practitioners.

Standardized Patients and Substance Abuse Counseling

Portraying patients with substance related problems for substance abuse counselors should be more challenging than acting out the same role with primary care clinicians. Unlike physician visits that average 16.3 – 20.4 minutes (52), assessment interviews with addictions counselors typically take up to two hours. Substance abuse client SP cases must contain enough information to withstand these lengthy assessment interviews. Client profiles for the current study, for example, included between twenty to thirty double-spaced pages of information ranging from basic demographics to client family members’ social histories. Additionally, SP information was not limited to facts and included substantial emphasis on affect. For example, client profiles included scripted verbal as well as emotional responses to commonly asked questions (e.g., the Drug Abuse Screening Test) and questions specific to the SP case (e.g., questions related to methamphetamine use).

In addition to issues of substance abuse client case complexity, counselors are more intimately attuned to the presentations of clients with substance related problems (e.g., physical symptoms of withdrawal) than are physicians. Substance abuse counseling interviews also generally address a wider variety of topics than do physicians (e.g., housing possibilities or child custody). Historically, training approaches to therapeutic interaction in addictions treatment also vary considerably compared to other clinical fields (53). This variation requires that SPs come equipped to any given interview with enough factual and affective material to support approaches from psychoanalytic to strictly cognitive behavioral. One SP portraying one complex case must be ready to remain consistent across interactions with counselors with backgrounds varying from volunteer faith-based outpatient work to Ph.D. level medical model work in a hospital. Finally, factors that differentiate one substance abuse counselor from another along with specific case content may inform therapeutic focus on differing aspects of the SP’s case. While one practitioner may focus and remain engaged on quantity and duration of substance use, another practitioner may focus on the relationship of the SP with family. These defining features of substance abuse clients and assessment interviews pose challenges for SP technology. Standardized patients must stay in role for hours, have a large amount of information to memorize and may need to demonstrate intense physical and emotional symptoms. Case complexity necessitates vigilance regarding case validity.

Standardized Patient Validity for Substance Abuse Practitioners

What constitutes a valid (i.e., authentic) SP case and portrayal? Answering this question poses methodological difficulties considering the absence of research on the topic. To the authors’ knowledge, validity of SPs portraying substance abuse clients has yet to be examined. Indeed, a literature review seeking an “authenticity” scale for SPs, in general, yielded only one result. A search for a scale examining the “realism” of SPs portraying substance abuse clients produced no results.

One dilemma in studying “SP authenticity” in the context of such a paucity of research on the topic is establishing a definition. For example, meeting quantitative thresholds associated with numeric indicators would provide consistent, systematic, and generalizable evidence of SP authenticity. These goals are achieved most commonly and accurately using pre-determined scales that limit raters’ responses, assuming a universally accepted set of dimensions constituting authenticity.

However, exclusive use of quantitative methods omits possibilities for descriptive accounts of subjective responses to SP performance. Due to the limited response options of many quantitative scales, SP developers, trainers and trainees may miss important perspectives that could contribute to maximizing SP potential, including understanding subtle critiques of particular cases. Those subtleties are particularly important given the complexity of real substance abuse client profiles. Additionally, defining authenticity of SP cases qualifies as a preliminary line of inquiry. Qualitative methodologies are particularly well-suited to preliminary exploration and discovery (54). However, just as relying solely on quantitative indicators may omit useful subjective contributions to a new line of inquiry, overemphasizing qualitative methods could risk overly complexified, non-generalizable, and fluctuating perspectives on developing a consensus view of what constitutes an authentic SP (55). Therefore a mixed method examination of authenticity should provide both a scaled quantitative assessment of SP case validity and qualitative answers to the questions “why and how” a case appears authentic or inauthentic.

A premise of this pilot study was that criticism of SPs portraying complex cases may be due to inadequate case development – the storyline – and inaccurate case portrayal or how it was depicted. In order to address those problems the authors developed, implemented and evaluated for authenticity four SP cases. This paper describes quantitative and qualitative data regarding SP authenticity on two of the four cases. The two cases included in this paper were evaluated by the largest number of substance abuse practitioners yielding the most data from which to assess authenticity. Details about the additional two cases are available from the authors.

METHODS

This section describes participants, procedures, and measures used during SP training and assessment interviews between two SPs and real substance abuse practitioners. The purpose of the interviews was for counselors to evaluate authenticity of the SPs on multiple dimensions. Brief descriptions of the two SP cases follow.

Case 1: “Debby Patterson”

Debby is a 20 year-old unemployed woman who along with her husband, Eric, was arrested one week ago at their home after the police responded to a domestic dispute. The police noted dilapidated conditions (no telephone or electricity) and a small amount of methamphetamine. Debby’s children were subsequently placed in protective custody. Debby knows she has a problem with methamphetamine and wants to quit but blames the addiction on her husband. Her co-occurring problem is intimate partner violence.

Case 2: “Mike Langsley”

Mike is a 36 year-old unemployed male who has been using heroin for approximately six months. He first started after he had a fight with his wife, Jen. Three days ago, Jen discovered that their savings account was almost completely depleted (Mike spent it on heroin) and then she found a syringe in his sock drawer. Jen confronted Mike with the syringe and the bank statement. Mike confessed to using heroin and Jen kicked Mike out of the house. Mike is adamant about entering detoxification immediately and is vaguely aware of possibilities for subsequent rehabilitation. Mike’s mental health co-morbidity is dysthymia.

Participants

The convenience sample of participating counselors consisted of 21 practicing clinicians (N = 15) or substance abuse counseling students (N = 5). One participant did not report on practicing versus student status. Participant recruiting was primarily by networking in the alcohol and drug counseling community in the Portland, Oregon metropolitan area, including contacting key personnel at two alcohol and drug treatment programs. Establishing contact occurred through either phone or email in order to arrange subsequent face to face meetings. Public discussion about the project, solicitation of feedback from persons working in the addictions treatment field, and further recruiting of participants occurred through attending the 2004 Northwest Institute of Addiction Studies (NIAS) conference. Participation was, in all cases, voluntary. Responses to items on a pre-interview questionnaire provided general demographic information and scores on items assessing participants’ beliefs about addiction (56), understanding of models explaining substance abuse (57), and views about treating dually diagnosed clients (58). Participant scores were compared to normative data using chi-squared and t-tests.

Participants were mostly female (74%), Caucasian (61%), average age 42 (minimum 24, maximum 63) with average income between $31,000 and $36,000. The majority of participants held a master’s degree (N = 13). In general, participants’ views about addiction and its treatment were similar to published scores for other samples of addictions practitioners. However, participants in this study tended to rate statements related to 12-step programs (e.g., “12-Step programs should be used more”) less favorably than normative study samples, statements related to treatment matching (e.g., “treatment matching should be used more often”), less favorably than norms and statements related to medication (e.g., “methadone maintenance should be used more”) more favorably than norms.

Thirty one assessment interviews were conducted with “Debby” and “Mike”. Eighteen counselors interviewed “Debby.” Thirteen counselors interviewed “Mike.” Ten counselors interviewed both SPs. Participating counselors were aware they were interviewing a standardized patient and that the purpose was to rate SP authenticity. Although SP cases were scripted, actors were coached to adjust responses and affect appropriately according to counselor prompts.

Personal acquaintanceship of counselors was beyond control of the investigators and may have been the case especially when counselors volunteered from the same treatment facility. However, effort was taken to avoid any counselor seeing or interacting with other counselors between SP sessions throughout the study. Counselors were also asked to avoid discussing their participation experience with other counselors in order to minimize participants’ prior knowledge or opinions about any SP they interviewed.

The standardized patients were professional actors recruited through flyers at a university acting department and via a third party referral. The candidate actors were screened for, among other qualities, ability to portray affect and physical dispositions consistent with presenting problems of substance abuse clients as well as any history of drug abuse. The SPs portraying Debby and Mike were paid for their time including time spent on training and homework as well as time spent in clinical interviews.

Procedures

SP Case Development

In order to develop adequate knowledge of what “typical” storylines occur for substance abuse clients, a literature review was conducted, and ideas were solicited from a series of experts including: drug and alcohol counselors, the heads of two addictions treatment centers, and psychiatrists. Particular attention was paid to learning about the variety of topics that were typically discussed, as well as how clients would usually discuss them (affect). Based on this feedback, each SP “script” included but was not limited to the following: social/medical/family/drug history, symptoms, a timeline, co-morbidities, work, insurance, and relationships with others. Specific SP comments were included as contingent on counselor triggers. Additionally, emotional responses were written into the scripts also contingent on particular topics broached by counselors. Care was taken to avoid including overly detailed SP responses to all counselor questions, as overly specific responses can appear rehearsed and do not always reflect reality. Therefore, some symptoms and issues on the timeline were described in detail, while others were purposely left vague.

SP Training and Case Refinement

SPs trained 15–25 hours. Much of this time was spent memorizing and/or being quizzed on the details of the case. Additionally, SPs practiced with a trainer and a psychiatrist on accurate portrayal of physical and psychological symptoms. Unsurprisingly, of all tasks practiced, SPs needed most corrections on delivering consistent emotional responses to counselor cues. SP performance evaluations were solicited from drug and alcohol counselors, physicians, and a focus group of people in recovery from addiction.

The focus group was held in order to receive feedback on the authenticity of the newly developed SP cases. The six participants (three females) who made up the focus group were all persons in long-term recovery from alcohol or drug abuse. Participants had previously interacted as an advisory group for an unrelated research project. Participants were paid for their time and provided dinner. Informed consent forms were completed and the session was audio-taped. Participants viewed videotapes of the two SPs – Mike and Debby – in a training session with one of the trainers portraying a substance abuse counselor. While viewing the tapes, participants were asked to rate each SP on a paper and pencil measure using a Likert-type scale assessing three questions, including, 1) How realistic is ___’s physical appearance, 2) How realistic were ___’s emotions, and 3) How realistic were ____’s reactions to the counselor? After viewing each SP and responding to the paper and pencil measure for that SP respectively, investigators facilitated a group discussion on the questions. Each question and the associated Likert-type scale appeared in large print on separate pieces of paper placed on the wall. Responses were tabulated, providing participants the opportunity to see how their responses related to others’ responses and to discuss their answers. Written and audio narrative materials were analyzed for themes running across formats. Adjustments to SP case content and portrayal were made accordingly.

In any discipline, SP case development and training is an iterative and ongoing process occurring throughout data collection. Random video footage of interviews occurring during data collection were reviewed and assessed for inconsistencies or other problems. Corrections were made accordingly. SP debriefing also occurred following each interview in order to obtain their perspectives on how the case worked and areas that might need additions or improvements. Adjustments were made accordingly.

Assessment Interviews

All interviews took place in a dedicated research space. Assessment interviews were videotaped using a camcorder placed on a tripod in the corner of the room where interactions took place. In all instances, the SP and interviewing clinician were the only occupants of the room during the video taping process. Data collection took place over a four month period in 2005.

Orientation of counselors prior to interviews included verbal description of the project, specifically, that they would be interviewing SP(s) in order to evaluate authenticity from a clinician’s perspective. Additionally, participants were informed that the SP’s were instructed to remain in role from their introduction to their departure from the interview. Counselors were instructed to behave as they normally would in an assessment interview. They were also instructed that they were not to behave as if they worked for any particular agency. These instructions clarified concerns about the extent to which counselors could offer certain services such as detoxification, residential or out-patient treatment. Counselors were also instructed that they could use materials such as check-lists if they brought them to the sessions. The clinicians were made aware that they would have one hour and 15 minutes to interview the SP and that, if they were not done by one hour and 10 minutes into the interview, a researcher would knock on the door to inform them they had five minutes left. If the clinicians ended their interview earlier than the allotted time, they were to close the interview naturally and either send the SP back to the research office (across the hall) or the clinicians could come themselves to inform the researchers they were done and research staff would usher the SP back to the research office. Clinicians were told that they would be videotaped during the meeting but that the camera would not be turned on until the session with the SP began. Clinicians were given the opportunity to ask questions and or express concerns. After filling out a pre-interview questionnaire acquiring demographic information and general theoretical and practical perspectives on substance abuse counseling, counselors were provided a brief written statement about the standardized patient case they were going to interview. Counselors were then introduced to the standardized patient as if he or she were a real client. Once the assessment interview was complete and the SP left the premises, clinicians were allowed time to complete the post interview questionnaire assessing diagnostic items and the Maastricht Assessment of Simulated Patients scale (59). Upon completion of the process, clinicians were asked to fill out a receipt and paid for their time.

Measures

Counselors were asked to complete a post-interview questionnaire containing the “authenticity” portion of the Maastricht Assessment of Simulated Patients (MaSP) (59). This ten-item subscale (shown in Table 1) was designed to measure the authenticity of a SP’s general performance. This scale asked counselors to respond to multiple items on a five-point Likert-type scale ranging from 1 – strongly agree to 5 – strongly disagree. It, combined with the second subscale (“feedback after the consultation”), has a reported reliability of.73. Two items were removed. The first item concerned SP performance during time-outs. The scenario was not applicable to this study. “Standardized patient appeared to withhold information unnecessarily” was also removed. This choice reflected intentional SP case content directing the actor to withhold information contingent on more or less counselor probes to a specific topic (e.g., interpersonal violence). With these items removed, the present study found a reliability of.76 (Cronbach’s alpha) for the “Authenticity” subscale.

Table 1.

Results of the Maastricht Assessment of Simulated Patients (Mean, Std. Dev.)

Maastricht Assessment of Simulated Patients	Mike (n=13)	Debby (n=18)
Standardized patient appeared authentic.	1.3, 0.7	1.3, 0.6
Standardized patient could be a real patient.	1.4, 0.7	1.4, 0.8
Standardized patient was clearly role-playing.*	1.9, 0.8	1.7, 0.9
Standardized patient stayed in his/her role the entire time.	1.3, 0.5	1.4, 0.5
Standardized patient simulated physical complaints unrealistically.*	2.2, 0.8	1.8, 0.8
Standardized patient’s appearance fit the role.	1.6, 0.7	1.5, 0.7
Standardized patient answered questions in a natural manner.	2.2, 1.2	1.8, 1.2
Standardized patient simulated psychological complaints realistically.	1.9, 1.0	2.0, 0.9
Overall Mean for the Maastricht Assessment	1.9, 0.7	1.6 0.5

Maastricht Assessment items rated on a 5 point, Likert-type scale with 1 = “Strongly Agree, 5 = “Strongly Disagree”

^*Reverse coded to conform to other items.

^*Item “Standardized patient appeared to withhold information unnecessarily” removed.

Participants were also provided the opportunity to contribute qualitative responses to the following statement. “Please provide any additional comments regarding this experience, (e.g., level of comfort in participating, level of difficulty communicating with the standardized patient, suggestions for future use of standardized patients in this context, etc.) below.”

Counselor ratings of SP authenticity on the MaSP were analyzed using SPSS 14.0. Mean scores and standard deviations were obtained for each SP on each questionnaire item. An overall mean score was obtained for each SP. Content analysis was conducted on subjective counselor responses to the open ended item “Please provide any additional comments regarding this experience” for each SP. Recurring themes were recorded as well as comments specific to individual counselors.

RESULTS

Overall mean scores for the MaSP indicated that counselors found both Debby and Mike highly authentic (see Table 1). Debby’s case and portrayal were rated only slightly higher on authenticity than Mike’s case and portrayal. Counselors rated Debby slightly more favorably than Mike on all items except “SP stayed in his/her role the entire time.” Similar mean scores for both SPs indicated similar responses to the SPs by interviewing counselors.

Although both SPs’ mean scores were similar for all the MaSP items, the overall lower mean score for Mike is substantiated by particular counselors’ subjective critiques in their responses to the qualitative item, “please provide any additional comments regarding this experience.” Those counselors reporting problems with certain aspects of Mike’s authenticity also scored the SP case and portrayal approximately one standard deviation below the mean. Table 2 provides mean scores and associated comments for counselors who rated Mike lower than the overall mean score.

Table 2.

Comments on Mike’s Authenticity for Counselors who provided an overall Mean Score higher than 1.9^*.

M	Comments
2.44	“The constant scratching and its intensity was way over the edge. Looked like poison ivy, not heroin withdrawal.” “The nodding along with frenetic pleading was also unrealistic.”
2.00	“Most patients are not this willing – want to ‘get clean’ when they’re sicker.”
2.56	“It is ridiculous to think information could be obtained from someone unable & unwilling to provide the necessary report to complete this post-interview questionnaire.” “Even if a client comes to my office under the influence he/she typically will provide more information than this patient did.”
2.56	“At first there was no eye contact from the ‘patient.’” “This appeared to be forced by the patient and I wasn’t feeling the authenticity.” “When he finally did make some eye contact, it felt more authentic.”
3.63	“Difficult to get information from this patient because he was focused on 2 things – getting his wife back and getting detox.” “Withdrawal from heroin not totally realistic.” “He psychologically & physically appeared to be in great pain from withdrawal.” “However, he constantly scratched his face – if real withdrawal he would have had tremors and shaking (which he did not) and have been scratching more than his face.”

^*Higher scores indicate lower authenticity

Mike’s case included positive comments from two counselors. For example, “The SP was so ‘real’ that I honestly worried about not giving out resources that I said ‘I will’ in the session.” A single neutral comment stating case content without opinion was provided by one counselor who said, “Patient had nothing going for him.” Negative comments were provided by five counselors. Example negative statements include, “SP more withholding than typical intoxicated patient,” “withdrawal unrealistic,” “six months not long enough for patient to want to quit (i.e., is willing),” and, “looked like he was dealing with poison ivy not heroin withdrawal.”

Nine counselors provided positive comments regarding Debby’s case including, for example, “Impressive and accurate use of emotion/behaviors,” “I talked to her ‘sisters’ a lot,” and “She was the real deal!” One counselor provided the neutral comment that the “client (was) evasive.” Although this comment was included as a neutral response to Debby’s case, it is worth noting that reviewing the tapes indicated that this counselor saw the SP’s “evasiveness” as consistent with her role. One counselor provided negative comments on Debby’s case. Example negative items from the one counselor included, “unusual for a couple to start using together,” “level of drug use seemed high (one gram in a 24 hour period),” and “patient seemed to understate her intelligence.” Eight counselors provided comments on the overall experience of interviewing Debby. Example responses following interviews with Debby included, “really good experience,” “experience more comfortable than expected,” “second interview more comfortable,” and “I use silence as a method to elicit client based discussion… did not work here.” Two counselors did not comment.

Tables 3 and 4 provide thematic responses to Mike’s and Debby’s cases and portrayals as they correspond to items on the MaSP. Mean scores and standard deviations for the MaSP are included for comparison purposes.

Table 3.

Comparing Quantitative and Qualitative Responses to “Mike”

MaSP Item	M, SD	Qualitative Response Theme
SP appeared authentic.	1.6, 1.2	Withdrawal and/or physicality unrealistic & inconsistent w/quantity and last reported use;
SP could be a real patient.	1.6, 1.0	Patient too intoxicated to conduct assessment; Odd match between wife & Mike; Lack of ANY eye contact felt unauthentic;
SP was clearly role playing.*	1.9, 0.8	SP so realistic counselor forgot SP was an actor playing a role/excellent actor; Too much scratching; Not enough scratching;
SP appeared to withhold information unnecessarily.*	2.4, 1.0	SP more withholding than typical intoxicated patient;
SP stayed in his/her role the entire time.	1.3, 0.5	n/a
SP simulated physical complaints unrealistically*	2.3, 0.9	Withdrawal and/or physicality unrealistic & inconsistent w/quantity and last reported use;
SP’s appearance fit the role	1.6, 0.7	n/a
SP answered questions in a natural manner.	2.3, 1.3	Too withholding;
SP simulated psychological complaints realistically.	2.1, 1.1	Excellent actor/he balked at the idea of treatment with or without wife;

5-point, Likert scale with 1 = “Strongly Agree, 5 = “Strongly Disagree”

^*Reverse coded to conform to other items.

Table 4.

Comparing Quantitative and Qualitative Responses to “Debby”

MaSP Item	M, SD	Qualitative Response Theme
SP appeared authentic.	1.3, 0.6	Authentic portrayal of meth, loss of children, emotions/behaviors;
SP could be a real patient.	1.4, 0.8	Unusual for couple to start using together; Level of drug use seemed high;
SP was clearly role playing.*	1.8, 1.0	Patient seemed to understate her intelligence; Authentic portrayal of meth, loss of children, emotions/behaviors;
SP appeared to withhold information unnecessarily.*	2.1, 0.8	n/a
SP stayed in his/her role the entire time.	1.4, 0.5	Authentic portrayal of meth, loss of children, emotions/behaviors;
SP simulated physical complaints unrealistically*	1.8, 0.8	n/a
SP’s appearance fit the role	1.5, 0.7	n/a
SP answered questions in a natural manner.	1.8, 1.1	Authentic portrayal of meth, loss of children, emotions/behaviors;
SP simulated psychological complaints realistically.	2.0, 0.9	Authentic portrayal of meth, loss of children, emotions/behaviors;

5-point, Likert scale with 1 = “Strongly Agree, 5 = “Strongly Disagree”

^*Reverse coded to conform to other items.

Counselors reported satisfaction with interviewing the SP(s). As part of the post-interview questionnaire, counselors responded to the item, “How helpful to you as a professional was the overall process of interviewing the SP, filling out pre and post interview questionnaires, and communicating with the researchers” on a Likert-type scale ranging from 1 = very unhelpful to 5 = very helpful (M = 1.71, SD = .74). The primary critique of the process concerned counselors desiring more information on the “agency” they were “working” for during the mock interviews as well as desiring knowledge of referral and interview closure options. For example, one counselor stated, “Wasn’t sure what options were available – what the policies of my pretend agency were – this colored my experience of participation.” Another counselor offered, “I wasn’t sure how to conclude the interview. Usually, we schedule groups, individual sessions and then give a urinalysis. It felt awkward to do this when nothing was available to work with.”

Finally, expenditures associated with these procedures were sub-divided into fixed and variable (i.e., per session) components. Fixed expenditures pertained to development of SP roles and to actor training, respectively. The fixed expenditures included: (a) researcher story development at $30 an hour for approximately 24 hours = $720 and (b) $15 an hour for 15 hours of training for two SPs = $450 for total fixed expenditure of roughly $1170. Variable (i.e., per interview) expenditures included (a) $15 an hour for an SP to participate in a two-hour session (= $30) as well as (b) two researchers at $30 an hour facilitating a two hour interview = $120 for an overall variable expenditure of roughly $150 per session.

DISCUSSION

Substance abuse counseling is grappling with decreasing resources and increased demands for implementing evidence based practices. Quality of care depends on effective counseling, beginning with accurate initial assessment of client needs and incorporating evidence based practices. This pilot study explored the authenticity of standardized patient (SP) cases as substance abuse treatment clients. Results inform the feasibility of using SPs as valid teaching and evaluation tools in substance abuse treatment. Research on the feasibility of creating and implementing authentic substance abuse SP cases is particularly important given that existing measures of clinician behaviors do not provide direct, observable, and consistent access to counselor performance. SPs also obviate problems with counselor compliance in providing documentation (e.g., tape recordings) of sessions.

This paper reported on findings from a pilot study that assessed the authenticity of two SP cases designed as substance abuse treatment clients. One SP case, “Mike,” was scripted as a heroin addicted 35 year old male with the mental health co-morbidity of dysthymia. A second SP case, “Debby,” was scripted as a methamphetamine addicted 24 year old female with the co-occurring problem of intimate partner violence. Substance abuse practitioners conducted assessment interviews with both SPs for purposes of assessing authenticity. Both SP cases received similarly high quantitative ratings for authenticity on multiple dimensions from the Maastricht Assessment of Simulated Patients (MaSP) scale (59). This finding supports future studies examining SP authenticity, particularly SPs as substance abuse clients. Further, these preliminary findings of SP authenticity support use of SP technology as a teaching and evaluation tool in substance abuse counseling.

A combined quantitative and qualitative approach to assessing authenticity appears more useful than either approach alone. Specifically, while both SPs received high overall scores on the MaSP, counselors’ lower authenticity ratings for Mike’s case combined with qualitative comments suggest that counselors paid attention to details and actively participated in the assessment process. Combined quantitative and qualitative assessments of Mike’s case, in particular, yielded crucial information for future case development.

Are some SP substance abuse cases more “feasible” than others? Although Mike’s case received high quantitative scores for authenticity, the qualitative comments by those counselors who scored him lower than the mean were quite critical, particularly comments centered on the authenticity of withdrawal. This finding supports the need for improved understanding of what constitutes “authenticity” for SPs as substance abuse clients and, therefore, which cases would best translate to applications in the field. Debby’s consistently high quantitative scores combined with consistently positive comments may indicate her case is feasible. Alternatively, critical comments for Mike’s case may not preclude its utility in the addictions treatment field, but rather provide yet another case development tool.

Mixed reviews of Mike’s case also substantiate the importance of investigating the issue of what constitutes authenticity in general. As with all mean scores, high overall quantitative marks for authenticity of Mike’s case were a combination of higher scores from some counselors and lower authenticity scores from others. The critical comments on Mike’s case came from the same counselors who scored the SP below the mean on the MaSP. This outcome may indicate that counselors either really liked Mike’s case or really did not like Mike’s case. Or, the qualitative comments could be indicative of overall disapproval of Mike’s case that could not be accessed through questions on the MaSP. In either instance, qualitative comments offer a valuable tool for developing better anchors for quantitative authenticity items. Further, the combined quantitative and qualitative information can inform which specific items measuring authenticity are adopted in future studies of SPs as substance abuse clients. Combined quantitative and qualitative evaluation of SP authenticity employed in this study may also serve as a launching point for refining items on future authenticity scales.

It is also worth noting that counselors overall were satisfied with the SP procedures. In addition, expenditures associated with the project were modest.

LIMITATIONS

This pilot study used the MaSP as the quantitative measure of SP authenticity. The MaSP appears to be one of the few, if not the only, quantitative assessment of simulated patients that evaluates several dimensions relevant to SPs portraying substance abuse clients. However, the MaSP was not specifically designed to assess authenticity of SPs portraying substance abuse clients. Rather, the scale was designed to capture authenticity of more typical uses of SPs for teaching and evaluation of physicians. Further, although the MaSP has been validated for SPs in an educational setting, the scale has not been validated specifically for assessing SPs as substance abuse clients. Given the mixed quantitative and qualitative findings in this study, it cannot be said that the MaSP captures necessary dimensions to measure SP case and portrayal features specific to substance abuse clients. Consistency in lower authenticity ratings for Mike’s case by the same counselors who provided critical qualitative comments does lend support for the scale’s ability to capture individual level assessments of authenticity. Future efforts should include defining and validating a quantitative assessment of SP authenticity specific to complex cases, including determining thresholds for what constitutes “good enough.”

The study was not designed to assess the utility of a mixed method evaluation of SP authenticity. Including the broad qualitative item, “please provide any additional comments regarding this experience” at the end of the post-interview questionnaire was intended to allow participants the opportunity to express opinions or thoughts on many topics. The findings that many counselors elaborated on SP authenticity and that lower authenticity quantitative indicators for one SP’s case were consistent with qualitative critique were not anticipated, yet crucially informative. It is possible that additional important qualitative information on counselors’ assessments of authenticity could have been obtained had questions been crafted specifically for that purpose. Future work should involve creation of a validated authenticity measure for complex SP cases as well as increasing validity of the cases by incorporating specially crafted qualitative questions that compliment a quantitative scale.

A convenience sample was used for this study that included counseling students. While an ideal research design would draw on a random sample of practitioners with similar work experience to assess SP authenticity, the educational, economic and geographic diversity that defines community based substance abuse treatment providers suggests a homogenously experienced sample may not represent the clinicians who would encounter a given client.

CONCLUSIONS

In summary, this pilot project assessing authenticity of two SPs as substance abuse clients found that it is possible to create an SP case that substance abuse counselors will perceive as authentic. Demonstrating SP validity is of paramount importance prior to widespread adoption of the technology for teaching and evaluation in substance abuse treatment. Mixed ratings of one case in this study point to the implications of by-passing the critical step of assessing authenticity. While actors may not be able to portray convincingly all clients seen in substance abuse treatment, the methodology does hold promise for training and education purposes.