Standardized Patient Walkthroughs in the National Drug Abuse Treatment Clinical Trials Network: Common Challenges to Protocol Implementation

Abstract

Background

Training research staff to implement clinical trials occurring in community-based addiction treatment programs presents unique challenges. Standardized patient walkthroughs of study procedures may enhance training and protocol implementation.

Objectives

Examine and discuss cross-site and cross-study challenges of participant screening and data collection procedures identified during standardized patient walkthroughs of multi-site clinical trials.

Method

Actors portrayed clients and “walked through” study procedures with protocol research staff. The study completed 57 walkthroughs during implementation of 4 clinical trials.

Results

Observers and walkthrough participants identified three areas of concern (consent procedures, screening and assessment processes, and protocol implementation) and made suggestions for resolving the concerns.

Conclusions and Scientific Significance

Standardized patient walkthroughs capture issues with study procedures previously unidentified with didactic training or unscripted rehearsals. Clinical trials within the National Drug Abuse Treatment Clinical Trials Network are conducted in addiction treatment centers that vary on multiple dimensions. Based on walkthrough observations, the national protocol team and local site leadership modify standardized operating procedures and resolve cross-site problems prior to recruiting study participants. The standardized patient walkthrough improves consistency across study sites and reduces potential site variation in study outcomes.

INTRODUCTION

The National Drug Abuse Treatment Clinical Trials Network (CTN) conducts randomized controlled clinical trials of medications and behavioral interventions for substance use disorders (1). Studies completed within community based drug and alcohol treatment programs (CTPs) recruit patients as research participants. Conducting clinical trials in real world treatment environments presents unique challenges to protocol implementation.

Participating study sites vary, often substantially, in organizational structure, culture, staffing and programmatic options. A CTP, for example, may serve primarily privately funded patients who are employed and have supportive social structures. Another site, in contrast, might serve court-mandated, publicly funded patients with substantial criminal histories, and tenuous social and economic supports. Each study site also differs in structure and supports for research. Program leaders, practitioners and research teams must coordinate clinical and research activity; a task more easily accomplished in some sites than others. These multi-faceted partnerships and client bases lead to differences in protocol implementation, yet data collection and intervention procedures must conform to strictly monitored quality control and consistency. National trainings for CTN trials are, therefore, intensive and emphasize consistency in obtaining data and delivering study interventions.

National protocol trainings vary by protocol and are primarily didactic. Occasionally, treatment practitioners role-play behavioral health interventions during the training phase. Role playing emphasizes development of clinical skills among practitioners delivering the study interventions. Research assistants who collect study data, however, are unlikely to participate in formal role plays during national trainings. Standardized patients provide enhanced training by providing more ‘real life” role plays for individuals implementing CTN clinical trials.

Standardized patients are trained actors who portray a patient in a clinical setting (2,3). Typically, the “patient” reports symptoms consistent with biomedical conditions and are used to train medical practitioners (4-6). Standardized patients, for example, trained third year medical students to conduct sexual history and HIV counseling (7) and portrayed an individual seeking peer support in order to evaluate available family to family services (8). Limited, yet expanding literature demonstrates possibilities for standardized patients portraying individuals with substance use disorders (9-18), including establishing authenticity of performances with substance abuse treatment practitioners conducting mock assessment interviews (19). Finally, standardized patient walkthroughs illustrate the utility of the methodology for enhanced training capable of identifying and ameliorating challenges to implementing protocol procedures (20).

Standardized patients permit rehearsal of protocol procedures while holding constant the “participant” role. Utilizing the same participant across multiple rehearsals of the same protocol enhances on-site staff training, provides CTPs running clinical trials and the national lead investigators of trials opportunities to identify common difficulties with protocol implementation, examines challenges between and within study sites, and finally, provides treatment program and lead national investigators the opportunity to modify study procedures. This paper reports common challenges identified during participant screening and baseline data collection with 57 standardized patient walkthroughs at 20 CTPs participating in four CTN trials.

MATERIALS AND METHODS

Standardized patient walkthroughs (n=57) of participant screening, consenting, baseline data collection and, less frequently, study interventions occurred at community based treatment programs (n=20) participating in four clinical trials for substance abuse treatment interventions: 1) HIV Rapid Testing and Counseling in Drug Abuse Treatment Programs in the U.S., 2) Stimulant Reduction Intervention using Dosed Exercise (STRIDE;), 3) Smoking-Cessation and Stimulant Treatment: Evaluation of the Impact of Concurrent Outpatient Smoking-Cessation and Stimulant Treatment on Stimulant-Dependence Outcomes (S-CAST), 4) Stimulant Abuser Groups to Engage in 12-Step: Evaluation of a Combined Individual-Group Intervention to Reduce Stimulant and Other Drug Use by Increasing 12-Step Involvement (STAGE-12).

Walkthrough participants

A typical walkthrough included three individuals: the research assistant(s) and/or interventionist on staff at the clinic, the standardized patient, and a visiting researcher. The guest researcher was present to facilitate walkthrough activities and debrief CTP research staff. The standardized patients (1 female and 1 male) were professional actors. For each protocol the actor’s role used scripts previously tested for authenticity (19), and modified to meet inclusion and exclusion criteria for the four studies. To maximize authenticity of clients at individual CTPs, the standardized patient research team conducted phone meetings with key personnel from each site. The phone calls included a range of content from gaining an understanding of how a specific CTP planned to implement protocol procedures, to details about CTP clients (i.e., typical demographics of the clientele, locations where clients “hang out,” why the clients are typically admitted to any particular CTP, etc.). These phone meetings allowed the researchers and the standardized patients to become familiar with the agency, discuss plans for protocol implementation, and begin to choreograph the events of the prospective SP walkthroughs. The meetings also provided the researchers with realistic challenges that CTP research staff may encounter during protocol implementation. Scripts were sufficiently detailed to provide content for two hours of assessment. Following visits to all CTPs participating in the same protocol, the facilitating researcher(s) generated a summary report outlining common challenges and recurrent observations across sites. The report was provided to the lead teams. The lead investigators and site study teams modified study procedures including but not limited to suggested approaches to obtaining accurate data on study measures and altering expectations of what should be obtained using particular measures, etc.

Standardized patient roles

Three trials tested interventions for stimulant users. The standardized patient portrayed an individual dependent on either methamphetamine or crack cocaine. Gender was based on the availability of the actors for scheduling walkthroughs and demographics of the patients at the clinics (2 served only females; 1 served only males). Substance use was based on the predominant drug used in the geographic location of the clinics. The fourth trial testing HIV rapid testing and standardized patients acted as if they tested positive for HIV.

Procedures

A typical walkthrough included introductions, pre-screening, consenting, screening, randomization, baseline data collection, and sometimes the study interventions. Table 1 lists protocol procedures included in most walkthroughs.

Table 1.

Common trial procedures and content for walkthroughs

Procedure	Content
Study Introduction	Script. Usually no longer than two paragraphs.
Pre-Screen	Brief locator questionnaire, substance use scales.
Informed Consent	Investigators, sponsors, brief study overview, study purpose, study procedures, data collection methods (e.g., audio recording), confidentiality, risks and discomforts, benefits, alternatives, costs, payment, liability, researcher contact information, signatures.
Screening	Locator questionnaire, demographics, DSM-IV questions, medical history, Addiction Severity Index-Lite, Composite International Diagnostic Interview, Medical history, depression scales, prior and current medications, psychological and behavioral scales, withdrawal scales, Fagerstrom test for nicotine dependence, urinalysis, physical examination, time line follow back substance use calendar, Short Form 36 Health Survey, Mini International Neuropsychiatric Interview, Self-Administered Comorbidity Questionnaire.
Randomization	Electronic determination of treatment arm.
Baseline data collection	Data collected during baseline can include any one of the items listed in the screening category mentioned above. Placement of measures and procedures in a CTN clinical trial varies depending on inclusion/exclusion criteria.
Study intervention	Individual and/or group behavioral counseling, medication dispensing and dosing.

¹This list represents a combination of procedures drawn from multiple CTN clinical trials and is not representative of any one individual study.

²Scales and procedures unique to only one protocol are not included.

The actor did not break role unless the visiting researcher instructed them to do so. The visiting researcher took detailed field notes, and led debriefing sessions. Debriefing included informal communication between the three individuals about strengths of the protocol implementation, areas of concern, recommendations, and dialog on the potential benefits of the recommendations.

Analysis

Field notes were edited, consolidated and used to generate summary reports for the national protocol team leading each study. Summary reports included an introduction to the walkthrough processes specific to a study and tables noting recurring areas of concern, associated recommendations, and reasons for the recommendations. “Recurring” was defined as problems that were noted at more than one study site participating in a protocol. The summary reports were reviewed for items appearing consistently across studies. Lessons learned and “cross-study” implementation concerns are provided in this paper.

RESULTS

Three themes (consent, screening and assessment, and protocol implementation) emerged across the 57 walkthroughs for four clinical trials. Table 2 lists the concerns and provides recommendations for addressing the concerns.

Table 2.

Example challenges with protocol procedures and associated recommendations

Consent
Challenge related themes	Recommendations for improvement
1. Describe the study	Visual aids (e.g., schematics of treatment arms, flow charts, bulleted points of importance) to assist in the participants understanding of the study’s purpose, safety, confidentiality, treatment options ,research arms and what is expected to participate.
2. Communicate key points	Highlight and demarcate key points. Summarize sections to extract critical details. Use simple words as much as possible. Make eye contact to make sure study participants are tracking. (e.g, screening is to see if the “study is a good fit for you” rather than whether the person is a good fit for the study).
3. Comprehension quiz	Use positive wording when correcting answers. Review which section of the consent refers to the item that was answered incorrectly.

Screening & Baseline Data Collection
Challenge related themes	Recommendations for improvement
4. Insufficient participant response to questionnaire items	Probe for a response capable of accurately answering the question on a screening instrument. For example, “I don’t know; I don’t have a drinking problem” is not an accurate enough response to the question “how many drinks do you consume in a day” to justify marking “none.”
5. Explain self-report instructions clearly	Practice explaining challenging instructions (e.g., differentiation between substance use in 30 days versus over a life time). Use visual aids when necessary.
6. Delivery of sensitive material/questions	Vary tone of voice, re-establish eye contact, take a break, and preface a question by noting its difficult or sensitive nature. Remind participant of protocol procedures in place to ensure confidentiality.

Protocol Implementation
Challenge related themes	Recommendations for improvement
7. Potential for breach of confidentiality	Avoid open office doors. Place a sign outside the office door asking for quiet.
8. Participant’s affect and reactions/responses influence the research staff’s affect and responses	Have a “bottom line” set of items to get across to the participant. Monitor affect in response to participant behavior. Observe that a participant is “ramped up” and, therefore, you remain calm.
9. Breaks	Pay attention to the participant’s body language and offer breaks as needed.

Consent

Clinical trials differ in complexity of study design. Consenting participants involves describing the study. Walkthroughs revealed that effective explanation of complex designs varied substantially. Consent forms often exceeded eight pages of single spaced text and involved language that exceeds reading comprehension levels for some clients. Some Institutional Review Boards seek to improve comprehension by requiring a verbatim reading of the document. The standardized patients reported that having the consent read to them decreased comprehension due to complex language and sentence structure, boredom, irritability, and fatigue. Finally, CTN consent procedures typically concluded with a comprehension “quiz.” Participants must score 100% on the quiz before signing the consent. The quiz ensures comprehension of critical components of the consent. Approaches to administering the quiz differed. The standardized patient walkthroughs highlighted the importance of not generating test anxiety while clearly correcting incorrect responses.

Screening and baseline data collection

Screening and baseline data collection instruments (see Table 1 for examples) elicit a variety of data (e.g., age, gender, education level vs. sexual risk behavior). Instructions range from simple memorable behaviors (e.g., reporting incarceration in the past 30 days) to complex behaviors (e.g., quantities of substances used in the past 30 days versus the amount of the same substance used in a lifetime [including the past 30 days]). Often, trials involve questions about illegal activity (e.g., drug use, child abuse, theft) that can provoke emotional responses (e.g., reporting the removal of children to state custody). Successful protocol implementation relies on research staff obtaining valid participant responses to questionnaire items, explaining instructions accurately, and applying interpersonal skills when probing sensitive topics.

Although all research staff received didactic training on data collection instruments, walking through screening and baseline data collection procedures attuned research staff to data collection challenges. For example, “timeline follow backs” ask participants to track daily use of multiple substances for up to three months previous to baseline data collection. This instrument is also used as the data collection method for primary outcomes in many trials. However, when standardized patients were scripted to realistically struggle with this task, researcher assistants often gave up, quickly became frustrated, and inserted best guesses. For example, when “the patient” said “I don’t know, I said I smoke a pack a day, isn’t that good enough.” The research assistant reported “24 cigarettes daily”. Debriefing provided the opportunity to suggest additional memory anchors (e.g., days the participant may have had minimal to no cash or other methods of procuring drugs) and approaches to probing further while minimizing participant irritation (e.g., offering breaks or agreeing that the task is getting difficult and moving to other procedures and returning to the timeline follow back).

Study Implementation

Every trial required attention to participant confidentiality, participant psychological and physical comfort, and overall participant attentiveness. Standardized patient walkthroughs identified common implementation problems (e.g., distracting ambient noise, potential for others to overhear confidential interviews, uncomfortable seating, and lack of rest breaks. Rest breaks are required because of lengthy (2 to 4 hours) procedures for consenting, data collection and randomization. During lengthier procedures, standardized patients reported difficulty tracking what the research assistant was asking. Like many study participants, the standardized patient needed breaks to smoke cigarettes and recover from tiresome assessment processes. Recommendations for small changes in procedures improved the standardized patient’s comfort and tolerance for assessment. Problematic assessment processes were reported to the lead investigators for each trial and modified as necessary.

DISCUSSION

Standardized patient walkthroughs of participant screening, consenting, and baseline data collection in four clinical trials conducted as part of the CTN identified problems with protocol implementation and improved procedures for patient consenting, assessment, and randomization. Our work extends a prior analysis based on a single standardized patient at one site for one clinical trial (20). The analysis highlights the use of standardized patient walkthroughs to capture issues with study procedures previously unidentified with didactic training alone. Additionally, field notes permitted examination of problems with implementing particular trial procedures across studies and treatment centers. Common implementation concerns included difficulties with: a) explaining the critical facets of a clinical trial to the participant, b) communicating key points of consent forms, c) navigating the consent quiz, d) accepting and documenting insufficient participant responses to questionnaire items, e) explaining difficult survey instructions, f) delivering “sensitive” interview material to participants, g) inadvertently providing opportunities for breaches in participant confidentiality (e.g., conducting research visits with clients in rooms where interview responses could be overheard by others, h) allowing the participant’s affect to shape the data collection experience, i) and appropriately managing breaks during lengthy study procedures.

Standardized patient walkthroughs have gained substantial popularity and momentum within the CTN. First employed in 2005 with one set of walkthroughs in a single treatment center for one study; well over 60 walkthroughs have now been conducted within the CTN for 7 separate protocols. Participating research and treatment center staff consistently report satisfaction with the process and on the benefits of real time interpersonal role play and feedback with an actor who is trained to remain consistent within and between performances. Use of a standardized patient has at least three advantages: 1) uninterrupted opportunities to identify issues with study flow, 2) learning the need to improve on procedures previously thought mastered, and 3) becoming better acquainted with participant scenarios. The national protocol teams report that identification of concerns with protocol implementation led to changes in protocol procedures and case report forms. Additionally, because CTN trials may be staged in separate waves, protocol modifications are made based on lessons learned during the initial implementation period. Standardized patient observations of problem areas in wave I, for example, have led to changes in wave II national trainings (e.g., inclusion of recommended changes in how to word questions posed to participants).

Currently, standardized patient walkthrough methodology necessitates transporting the researcher(s) and actor(s) to each site conducting a clinical trial. While possible to develop a centralized training option for clinical and research staff to conduct their own standardized patient walkthroughs, previous research demonstrates that realistic portrayal of any substance abuse treatment client requires trained professionals (19). Availability of professional actors capable of realistically portraying substance abuse treatment clients likely varies across communities. Conversations with each study site prior to walkthroughs allows for maximizing local authenticity while working within the parameters of the same role with the same actor. Without role and actor consistency, differences in actors and rehearsal procedures complicate the potential to discern challenges specific to one site versus those that occur across sites. Research assistants, for example, ask the same questions during screening and baseline procedures. Individual approaches to inquiries, however, varied substantially. The actor must not only remember observations for debriefing while staying in role, but also provide consistent responses with consistent affect regardless of inconsistencies in how questions were asked.

Implications

Utilizing standardized patients as substance abuse treatment clients is relatively new terrain (19). Even more novel are applications of the methodology to enhance training and identify potential for improvement in protocol procedures in clinical trials of interventions for substance use disorders (20). This report identifies cross-study and cross-site common concerns in protocol implementation and recommendations for resolutions to those concerns. Future research may seek to demonstrate improved protocol implementation at treatment centers benefiting from standardized patient walkthroughs versus sites without the walkthroughs. Additional research may examine how single site clinical trials could benefit from walkthroughs by obtaining and modifying detailed information on study procedures from the participant perspective at multiple time points, providing a unique approach to single site clinical trial quality control. Ultimately, standardized patient walkthroughs provide a distinctive approach to improved consistency in the integrity of key procedures that may affect primary outcomes in clinical trials.