Feasibility and acceptability of the DSM‐5 Field Trial procedures in the Johns Hopkins Community Psychiatry Programs

Abstract

The Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM‐5) contains criteria for psychiatric diagnoses that reflect advances in the science and conceptualization of mental disorders and address the needs of clinicians. DSM‐5 also recommends research on dimensional measures of cross‐cutting symptoms and diagnostic severity, which are expected to better capture patients' experiences with mental disorders. Prior to its May 2013 release, the American Psychiatric Association (APA) conducted field trials to examine the feasibility, clinical utility, reliability, and where possible, the validity of proposed DSM‐5 diagnostic criteria and dimensional measures. The methods and measures proposed for the DSM‐5 field trials were pilot tested in adult and child/adolescent clinical samples, with the goal to identify and correct design and procedural problems with the proposed methods before resources were expended for the larger DSM‐5 Field Trials. Results allowed for the refinement of the protocols, procedures, and measures, which facilitated recruitment, implementation, and completion of the DSM‐5 Field Trials. These results highlight the benefits of pilot studies in planning large multisite studies. Copyright © 2013, American Psychiatric Association. All rights reserved.

Introduction

As part of the overall development and research strategy for the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM‐5), the DSM‐5 Research Team proposed field trials to determine feasibility, clinical utility, reliability, and where possible, validity of the proposed revisions. Because the underlying rationale, design, and methods envisioned for the field trials were a significant departure from previous DSM trials (Clarke et al., 2013; Kraemer et al., 2010; Kraemer et al., 2012; Regier et al., 2013), pilot studies (i.e. smaller versions of the main study to test whether the components of the main study can work together [van Teijlingen and Hundley, 2002; Arain et al., 2010]) of the proposed methods and measures were planned in adult and pediatric samples.

There were several unique aspects to the proposed DSM‐5 Field Trials. First, they were to be conducted in clinical settings with clinicians of diverse clinical backgrounds. No structured research diagnostic interviews were required, and training in the use of DSM‐5 would be comparable to that expected for all practicing clinicians. A stratified sampling method was proposed to generate sufficient sample sizes for low prevalence disorders. Consequently, sampling weights were required in the field trials to estimate the parameters of interest, such as frequency, mean score, and reliability. Also, several novel aspects were incorporated into the diagnostic assessment process, including use of clinician‐ and patient‐rated dimensional assessments of symptoms, disorders, and disability. Using the clinical utility framework proposed in First (2006) and Smart (2006), the DSM‐5 Research Team developed research questionnaires for both clinicians and patients to assess ease of use and clinical usefulness of the new diagnostic criteria and dimensional measures. The DSM‐5 Field Trials also proposed to videotape a random 20% of the patient diagnostic clinical interviews, to be used for examination of convergent validity. An electronic data capture system was proposed to collect patient, clinician, and research coordinator (RC) data and to enable central management of the study that would involve academic clinical sites across the United States and in Canada.

As is the purpose of pilot studies (van Teijlingen and Hundley, 2002; Arain et al., 2010), it was imperative to test small‐scale versions of the DSM‐5 Field Trials prior to implementation to identify and correct any procedural problems. Therefore, the DSM‐5 pilot studies aimed to provide guidance for the DSM‐5 Field Trials, the results of which can be informative to other researchers designing similar complex multisite studies in the future. Specifically, the DSM‐5 pilot studies aimed to:

1. Test the proposed methods and procedures for the field trials, such as examining the:

   1. feasibility of implementing a stratified sampling protocol to recruit patients in busy clinical settings;

   2. logistical feasibility, including Institutional Review Board (IRB) process, recruiting a sufficient sample size, collecting information to calculate the sampling weights, and integrating the study protocol into busy clinics;

   3. feasibility of videotaping a random 20% of diagnostic interviews;

   4. adequacy of the proposed trainings in preparing clinicians and RCs for their roles in the field trials.

2. Test the ease of implementation and use of the electronic data capture system, designed to collect and store study data.

3. Identify any major problems in the newly developed research forms that would prevent their successful implementation in the field trials.

These aims were tested in both adult and pediatric psychiatric samples. This article reports on the implementation and findings of the pilot studies.

Methods

Study location, design, sampling and recruitment process, and stratification

The DSM‐5 pilot studies took place at the Johns Hopkins Medical Institution (JHMI) in the Adult Community Psychiatry Outpatient Program of the main campus and in the Child and Adolescent Outpatient Program of JHMI's Bayview Medical Center. The sites were selected because of similar characteristics to the large academic medical centers which were to be recruited for the DSM‐5 Field Trials, including high patient volume, experience conducting clinical research studies, and existing research infrastructure to facilitate conducting the study in a short timeframe. The proximity of JHMI to American Psychiatric Association (APA) headquarters facilitated training and site visitations at minimal additional cost.

The pilot study protocol was centrally developed and monitored by the DSM‐5 Research Team at the APA and reviewed and approved by IRBs at APA and JHMI. The JHMI IRB mandated the submission of a HIPAA waiver to allow the sites to collect demographic and DSM‐IV diagnostic information on all patients seen in the clinics during the study period using the DSM‐5 Patient Recruitment Screening Form (PRSF), without consent, to enable examination of the feasibility of developing sampling weights, which were necessary in the field trials.

The pilot studies were conducted over an eight‐week period with recruitment goals of 20 patients per diagnostic stratum in the adult pilot and 10 patients per diagnostic stratum in the child pilot. The study included two study visits, for test and retest, scheduled at least four hours but no greater than two weeks apart. Interim and follow‐up debriefing sessions were conducted with the principal investigators (PIs), study clinicians, and RCs to identify and discuss potential problems before the DSM‐5 Field Trials were implemented. Also, the pilot study involved videotaping a random 20% of diagnostic patient interviews to determine the feasibility of obtaining consent for this component of the study.

Consecutive new and existing patients seen in the clinics for initial or routine appointments were screened for interest and potential eligibility using the study's PRSF and then stratified into one of the targeted diagnostic strata based on their current DSM‐IV diagnoses or, for newly proposed diagnoses, on the presence of symptoms that had a high probability of indicating the new diagnoses, as indicated by their treating or intake clinicians without systematic verification. The PRSF asked for date of visit, patient's name, status (new or existing patient), existing DSM‐IV diagnoses and/or the presence of other psychiatric symptoms, current symptomatic status of existing DSM‐IV diagnoses (i.e. currently having symptoms meeting criteria for the disorder), and whether the individual was interested in learning more about the study. Targeted diagnostic strata were as follows:

1. For the adult pilot, targeted diagnoses were Major Depressive Disorder (MDD), Bipolar Disorder (BPD), Schizophrenia, Schizoaffective Disorder, and “Other” (i.e. diagnosis other than MDD, BPD, Schizophrenia, and Schizoaffective Disorder) for a total of 100 patients.

2. For the child pilot, targeted diagnoses were MDD, Disruptive Mood Dysregulation Disorder (DMDD), Oppositional Defiant Disorder (ODD), Generalized Anxiety Disorder (GAD), and “Other” (i.e. diagnosis other than MDD, DMDD, ODD, and GAD) for a total of 50 patients.

During the recruitment period, each patient seen in the clinic was informed about the study by the treating or intake clinician, who completed a PRSF. Interested patients were referred to the RCs for assessment of eligibility. Eligible adult patients included those who were currently symptomatic and were able to read and communicate in English. Eligible child patients were ages six to 17 years, with a parent/legal guardian who could read and communicate in English (e.g. to complete measures and participate in clinical interviews). Written informed consent and child/adolescent assent was sought from interested and eligible patients. Separate consent was obtained for videotaping. Each adult patient and parent/guardian received remuneration of $40 per study visit and each child/adolescent patient received a $25 gift card.

Study personnel recruitment and training

All research staff completed Web‐based training in human subjects protection and research ethics, with knowledge assessments. RCs received a three‐hour training session that included training on the study procedures and orientation to the RC, patient and clinician components of the study and data collection system, to enable effective study coordination.

The site PIs were asked to recruit clinicians of varied mental health disciplines to reflect the diversity of clinicians who use the DSM in clinical practice. Sites were offered clinician remuneration of $100 (adult pilot) or $150 (child/adolescent pilot) per patient interview. Clinicians were also offered Continuing Medical Education (CME) credits or certificates toward Continuing Education Units (CEUs), and acknowledgment in DSM‐5.

All participating clinicians attended a mandatory 2–3 hour in‐person training session that included orientation to major diagnostic changes to DSM‐5, the dimensional measures, and the data collection system. A mock clinical interview was carried out to demonstrate the diagnostic procedures and data entry using the data collection system.

Data collection and study visit procedures

The Research Electronic Data Capture (REDCap) system developed by Vanderbilt University with funding from the National Institutes of Health (Harris et al., 2009) was modified to meet the needs of the DSM‐5 Field Trials. Modifications included the development of (1) an RC component for coordination of the study within sites, (2) a patient component for the collection of data simultaneously across sites with “real‐time” scoring and transmission to clinicians, (3) a clinician component to allow immediate access to patient‐reported information, and (4) a central management component to enable the overall coordination of the field trials by the APA. The plan was to have a three component system in which the RC component would communicate with the patient and clinician components thereby allowing the patient‐completed measures to be scored and the results and interpretations transmitted to the clinician‐component of the system immediately after completion. This would enable the clinician to review the patient/informant‐reported information before the clinical interview. However, the system was still in development during the pilot studies and the patient, clinician, and RC components were only manually linked, which prevented “real‐time” scoring and transmission of patient scores to the clinician for review. This meant that for each patient seen, the RC printed the results of the patient‐completed measures and provided them to the clinician for review before seeing the patient (i.e. results were not electronically transmitted).

The patient component consisted of adult (age 18+), child/adolescent (age 11–17), and parent/guardian‐rated versions (for child age 6–17) of DSM‐5 cross‐cutting symptom measures (Narrow et al., 2013), which assessed the presence and severity of mental health symptoms common to most psychiatric disorders, such as depression, anxiety, anger, sleep problems, and substance use. It also included the World Health Organization Disability Assessment Schedule (WHODAS II), which assessed disability in areas such as communicating and understanding, mobility, self‐care, and life activities (Üstün et al., 2010; WHO, 2010). Finally, the adult (age 18+) and child/adolescent (age 11–17) versions of the 36‐item Personality Inventory for DSM‐5 (PID‐5), developed and tested by the DSM‐5 Personality and Personality Disorders Work Group (Krueger et al., 2012), were included. The patient component of the system used unique patient identification numbers (PIDs) to register each patient in the study. The PID was used to open the battery of DSM‐5 measures for the patient (i.e. adults and children age 11–17) and/or parent/guardian (for all child patients) to complete. Use of the PID ensured confidential completion of the measures.

All available DSM‐5 diagnostic checklists and clinician‐rated measures were included in the clinician component of the DSM‐5 REDCap system, including DSM‐5 developed clinician ratings of: Suicide Risk in Teens, Suicide Concerns in Adults, Psychosis, Early Development and Home Background, Clinical Utility, and the six‐item clinician WHODAS. A unique clinician identification number (CID) was assigned to each clinician, which was used to access the clinician component of the system. The system and use of the CID allowed multiple clinicians to confidentially access the system at the same time to complete clinician‐rated measures and access the results on assigned patients, based on PIDs, while maintaining blindness to each other's ratings.

The RC component included the PRSF and patient tracking forms. The system enabled patient tracking via the PID, and assignment to independent clinicians, via the CID, to conduct diagnostic interviews. The RC component used in the pilot study was developed to help identify if additional fields were needed to make the recruitment and tracking processes more manageable in the DSM‐5 Field Trials (i.e. did the RC for the system advocate efficient as is or suggest modification to make it more user‐friendly and make the management of the study easy).

At each study visit, the patient (or, for children, the patient and parent/guardian), with the assistance of the RC, completed the pre‐measure questions on age, sex, and reading comprehension level. Reading comprehension was assessed by asking whether the patient could read and understand written material independently, with minimal assistance from the RC, with major assistance for the RC, or not at all (for child patients). Reading comprehension level was also determined during the consent process by asking the patient (and/or parent/guardian) what he/she understood about the study and what was being asked of him/her. The assessment of reading comprehension level was important since it determined the set‐up for completion of the DSM‐5 measures and the level of assistance the RC needed to provide. At each study visit, the DSM‐5 measures, including the DSM‐5 Level 1 and associated Level 2 cross‐cutting symptom measures, the WHODAS, and, for adults and children ages 11 and older, the 36‐item PID‐5 Scale, were completed.

Upon completion of the measures, the RC downloaded and printed a copy of the results (child and adult pilot) and interpretations (child pilot) of the patient ratings (e.g. score on the depression domain indicates, on average, mild depression over the past week) for the clinician to review prior to the diagnostic interview. After the diagnostic interview, the RC met with the patient to complete the Patient Utility Questionnaire (PUQ) that assesses the feasibility of using the DSM‐5 measures and the usefulness of the information in helping to describe his or her symptoms to the clinician.

Data analysis

For purposes of assessing DSM‐IV prevalences, individuals with comorbid diagnoses were counted for each diagnosis they had. However, for sampling, a patient with comorbid diagnoses was assigned to the stratum for whichever of his/her diagnoses was rarest at the site. Sampling weights were calculated for each site (adult and child) and each stratum using information collected on the PRSFs. The sampling weight for each stratum was the proportion of those in the sampling frame assigned for sampling to that stratum. Thus, individuals with comorbid diagnoses only contributed to the sampling weight for the stratum to which they were assigned. Descriptive analyses (mean, standard deviation, and frequency distributions) were conducted using SAS statistical software and SUDAAN.

Results

The pilot study was successfully implemented in both settings with some sample size problems. Sixty‐eight patients (of the planned 100), mostly females (63.7%) age 44.8 years old (standard deviation [SD] = 11.7), completed both visits in the adult pilot study (Table 1). For the child pilot, 36 (of the planned 50) patients age 11.8 years old (SD = 3.1) and mostly males (61.1%) completed visit one and 35 completed both study visits (Table 2). To obtain the final sample sizes, the initial four‐week study recruitment period was extended to six‐weeks. Despite this extension, recruitment fell short of our goal. However, once patients consented to participate and completed the first study visit they also completed the second study visit (i.e. 100% and 95% retention rates for the adult and child sites respectively).

Table 1.

Socio‐demographic and clinical characteristics of the patients in the adult pilot study at JHMI, Adult Community Psychiatry Outpatient Program in the Department of Psychiatry and Behavioral Sciences on the main medical campus (weighted analyses, unweighted sample size)

		Assigned DSM‐IV stratum
	Total (n = 68)	Major depression (n = 18)	Bipolar I & II (n = 16)	Schizophrenia (n = 20)	Schizoaffective (n = 8)	“Other” disorders (n = 6)	p‐Value
Age							0.01
Mean (SD)	44.8 (11.7)	43.1 (10.2)	42.2 (11.2)	47.8 (11.9)	52.6 (9.6)	45.5 (14.6)
Sex
% Female	63.7 (n = 42)	83.3 (n = 15)	62.5 (n = 10)	40.0 (n = 8)	50.0 (n = 4)	83.3 (n = 5)	0.04
Marital status
% Never married	51.5 (n = 33)	55.6 (n = 10)	37.5 (n = 6)	60.0 (n = 12)	25.0 (n = 2)	50.0 (n = 3)	<0.01
Race/ethnicity							<0.01
% Non‐Hispanic Whites	22.6 (n = 15)	27.8 (n = 5)	43.8 (n = 7)	5.0 (n = 1)	25.0 (n = 2)	—
% Blacks/African Americans	73.3 (n = 49)	72.2 (n = 13)	43.8 (n = 7)	95.0 (n = 19)	62.5 (n = 5)	83.3 (n = 5)
% Other	4.1 (n = 4)	—	12.5 (n = 2)	—	12.5 (n = 1)	16.7 (n = 1)
Highest level of education							<0.01
% Some high school or less	19.2 (n = 15)	11.1 (n = 2)	12.5 (n = 2)	30.0 (n = 6)	50.0 (n = 4)	16.7 (n = 1)
% High school graduate	38.0 (n = 25)	50.0 (n = 9)	50.0 (n = 8)	15.0 (n = 3)	25.0 (n = 2)	50.0 (n = 3)
% Greater than high school	34.3 (n = 22)	27.8 (n = 5)	37.5 (n = 6)	45.0 (n = 9)	12.5 (n = 1)	16.7 (n = 1)
Annual household income
% Less than $10,000	73.8 (n = 51)	61.1 (n = 11)	75.0 (n = 12)	90.0 (n = 18)	87.5 (n = 7)	50.0 (n = 3)	0.31
Current employment status							0.01
% Unemployed	32.8 (n = 22)	27.8 (n = 5)	37.5 (n = 6)	35.0 (n = 7)	12.5 (n = 1)	50.0 (n = 3)
% Permanently disabled	35.4 (n = 24)	38.9 (n = 7)	25.0 (n = 4)	40.0 (n = 8)	50.0 (n = 4)	16.7 (n = 1)
% Other	31.8 (n = 22)	33.4 (n = 6)	37.5 (n = 6)	25.0 (n = 5)	37.5 (n = 3)	33.3 (n = 2)
DSM‐5 diagnoses
% Generalized Anxiety Disorder	31.6 (n = 20)	44.4 (n = 8)	43.8 (n = 7)	10.0 (n = 2)	12.5 (n = 1)	33.3 (n = 2)	0.05
% Major Depression	29.8 (n = 17)	68.4 (n = 13)	11.8 (n = 2)	—	—	28.6 (n = 2)	<0.01
% Alcohol Use Disorder	18.0 (n = 11)	27.8 (n = 5)	25.0 (n = 4)	5.0 (n = 1)	12.5 (n = 1)	—	0.04
% Other Substance Use Disorder	13.6 (n = 10)	11.1 (n = 2)	18.8 (n = 3)	10.0 (n = 2)	12.5 (n = 1)	33.3 (n = 2)	0.82

Table 2.

Socio‐demographic and clinical characteristics of the patients in the child/adolescent pilot study at JHMI, Child and Adolescent Outpatient Program in the Department of Psychiatry and Behavioral Sciences at Bayview Medical Center (weighted analyses, unweighted sample size

Child study		Assigned DSM‐IV stratum
	Total sample (n = 36)	Major Depressive Disorder (n = 3)	Oppositional Defiant Disorder (n = 7)	Disruptive Mood Dysregulation Disorder (n = 11)	Generalized Anxiety Disorder (n = 8)	“Other”disorders (n = 7)	p‐Value
Age	11.8 (3.1)	14.0 (4.2)	11.1 (3.7)	12.0 (2.8)	9.4 (2.3)	13.4 (3.0)	<0.01
Mean (SD)
Sex	38.9 (n = 14)	66.7 (n = 2)	28.6 (n = 2)	45.4 (n = 5)	37.5 (n = 3)	28.6 (n = 2)	0.83
% Female
Race/ethnicity							0.04
% Non‐Hispanic Whites	64.9 (n = 23)	100.0 (n = 2)	57.1 (n = 4)	63.6 (n = 7)	62.5 (n = 5)	71.4 (n = 5)
% Blacks/African Americans	28.7 (n = 10)	—	28.6 (n = 2)	27.3 (n = 3)	37.5 (n = 3)	28.6 (n = 2)
% Other	6.4 (n = 2)	—	14.3 (n = 1)	9.1 (n = 1)	—	—
Level of education							<0.01
% Kindergarten	30.6 (n = 11)	33.3 (n = 1)	33.3 (n = 2)	36.4 (n = 4)	50.0 (n = 4)	—
% Elementary/grade	40.2 (n = 13)	—	33.3 (n = 2)	45.4 (n = 5)	37.5 (n = 3)	42.9 (n = 3)
% Junior high/middle	26.9 (n = 10)	66.7 (n = 2)	16.7 (n = 1)	18.2 (n = 2)	12.5 (n = 1)	57.1(n = 4)
Annual household income	50.3 (n = 17)	33.3 (n = 1)	66.7 (n = 4)	54.6 (n = 6)	37.5 (n = 3)	42.9 (n = 3)	0.82
Less than $10,000
Living situation							0.24
% Living with one parent	55.1 (n = 20)	33.3 (n = 1)	71.4 (n = 5)	45.4 (n = 5)	50.0 (n = 4)	71.4 (n = 5)
% Living with both parents	35.9 (n = 12)	66.7 (n = 2)	14.3 (n = 1)	54.6 (n = 6)	25.0 (n = 2)	14.3 (n = 1)
DSM‐5 diagnoses (%)
% Disruptive Mood Dysregulation Disorder	24.9 (n = 9)	—	42.9 (n = 3)	27.3 (n = 3)	37.5 (n = 3)	—	0.01
% Major Depressive Disorder	12.1 (n = 3)	—	—	27.3 (n = 3)	—	—	0.06
% Separation Anxiety Disorder	27.6 (n = 10)	33.3 (n = 1)	14.3 (n = 1)	36.4 (n = 4)	50.0 (n = 4)	—	0.01
% Attention Deficit Hyperactivity Disorder	77.6 (n = 29)	100.0 (n = 3)	71.4 (n = 5)	72.7 (n = 8)	100.0 (n = 8)	71.4 (n = 5)	0.05

The feasibility of implementing a stratified sampling protocol

To make the stratification possible, RCs continually reviewed the PRSFs to keep abreast of rate of patient flow while being mindful of clinicians' available time slots and last minute schedule changes. This information was crucial to collect information for calculating the sampling weights, identifying eligible patients, identifying their eligible diagnostic strata, assigning each to the appropriate single stratum for sampling, and scheduling study visits within a two week window to reduce the likelihood of remission or onset of a new disorder. This component of the study was most challenging. The coordination of patient and clinician schedules was difficult given limited clinician time slots and priority that had to be given to scheduling emergency and routine patient visits. This was more problematic in the child pilot because both the parent/guardian's work schedule and the child's school schedule had to be considered and coordinated with clinician availability.

In the adult pilot, clinicians provided an average of 20 time slots per week. Since each clinical diagnostic interview took approximately 60 minutes, the 20 time slots per week meant that about 20 patients could be seen weekly. For the child study, each participating clinician was asked to provide 11 90‐minute time slots for the study period. Not all clinicians were able to do so, and others provided more. At the outset, available time slots were quickly filled with baseline visits, resulting in follow‐up visits occurring outside the two‐week time frame for 41.1% of the adult patients and 5.6% of the child patients. Lack of open slots was also problematic when parents or guardians sought to reschedule appointments missed due to family illnesses, inclement weather, or school testing (5.3% of the patients).

Two part‐time RCs with a one‐day overlap in schedule were hired for the pilot studies. This level of staffing proved insufficient and impacted the ability to implement the stratified sampling design. Total RC time spent per study visit ranged from 45–60 minutes for the adult pilot and 45–75 minutes for the child pilot. After the study was implemented, it became clear that more RC time was needed to provide support for patients and clinicians during study visits and complete other study‐related tasks.

Logistical issues

Clinician recruitment and retention

Twenty‐two licensed clinicians in the adult pilot (five board certified psychiatrists [MDs], five licensed clinical social workers [LCSW‐Cs], and eight licensed professional counselors [LCPCs]) and 19 in the child pilot (six board certified child and adolescent psychiatrists [MDs], five LCSW‐Cs, six LCPCs, and two advanced practice registered nurses [APRNs]) were eligible to participate. Of the 22 clinicians (13 female, nine male) at the adult site, 10 (four LCSW‐C, four LCPC, two MDs) volunteered to participate. Most were female (60%), which reflected the proportion of female clinicians in the Adult Community Psychiatry Outpatient Program at JHMI. The clinicians had on average 12.3 years (SD = 6.1 years) in practice compared to an average 14.7 years (SD = 6.8 years) in practice, which is not statistically different (t = 0.9837; p = 0.34). All 10 clinicians participated in the study from start to completion, each conducting seven to 22 of the 136 diagnostic interviews for the 68 patients.

Of the 19 eligible clinicians (18 female, one male) in the child pilot, 14 (four LCSW‐Cs, three LCPCs, one APRN, five MDs) agreed to participate, but only 13 provided time slots. Of the 13 clinicians who provided time slots, 92% were female, which reflected the proportion of female clinicians in the clinic. The 13 clinicians had 11.5 mean years in practice (SD = 10.3 years), similar to the total sample of clinicians who were available to participate at the site (mean = 11.5; SD = 10.0: t = −0.003, p = 0.99), and participated in the study from start to completion. Each clinician conducted two to 16 of the 74 diagnostic interviews for the 38 patients.

Patient recruitment and retention

In the adult site, 615 currently symptomatic patients were seen during the study period, thus forming the sampling frame for the site (Figure 1). The distribution of current DSM‐IV diagnoses in the clinic population was: 33% MDD, 25.4% Schizophrenia, 22.4% BPD, 7.5% Schizoaffective Disorder, and 13.3% “Other” non‐targeted conditions. Ninety‐two eligible patients gave written informed consent (28 with MDD; 21 with Schizophrenia; 25 with BPD; nine with Schizoaffective Disorder; and nine with Other diagnosis). Of these, 14 patients were recruited but were ultimately unable to participate (13 = strata was already full; 1 = unable to reach to schedule the study visit). Of the 78 patients who were scheduled for the study, 10 missed their baseline visit (e.g. due to illness or other unexpected life events). The remaining 68 patients completed both study visits.

Figure 1.

Patient recruitment process for the DSM‐5 Adult Pilot Study at the Adult Community Psychiatry Outpatient Program in the Department of Psychiatry and Behavioral Sciences on the main medical campus, JHMI. (*Note: Numbers add to greater than 615 due to comorbidity.)

Figure 2 illustrates that 141 currently symptomatic child/adolescent patients were seen. Of these, 58 were currently symptomatic for the diagnoses of interest or the “Other” group, had a parent/guardian who could read and communicate in English, and expressed interest in the study. Nineteen could not be enrolled due to scheduling conflicts, hospitalization, or were withdrawn by their parents before the baseline visit. Written consent was obtained from 39 participants and guardians for the pilot study, of whom 38 completed Visit 1, but one patient was withdrawn by the study team. Two patients in the DMDD stratum did not attend Visit 2 due to schedule conflicts. Therefore, 35 patients completed Visits 1 and 2 in the child pilot study. This number was far short of what was projected and, like the adult pilot, indicated the need for a longer patient recruitment and data collection period. Comorbidity was an issue at both sites as many patients qualified for assignment to more than one diagnostic group (i.e. on average, each patient had two DSM‐IV diagnoses). However, each patient was assigned for sampling to only one group.

Figure 2.

Recruitment process and sampling of patients in the Child Pilot Study Site in the Child and Adolescent Outpatient Program in the Department of Psychiatry and Behavioral Sciences at Bayview Medical Center at JHMI. (*Note: Numbers add to greater than 141 due to high level of comorbidity.)

The feasibility of obtaining videotapes of 20% of the study's diagnostic interviews

A proposed goal of the DSM‐5 Field Trials was to videotape a random 20% of the clinical interviews per stratum to assess convergent validity and the feasibility of this was examined in the pilot studies. Of the 68 adult patients who completed the study, 38.2% consented to the videotaped component. The reasons for refusal were not documented. Of the 39 child patients, 17 (44.7%) consented to and completed the videotaped component. Those who refused either gave no reason or expressed some form of discomfort about being videotaped (e.g. “too shy”). The response rates for the videotaped component were based on the RCs seeking consent from all eligible patients.

The adequacy of clinician and research coordinator training

Seven of the 10 adult study clinicians completed training evaluation. All agreed or strongly agreed that the training session provided useful general information and that, with sufficient practice, they would be comfortable using the results of the patient‐ and/or informant‐completed measures and the REDCap system. Despite this, clinicians specifically suggested the single, 2–3 hour training session was not sufficient. They also felt that it was important that refresher training sessions be available for clinicians as necessary so as to reduce added burden on the RCs, who often had to assist clinicians while managing patient recruitment and workflow.

From the debriefing sessions at both sites, the pilot study clinicians reported that the mock interview session was very useful for the illustration of the diagnostic interview process and the procedure for entering information into the diagnostic checklist in the REDCap system during the training session. They encouraged the use of mock interviews in field trial training but recommended that the mock interviews be conducted by the site clinicians instead of the APA staff.

From the debriefing session, RCs reported the training enabled them to understand all study procedures and how to troubleshoot if problems arose with the REDCap system. However, they reported that the fact that the REDCap system components for patients, clinicians, and RCs were unlinked hindered management of the workflow of the study. Therefore, two weeks into the study they developed and used an electronic calendar, separate from the REDCap system, to coordinate the clinicians' schedules, patient clinic visits, and patient study visits, thereby enhancing recruitment and retention efforts.

Tests of the DSM‐5 REDCap system

The pilot studies suggested that reformatting and rewording of some of the REDCap forms was necessary (e.g. the Clinical Utility Questionnaire (CUQ)). Some clinicians suggested that the system utilize more automated features to assist the clinicians in the completion of the DSM‐5 diagnostic checklists. However, they also emphasized that the system still needed to preserve the role of clinician judgment in determining diagnosis and severity.

Problems in the newly developed research forms

The final implementation issue related to the originally proposed PRSF used in the adult version of the pilot study. Initially, the form included a single question about whether the patient was currently symptomatic for his or her diagnosis, without a clear definition of “currently symptomatic”. This item posed a problem for individuals with comorbidities since the RC had difficulties determining for which disorder the patient was currently symptomatic. It also posed a problem since clinicians relied on their own judgment as to the meaning of “currently symptomatic”. The PRSF did not include a section for the treating or intake clinician to print their names, which resulted in the RCs spending time trying to decipher clinicians' signatures if there was ambiguous information on the form. Modifications were made to the PRSF between the adult pilot study and the start of the child pilot study. This enabled the RCs for the child pilot study to more readily compare the completed number of PRSFs and the list of patients who were scheduled for clinic visits each day, which facilitated the recruitment process. The RCs indicated that the modified PRSF could be further modified to include information on patient's age, sex, and contact information to aid in the recruitment process.

Problems in the DSM‐5 proposals

Participating clinicians found incorporation of DSM‐5 cross‐cutting measures into their clinical evaluation relatively easy and found the information moderately to very helpful. However, from the debriefing session, the general consensus was that the information was more useful for some disorders than for others. In the adult pilot study, some clinicians indicated during the debriefing session that patients with psychotic disorders tended to over‐endorse items because of distortions or misunderstanding of the time‐frame of the cross‐cutting measures. For the child pilot study, clinicians reported under‐endorsement of depressive symptoms by patients with internalizing disorders. Clinicians thought that the children/adolescents had difficulty responding to questions about internalizing symptoms when they were presented as the first items on the questionnaire. It was recommended that the questionnaire be re‐ordered so that externalizing items precede the internalizing items.

The pilot study clinicians helped to identify diagnostic criteria that were problematic and might be in need of re‐wording or clarification. For example, Criterion D in the initial draft of the DSM‐5 proposal for ODD asked if the oppositional defiant “behaviors were confined to one setting or in more severe cases in multiple settings” in a single item/question, which was confusing. Similarly, they found the initially proposed Criterion F for DMDD provided a lower but no upper age limit, which implied that an individual over age 18 could be diagnosed with DMDD for the first time. In addition, Criterion H for DMDD was described as wordy and confusing. Clinicians' comments were provided to the DSM‐5 Work Groups for consideration.

Discussion

Pilot studies can give researchers warnings about procedural problems that warrant consideration prior to expending resources, such as (1) whether a study would fail to obtain adequate sample sizes, (2) whether the study protocol would be followed and which factors could affect protocol compliance, and (3) whether the study methods and instruments are easy to understand and use (van Teijlingen and Hundley, 2002; Arain et al., 2010). These advantages were clearly observed in the DSM‐5 pilot study. The results of these pilot studies allowed for the correction of a number of procedural issues associated with the planned DSM‐5 Field Trials and enhanced their feasibility and successful completion.

There were several main findings of the DSM‐5 pilot study. First, clinicians and patients were willing to participate in the pilot study across both sites, which indicated that the planned field trials were feasible in a busy clinical setting. However, the study fell short of the targeted patient recruitment goal. The lower than expected recruitment rate in the pilot study, even with an extension of the recruitment period, indicated that recruitment rates would be a problem in the field trials, especially in the child sites, and that efforts would be needed to ramp up recruitment. For instance, the patient recruitment and data collection period proposed for the DSM‐5 Field Trials clearly needed to be lengthened. As a result, the initial 4.5‐month recruitment period for the field trial was extended by an additional 3.5 months. Importantly, despite difficulties in recruitment rates, the pilot studies demonstrated high patient retention rates after the first study visit. This suggested the field trials should utilize similar techniques to obtain a high retention rate. Therefore, we encouraged field trial sites to implement such strategies as scheduling baseline and first study visits as close together as possible; sending multiple patient appointment reminders; and scheduling evening and weekend appointments to accommodate patients' schedules. The second important finding is that videotaping of interviews at both sites exceeded the 20% target. However, due to individuals changing their mind about the videotaping, the field trials targeted videotaping greater than 20% of all interviews to obtain the desired amount.

After training sessions, adult pilot staff felt prepared to implement the pilot study, which was informative for the planned training session for the clinicians in the field trials. Furthermore, feedback from the pilot study clinicians was used to improve the training for clinicians in the DSM‐5 Field Trials. For instance, clinicians' feedback on the insufficiency of a single two‐to‐three‐hour training session and the need for refresher sessions resulted in a two‐part training session for all field trial clinicians (Clarke et al., 2013) and refresher training sessions via Webinars and training videos. Feedback from the pilot study clinicians was also used to improve clinician component of the REDCap system that was used in the field trials. For instance, based on the pilot study clinicians suggestion that the system be more automated while preserving the ability for diagnosis and rating of severity to be based on clinical judgment, the clinician component of the system with the DSM‐5 diagnostic checklist was equipped with more automated features, such as fields that calculated whether a certain number of criteria were endorsed. Final decision as to whether the diagnostic criteria were met for a particular diagnosis was not automated but left to the clinician to answer. Fields were also added so that clinicians could explain discrepancies between their own impressions and checklist results.

Lessons learned from the RCs' experiences in the DSM‐5 pilot studies influenced development of the RC component of the DSM‐5 Field Trial REDCap system and modifications to the patient and clinician components. For instance, efforts were made to ensure that the three components of the REDCap system were linked and operational. The system was also equipped with a function that allowed the tracking of patients from recruitment to completion of the study. Feedback from the RCs was also used in the development of the training session for RCs across DSM‐5 Field Trial sites. In addition, to address the issue regarding the PRSF, modifications were made prior to the start of the field trials that allowed the treating or intake clinician who completed the form on a patient to comment on whether or not a patient was symptomatic for each disorder separately and to indicate the patient's age, sex, interest, and contact information, which was useful to the field trial RCs. These changes in procedure allowed for greater ease in collecting data during the field trials.

Incompatibilities in clinician and patient availability for study visits also proved to be an obstacle. To address the workflow difficulties of matching clinician time slots with patient availability, more flexible time slots were necessary, which could be accomplished by having a larger pool of study clinicians. In addition, recruitment of clinicians for hard‐to‐fill slots over the course of the week and on weekends was necessary especially for child sites. In the field trials, evening and weekend clinics were encouraged and implemented in some field trial sites.

Additionally, results indicated that the time demands imposed on the RCs to maintain the workflow of the study were far greater than originally anticipated. As a result, across participating field trial sites additional RC time was funded to ensure that patient enrollment and study visits were completed in a timely manner. This was especially necessary to accommodate RC coverage for the evening and weekend clinics that were implemented.

Results from the pilot studies helped the DSM‐5 Work Groups, Study Groups, and Research Group determine if the proposed measures and diagnostic criteria were clinically useful to clinicians and patients, were easy to implement in busy clinical setting, and whether any modifications were needed. For instance, results showed that a majority of adults and parent/guardians felt that the cross cutting symptom measures would be clinically helpful and liked that the measures were completed electronically. In addition, clinicians found the cross cutting measures to be helpful in their diagnostic decision‐making, though this varied across disorders. In addition, suggestions for the re‐ordering of some of the items on the child and parent/guardian version of the measure were presented to the DSM‐5 Diagnostic Assessment Instruments Study Group and Child and Adolescent Disorders Work Group for consideration and were accepted. These changes were made prior to the use of the instruments in the field trials. Specifically, in the versions of the child‐ and parent‐rated cross cutting measures that were used in the field trials, the items on inattentiveness, somatic symptoms, and sleep disturbance represented the first four items on the measures (APA, 2013, pp. 738–741 and online at http://www.psychiatry.org/dsm5) instead of the depression, anxiety, and mania items, as was initially proposed.

The pilot study clinicians' feedback also helped identify problematic diagnostic criteria before the start of the field trials. For example, Criterion D for ODD, which initially asked if the oppositional‐defiant “behaviors were confined to one setting or in more severe cases in multiple settings” in a single item/question, was clarified and treated as two separate questions (Criteria D1 [i.e. behaviors confined to one setting] and D2 [i.e. behaviors observed in multiple settings]) before the field trial was initiated. Similarly, Criterion F for DMDD was re‐worded and clarified to include an upper age limit of 18 years before the start of the field trials.

In summary, the pilot studies allowed for the refinement of the DSM‐5 Field Trial study protocols that improved recruitment, training, and implementation and enhanced its feasibility and successful completion. These results highlight the need and benefits of well‐designed pilot studies in the planning of large multisite studies in psychiatry and other fields of study.