Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2013 Aug 19.
Published in final edited form as: AIDS Care. 2009 Jan;21(1):103–108. doi: 10.1080/09540120802033864

Assessing the need for an online decision-support tool to promote evidence-based practices of psychosocial counseling in HIV care

Rita Kukafka a,b,*, Mari Millery b, Connie Chan a, William LaRock c, Suzanne Bakken a,c
PMCID: PMC3746314  NIHMSID: NIHMS489429  PMID: 19085226

Abstract

Psychosocial counselors have a vital and challenging role in supporting persons living with HIV/AIDS (PLWH/A) to better manage their disease. However, gaps in training, education, and skills limit the effectiveness of counselors’ efforts. We propose that the use of a decision-support tool for counselors at the point of care can support them in their work as well as help alleviate many training and practice gaps. Decision-support tools aimed at reducing knowledge and practice gaps are used extensively to assist clinical providers at the point of care; however, there is a need for decision-support tools designed specifically for HIV/AIDS counselors. To identify requirements for such a tool, we conducted a needs assessment through interviews of 19 HIV/AIDS clinic counselors who provide 20 or more hours per week of psychosocial support to PLWH/A. The assessment explored their education and training backgrounds, the extent to which evidence-based practices are implemented, and how a decision-support tool can support counselor work practices. Qualitative analysis was organized around seven main categories: counselor characteristics, patient characteristics, barriers, definitions of key concepts, use of guidelines, client assessments, and resources. The resulting coding schemes revealed knowledge and practice gaps among the interviewees, as well as barriers and challenges of counseling. Education and training background of the counseling staff varied widely. When asked to define five key concepts related to HIV counseling, 26–47% of respondents were unable to articulate an adequate definition. Less than half of the interviewees recalled sources of guidelines used in their work and specific models of care introduced during trainings. Interviews identified environmental barriers, language and literacy, patient education, and patient communication as the most prominent challenges to counseling work. The results from this study inform the need for and development of a decision-support tool to support the training and practices of HIV/AIDS counselors.

Keywords: care, decision-support system, evidence-based medicine, counseling, needs assessment

Supporting the psychosocial needs of persons living with HIV/AIDS (PLWH/A) requires a wide range of knowledge and comfort with counseling on sensitive and complex issues. Counselors play a vital role in supporting and promoting treatment adherence and effective disease management among PLWH/A. In this paper, “counselor” comprises health professionals who provide 20 or more hours per week of psychosocial support and/or other counseling services to PLWH/A. These counseling services include, but are not limited to, addressing mental illnesses, adherence counseling, substance abuse counseling, case management, and social work services. Although diverse, these counseling roles share a common base of information needs around HIV, psychosocial issues, and best practices (Britton, Rak, Cimini, & Shepherd, 1999; Hunt, 1996). Training, education, and skill gaps in counselor practices can potentially be filled by a decision-support tool. Decision-support tools are interactive computer-based tools that aid users in making decisions, by providing a knowledge base constructed from research-based and practice-based evidence.

Different levels of knowledge can be embedded into the tool to meet the diverse information needs of those in counseling roles. We report findings from a study to identify the need for and the requirements of a decision-support tool to enhance psychosocial counseling efforts in HIV/AIDS care.

Education and training gaps

A foundational base of knowledge, confidence, skills, and training has been shown to be a necessary requisite of effective psychosocial counseling (Burke & Fair, 2003; Kerwin et al., 2006), and the same has been found specifically for counseling of HIV patients (Britton et al., 1999; Reif, Smith, & Golin, 2003). However, studies reveal continued training and education needs (Burke & Fair, 2003; House, Eicken, & Gray, 1995; Hunt, 1996; Shelton, Golin, Smith, Eng, & Kaplan, 2006; The Annapolis Coalition on the Behavioral Health Workforce, 2007), and numerous barriers that complicate counselors’ everyday work (Shelton et al., 2006). Studies have additionally documented that counselors are interested in and recognize the need for more HIV/AIDS training (House et al., 1995; Shelton et al., 2006). Access to up-to-date and tailored evidence-based practice guidelines can supplement counselors’ knowledge of relevant issues and recommended best practices. However, low rates of adherence to evidence-based practices can result from counselors’ perception of the gap between research and practice, as well as from practice barriers (Santisteban et al., 2006). Providers generally support Internet-based tools to learn more about relevant guidelines (Campbell, Catlin, & Melchert, 2003) and have favorable views toward Internet-based HIV/AIDS-related counseling tools (DeGuzman & Ross, 1999).

Role of decision-support tools in providing access to training and guidelines

Decision-support systems can facilitate access to training and guideline resources that raise awareness of evidence-based practices. Most decision-support systems are designed around the needs of physicians and other medical professionals, even though other staff within the multidisciplinary care team could benefit greatly from them as well. In particular, there is a need for decision-support tools for counseling professionals.

Examples of support systems for health professionals have been successfully implemented and evaluated (Davis, 1998; Dzewaltowski, Glasgow, Klesges, Estabrooks, & Brock, 2004; Morgenstern, Morgan, McCrady, Keller, & Carroll, 2001; Riley et al., 2007; Sholomskas & Carroll, 2006; Sim et al., 2002; Weingardt, 2004), providing evidence that technology tools can indeed help bridge the gap between research and practice. These studies document valuable lessons learned that should inform tool design efforts, particularly the development of tools for counselors.

HIV Tailored Information Provider Services (TIPS) (Messeri, Kukafka, Millery, Schluter, & Ferat, 2003) is a decision-support tool to provide web-based access to relevant information, resources, and case-specific guidelines, to promote evidence based practices. The findings from a four-year randomized trial to evaluate HIV TIPS revealed that adherence counselors need decision support and may be more inclined to use such a support tool than prescribing providers (Messeri et al., 2005). These findings motivated the current needs assessment study that would inform the enhancement of the HIV TIPS functionality to better support the needs and requirements of HIV counselors. The needs assessment was designed to explore the extent that counseling work practices are informed by theory and evidence-based practices and to reveal knowledge gaps, barriers and challenges to counseling that a decision-support tool can potentially address.

Methods

Nineteen semi-structured interviews were conducted with a sample of counseling staff working with PLWH/A, recruited through HIV care agencies that participated in an evaluation study of the HIV TIPS tool (Messeri et al., 2005). A letter introducing the interview study was sent to medical directors at 13 agencies to invite counseling staff to participate in the interviews. The plan was to conduct approximately 20 interviews, which were expected to yield saturation for the qualitative data analyses. The sample consists of the first 19 counselors reached through the recruitment efforts. The interview questions were developed by a team of investigators with expertise in HIV/AIDS counseling, nursing, public health, and informatics. The questions were reviewed by HIV/AIDS domain experts and pre-tested on a small sample of counselors. The Columbia University Institutional Review Board approved the conduct of this study.

The interview questions were designed to elicit information about the counselors’ educational and training backgrounds and the degree to which their work practices are based on models grounded in theory and evidence-based practices. The seven major categories of interest include: counselor characteristics, patient characteristics, barriers to counseling work, definitions of key concepts related to psychosocial HIV counseling, use of and source of guidelines, client assessments, and useful resources that could be incorporated in a decision-support tool. The results from the “client assessments” category were further coded in terms of the other major categories of “guidelines” “resources” and “barriers”, and hence have been reported in these categories. Interviews were audio taped, transcribed, then coded using Atlas.ti software (Muhr, 2004).

The first broad round of coding was based on the seven major categories of interest listed above. Researchers collaboratively created sub-categories in a second round of coding to capture specific themes within each of the major categories.

In the “definitions” category, interview respondents were asked to define in their own words five key concepts related to psychosocial HIV counseling: harm reduction, motivational interviewing, depression, stigma, and stages of readiness. The selection of these particular concepts was based on what the investigators viewed as key concepts related to psychosocial HIV counseling, reflecting their own professional experience. A more detailed coding scheme to assess the completeness and accuracy of the definitions was developed through investigator’s knowledge originating from formal textbook type definitions: “inadequate” (score of 1), “partially adequate” (2), and “fully adequate” (3). An additional dimension, labeled as “practice-based” and coded separately from the three levels above, expanded the coding to accommodate responses where the concepts were defined using case examples and illustrations based on work experiences. Three researchers independently ranked the concept definitions based on the codebook, then came together to discuss the level of agreement in their rankings. Post discussion kappas were calculated to assess inter-rater agreement.

Results

Counselor characteristics

Half of the counselor sample (n=19) had completed a graduate degree (33%) or professional degree (17%). Graduate degrees included Master’s degrees in public health, counseling, psychology, and social work, and professional degrees included physician and nurse. The other half consisted of 22% with college degrees, 22% with some college, and 6% (one person) with high school education.

Professional roles of the interviewees varied; the titles included: doctor, HIV nurse, nutritionist, social worker, case manager, physician assistant, addiction specialist, community outreach coordinator, HIV tester, HIV test counselor, and harm reduction counselor. Participants described additional responsibilities that extended beyond their titles to more fully reflect work roles; these responsibilities included: addiction counseling, supervising clinic compliance, substance abuse counseling, treatment adherence counseling, intake for HIV population, mental health assessment, patient education, mobile health counseling services, and social services support.

When asked about training received to work with PLWH/A, each participant listed from one to three sources of training, for a total of 34 references to training sources. The primary source, 44% of the 34 sources listed, was institutional training sessions, which were provided, for example, by the AIDS Institute of the New York State Department of Health. Work experience, which constituted 35% of Training source references, was the second most frequent. In addition, 21% of participants listed educational institutions or degree-granting training programs. The interview also asked about training in specific counseling approaches in four areas: 58% of participants reported training on mental health assessment, 79% on substance abuse assessment, 74% on adherence counseling, and 72% on prevention Counseling with HIV-positive clients.

Patient characteristics

Counselors were asked about the patient populations they serve in the clinics where they work. The patients were primarily African Americans and Latinos and also included some Caribbeans, Asians, and Caucasians. Age, severity of illness, and treatment adherence levels of patients ranged widely. A majority of counselors (68%) estimated low or very low computer literacy among their patients; most estimated 10–30% of patients being familiar with using computers.

Guidelines and models

Almost half of the counselors (nine of 19) reported using specific guidelines in their work. As for their awareness of sources of specific guidelines, seven counselors cited New York State AIDS Institute, four cited US Department of Health and Human Services (DHHS), and one cited the Centers for Disease Control and Prevention (CDC) as sources of guidelines. Regarding introduction to specific models during trainings, 53% did not recall learning any specific models, 26% indicated “yes” but could not identify any specific model, and 21% identified a specific model, with stages of change the most frequently cited example.

Barriers

The issues most often cited as barriers to counseling work were coded under the following themes: environment, language and literacy, patient education, and patient communication. Fifty-three percent of interviewees mentioned environmental barriers, including problems of limited time and space, access to up-to-date resources, and workload. For example, one interviewee highlighted the overload of forms and unwillingness to take on additional tasks: “I’m bombarded with that already, I’m not willing to see another form”.

Sixty-three percent of interviewees cited language and literacy barriers, which refer to the problems of low patient literacy or necessity to communicate with patients in languages other than English. A lack of patient education resources to help patients stay informed and to allay patient misconceptions was mentioned by 21% of interviewees.

The most common barriers were in patient communication, indicated in 16 of 19 (84%) interviews. This barrier includes developing an open and trusting relationship with the patient, and engaging and motivating patients. The following example reflects the challenge of honesty in communication:

So it’s hard because you can’t say you’re effective in doing this if the people’s behavior is not changing and they’re still reporting back to you that their behaviors are changing. So the only battle I think I won is one of honesty – that at least they’re honest to tell me that what I’m doing isn’t working so well.

Another response illustrates the barrier of patient attitude in communication:

Um, really the guidelines is based on patient needs. And um – ‘cause they’re in-in the driver’s seat in a sense – where, um … you can’t do anything with a patient if a patient is not gonna allow you, or is not gonna be receptive to it. ‘Cause you can sit here – they’ll just “yes” you all day, and nothing will get done.

Definitions of concepts

The distribution of scores for each of the five concepts is presented in Table 1. The majority of definitions were found to be either “inadequate” (score of 1) or “partially adequate” (score of 2). Harm reduction had the most “fully adequate” (score of 3) definitions (44%), while stigma and stages of readiness had 33% each, and motivational interviewing and depression had the least (16% each). Motivational interviewing had the highest proportion of inadequate definitions (47%). As also shown in Table 1, 21–47% of the definitions, depending on concept, were coded as “practice based”. Depression (47%) and harm reduction (44%) had the highest rates of practice-based definitions. Following is an example of a definition of harm reduction coded as “practice-based” and “fully adequate”:

Harm reduction is th-the client, um, comes and talk to me and says, “I got high yesterday. I only used … two bags [of heroin]”, when he’s usually using four. And y’know, I will give him suggestions about needle exchange, and I will acknowledge, most of all, the accomplishment that they have done.

Table 1.

Scoring of definitions

Score
1
Inadequate		 2
Partially Adequate		 3
Fully Adequate		 P
Practice-based
Harm reduction (n=18) 6 (33%) 4 (22%) 8 (44%) 8 (44%)
Motivational interviewing( n=19) 9 (47%) 7 (37%) 3 (16%) 4 (21%)
Depression (n=19) 5 (26%) 11 (58%) 3 (16%) 9 (47%)
Stigma (n=18) 6 (33%) 6 (33%) 6 (33%) 4 (22%)
Stages of readiness (n=18) 6 (33%) 6 (33%) 6 (33%) 6 (33%)
Open in a new tab

Due to rounding, percentage totals for each definition do not add up to 100%

The kappa coefficients for interrater reliability varied from 0.92 to 0.94 across the five concepts, indicating a high level of agreement among the three coders.

The associations between education level and definition scores were examined. Education level was divided into two groups: high school up to some college vs. college/graduate/professional degrees. Higher education appeared to be associated with higher definition scores. For example, 71% of the lower education group vs. 9% of the higher education group gave inadequate definitions for stigma. Chi-square statistics were calculated to test these associations and found significant for stigma, χ2(2, N=18)=7.5, p=0.02, and stages of readiness, χ2(2, N=18)=7.5, p=0.02. The other three concepts showed the same trend, although not statistically significant. Practice-based definitions were found to be equally frequent across the two education levels.

Although at some point most counselors may have been exposed to the theories and research behind best practices, our findings suggest they may not always be able to recall them to carry out at the point of care. The following quote is from a counselor who had received training but struggled to articulate a definition of motivational interviewing:

Motivational Interviewing – we actually took a training on that … um … where we – with the Motivational Interviewing on – to the topic “Smoking.” That’s what the – we did. Um, kind of like to … get the patients to change their behaviors with smoking or condom use, or alcohol use, drug use. Um, trying to reduce … more risk

This definition was scored as “partially adequate”.

Resources

Suggestions for augmenting existing resources were offered for format and content. Respondents proposed that online tools, such as forms, assessment tools and listings of local referral resources, would be helpful. They suggested that simple and attractive diagrams would be engaging and descriptive in presenting information. A need for resources was highlighted for three content areas: housing assistance, patient education materials and updates on HIV medications. Many responses referred to information resources already used by counselors that could be made web-based.

Discussion

The results from the interviews reveal knowledge and training gaps among counseling staff working with PLWH/A as well as gaps in implementation of evidence-based practices. Overall, counselors demonstrated deficiency in definitions of basic theoretical concepts and limited utilization and awareness of guidelines. Levels of education and training appear to be associated with gaps in knowledge. These results echo the educational, training, and practice gaps revealed in other studies (Burke & Fair, 2003; House et al., 1995; Hunt, 1996; Shelton et al., 2006; The Annapolis Coalition on the Behavioral Health Workforce, 2007).

A substantial proportion of counselors reported having received no training to conduct four tasks central to HIV counseling-substance abuse assessment, prevention counseling, mental health assessment, and adherence counseling. This is consistent with the fact that very limited federal and local resources are dedicated to continuing education of counseling staff, in contrast with medical professionals who are targeted by such programs as the AIDS Education and Training Centers funded by the Ryan White Program. Additionally, many interviewees were unable to recall models presented in trainings, which suggests that current training efforts may be unsustainable or insufficiently effective. The results suggest that training has been unable to provide counselors with adequate skills and knowledge to perform their work and apply evidence-based practices.

Low numbers of “fully adequate” definitions is of particular concern because the five concepts used in this study are central to HIV-related counseling. The assessment revealed that instead of textbook style abstract definitions, counseling staff often articulated their definitions of theory-based concepts, such as stages of readiness, by using examples from their own work practice. This indicates that the way interviewees think about these concepts is embedded in case-based practice, which suggests that case-based presentation may be an effective way of structuring such concepts within a decision-support tool.

Our findings are consistent with the literature showing that counselors are supportive of computer-based tools to fill resource gaps and to aid their work with patients, specifically through access to up-to-date and tailored evidence-based practice guidelines and counseling tools (Campbell et al., 2003; DeGuzman & Ross, 1999). A decision-support tool can help bridge the gaps uncovered by this needs assessment. Our needs assessment has informed how to further develop resources such as HIV TIPS (http://www.hivtips.org), which are currently tailored for medical providers, to extend functionality to better serve HIV/AIDS counselors. The barriers and resource challenges identified by interviewees can help inform the design of necessary functionalities. A tool for counselors should include online assessment tools, guidelines, local treatment and social support resources, patient education resources, and other functionalities as voiced in the interview responses. Tool development should also address the language and literacy barriers cited by the counselors. There are limitations to the extent to which our assessment results can be generalized. As the interview sample was limited to the New York City area, some of the barriers and resource challenges could be specific to the location. The results are based on interviews with a convenience sample of 19 participants, which limits generalizability. Furthermore, the interview method is limited in scope – no observations were conducted to examine how counselors actually practice and implement evidence and knowledge in their everyday work. The interview method is also limiting because individuals may not be able to articulate their knowledge when defining a concept or recalling knowledge.

This needs assessment highlights the need for decision-support tools in the public health domain, particularly among counselors working with PLWH/A. By providing enhanced access to relevant guidelines at the point of care, such a tool can help supplement knowledge gaps and facilitate implementation of evidence-based practices. The results indicate that a similar study conducted on a larger scale is justified and could reveal additional insights into these needs, as well as further inform the development of an effective decision-support tool for psychosocial counseling in HIV care settings.

Acknowledgements

The survey conducted in this study was funded in part by The Center for Interdisciplinary Research on Antimicrobial Resistance, CIRAR, http://www.cumc.columbia.edu/dept/nursing/CIRAR/, funded by The National Center for Research Resources, P20 RR020616. The research team expresses thanks to Peter Messeri, the Principal Investigator of the HIV TIPS project, all the participants in this study, and the AIDS Education and Training Center. Connie Chan is supported by NLM predoctoral fellowship T15-LM007079.

Footnotes

Publisher's Disclaimer: Full terms and conditions of use: http://www.tandfonline.com/page/terms-and-conditions

This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae, and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand, or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.

References

  1. Britton PJ, Rak CF, Cimini KT, Shepherd JB. HIV/AIDS education for counselors: Efficacy of training. Counselor Education and Supervision. 1999;39(1):53–65. [Google Scholar]
  2. Burke LE, Fair J. Promoting prevention: Skill sets and attributes of health care providers who deliver behavioral interventions. Journal of Cardiovascular Nursing. 2003;18(4):256–266. doi: 10.1097/00005082-200309000-00004. [DOI] [PubMed] [Google Scholar]
  3. Campbell TC, Catlin LA, Melchert TP. Alcohol and other drug abuse counselers’ attitudes and resources for integrating research and practice. Journal of Drug Education. 2003;33(3):307–323. doi: 10.2190/VUHN-X36D-KH56-2P77. [DOI] [PubMed] [Google Scholar]
  4. Davis KA. Development and evaluation of computer-based training for pre/post human immunodeficiency virus test counseling. Journal of Nursing Staff Development. 1998;14(2):69–72. [PubMed] [Google Scholar]
  5. DeGuzman MA, Ross MW. Assessing the application of HIV and AIDS related education and counseling on the Internet. Patient Education & Counseling. 1999;36(3):209–228. doi: 10.1016/s0738-3991(98)00096-2. [DOI] [PubMed] [Google Scholar]
  6. Dzewaltowski DA, Glasgow RE, Klesges LM, Estabrooks PA, Brock E. RE-AIM: Evidence based standards and a web resource to improve translation of research into practice. Annals of Behavioral Medicine. 2004;28(2):75–80. doi: 10.1207/s15324796abm2802_1. [DOI] [PubMed] [Google Scholar]
  7. House RM, Eicken S, Gray LA. A national survey of AIDS training in counselor education programs. Journal of Counseling & Development. 1995;74:5–11. [Google Scholar]
  8. Hunt B. HIV/AIDS training in CACREP-approved counselor education programs. Journal of Counseling & Development. 1996;74:295–299. [Google Scholar]
  9. Kerwin ME, Walker-Smith K, Kirby KC, Kerwin ME, Walker-Smith K, Kirby KC. Comparative analysis of state requirements for the training of substance abuse and mental health counselors.[ see comment] Journal of Substance Abuse Treatment. 2006;30(3):173–181. doi: 10.1016/j.jsat.2005.11.004. [DOI] [PubMed] [Google Scholar]
  10. Messeri P, Ancker J, Kukafka R, Millery MP, Nader N, Schluter D, et al. Organizational and provider factors influencing use of an internet-based clinical support system; Paper presented at the American Public Health Association Annual Meeting; December; Philadelphia, PA. 2005. [Google Scholar]
  11. Messeri P, Kukafka R, Millery MP, Schluter D, Ferat R. Tailored information provider service: An interactive, web-based system to support clinical management of HIV; Paper presented at the American Public Health Association Annual Meeting; November; San Francisco, CA. 2003. [Google Scholar]
  12. Morgenstern J, Morgan TJ, McCrady BS, Keller DS, Carroll KM. Manual-guided cognitive- behavioral therapy training: A promising method for disseminating empirically supported abuse treatments to the practice community. Psychology of Addictive Behaviors. 2001;15(2):83–88. [PubMed] [Google Scholar]
  13. Muhr T. User’s manual for ATLAS.ti 5.0 (version 5.0) Berlin, Germany: ATLAS.ti Scientific Software Development GmbH; 2004. [Google Scholar]
  14. Reif S, Smith S, Golin CE. Medication adherence practices of HIV/AIDS case managers: A statewide survey in North Carolina. AIDS Patient Care and STDs. 2003;17(9):471–481. doi: 10.1089/108729103322395500. [DOI] [PubMed] [Google Scholar]
  15. Riley WT, Schumann MF, Forman-Hoffman VL, Mihm P, Applegate BW, Asif O. Responses of practicing psychologists to a web site developed to promote empirically supported treatments. Professional Psychology-Research and Practice. 2007;38(1):44–53. [Google Scholar]
  16. Santisteban D, Vega RR, Suarez-Morales L, Santisteban D, Vega RR, Suarez-Morales L. Utilizing dissemination findings to help understand and bridge the research and practice gap in the treatment of substance abuse disorders in Hispanic populations. Drug & Alcohol Dependence. 2006;84(Suppl. 1):S94–S101. doi: 10.1016/j.drugalcdep.2006.05.011. [DOI] [PubMed] [Google Scholar]
  17. Shelton RC, Golin CE, Smith SR, Eng E, Kaplan A. Role of the HIV/AIDS case manager: Analysis of a case management adherence training and coordination program in North Carolina. AIDS Patient Care STDS. 2006;20(3):193–204. doi: 10.1089/apc.2006.20.193. [DOI] [PubMed] [Google Scholar]
  18. Sholomskas DE, Carroll KM. One small step for manuals: Computer-assisted training in twelve-step facilitation. Journal of Studies on Alcohol. 2006;67(6):939–945. doi: 10.15288/jsa.2006.67.939. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Sim I, Sanders GD, McDonald KM, Sim I, Sanders GD, McDonald KM. Evidence-based practice for mere mortals: The role of informatics and health services research. Journal of General Internal Medicine. 2002;17(4):302–308. doi: 10.1046/j.1525-1497.2002.10518.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. The Annapolis Coalition on the Behavioral Health Workforce. An action plan for behavioral health workforce development. Annapolis, MD: Author; 2007. [Google Scholar]
  21. Weingardt KR. The role of instructional design and technology in the dissemination of empirically supported, manual-based therapies. Clinical Psychology- Science and Practice. 2004;11(3):313–331. [Google Scholar]

RESOURCES