Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2013 Jun 1.
Published in final edited form as: J Subst Abuse Treat. 2011 Nov 23;42(4):356–365. doi: 10.1016/j.jsat.2011.09.004

Is Exposure to an Effective Contingency Management Intervention Associated with More Positive Provider Beliefs?

Kimberly C Kirby a,b,*, Carolyn M Carpenedo a, Maxine L Stitzer c, Karen L Dugosh a, Nancy M Petry d, John M Roll e, Michael E Saladin f, Allan J Cohen g, John Hamilton h, Karen Reese i, Gina R Sillo j, Patricia Quinn Stabile k, Robert C Sterling l
PMCID: PMC3319812  NIHMSID: NIHMS340520  PMID: 22116009

Abstract

This study empirically examined opinions of treatment providers regarding Contingency Management (CM) programs while controlling for experience with a specific efficacious CM program. In addition to empirically describing provider opinions, we examined whether the opinions of providers at the sites that implemented the CM program were more positive than those of matched providers at sites that did not implement it. Participants from 7 CM treatment sites (n = 76) and 7 matched non-participating sites (n = 69) within the same nodes of NIDA's Clinical Trials Network completed the Provider Survey of Incentives (PSI), which assesses positive and negative beliefs about incentive programs. An intent-to-treat analysis found no differences in the PSI summary scores of providers in CM program vs. matched sites, but correcting for experience with tangible incentives showed significant differences, with providers from CM sites reporting more positive opinions than those from matched sites. Some differences were found in opinions regarding costs of incentives and these generally indicated that participants from CM sites were more likely to see the costs as worthwhile. The results from the study suggest that exposing community treatment providers to incentive programs may itself be an effective strategy in prompting the dissemination of CM interventions.

Keywords: Contingency Management, Treatment Dissemination, Translational Research, Cocaine Dependence, Methamphetamine Use Disorders

1. Introduction

The efficacy of contingency management (CM) or incentive programs that provide tangible incentives contingent upon providing objective evidence of drug abstinence is documented in controlled clinical trials (see Higgins et al., 2002, 2004 for reviews) and has been recognized in independent reviews (Chambless & Ollendick, 2001; McGovern & Carroll, 2003) and meta-analyses(e.g., Castells et al., 2009; Lussier et al., 2006) of empirically-supported treatments for drug use disorders. Still, community treatment programs in the United States have been reluctant to adopt these programs (cf. McGovern et al., 2004; Willenbring et al., 2004). In one survey of treatment providers (McGovern et al., 2004), 38% of clinic directors and 48% of clinicians reported that they were not familiar with CM and were less ready to adopt it in comparison to four of five other non-pharmacological treatments (Motivational Interviewing, Cognitive Behavior Therapy, Relapse Prevention and Twelve-step Facilitation). In other surveys (Herbeck et al., 2008), 27% of treatment providers reported any use of CM and only 6% (Willenbring et al., 2004) indicated that they had a high level of CM use in their clinic. Similarly, we found that only 17% of treatment providers reported having any prior experience with tangible incentive programs, but over half (54%) indicated that they would be willing to have their clinic add an incentive treatment that provided tangible rewards (e.g., prizes, gift certificates; Kirby et al., 2006). A similar Australian survey found that very few providers reported experience with tangible incentive programs and fewer than half (42%) indicated willingness to adopt one (Ritter & Cameron, 2007). More recently, in a survey of treatment providers across Pennsylvania, we found that fewer treatment providers believed that CM should be implemented or were willing to implement it compared to two other empirically-based treatments (Relapse Prevention and Motivational Interviewing) and to twelve-step approaches (Benishek et al., 2010).

Researchers have suggested that barriers to implementing CM may include difficulty of implementation, lack of staff time, lack of adequate training, political and philosophical opinions held by treatment providers, or organizational characteristics (e.g., Benishek et al., 2010; Bride et al., 2011; Ducharme et al., 2010; Kirby et al., 1999, 2006; McGovern et al., 2004; McLellan, 2001; Petry, 2001; Petry & Simcic, 2002; Roll et al., 2009). Providers have also objected to CM programs that targeted only one behavior, required close tracking of behavior, or mandated frequent urinalysis testing (Kirby et al., 2006). Also, worries regarding possible negative side-effects of CM programs were common, and included concerns about jealousy between clients, clients arguing about rewards, or undermining of the treatment process (Kirby et al., 2006). We found only two studies that explicitly asked about the reasons substance abuse counselors and administrators object to or fail to use CM (Benishek et al., 2010; Willenbring et al., 2004). Frequently endorsed barriers included: lack of knowledge, skills, or training necessary to implement the intervention, lack of staff time, low priority or demand for CM, and lack of support by staff and administration. Cost is also a very prevalent criticism of CM programs. In a previous study (Kirby et al., 2006) we found that the majority of providers surveyed believed that their programs could not afford incentive programs that cost $50 or more per patient per month.

Finally, it appears that opinions about CM interventions may vary with treatment provider characteristics (Ducharme et al., 2010; McGovern et al., 2004; Kirby et al., 2006). We found that treatment providers who reported experience with tangible incentive programs displayed more positive beliefs about them than did those who reported no experience or experience only with social incentives (e.g., praise, recognition; Kirby et al., 2006). While a description of tangible incentives was given to treatment providers at the beginning of the survey, there is considerable variety in the types and quality of tangible CM interventions that participants might experience and our study did not control for exposure to a specific, efficacious CM intervention.

Fortunately, two studies conducted by the National Institute of Drug Abuse (NIDA) Clinical Trials Network (CTN) testing the efficacy of a prize-based incentive program provided an excellent opportunity for us to examine the opinions of treatment providers who had or had not experienced a specific, effective tangible CM intervention. Eight community-based psychosocial outpatient programs and 6 community-based methadone maintenance programs participated in the Motivational Incentives to Enhance Drug Abuse Recovery (MIEDAR) studies. The procedures and results of these studies have been reported in detail elsewhere (Petry et al., 2005; Peirce et al., 2006). Stimulant abusing clients were randomly assigned to receive either standard care alone or standard care with an added prize-based incentive program. The average cost of prizes for participants who achieved at least 2 weeks of continuous abstinence was $68 dollars per patient per month in the psychosocial setting (Petry et al., 2005) and $40 per patient per month in the methadone setting (Peirce et al., 2006). Participants in the incentive condition submitted a significantly greater number of stimulant-negative samples and had longer durations of continuous abstinence than the standard group, but even so, given cost and other objections expressed in previous studies, it was not clear that treatment providers would consider the program worth implementing.

The general goal of this study was to explore provider opinions regarding tangible incentives while controlling for exposure to MIEDAR – an effective CM intervention. Our specific purposes were twofold. The first purpose was to empirically describe opinions about tangible incentive programs among treatment providers that were or were not exposed to MIEDAR in order to indicate the most prevalent objections. The second purpose was to examine whether the opinions of treatment providers employed at the sites that implemented the MIEDAR intervention were more positive than those of matched providers at sites that did not implement it. Because being employed at a MIEDAR or a matched site did not guarantee that a treatment provider did or did not (respectively) have direct experience with the MIEDAR program, we also further controlled for exposure to the MIEDAR intervention by repeating analyses including only MIEDAR providers that reported personal experience implementing tangible incentives and matched participants reporting no personal experience with tangible incentive procedures.

2. Methods

2.1 Participants and Procedures

Six of the 7 CTN nodes (86%) involved in the MIEDAR studies agreed to participate in this study; however, data from one node were eliminated because no matched site was available, resulting in a sample drawn from 5 of the 7 (71%) nodes. The participants were recruited from treatment sites that participated in the MIEDAR studies and matched sites that did not participate but were within the same CTN nodes. Primary matching criteria were outpatient program modality (drug free or methadone) and program size as determined by number of counselors at the treatment program. Two of the participating nodes provided data from both psychosocial and methadone clinics. Table 1 shows the number of participants by treatment modality and node across 7 MIEDAR and 7 Matched recruitment sites.

Table 1.

Clinic and Participant Characteristics

Characteristic MIEDAR Matched p
Clinic Characteristics
Number of Counselors (n) 76 69 --
    Psychosocial Clinic 1 7 7
    Psychosocial Clinic 2 3 4
    Psychosocial Clinic 3 5 3
    Psychosocial Clinic 4 8 11
    Methadone Clinic 5 16 14
    Methadone Clinic 6 14 9
    Methadone Clinic 7 23 21
Methadone Modality %(n) 70% (53) 64% (44) .46
Node Location (n) --
    Delaware Valley 23 21
    Mid-Atlantic 17 13
    New York 23 21
    Pacific 5 3
    South Carolina 8 11
Participant Characteristics
Position %(n) .33
    Counselor/Counseling Supervisor 80% (61) 75% (52)
    Program Director/Administrator 19% (14) 19% (13)
    Other 1% (1) 6% (4)
Education %(n) .19
    Post-graduate (MD, PhD, Masters) 50% (38) 54% (37)
    BA/BS 38% (29) 26% (18)
    Other (e.g., CAC, High School) 12% (9) 20% (14)
Experience in Years (mean ± sd) 9.4 ± 14.9 10.7 ± 8.7 .52
Recovery Status %(n) 25% (19) 22% (15) .65
Previous Experience with Tangible Incentives %(n) 54% (41) 28% (19) .002
Open in a new tab

CTN research assistants contacted program directors at the MIEDAR sites and at matched sites to invite their staff to participate in the survey. They suggested that the survey be conducted during or immediately after a staff meeting where all the counselors, the program director and other treatment delivery staff would be present; however, when this was not possible the survey was conducted at a time that was convenient for the clinic.

The protocol was conducted with approval from the University of Pennsylvania Institutional Review Board and from local IRBs associated with each CTN node. The purpose of the study, the requirements, the risks and benefits, and the right to refuse participation or withdraw without penalty was described to participants. Participants were also told that no personal identifying information would be collected. They were provided either with written instructions or with a consent form, depending on whether the relevant local IRB waived or required written informed consent. After completing the consent or instruction procedure, the surveys were distributed and the participants completed them. Participants were paid $10 for participating in the study. Table 1 provides basic descriptive information for this sample.

2.1.1 Provider Survey of Incentives (PSI)1

Details regarding the development and structure of the PSI are described elsewhere (Kirby et al., 2006). Most survey items fall into five basic themes: positive opinions regarding incentives; limitations of incentive programs; negative side effects; impracticality; and moral or ethical objections. Participants indicate degree of agreement with items on a 5-point Likertscale (1 = strongly disagree; 2 = disagree; 3 = neutral; 4 = agree; 5 = strongly agree). Twenty-eight of the 44 items assess beliefs about both tangible and social incentives. Participants are asked to indicate degree of agreement with each of these items twice, once for use of tangible incentives and once for use of social incentives. In addition to the 28 parallel items, there are 10 items that pertain uniquely to tangible incentive programs (e.g., cost issues; clients will sell incentives) and six items that are unique to social incentives or ask about other approaches. For the purpose of this study, only the 38 items addressing tangible incentives were examined. Tangible incentives were defined in the survey instructions as goods or services typically ranging in value from $1 to $100 and provided in the form of retail items or gift certificates. Examples of the items are provided in Table 2.

Table 2.

Proportion of Providers Agreeing with Statements about Tangible Incentives

Combined MIEDAR Matched
N = 145 n = 76 n = 69
Positive Opinions (10 items; 9 parallel and 1 unique)
1 Any source of abstinence motivation, not just internal motivation, is a good thing for treatment 80 75 86
2 Incentives are more likely to have positive effects on the client than to have negative effects 75 75 75
3 Incentives are useful if they reward clients for fulfilling treatment goals other than just providing a clean urine, such as for regular attendance 72 71 74
4 Incentives can be useful whether or not they address the underlying issues of addiction 72 74 71
5 An advantage of incentive programs is that they focus on what is good in the client's behavior (e.g., abstinence), not what went wrong 68 71 65
6 Overall, I would be in favor of adding an incentive program to my treatment program 70 74 65
7 Tangible incentives are worthwhile because they can get clients in the door for treatment. 66 70 61
8 Giving incentives for drug-free urine samples helps the client become abstinent 64 62 67
9 Overall, incentives are good for the client/counselor relationship 63 62 64
10 Incentives help the client achieve sobriety, allowing the counselor to focus on helping them make other life changes 61 64 59
Objections and Limitations (15/22 items†; 19 parallel and 3 unique)
11 Incentives don't address the underlying issues of addiction 51 57 45
12* Clients who sell their tangible incentives will use the money to continue their substance abuse. 46 46 46
13 It's not right to give an incentive to clients for being clean when they aren't fulfilling other treatment goals (e.g., attending group) 46 42 51
14 It's not right to give an incentive for goals such as attendance if they are not testing drug negative 41 37 46
15 Incentives will cause jealousy among clients who don't get them 39 38 41
16 If you give a tangible incentive to clients who've earned them, but not to others, it will result in clients arguing about rewards 32 30 35
17 If a client is abstinent just to get the incentive, it could hurt the treatment process 26 26 26
18 Incentives are a bribe 23 28 17
19 Incentives are not useful for short-term treatments (e.g., one month or less) 22 20 25
20 Incentive programs that require close tracking of client behavior are too labor intensive to incorporate into our program 22 30 26
21 A problem with incentives is that abstinence will only last for as long as incentives are given 17 22 10
22* Incentive programs that require ≥1 urinalysis a week are not practical because most programs do not take weekly urines on all clients 17 12 13
23 Incentive programs are not consistent with my philosophy of treatment 17 17 17
24 Most clients would sell the tangible incentives they receive. 17 20 13
25 Incentives will stop the client from seeing beyond the external reward and prevent them from realizing their internal motivation 16 13 19
Open in a new tab

Italics indicate > 1 s.d. difference between MIEDAR and Matched means; 6 items focusing on costs are shown only in Figure 1

*

indicates item not included in scale calculations

†

7 items endorsed by fewer than 15% of the providers were not included in the table, but available from authors

Summary scores indicating overall positive or negative beliefs regarding tangible incentives are calculated by reverse-scoring scale items associated with negative opinions so that higher ratings on the Likert scale uniformly indicate more positive beliefs about incentive programs. Five of the 38 items displayed item-total correlations below .20 and were removed from scale calculations (see items with asterisks in Table 2 and Figure 1 caption). This resulted in a tangible scale of 33 items with a coefficient alpha of .92.

Figure 1.

Figure 1

Open in a new tab

Proportion of treatment providers agreeing with 1) the negative statement “My treatment program could not find funds for tangible incentives that cost ($10, 50*, or 150*) per client per month” (left hand panel); and 2) the positive statement “Tangible incentive programs that cost ($10, 50, or 150*) per client per month are worth it considering how effective they are” (right hand panel). There were no significant main effects of MIEDAR vs. matched group for either statement but there was a significant main effect of amount for both statements (X22= 42.15, p < .0001 and X22 = 20.21, p < .0001, respectively). Results of posthoc tests are notated with lowercase letters; bars with no overlapping letters differed significantly from each other. There was also a positive interaction effect between group and amount for the second question regarding worth (X22 = 7.68, p < .02); there was a significant MIEDAR vs. matched difference only at the $50 incentive value. Asterisks indicate items that were not included in the 33-item tangible scale (see Figure 2).

2.2 Data Analysis

Participant responses were examined in three ways. First, for descriptive purposes only, we calculated percentages of participants agreeing or strongly agreeing with individual items. Next, statistical analyses were conducted. Since initial Chi Squares and ANOVAs demonstrated no demographic differences between MIEDAR and matched groups, and inclusion of node as a covariate did not substantively change the results, covariates were not included in the analyses presented. Because responses to the Likert-scaled cost items were not normally distributed, rates of agreement (i.e., agree or strongly agree) to the six items specifically examining cost issues: [(1) “My treatment facility could not find funds for tangible incentives that cost $ _ (10, 50, or 150) per client per month” and (2) “Tangible incentive programs that cost $ _ (10, 50, or 150) per client per month are worth it considering how effective they are”] were examined using Generalized Estimating Equations (GEE; Diggle, Heagerty, Liang, & Zeger, 2002). For each analysis, the dependent variable was the recoded binary response for each amount and terms for MIEDAR group, incentive amount, and their interaction were included. The analyses specified an exchangeable correlation matrix and utilized empirically-based standard errors. Finally, PSI summary scores were analyzed. An ANOVA was conducted in which employment at a MIEDAR vs. matched clinic was the independent variable and the tangible scale score was the dependent variable. Because assignment to a MIEDAR (or matched) site did not completely control whether the participant had (or had not) implemented tangible incentives in practice, analyses were also conducted comparing responses of only the participants from MIEDAR clinics who reported using tangible incentives to those matched participants who reported that they had not utilized them. These analyses provided a truer test of actual exposure to a specific effective incentive program on attitudes regarding CM.

3. Results

The PSI was administered to 145 participants: 76 from sites that had completed the intervention phase of the MIEDAR study and 69 from matched sites. The sample was comprised of 113 counselors or counseling supervisors, 27 program directors or program administrators, and 5 other clinical or support staff (e.g., medical staff, intake worker, unknown). More participants were from methadone clinics (n = 97) than from the psychosocial clinics (n = 48). There were no significant differences between the MIEDAR and matched groups with respect to position, education, experience in the field of addictions, or recovery status (Table 1), and MIEDAR and Matched sites were similar in size, matching on average within 2.1 counselors (range = 0-5) of each other.

We examined the participants’ reports of experience with incentives to verify that there was a significant difference between MIEDAR and matched sites with respect to experience with tangible incentives. Table 1 shows that while there was a statistically significant difference, counselor employment in a MIEDAR versus matched treatment clinic did not completely control for experience: only 54% of the participants from MIEDAR sites indicated they had experience with tangible incentive programs and only 72% of the participants from matched sites reported no experience with them (i.e., 28% reported experience with tangible incentives).

3.1 Description of Providers’ Opinions Regarding CM

Table 2 shows the percent of participants endorsing 25 of the 38 items. Seven negative opinion items endorsed by fewer than 15% (range = 7% - 12%) of the participants and the six cost items analyzed using GEE are not shown. Overall, a greater proportion of participants agreed with the positive opinions than with the objections and limitations. Agreement rates for each of the positive opinion items were over 60% (M = 69%; S.D. = 5.9). In contrast, all but one of the objections and limitations items (Table 2, item #11) were endorsed by fewer than 50% of respondents (M = 23%; S.D. = 13.6). The two most highly endorsed positive items were 1) “Any source of abstinence motivation, not just internal motivation, is a good thing for treatment” (80% agreement) and 2) “Incentives are more likely to have positive effects on the client than to have negative effects” (75%). Additionally, the majority of participants indicated they would be in favor of adding a tangible incentive program to their treatment program (70%).

The most highly endorsed objection was that tangible incentives do not address the underlying issue of addiction (51%), although 72% of the participants agreed that tangible incentives are useful whether or not they address the underlying issues of addiction (item #4). The second most endorsed objection was that clients who sell their incentives would use the money to buy drugs (46%); however, only 17% of the providers believed that clients would in fact sell the incentives (item #24). The next two most highly endorsed objections listed in Table 2 were related to each other: 1) it wouldn't be right to give tangible incentives to clients for being clean when they aren't fulfilling other treatment goals, such as attending group (46%) and 2) it wouldn't be right to give a tangible incentive for goals such as attendance if they are not testing drug negative (41%).

3.2 Comparison of MIEDAR vs. Matched Clinics

We examined the tangible items in Table 2 (plus the 7 items not shown) comparing responses from MIEDAR and Matched participants. Since statistically comparing individual items would have greatly inflated the likelihood of a Type I error, our comparison is descriptive. Overall, the differences in percent agreement between the two groups were small for most items. Items shown in bold italic had absolute differences between the group means at or above the 80th percentile (i.e., ≥ 9) of differences. Focusing on the items with the largest differences in agreement shows an inconsistent pattern; MIEDAR participants appeared more likely to agree with only two of three positive opinions, and less likely to agree with only two of five objections and limitations.

3.2.1 Cost items

The left panel at the top of Figure 1 shows the proportion of participants agreeing or strongly agreeing with the statement “My treatment facility could not find funds for tangible incentives that cost $ _ (10, 50, or 150) per client per month.” There were no significant between group differences or interaction effect on these three items, but there was a main effect of incentive amount with fewer participants agreeing that their program could afford incentives as the cost per client per month increased (X22= 42.15, p<.0001). With regard to agreement with the statement, “Tangible incentive programs that cost $ _ (10, 50, or 150) per month are worth it considering how effective they are”, there was no main effect of group but there was a main effect of amount (X22 = 20.21, p<.0001) and a significant MIEDAR by amount interaction (X22 = 7.68, p = .02). Contrasts indicated that matched participants perceived that $10 and $50 incentives had significantly greater worth than $150 incentives. MEIDAR participants, on the other hand, perceived $50 incentives to have significantly greater worth than $10 and $150 incentives and $10 incentives to have significantly greater worth than $150 incentives. Finally, participants in the MIEDAR group reported $50 incentives as having more worth than participants in the matched group.

Because assignment to a MIEDAR (or matched) site did not completely control whether the participant had (or had not) had experience with tangible incentives, we repeated the analysis comparing responses of only the participants from MIEDAR clinics who reported using tangible incentives to those matched participants who reported that they had no experience with them. Results were very similar to the analysis presented above, except that for the question regarding whether incentives were worth the cost, a significant main effect by group emerged (X21 = 7.11, p<.001), with the MIEDAR group being more likely to view the incentives as being worth the cost.

3.2.2 PSI summary scores

The left panel of Figure 2 displays the mean scale scores for tangible incentives by MIEDAR condition. Mean scale scores indicated that participants from clinics that participated in MIEDAR (n = 75; 3.54 ± 0.49) had slightly more positive opinions than matched participants (n = 66; 3.43 ± 0.50); however, analysis of variance indicated no significant differences between groups (F1,139 = 2.13, p = 0.15).

Figure 2.

Figure 2

Open in a new tab

Mean scale scores (calculated from 33 tangible items) by MIEDAR condition. The bars on the left display mean scores for the intent-to-treat analysis, ignoring prior experience with incentives. Analysis of variance (n=141) indicated no significant differences between the MIEDAR and matched participants (F2,138 = 1.20, p=0.31). The bars on the right display mean scale scores by MIEDAR condition when 16 matched participants who reported prior experience with tangible incentives and 34 MIEDAR participants who reported no prior experience with incentives were removed from the analysis. An ANOVA (N=90) indicated that this difference was statistically significant (F2,87 = 4.09, p=0.02). Error bars show the standard errors (from left to right SDs = .49, .50, .50, and .50).

Again, to better control whether the participant had (or had not) implemented tangible incentives in practice, we repeated the analysis with only the participants from MIEDAR clinics who reported using tangible incentives and those matched participants who reported that they had not utilized them. The right panel of Figure 2 shows the mean scale scores for MIEDAR (n=41) and matched (n=49) participants correcting for experience. MIEDAR participants with prior incentive experience had higher scale scores (3.63 ± .50) than matched participants without prior experience (3.33 ± .50). The ANOVA indicated that this difference was statistically significant (F1,88 = 7.85, p=0.006).

4. Discussion

4.1 Providers’ Opinions Regarding CM

A higher proportion of treatment providers within NIDA's CTN endorsed positive as compared to negative statements regarding incentive interventions. On average, positive statements were endorsed by 69% of the providers while 23% endorsed negative items. This tendency for greater agreement with positive vs. negative statements replicates the results of Kirby et al. (2006) and Ritter and Cameron (2007), although somewhat larger proportions of providers in the CTN endorsed the positive statements compared to the treatment providers in the other two studies. The results of both of the American studies contrast with those of Ritter & Cameron (2007) who noted that Australian treatment providers demonstrated less support and greater concern regarding incentive programs.

The costs of incentives continue to be a significant concern for treatment providers. The items endorsed by the greatest proportion of participants were that their treatment program could not afford incentive programs that cost $50 and $150 per client per month. The next most prevalent objection or limitation of tangible CM programs was that incentives do not address the underlying issues of addiction, followed by concern that clients who sell their incentives would use the money to continue their substance use. It is not clear whether the latter concern would curtail treatment providers’ enthusiasm for using CM, since few providers (17%) believed that clients would in fact sell the incentives. Finally, providers expressed some concern about reinforcing one behavior while another important behavior was not being performed satisfactorily. These results are also consistent with those of Kirby et al. (2006); in fact when the unique tangible incentive items in the 2006 study are considered, the 10 most prevalent objections and limitations endorsed in both studies are the same.

4.2 Differences between MIEDAR vs. Matched Clinics

There were no statistical differences in summary scores reflecting beliefs about incentive programs seen for the MIEDAR versus matched providers when we included all the participants employed by the respective clinics. This is likely due to imprecision in our independent variable. Specifically, surveying counselors at MIEDAR versus Matched sites did not completely control for experience with a specific incentive program since a substantial proportion (46%) of the MIEDAR participants who should have had experience with the MIEDAR incentive protocol reported that they had no experience with tangible incentives. Additionally, 28% of the matched participants reported having experience with a tangible incentive program (although it may or may not have been the MIEDAR program). In community treatment programs, clinical staff turnover can be as high as 50% in a year (McLellan, Carise & Kleber, 2003) and may account for the imprecision in our independent variable. When we conducted a secondary analysis correcting for experience with tangible incentives, we did find small but significant differences, with the MIEDAR participants reporting more positive opinions than the matched participants.

Examination of specific items indicated no clear differences between MIEDAR and matched groups in pattern of agreement with positive statements about CM: both groups were more likely than not to agree with positive statements. In cases where there were larger differences in agreement rates between MIEDAR and matched participants regarding positive items, MIEDAR participants were more likely to indicate that they believed tangible incentives were worthwhile and that they would be willing to add them to their treatment program. Somewhat paradoxically, they were also less likely to agree that any source of abstinence motivation, not just internal motivation, is a good thing for treatment. The implications of this difference are difficult to interpret because we have no information regarding other sources of motivation that the participants may have been considering and because although the proportion of participants endorsing this item was lower compared to matched participants, this was one of two statements that the greatest proportion of MIEDAR participants agreed with.

There were 5 objections and limitations that had larger differences in agreement between the MIEDAR and matched groups. MIEDAR participants less frequently agreed that it is not right to give incentives for one behavior when the client is failing at another desirable behavior (as evidenced by items 13 and 14 in Table 2). However, MIEDAR participants were more likely to agree that incentives don't address the underlying issues of addiction; that abstinence will last only as long as incentives are in place; and that they constitute a bribe. The first two of these statements reflect realistic concerns. With respect to the first concern, although incentives are designed to counteract drug reinforcement, which is supported scientifically as a primary factor underlying addiction (Griffiths, Bigelow & Henningfield, 1980), they do not directly address cognitive and life-skills issues deficits or other multiple problems that are prevalent in drug dependent populations. This particular limitation alone may not seriously inhibit providers from using incentive programs, since the majority of providers (74% MIEDAR; 68% Matched) thought that incentives were useful despite this limitation.

With respect to the second concern, CM is like all medical and behavioral treatments of addiction and other behavioral disorders in that maintaining long-term behavior change is difficult when treatment is stopped. Although participants who achieve longer durations of abstinence during CM are more likely to maintain abstinence after incentives are withdrawn (Carpenedo, Kirby, Dugosh, Rosenwasser & Thompson, 2010; Higgins, Badger & Budney, 2000; Silverman, Robles, Mudric, Bigelow & Stitzer, 2004), the effects of CM are similar to those of other effective medical treatments for chronic conditions (e.g., medication for high blood pressure) in that once the treatment is stopped, the chronic condition that it is effectively treating is likely to return.

The third item where MIEDAR and matched participants showed larger differences stated that incentives are a bribe. It is somewhat surprising that a great proportion of MIEDAR participants agreed with this statement, because the word “bribe” may have a negative connotation; however, MIEDAR participants were slightly more positive about incentives overall. Although bribes were considered a negative feature of CM by the providers who assisted in the development of the PSI items (see Kirby et al., 2006), it is possible that those views are not consistent across providers or have since changed. Provider participants in the current study may have considered the incentives to be a bribe only because they serve to induce or influence behavior and as such, greater endorsement of this item by MIEDAR participants may have indicated greater awareness of the potential of incentives to positively influence patient behavior independent of a moral or value judgment. Further study would be needed to determine if this is a robust finding and whether providers truly view the influence of incentives negatively or consider it a positive aspect of CM.

Finally, while the MIEDAR and matched participants were equally likely to believe that their program could not afford higher cost incentives, a greater proportion of MIEDAR participants believed that programs that cost $50 per client per month were worthwhile given how effective they are. This amount is similar to the actual cost of the MIEDAR program, although it is unlikely that the MIEDAR participants were aware of the cost of the program at the time they completed the survey. The differences between MIEDAR and matched clinicians provide support for the contention that experience with these interventions promotes more positive attitudes toward them.

4.3 Implications

The implications of this study should be considered in the context of several limitations. First, the MIEDAR study protocol did not require that the treatment providers themselves administer the incentives. It did specify that the counselors at the clinic be encouraged to become involved in the program by helping to guide clients in selection of prizes and to be present during prize-bowl drawings, but the extent to which this occurred was not monitored. More direct control of exposure would have been preferable.

A second limitation is that like many studies, the generality of the findings is unknown. Provider response rates at each clinic were not systematically tracked, which raises the possibility of sample bias; however, very few providers refused to participate in the study and as such it is unlikely that the sample is systematically biased in ways that would render it unrepresentative. The fact that the survey was conducted across multiple geographically diverse sites increases the likelihood that it represents a wide variety of treatment providers. Although all participants were members of the programs that belonged to the CTN sites and sites volunteered to participate in the MIEDAR study, it should be noted that node investigators and clinic administrators made the decision to volunteer, not the counselors that comprised the majority of the MIEDAR sample. Still, because MIEDAR was not randomly assigned to sites, we cannot rule out the possibility of pre-existing differences. We can assume that these findings do not generalize to counselors globally, since the data from the Australian study (Ritter & Cameron, 2007) suggest that treatment providers in America may not have views similar to those in other countries. One factor that reflects positively on generality, however, is the similarity between the American sample in this study and that of Kirby et al. (2006). This systematic replication of those findings helps to build confidence in their validity.

It is also important to note that although differences between the two groups were statistically significant, they were small and it is not clear whether the differences in PSI scores would translate into differences in clinical practice. As such, we cannot assume that the PSI could be used to predict the likelihood that a group of practitioners would successfully adopt a CM program. Further research would be needed to determine if the PSI can serve this function. Adoption of CM interventions may or may not depend on the nature or extent of positive and negative attitudes endorsed by clinical staff. For example, the Theory of Planned Behavior (Norman, Armitage & Quigley, 2007; Armitage & Conner, 2001) suggests that a person's intention to engage in a behavior has three determinants: 1) attitude toward the behavior (overall positive or negative evaluation of the behavior), 2) perception of social pressure to perform the behavior, and 3) perceived behavioral control, which is determined by both internal (e.g., skills) and external (e.g., resources and constraints) factors. As such, if CM is adopted as a system-wide intervention (e.g. Kellogg et al, 2005), then this external factor may require counselors to use the technique independent of their personal views. Bride et al. (2011) found that organizational factors such as a general supportive therapeutic approach, being research friendly, offering outpatient care, and serving drug-court populations were associated with organizational CM adoption. Conversely, if resources are not available to implement CM, then counselors may not be able to implement it despite positive views. Nevertheless, adoption may be facilitated by a prevalence of favorable opinions. Studies have shown that direct measures of attitude make a significant contribution toward physicians’ intention to implement a procedure (Khanna et al., 2009, B= .620, p< .001; Sable et al., 2006, r = .62, p< .001), although a meta-analysis of the efficacy of the Theory of Planned Behavior noted that it accounts for less variance when predicting observed behavior than when predicting self-reported behavior (R2 = .21 vs .32, respectively; Armitage & Conner, 2001).

Because attitudes or beliefs about CM may contribute toward treatment providers’ intention to use CM, it may be worthwhile to examine methods for addressing some of the objections to it. After cost issues, the most prevalent objections that providers endorsed was that incentives do not address the underlying issues of addiction and that clients who sell incentives would use the money to buy drugs. We've noted that responses to other questions suggest that these objections may not seriously inhibit providers from using incentive programs. The next most prevalent objections pertained to the tendency for CM programs to focus on a single behavior. Clinicians would like to target more than one behavior simultaneously (e.g. attendance and abstinence) and some prior research has suggested ways that more than one behavior might be addressed without significant drops in efficacy (e.g., Ghitza et al., 2008; Kirby et al., 2008; Petry & Martin, 2002). Addressing multiple behaviors may improve implementation of CM.

Finally, the cost of incentives needs to be taken into consideration in adoption efforts. Over half of the clinicians indicated that their treatment program could not afford incentives costing $50 per client per month. This is the level most consistent with the cost of the MIEDAR program (i.e., $40 - $68 per client per month). Cost-effectiveness analysis has begun to provide some useful perspectives on the benefits to be gained in relation to the cost of these types of interventions (Olmstead, Sindelar & Peirce, 2007; Olmstead, Sindelar & Petry, 2007a, 2007b), with one study finding that higher magnitude CM treatments produce outcomes at a lower cost per unit compared to lower cost CM (Sindelar, Elbel, & Petry, 2007). These studies may help to put the costs of incentives in a more favorable context for treatment providers.

It may not be necessary to address every provider objection to facilitate use of CM. Several studies have reported successful dissemination of CM programs implemented by community treatment providers (e.g., Henggeler et al., 2008b; Kellogg et al., 2005; Ledgerwood et al., 2008) and at least one country has advocated its use in addiction treatment (Department of Health [England], 2007). Henggeler et al. (2008a) reported that 91% of the public sector substance abuse provider organizations in a statewide study supported a 1-day workshop training a CM intervention and over half of the therapists trained in CM reported adopting its use with at least one patient during the following 6 months (Henggeler et al., 2008b).

While the findings from this study should be viewed cautiously, its results are consistent with those of Ducharme et al., (2010), suggesting the possibility that exposing community treatment providers to CM programs may itself be a helpful strategy for encouraging the dissemination of CM interventions. While the provider participants’ support for CM was not unanimous, it was more positive than negative, and in comparison to matched treatment providers, the MIEDAR providers who were exposed to CM reported more favorable opinions regarding incentive programs. While they also occasionally expressed greater concerns regarding the limitations of the CM, these concerns were realistic. Kellogg et al. (2005) reported that the adoption of CM by five treatment programs in New York City was the direct result of a NIDA Blending conference that featured the MIEDAR study. To this end, the results of this study and those of Kellogg et al. (2005) suggest that NIDA's CTN may be making progress toward the goal of encouraging community treatment providers to adopt this evidence-based practice.

Acknowledgements

This study was supported by the National Institute of Drug Abuse through grants U10 DA13034, U10 DA13043, and R01 DA 015991. Portions of the data were presented at a National Institute on Drug Abuse CTN Steering Committee meeting in Washington, DC in October 2009.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

1

The PSI and information regarding the methods for developing it are available from the corresponding author, at http://www.tresearch.org/bev_int/past_projects.htm (also see assessment instruments in the webpage resources section).

References

  1. Armitage CJ, Conner M. Efficacy of the Theory of Planned Behaviour: A meta-analytic review. British Journal of Social Psychology. 2001;40:471–99. doi: 10.1348/014466601164939. [DOI] [PubMed] [Google Scholar]
  2. Benishek LA, Kirby KC, Dugosh KL, Padovano A. Beliefs about the empirical support of drug abuse treatment interventions: a survey of outpatient treatment providers. Drug & Alcohol Dependence. 2010;107(2-3):202–208. doi: 10.1016/j.drugalcdep.2009.10.013. [DOI] [PubMed] [Google Scholar]
  3. Bride BE, Abraham AJ, Roman PM. Organizational factors associated with the use of contingency management in publicly funded substance abuse treatment centers. Journal of Substance Abuse Treatment. 2011;40:87–94. doi: 10.1016/j.jsat.2010.08.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Carpenedo CM, Kirby KC, Dugosh KL, Rosenwasser BJ, Thompson DL. Extended voucher-based reinforcement therapy for long-term drug abstinence. American Journal of Health Behavior. 2010;34(6):776–87. doi: 10.5993/ajhb.34.6.12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Castells X, Kosten TR, Capella D, Vidal X, Colom J, Casas M. Efficacy of opiate maintenance therapy and adjunctive interventions for opioid dependence with comorbid cocaine use disorders: A systematic review and meta-analysis of controlled clinical trials. American Journal of Drug & Alcohol Abuse. 2009;35(5):339–49. doi: 10.1080/00952990903108215. [DOI] [PubMed] [Google Scholar]
  6. Chambless DL, Ollendick TH. Empirically supported psychological interventions: Controversies and evidence. Annual Review of Psychology. 2001;52:685–716. doi: 10.1146/annurev.psych.52.1.685. [DOI] [PubMed] [Google Scholar]
  7. Department of Health (England) and the devolved administrations . Drug Misuse and Dependence: UK Guidelines on Clinical Management. Department of Health (England), the Scottish Government, Welsh Assembly Government and Northern Ireland Executive; London: 2007. http://www.nta.nhs.uk/publications/documents/clinical_guidelines_2007.pdf. [Google Scholar]
  8. Diggle PJ, Heagerty PJ, Liang K-Y, Zeger SL. Analysis of Longitudinal Data (2nd Edition), Oxford Statistical Science Series #25. Oxford University Press Inc.; New York: 2002. [Google Scholar]
  9. Ducharme LJ, Knudsen HK, Abraham AJ, Roman PM. Counselor Attitudes toward the use of motivational incentives in addiction treatment. The American Journal on Addictions. 2010;19:496–503. doi: 10.1111/j.1521-0391.2010.00081.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Ghitza UE, Epstein DH, Schmittner J, Vahabzadeh M, Lin JL, Preston KL. Effect of reinforcement probability and prize size on cocaine and heroin abstinence in prize-based contingency management. Journal of Applied Behavior Analysis. 2008;41(4):539–549. doi: 10.1901/jaba.2008.41-539. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Griffiths RR, Bigelow GE, Henningfield JE. Similarities in animal and human drug taking behavior. In: Mello NK, editor. Advances in Substance Abuse: Behavioral and Biological Research. JAI Press; Greenwich, CT: 1980. [Google Scholar]
  12. Henggeler SW, Chapman JE, Rowland MD, Halliday-Boykins CA, Randall J, Schakelford J, Schoenwald SK. If you build it, they will come: Statewide practitioner interest in contingency management for youths. Journal of Substance Abuse Treatment. 2008a;32:121–131. doi: 10.1016/j.jsat.2006.07.006. [DOI] [PubMed] [Google Scholar]
  13. Henggeler SW, Chapman JE, Rowland MD, Halliday-Boykins CA, Randall J, Schakelford J, Schoenwald SK. State-wide adoption and initial implementation of contingency management for substance-abusing adolescents. Journal of Consulting and Clinical Psychology. 2008b;76:556–567. doi: 10.1037/0022-006X.76.4.556. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Herbeck DM, Hser YI, Teruya C. Empirically supported substance abuse treatment approaches: A survey of treatment providers’ perspectives and practices. Addictive Behaviors. 2008;33(5):699–712. doi: 10.1016/j.addbeh.2007.12.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Higgins ST, Alessi SM, Dantona RL. Voucher-based incentives. A substance abuse treatment innovation. Addictive Behaviors. 2002;27(6):887–910. doi: 10.1016/s0306-4603(02)00297-6. [DOI] [PubMed] [Google Scholar]
  16. Higgins ST, Badger GJ, Budney AJ. Initial abstinence and success in achieving longer term cocaine abstinence. Experimental & Clinical Psychopharmacology. 2000;8(3):377–86. doi: 10.1037//1064-1297.8.3.377. [DOI] [PubMed] [Google Scholar]
  17. Higgins ST, Heil SH, Lussier JP. Clinical implications of reinforcement as a determinant of substance use disorders. Annual Review of Psychology. 2004;55:431–461. doi: 10.1146/annurev.psych.55.090902.142033. [DOI] [PubMed] [Google Scholar]
  18. Kellogg SW, Burns M, Coleman P, Stitzer M, Wale JB, Kreek MJ. Something of value: The introduction of contingency management interventions into New York City health and hospital addiction treatment service. Journal of Substance Abuse Treatment. 2005;28:57–65. doi: 10.1016/j.jsat.2004.10.007. [DOI] [PubMed] [Google Scholar]
  19. Khanna R, Kavookjian J, Scott VG, Kamal KM, Miller LA, Neal WA. Using the theory of reasoned action to determine physicians’ intention to measure body mass index in children and adolescents. Research in Social & Administrative Pharmacy. 2009;5(2):170–81. doi: 10.1016/j.sapharm.2008.06.006. [DOI] [PubMed] [Google Scholar]
  20. Kirby KC, Benishek LA, Dugosh KL, Kerwin MLE. Substance abuse treatment providers’ beliefs and objections regarding contingency management: Implications for dissemination. Drug and Alcohol Dependence. 2006;85:19–27. doi: 10.1016/j.drugalcdep.2006.03.010. [DOI] [PubMed] [Google Scholar]
  21. Kirby KC, Kerwin ME, Carpenedo CM, Rosenwasser BJ, Gardner RS. Interdependent group contingency management for cocaine-dependent methadone maintenance patients. Journal of Applied Behavior Analysis. 2008;41(4):579–595. doi: 10.1901/jaba.2008.41-579. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Kirby KC, Amass L, McLellan AT. Disseminating contingency-management research to drug abuse treatment practitioners. In: Higgins ST, Silverman K, editors. Motivating behavior change among illicit-drug abusers: Contemporary research on contingency-management interventions. American Psychological Association; Washington, D.C.: 1999. pp. 327–344. [Google Scholar]
  23. Ledgerwood DM, Alessi SM, Hanson T, Godley MD, Petry NM. Contingency management for attendance to group substance abuse treatment administered by clinicians in community clinics. Journal of Applied Behavior Analysis. 2008;41:517–526. doi: 10.1901/jaba.2008.41-517. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Lussier JP, Heil SH, Mongeon JA, Badger GJ, Higgins ST. A meta-analysis of voucher-based reinforcement therapy for substance use disorders. Addiction. 2006;101(2):192–203. doi: 10.1111/j.1360-0443.2006.01311.x. [DOI] [PubMed] [Google Scholar]
  25. McGovern MP, Carroll KM. Evidence-based practices for substance use disorders. Psychiatric Clinics of North America. 2003;26(4):991–1010. doi: 10.1016/s0193-953x(03)00073-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. McGovern MP, Fox TS, Xie H, Drake RE. A survey of practice and readiness to adopt evidence-based practices: Dissemination research in an addiction treatment system. Journal of Substance Abuse Treatment. 2004;26:305–312. doi: 10.1016/j.jsat.2004.03.003. [DOI] [PubMed] [Google Scholar]
  27. McLellan AT, et al. Moving toward a “third generation” of contingency management studies in the abuse treatment field: Comment on Silverman. Experimental and Clinical Psychopharmacology. 2001;2001;9(1):29–32. doi: 10.1037/1064-1297.9.1.29. discussion 35-29. [DOI] [PubMed] [Google Scholar]
  28. McLellan AT, Carise D, Kleber HD. Can the national addiction treatment infrastructure support the public's demand for quality care? Journal of Substance Abuse Treatment. 2003;25:117–121. [PubMed] [Google Scholar]
  29. Norman P, Armitage CJ, Quigley C. The theory of planned behavior and binge drinking: Assessing the impact of binge drinker prototypes. Addictive Behaviors. 2007;32(9):1753–1768. doi: 10.1016/j.addbeh.2006.12.009. [DOI] [PubMed] [Google Scholar]
  30. Olmstead TA, Sindelar JL, Peirce JM. Cost-effectiveness of prize-based contingency management in methadone maintenance treatment programs. Addiction. 2007;102(9):1463–71. doi: 10.1111/j.1360-0443.2007.01913.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Olmstead TA, Sindelar JL, Petry NM. Clinic variation in the cost-effectiveness of contingency management. The American Journal on Addictions. 2007a;16(6):457–60. doi: 10.1080/10550490701643062. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Olmstead TA, Sindelar JL, Petry NM. Cost-effectiveness of prize-based incentives for stimulant abusers in outpatient psychosocial treatment programs. Drug and Alcohol Dependence. 2007b;87:175–82. doi: 10.1016/j.drugalcdep.2006.08.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Peirce JM, Petry NM, Stitzer ML, Blaine J, Kellogg S, Satterfield F, Schwartz M, Krasnansky J, Pencer E, Silva-Vazquez L, Kirby KC, Royer-Malvestuto C, Roll JM, Cohen A, Copersino M, Kolodner K, Li R. Lower-cost incentives increase stimulant abstinence in methadone maintenance treatment: A National Drug Abuse Treatment Clinical Trials Network study. Archives of General Psychiatry. 2006;30(3):173–81. doi: 10.1001/archpsyc.63.2.201. [DOI] [PubMed] [Google Scholar]
  34. Petry NM. Challenges in the transfer of contingency management techniques: Comment on Silverman et al. (2001). Experimental and Clinical Psychopharmacology. 2001;9(1):24–26. doi: 10.1037/1064-1297.9.1.24. discussion 35-29. [DOI] [PubMed] [Google Scholar]
  35. Petry NM, Martin B. Low-cost contingency management for treating cocaine- and opioid-abusing methadone patients. Journal of Consulting and Clinical Psychology. 2002;70(2):398–405. doi: 10.1037//0022-006x.70.2.398. [DOI] [PubMed] [Google Scholar]
  36. Petry NM, Peirce JM, Stitzer ML, Blaine J, Roll JM, Cohen A, Obert J, Killeen T, Saladin ME, Cowell M, Kirby KC, Sterling R, Royer-Malvestuto C, Hamilton J, Booth RE, Macdonald M, Liebert M, Rader L, Burns R, DiMaria J, Copersino M, Stabile PQ, Kolodner K, Li R. Effect of prize-based incentives on outcomes in stimulant abusers in outpatient psychosocial treatment programs: a national drug abuse treatment clinical trials network study. Archives of General Psychiatry. 2005;62(10):1148–56. doi: 10.1001/archpsyc.62.10.1148. [DOI] [PubMed] [Google Scholar]
  37. Petry NM, Simcic F., Jr. Recent advances in the dissemination of contingency management techniques: Clinical and research perspectives. Journal of Substance Abuse Treatment. 2002;23(2):81–86. doi: 10.1016/s0740-5472(02)00251-9. [DOI] [PubMed] [Google Scholar]
  38. Ritter A, Cameron J. Contingency management: Perspectives of Australian service providers. Drug and Alcohol Review. 2007;26(2):183–189. doi: 10.1080/09595230601184653. [DOI] [PubMed] [Google Scholar]
  39. Roll JM, Madden GJ, Rawson RR, Petry NM. Facilitating the adoption of contingency management for the treatment of substance use disorders. Behavior Analysis in Practice. 2009;2(1):4–13. doi: 10.1007/BF03391732. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Sable MR, Schwartz LR, Kelly PJ, Lisbon E, Hall MA. Using the theory of reasoned action to explain physician intention to prescribe emergency contraception. Perspectives on Sexual & Reproductive Health. 2006;38(1):20–7. doi: 10.1363/psrh.38.020.06. [DOI] [PubMed] [Google Scholar]
  41. Silverman K, Robles E, Mudric T, Bigelow GE, Stitzer ML. A randomized trial of long-term reinforcement of cocaine abstinence in methadone-maintained patients who inject drugs. Journal of Consulting & Clinical Psychology. 2004;72(5):839–54. doi: 10.1037/0022-006X.72.5.839. [DOI] [PubMed] [Google Scholar]
  42. Sindelar J, Elbel B, Petry NM. What do we get for our money? Cost-effectiveness of adding contingency management. Addiction. 2007;102(2):309–316. doi: 10.1111/j.1360-0443.2006.01689.x. [DOI] [PubMed] [Google Scholar]
  43. Willenbring ML, Kivlahan D, Kenny M, Grillo M, Hagedorn H, Postier A. Beliefs about evidence-based practices in addiction treatment: A survey of Veterans Administration program leaders. Journal of Substance Abuse Treatment. 2004;26(2):79–85. doi: 10.1016/S0740-5472(03)00161-2. [DOI] [PubMed] [Google Scholar]

RESOURCES