Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2013 Nov 1.
Published in final edited form as: AIDS Behav. 2012 Nov;16(8):2257–2266. doi: 10.1007/s10461-011-0072-2

HIV-infected individuals with co-occurring bipolar disorder evidence poor antiretroviral and psychiatric medication adherence

David J Moore a, Carolina Posada b, Mili Parikh a, Miguel Arce a, Florin Vaida c, Patricia K Riggs a, Ben Gouaux a, Ronald J Ellis d, Scott L Letendre e, Igor Grant a, J Hampton Atkinson a; the HIV Neurobehavioral Research Program (HNRP)
PMCID: PMC3351543  NIHMSID: NIHMS341070  PMID: 22041931

Abstract

The contribution of bipolar disorder (BD), a prevalent serious mental illness characterized by impulsivity and mood instability, to antiretroviral (ART) and psychiatric medication adherence among HIV-infected (HIV+) individuals is unknown. We examined medication adherence among 44 HIV+/BD+ persons as compared to 33 demographically- and medically-comparable HIV+/BD− persons. Classification of adherent (≥90%) or non-adherent (<90%) based on proportion of correctly taken doses over 30 days was determined using electronic medication monitoring devices. HIV+/BD+ persons were significantly less likely to be ART adherent (47.7%) as compared to HIV+/BD− (90.9%) persons. Within the HIV+/BD+ group, mean psychiatric medication adherence was significantly worse than ART medication adherence, although there was a significant correlation between ART and psychiatric adherence levels. Importantly, 30-day ART adherence was associated with plasma virologic response among HIV+/BD+ individuals. Given the high overlap of HIV and BD, and the observed medication adherence difficulties for these persons, specialized adherence improvement interventions are needed.

Keywords: Medication Adherence, HIV/AIDS, Bipolar Disorder

INTRODUCTION

Successful suppression of HIV RNA in HIV-infected patients is highly dependent on adherence to antiretroviral treatment (ART) regimens (1–3). Poor adherence to ART is associated with overall worse disease outcomes, development of drug-resistant viral mutations (4, 5) and increased likelihood of HIV transmission (6).

Psychiatric illness is prevalent in HIV-infected individuals, with perhaps up to 50% of persons having a lifetime history of mood or anxiety disorder, and a similar proportion reporting alcohol or other substance use disorder (e.g., 7–9). A body of research has begun to link psychiatric disorders to medication non-adherence (10, 11), and depressive symptoms have been related to suboptimal ART adherence (12–14). Treatment with antidepressant medication appears to improve ART adherence among those with a current mental health problem, especially those with more complex medication regimens (15).

One potentially important psychiatric syndrome, however, has been relatively overlooked. Bipolar disorder (BD), a chronic and serious mental illness characterized by mood instability (manic and depressive episodes), impulsivity, risky sexual behavior, and substance abuse, affects approximately 2% of the general population (16). Bipolar disorder and HIV infection co-occur at much higher rates than either would be expected in the general population. The prevalence of HIV is particularly elevated among persons with BD, perhaps approaching 10% (17–21). Similarly, the prevalence of BD among HIV-infected persons ranges from 2.6% to 9.1% (22–24). As a result, there have been recent calls to focus research on HIV-infected persons with co-occurring psychiatric and substance use conditions (e.g., 25).

In addition to the features listed above, BD is often associated with poor medication adherence. It has been estimated that 40% of BD patients do not take their psychiatric medications (e.g., mood stabilizers) as prescribed, and one third take less than 30% of their medication (26). Individuals with BD who are non-adherent to their psychiatric medications are at greater risk for relapse and recurrence of manic and depressive episodes, and psychiatric hospitalization (26–29). Non-adherence in this group is also strongly associated with higher health care costs (27).

Despite this overlap of HIV and bipolar disorder, and a background of high rates of non-adherence to psychiatric treatment, previous research suggesting poor adherence in this subgroup has been sparse. In one of the few studies to use electronic tracking of ART medication in a group of HIV+ individuals with serious mental illness, adherence varied widely, with 40% of participants showing adherence rates of greater than 90%, whereas 31% had adherence rates of less than 50% (30). Previous research in this area has focused on mixed psychiatric samples (e.g., 30–32), has used administrative (Medicare) datasets (e.g., 32–34), and has used psychiatric diagnoses from community providers (e.g., 31) or medical charts (e.g., 35). None of these reports focused exclusively on HIV+/BD+ individuals, utilized rigorous research-assigned standardized psychiatric diagnoses, or tracked medications with electronic monitoring for as long as 30-days.

In order to address this relative gap in the literature, the present study was designed to assess the medication adherence rates of HIV+/BD+ persons as compared to HIV+/BD− persons. We hypothesized that HIV+/BD+ persons would have significantly worse adherence to their ART medications than those in the comparison group. We also hypothesized that those HIV+/BD+ participants in a depressed or manic state would show significantly worse adherence to both ART and psychiatric medications as compared to those who were euthymic.

METHODS

Participants

Participants were recruited from ongoing studies at the HIV Neurobehavioral Research Center (HNRC). Participants in this parent study underwent comprehensive neuromedical assessment, which included HIV disease and treatment history, and laboratory evaluation (e.g., HIV plasma RNA, testing for hepatitis C virus infection). Forty-four HIV+/BD+ and 33 HIV+/BD− participants drawn from this cohort consented, enrolled and underwent a comprehensive adherence assessment. All participants recruited into the HIV+/BD+ group were required to have rigorously diagnosed BD and be taking medications to treat both HIV infection and bipolar disorder. Individuals with either bipolar disorder type I (i.e., requires evidence of a frank manic episode) or bipolar disorder type II (i.e., characterized by hypomania and major depression) were enrolled, since mood instability and impaired function is a feature of both conditions. HIV+/BD− comparison participants were required to be without BD and be taking medication to treat their HIV illness. Given the high proportion of HIV+ individuals with major depressive disorder (MDD) and the opportunity to compare psychiatric medication adherence between HIV+/BD+ (n=39) and HIV+/BD− (n=7) in the subset of participants who were taking psychiatric medications, we chose to allow inclusion of comparison participants (i.e., HIV+/BD− participants) with a current or past diagnosis of MDD.

Participants were excluded from the study if they met DSM-IV for psychotic spectrum disorder (e.g., schizophrenia), mood disorder due to a General Medical Condition, or if they had neurological condition known to impact cognitive functioning such as Huntington’s Disease, stroke, traumatic brain injury, or a closed head injury with loss of consciousness for more than 30 minutes. Individuals with a mood disorder with psychotic features were also excluded (e.g., BD or MDD with psychotic features).

The UCSD Human Research Protection Program approved the current study. After meeting study inclusion/exclusion requirements, interested participants provided written informed consent to participate. Participants received monetary compensation for both the initial and follow-up assessments.

Procedure

Psychiatric and Substance Use Assessment

Psychiatric diagnoses were assigned using the Structured Clinical Interview for DSM-IV (SCID), which was administered by a licensed clinical psychologist or research associate with a master’s degree in psychology supervised by a licensed clinical psychologist and a psychiatrist with extensive experience in HIV; ambiguous diagnoses were resolved via case discussion with a diagnostic expert. This diagnostic interview also determined the person’s current mood as either euthymic, depressive, hypomanic, or manic episode, or mixed (i.e., concurrent manic and depressive symptoms). We also assessed mood symptomatology by evaluating severity of current manic symptoms with the Young Mania Rating Scale (YMRS) (36), and depressive symptoms with the Beck Depression Inventory-II (BDI-II) (37). Scores of less than 14 on each of these measures are considered minimal (36, 37). Finally, an overall estimate of impairment in daily functioning was established with the Global Assessment of Functioning (GAF) score, a clinician-rendered judgment ranging from 1 = Severe Impairment (e.g., imminent suicidality or inability to perform basic self-care) to 100 = Asymptomatic (i.e., superior functioning over a wide range of activities) (38).

Medication Adherence Assessments

Tracking of Medication Adherence

The Medication Event Monitoring System (MEMS, AARDEX, Sion, Switzerland) was used to track medication adherence to both ART and psychiatric medications over the study period. The median number of days tracked was 30 (Range = 21 to 30). Because the decision to also track psychiatric medications was made shortly after the start of the study, only 39 of the 44 HIV+/BD+ and 7 of the 33 HIV+/BD− participants have both ART and psychiatric medications tracked. MEMS TrackCaps provide an electronic record of the date and time when the cap is removed. It has been suggested that MEMS may provide a more accurate estimate of adherence than self-report or pill counts (both of which tend to overestimate medication adherence) (39–41), and MEMS have been used successfully with difficult to track HIV-infected subpopulations (42).

Selection of Sentinel Medications

We used a “sentinel” strategy to select the antiretroviral (ART) and psychiatric medication to be tracked via MEMS. For ARTs, the sentinel drug for MEMS monitoring was the participant’s protease inhibitor (PI), since this is the agent most critically sensitive to non-adherence. If persons were not prescribed a PI, we selected the most frequently dosed non-nucleoside reverse transcriptase inhibitor (NNRTI) or nucleoside reverse transcriptase inhibitor (NRTI) as the sentinel drug to track in our sample. For persons on fixed-dose combination tablets (e.g., efavirenz/emtricitabine/tenofovir disoproxil fumarate) we tracked this medication. For psychiatric medications, the sentinel medication was the primary mood stabilizer defined according to published clinical guidelines (e.g., 43). If multiple mood stabilizing medications were prescribed we used the following hierarchy in choosing the mood stabilizing medication (lithium > valproate > carbamazepine > lamotrigine > most frequently dosed selective serotonin or noradrenaline reuptake inhibitor). Again, these decisions were based on estimates of the agent’s sensitivity to missed dosing.

MEMS outcome variable

The primary outcome variable for analysis was MEMS-derived percent adherence over the last 30 days (# of bottle openings divided by # of prescribed doses multiplied by 100). We also dichotomized this variable into adherent (≥90%) or non-adherent (<90%). Finer-grained and time dependant MEMS variables do not appear to add additional information above and beyond this variable (44). This 90% cutscore has shown sensitivity to adverse events in previous studies of HIV infection (45) as well as studies of HIV infection in severe mental illnesses (SMI) (30). In order to control for the possibility of MEMS overestimating adherence, we adjusted adherence to 100% for those participants who had more MEMS cap openings than was indicated by the sentinel medication being tracked. This can occur, for example, when a participant opens a cap to count the number of remaining pills.

Instructing participants to use MEMS

Participants were instructed to: 1) remove a single dose at a time, 2) open the bottle only when taking a dose, and 3) use the MEMS bottle only to dispense the sentinel medication. A separate MEMS cap was provided for one sentinel ART and one sentinel psychiatric medication (where indicated). Participants were informed of the recording mechanism in the bottle cap, however, they were not told that the cap would record the date and time of opening. The examiner filled the MEMS bottle with the appropriate medication(s) and demonstrated its use. Participants were asked to demonstrate how they would take medications. All participants were able to demonstrate ability to use the MEMS to the examiner. Participants were also taught how to incorporate the MEMS in conjunction with their medisets and/or pill organizers if needed. They were told to place the MEMS device in front of their organizer, and to open it every time they would have opened the organizer. If this seemed too difficult, we suggested they placed a cue (e.g., small piece of hard candy) in the organizer as a reminder to open the MEMS bottle.

Plasma viral load assessment

Plasma HIV RNA viral load (VL) was measured by reverse transcriptase-polymerase chain reaction (Roche Amplicor, v. 1.5, lower limit of quantitation 50 copies per milliliter). The difference between values from prior to and after MEMS tracking was recorded. Virologic response in participants was classified as “Improved,” “No Change,” or “Worsened” based on a greater than +/− 0.5 log change in plasma viral load (46). Given that participants were drawn from a larger parent study that provided initial VL measures prior to enrollment in the present study, the time elapsed between initial VL and the VL obtained at the completion of MEMS tracking had a median duration of 56 days (IQR = 48.5, 70).

Statistics

All data were examined to assess for normality and skewness. Non-parametric equivalents of traditional parametric statistics were used where appropriate (e.g., Wilcoxon rank sum test for comparing continuous variables between groups). The baseline characteristics were compared between the HIV+/BD+ and HIV+/BD− groups using t-tests and Wilcoxon rank sum tests for continuous variables (e.g., age, education, etc.), and using Fisher’s exact tests for dichotomous or nominal variables (e.g., gender, race/ethnicity, etc.). Medication adherence was compared between groups using Wilcoxon rank sum test (0–100%, continuous scale), and Fisher’s exact test (dichotomous scale, i.e., using the 90% adherence cutoff). The comparison of ART versus psychiatric medication adherence used the paired Wilcoxon rank sum test. The correlations between adherence and individual variables were evaluated using Spearman’s rank correlation. The comparison of adherence between the three mood states and the three virologic response levels used the Kruskal-Wallis test. ANCOVA was used to assess for the influence of both time between VL assessments and VL change categories on adherence. The calculations used the R statistical package (47) and JMP version 8.0.2 (48).

RESULTS

The demographic and clinical characteristics of the study participants are presented in Table 1. The study groups were comparable on descriptive and HIV disease characteristics. There was a significantly higher proportion of persons with lifetime methamphetamine abuse/dependence diagnoses in the HIV+/BD+ group as compared to the HIV+/BD− group (Chi Square Likelihood Ratio = 9.2; p < 0.01); no other significant differences in lifetime substance disorder were observed between the two groups. HCV status was not different between the two groups and rates of current substance abuse or dependence were low (i.e., less than 8% for all listed substances across both groups; data not shown). There was a trend toward less frequent dosing among HIV+/BD+ participants; the vast majority of HIV+/BD+ participants (95.5%) had once daily dosing as compared to the HIV+/BD− participants (81.8%; Chi Square Likelihood Ratio = 3.8, p = 0.05). All other participants had twice daily dosing of their sentinel ART medication. Regimens between groups were similar (Table 2). Regarding the prevalence of BD Type I and Type II, only three individuals in the HIV+/BD+ group had BD Type II.

Table 1.

Descriptive characteristics of the study groups

HIV+/BD−
(n = 33)		 HIV+/BD+
(n = 44)		 Test Statistic p-value
Descriptive
Age; mean (SD) 46.2 (9.3) 44.0 (7.4) t = 1.11, df = 59.7 0.27
Education; mean (SD) 13.2 (2.2) 13.3 (2.3) t = −0.32, df = 71.7 0.75
Male; % (#) 79 (26) 82 (36) Fisher’s Exact Test (FET) 0.78
Caucasian; % (#) 67 (22) 72 (31) FET 0.27
HCV infected; % (#) 13 (4) 19 (7) FET 0.53
Lifetime Substance Abuse/Dependence; % (#)
Alcohol 45 (15) 56 (24) FET 0.49
Marijuana 15 (5) 33 (14) FET 0.11
Cocaine 18 (6) 40 (17) FET 0.08
Opioid 3 (1) 14 (6) FET 0.13
Methamphetamine 18 (6) 51 (22) FET 0.004
Major Depressive Disorder; % (#)
Current 15 (5) N/A N/A N/A
Lifetime 58 (19) N/A N/A N/A
HIV Disease Characteristics
CD4 Count; mean (SD) a 542.3 (383.4) 626.7 (368.6) t = −0.95, df = 63.3 0.35
Nadir CD4 Count; mean (SD) a 333.0 (271.1) 349.7 (261.0) t = −0.27, df = 65.5 0.78
HIV RNA plasma (log copies/ml); mean (SD) a 1.9 (0.8) 2.1 (1.0) Wilcoxon rank-sum test z = −0.79 0.43
Proportion undetectable (<50 cp/ml); % (#) a 81 (26) 74 (32) FET 0.58
AIDS % (#) 72 (23) 66 (27) FET 0.58
Medication Information
Number Prescribed Medications; mean (SD) 7.8 (4.1) 8.8 (4.1) t = 1.1, df = 69.3 0.27
Number of Prescribed Medication Doses per Day; mean (SD) 14.0 (9.5) 14.4 (8.5) t = 0.3, df = 64.6 0.85
Number Psychiatric Medications; mean (SD) 2.4 (1.7) 3.1 (1.7) Wilcoxon Rank Sum Test z = −1.9 0.06
Psychiatric
YMRS; mean (SD) 3.5 (3.3) 7.7 (6.7) t = −3.6, df = 66.0 <0.001
BDI-II Total; mean (SD) 11.2 (10.8) 15.1 (10.4) t = −1.53, df = 67.5 0.13
Global Assessment of Functioning; mean (SD) b 75.9 (12.9) 64.5 (16.1) t = −3.4, df = 69.5 0.001
Open in a new tab

Key:

a

data available for 32 HIV+/BD− and 43 HIV+/BD+;

b

data available for 30 HIV+/BD− and 42 HIV+/BD+;

BDI-II = Beck Depression Inventory-II; HCV = Hepatitis C Virus; YMRS = Young Mania Rating Scale; Nadir CD4 count is calculated as the lowest of self reported or laboratory generated value.

Table 2.

Class of tracked medications by group

HIV+/BD−
(n = 33)		 HIV+/BD+
(n = 44)
Proportion of ART tracked; % (#) 100 (33) 100 (44)
    PI 9 (3) 9 (4)
    NNRTI 15 (5) 11 (5)
    NRTI 61(20) 66 (29)
    Combination Drugs 15 (5) 14 (6)
Proportion of Psychiatric Tracked; % (#) 24 (7) 88 (39)
    Mood Stabilizer 14 (1) 31 (12)
    Antidepressant 57 (4) 21 (8)
    Antipsychotic 29 (2) 49 (19)
Open in a new tab

Key: PI = protease inhibitor; NNRTI = non-nucleoside reverse transcriptase inhibitor; NRTI = nucleoside reverse transcriptase inhibitor

As expected, the HIV+/BD+ group also had higher current symptoms of mania (t = 3.6; p<001; Table 1). In the HIV+/BD+ group at baseline 25% (11/44) were diagnosed with a current hypomanic, manic, or mixed episode, 27% (12/44) were diagnosed with a depressive episode and 48% (21/44) were deemed euthymic (i.e., normal mood). Among the HIV+/BD− persons, a subset met criteria for a current major depressive episode (15%, 5/33), and over half met criteria for lifetime major depressive disorder (58%, 19/33). Current mood diagnoses were used for comparisons of the impact of mood state on adherence as presented below. The HIV+/BD− group had higher (i.e., better) Global Assessment of Functioning Scores (t=3.3, p = 0.001) and there was a trend toward a higher number of psychiatric medications prescribed in the HIV+/BD+ group (Table 1).

Group adherence comparisons

In terms of medication adherence, HIV+/BD+ individuals had significantly worse ART adherence as compared to HIV+/BD− persons. The mean MEMS adherence for the HIV+/BD+ group was 75.1% (SD = 30.5) compared to 94.6% (SD = 13.3) for the HIV+/BD− group (Wilcoxon Rank Sum Test z=3.5, p<0.001; Cohen’s D = .83, Figure 1A). After dichotomizing adherence into adherent (≥ 90%) and non-adherent (<90%), 47.7% of the HIV+/BD+ group was adherent to their ART medication as compared to 90.9% of HIV+/BD− group (p < 0.001, Fisher’s Exact Test). Group comparisons of psychiatric medication adherence revealed a mean 30-day adherence of 67.2% (SD = 31.1; n = 39) for the HIV+/BD+ group as compared to 83.3% (SD = 25.2; n = 7) for the HIV+/BD− group (Wilcoxon Rank Sum Test z = 1.6, p = 0.10; Cohen’s D = .57, Figure 1B). When examining proportion adherent (i.e., >90% over 30-days), we found that 35.9% of the HIV+/BD+ group was adherent to psychiatric medications as compared to 71.4% of the HIV+/BD− group (p = 0.11, Fisher’s Exact Test).

Figure 1.

Figure 1

Open in a new tab

HIV+/BD+ participants evidence worse 30-day MEMS antiretroviral adherence rates as compared to HIV+/BD− participants. Note: Thick bar represents Median; error bars include 95% of the distribution; significance determined via Wilcoxon Rank Sum Test.

Among HIV+/BD+ participants who had MEMS data for both ART and psychiatric medications (n = 39), mean adherence was significantly worse for psychiatric medications (67.2%, SD = 31.1) as compared to ART medications (75.2%, SD = 29.5; Paired Wilcoxon Rank-Sum Test V = 349, p = 0.02). Although adherence was worse for psychiatric medications, there was a significant correlation between ART and psychiatric medication adherence within HIV+/BD+ group (Spearman ρ = 0.77; p < 0.001; n = 39). A similar result was found within the HIV+/BD− group; however, few participants in this group had both ART and psychiatric medications tracked (Spearman ρ = 0.55; p = 0.20; n = 7).

Mood State and Adherence

Given that HIV+/BD+ group evidenced worse ART adherence than the HIV+/BDgroup, we explored whether poor adherence was associated with the presence of a current episode of a mood disorder on the first day of the 30-day medication tracking. Within the HIV+/BD+ group, the following MEMS 30-day adherence rates [mean (SD), n] were found for 1) ART: euthymic = 77.5 (25.6), n = 18; depressive = 76.9 (32.7), n = 12; manic/hypomanic/mixed = 68.5 (38.2), n = 10; and 2) psychiatric medications: euthymic = 72.5 (24.5), n = 15; depressive = 62.3 (34.7), n = 12; manic/hypomanic/mixed = 63.9 (38.1), n = 9. These findings did not support our hypothesis that current mood episode would be related to differences in ART (Kruskal-Wallis Χ2 = 0.36, p = 0.84) or psychiatric medication (Kruskal-Wallis Χ2 = 0.37; p = 0.83) adherence rates.

We further examined the role of mood at the initial visit by dichotomizing the groups into euthymic or non-euthymic based on their current mood episode on the SCID regardless of whether they were assigned to the HIV+/BD+ or HIV+/BD− group. The euthymic group (n = 49) had a mean ART adherence of 86.8% while the non-euthymic (n = 28) had a mean adherence of 77.6% (Wilcoxon rank-sum test z = −0.24; p = 0.81). Using the 90% ART adherence cut-scores, 69.4% of the euthymic group was ART adherent and 60.7% of the non-euthymic group was adherent (Fisher’s Exact Test, p = 0.46). Among participants in the HIV+/BD− group, there were no significant differences between persons with a current or past diagnosis of MDD versus those without the corresponding diagnosis in terms of ART or psychiatric medication adherence (all p’s > 0.5). Associations of adherence with scores on the BDI and YMRS were similarly nonsignificant.

Adherence and Plasma Virologic Response

We examined whether ART adherence was associated with a 0.5 log change in plasma viral load prior to and after medication tracking. In examining whether participants’ virologic response was classified as Improved, Worsened or No Change, we found: 1) within the HIV+/BD+ group (n = 40), 12.5% Improved, 7.5% Worsened, and 80.0% No Change; 2) within the HIV+/BD− group (n = 30), 16.7% Improved, 6.7% Worsened and 76.7% No Change. When examining the MEMS 30-day ART values by Improved, Worsened, and No Change categorizations, those in the HIV+/BD+ group evidenced an association between medication adherence and virologic response classification in the expected direction (Kruskal-Wallis Test Χ2 = 6.5, p = 0.04; Table 3); this association was not observed in the HIV+/BD− group (Kruskal-Wallis Test, Χ2 = 2.8, p = 0.25; Table 3). Within the HIV+/BD+ group there was an association between the time between VL assessment and virologic change (Kruskal-Wallis Test, Χ2 = 7.5, p = 0.02) such that the Worsened Group had a longer interval between VL assessments than the Improved group. In order to control for the possibility that the interval was more strongly associated with adherence than VL change categories, we conducted an ANCOVA analysis that showed that the association between viral load change and adherence persists (F = 3.48, df = 2,35, p = 0.04) where the interval between VL assessment was not a significant predictor of adherence (t = −0.69, df = 35, p = 0.49) suggesting that VL change is associated with adherence above and beyond interval.

Table 3.

MEMS 30-days ART adherence percentages as tracked for the two groups (HIV+/BD+ versus HIV+/BD−) stratified by plasma viral load changes (± 0.5 log) over the course of the study.

Mean (SD) MEMS Adherence
HIV+/BD+ (n = 40)		 MEMS Adherence
HIV+/BD− (n = 30)
Improved plasma VL 86.6 (16.2); n = 5 100 (0); n = 5
No Change plasma VL 79.3 (27.3); n = 32 95.0 (11.4); n = 23
Worsened plasma VL 24.7 (21.7); n = 3 98.3 (2.4); n = 2
Open in a new tab

Key: VL=plasma HIV RNA viral load

Examination of Other Variables on Adherence

MEMS 30-day ART adherence was positively correlated with age across all participants (Spearman ρ = 0.29, p = 0.01). There was no difference in adherence between those participants found to have a lifetime methamphetamine abuse or dependence diagnosis and ART and psychiatric medication adherence rates. We controlled further for the possible influence of lifetime cocaine and methamphetamine abuse/dependence on adherence by computing adjusted analyses using a stratified Wilcoxon rank-sum test, stratified by the 4 different groups (Meth/Cocaine = Yes/Yes, Yes/No, No/Yes, No/No). The results of these analyses are consistent with the HIV+/BD+ versus HIV+/BD− results prior to stratification (ART adherence, HIV+/BD+ vs. HIV+/BD−: Stratified Wilcoxon z =3.34, p<0.001; PSY adherence, HIV+/BD+ vs. HIV+/BD−: Stratified Wilcoxon z = 1.57, p=0.12). GAF score was significantly correlated with both ART (Spearman ρ = 0.25, p = 0.03) and psychiatric medication adherence (Spearman ρ = 0.40, p = 0.006) across all participants in the expected direction (i.e., better GAF scores were associated with better adherence).

There was no effect of overall number of medications or number of doses on adherence (both ps > 0.05). Also, no significant ART or psychiatric medication adhere differences were found between men and women across the whole cohort or within each of the groups (all ps > 0.05). The small proportion of individuals with BD II precluded formal statistical analysis of the impact of this diagnosis on adherence within the HIV+/BD+ group, but mean adherence rates were generally similar [e.g., mean ART adherence BD-I = 73.9% (40.0) vs. BD-II = 82.8% (29.8)]. Other variables presented in Table 1 were similarly not associated with adherence.

DISCUSSION

Our findings suggest that HIV-infected persons with co-occurring bipolar disorder have significantly worse mean ART medication adherence than a comparable group of HIV+ individuals without bipolar disorder, but who nevertheless have lifetime histories of other psychiatric and substance use disorders, as expected in HIV populations. The proportion of persons with an ART adherence level above 90% was nearly two-fold higher in the HIV+ persons without bipolar disorder. Mean psychiatric medication adherence rates, and the proportion adherent, were not significantly different for the two groups (i.e., HIV+/BD+ v. HIV+/BD−) most likely due to the small number of HIV+/BD− participants with tracked psychiatric medications (n = 7); effect sizes were medium and in the direction of worse psychiatric medication adherence for the HIV+/BD+ group. Among HIV+/BD+ persons, mean percent adherence was significantly worse for psychiatric medications as compared to ART medications, and there was a significant correlation between adherence to ART and psychiatric medications.

As would be expected, ART adherence was significantly associated with virologic response over the study period for the HIV+/BD+ group. Given the worse adherence rates among the HIV+/BD+ group, these participants likely had more opportunity to improve virologically with improved adherence. Particularly notable is that the three HIV+/BD+ persons who showed Worsened VL had extremely poor adherence rates suggesting adherence may be the cause of worsening virologic control. The two HIV+/BD− individuals who had worsened virologic control still showed good ART adherence possibly suggesting treatment resistance.

No significant differences were observed between the study baseline mood episode and medication adherence. Euthymic individuals had the best adherence for both ART and psychiatric medications but adherence rates were not significantly better than those of depressed or manic/hypomanic/mixed participants. Although no significant differences were observed, psychiatric symptomatology in the groups was, on average, minimal (i.e., BDI scores < 14 and YMRS scores < 14). Our data suggest that trait characteristics of bipolar disorder may be more important for medication non-adherence than state characteristics; at least within the range of mood fluctuations observed in this predominantly euthymic HIV+/BD+ cohort.

Our study highlights the importance of taking psychiatric diagnosis into account when discussing medication adherence as well as the critical need to address adherence to different medication classes (e.g., ART and psychiatric medications) rather than focusing on a more global concept of medication adherence. Although ART adherence was significantly correlated with psychiatric medication adherence among HIV+/BD+ individuals, adherence to ART medication was better than adherence to psychiatric medications. The motivating factors supporting ART and psychiatric medication adherence may be different. Patients may prioritize physical over mental health, discounting the possibility that non-adherence to psychiatric medications may lead to mood dysregulation, which in turn may lead to poor ART adherence and serious health consequences associated with HIV infection.

It is important to reemphasize that our HIV+/BD+ group had many factors that predispose them to better adherence rates than what might be expected of the a greater cross section of HIV+/BD+ persons. We might expect worse adherence rates among HIV+/BD+ persons who are less psychiatrically stable, who are actively using substances of abuse, whose adherence is followed for a longer period of time, and who are not willing to participate in research. Additionally, the current study only recruited and assessed HIV+/BD+ persons who were currently being treated for their bipolar disorder. Given that an accurate diagnosis of bipolar disorder often can take as long as 10 years (49), it is likely that ART medication adherence would be worse among HIV+/BD+ individuals who were not receiving treatment for their BD.

We recognize that there are limitations to the present study. Medication adherence was determined using MEMS which is restricted because participants know that medications are being tracked; medication adherence is only recorded when the cap is removed, not when medication is ingested; and this method may underestimate adherence (50); however, MEMS technology is widely used in HIV research (e.g., 51) and is considered to be the gold standard in medication adherence studies. Our study is also limited by a relatively small sample size, but to our knowledge, our study represents the largest sample of HIV+/BD+ participants whose medication adherence has been evaluated in detail. An additional limitation is that we included individuals with both bipolar disorder I and II in our study; because of the small sample of persons with bipolar disorder II (n=3), we were not able to assess whether there were significant adherence differences between these two diagnoses. Finally, we acknowledge that other comorbid psychiatric conditions such as anxiety disorders or Axis II personality disorders, for which we did not directly assess, may have also had an effect on medication adherence in the present study.

CONCLUSIONS

Given the poor adherence rates of the HIV+/BD+ group, it is important that we consider innovative interventions to improve adherence among these individuals. These interventions may include intensive case management and technological interventions such as daily cell phone or text messaging reminders. Day-to-day tracking of mood may be particularly important given that frequent mood alterations of persons with bipolar disorder may have possible implications for poor adherence among HIV+/BD+ persons, and may be a better indicator than the baseline mood measures used in the present study. HIV+ persons with co-occurring psychiatric disorders may be a subgroup of individuals who are a possible vector for continued HIV transmission given the propensity of HIV+/BD+ persons to engage in risk behaviors. In addition, chronic partial adherence to ART can foster development of drug resistant HIV, further compromising HIV control efforts. Therefore, as a matter of public health policy, a greater effort to diagnose BD, and to develop enhanced adherence strategies for this subset of the HIV-infected population is warranted.

Figure 2.

Figure 2

Open in a new tab

30-day MEMS psychotropic adherence rates for HIV+/BD+ participants are lower, but not significantly worse as compared to HIV+/BD− participants. Note: Thick bar represents Median; error bars include 95% of the distribution; significance determined via Wilcoxon Rank Sum Test.

ACKNOWLEDGEMENTS

This work was supported by National Institute of Mental Health R03 MH078785 and the California HIV/AIDS Research Program IDEA Award ID06-SD-201. The HIV Neurobehavioral Research Center (HNRC), supported by National Institute of Mental Health Center Award P30 MH 62512, and P01 DA 012065, supported by the National Institute of Drug Abuse, also contributed participant data to this study.

Footnotes

* The San Diego HIV Neurobehavioral Research Center [HNRC] group is affiliated with the University of California, San Diego, the Naval Hospital, San Diego, and the Veterans Affairs San Diego Healthcare System, and includes: Director: Igor Grant, M.D.; Co-Directors: J. Hampton Atkinson, M.D., Ronald J. Ellis, M.D., Ph.D., and J. Allen McCutchan, M.D.; Center Manager: Thomas D. Marcotte, Ph.D.; Jennifer Marquie-Beck, M.P.H.; Melanie Sherman; Neuromedical Component: Ronald J. Ellis, M.D., Ph.D. (P.I.), J. Allen McCutchan, M.D., Scott Letendre, M.D., Edmund Capparelli, Pharm.D., Rachel Schrier, Ph.D., Terry Alexander, R.N., Debra Rosario, M.P.H., Shannon LeBlanc; Neurobehavioral Component: Robert K. Heaton, Ph.D. (P.I.), Steven Paul Woods, Psy.D., Mariana Cherner, Ph.D., David J. Moore, Ph.D., Matthew Dawson; Neuroimaging Component: Terry Jernigan, Ph.D. (P.I.), Christine Fennema-Notestine, Ph.D., Sarah L. Archibald, M.A., John Hesselink, M.D., Jacopo Annese, Ph.D., Michael J. Taylor, Ph.D.; Neurobiology Component: Eliezer Masliah, M.D. (P.I.), Cristian Achim, M.D., Ph.D., Ian Everall, FRCPsych., FRCPath., Ph.D. (Consultant); Neurovirology Component: Douglas Richman, M.D., (P.I.), David M. Smith, M.D.; International Component: J. Allen McCutchan, M.D., (P.I.); Developmental Component: Cristian Achim, M.D., Ph.D.; (P.I.), Stuart Lipton, M.D., Ph.D.; Participant Accrual and Retention Unit: J. Hampton Atkinson, M.D. (P.I.), Rodney von Jaeger, M.P.H.; Data Management Unit: Anthony C. Gamst, Ph.D. (P.I.), Clint Cushman (Data Systems Manager); Statistics Unit: Ian Abramson, Ph.D. (P.I.), Florin Vaida, Ph.D., Reena Deutsch, Ph.D., Anya Umlauf, M.S., Tanya Wolfson, M.A.

References

  • 1.Gifford AL, Bormann JE, Shively MJ, Wright BC, Richman DD, Bozzette SA. Predictors of self-reported adherence and plasma HIV concentrations in patients on multidrug antiretroviral regimens. J Acquir Immune Defic Syndr. 2000 Apr 15;23(5):386–395. doi: 10.1097/00126334-200004150-00005. [DOI] [PubMed] [Google Scholar]
  • 2.Descamps D, Flandre P, Calvez V, et al. Mechanisms of virologic failure in previously untreated HIV-infected patients from a trial of induction-maintenance therapy. Trilege (Agence Nationale de Recherches sur le SIDA 072) Study Team) JAMA. 2000 Jan 12;283(2):205–211. doi: 10.1001/jama.283.2.205. [DOI] [PubMed] [Google Scholar]
  • 3.Montaner JS, Reiss P, Cooper D, et al. A randomized, double-blind trial comparing combinations of nevirapine, didanosine, and zidovudine for HIV-infected patients: the INCAS Trial. Italy, The Netherlands, Canada and Australia Study. JAMA. 1998 Mar 25;279(2):930–937. doi: 10.1001/jama.279.12.930. [DOI] [PubMed] [Google Scholar]
  • 4.Race E, Dam E, Obry V, Paulous S, Clavel F. Analysis of HIV cross-resistance to protease inhibitors using a rapid single-cycle recombinant virus assay for patients failing on combination therapies. AIDS. 1999 Oct 22;13(15):2061–2068. doi: 10.1097/00002030-199910220-00008. [DOI] [PubMed] [Google Scholar]
  • 5.Harrigan PR, Wynhoven B, Brumme ZL, et al. HIV-1 drug resistance: degree of underestimation by a cross-sectional versus a longitudinal testing approach. J Infect Dis. 2005 Apr 15;191(8):1325–1330. doi: 10.1086/428852. [DOI] [PubMed] [Google Scholar]
  • 6.Kalichman SC, Cherry C, Amaral CM, et al. Adherence to antiretroviral therapy and HIV transmission risks: implications for test-and-treat approaches to HIV prevention. AIDS Patient Care STDS. 2010 May;24(5):271–277. doi: 10.1089/apc.2009.0309. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Atkinson JH, Heaton RK, Patterson TL, et al. Two-year prospective study of major depressive disorder in HIV-infected men. J Affect Disord. 2008 Jun;108(3):225–234. doi: 10.1016/j.jad.2007.10.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Bing EG, Burnam MA, Longshore D, et al. Psychiatric disorders and drug use among human immunodeficiency virus-infected adults in the United States. Arch Gen Psychiatry. 2001 Aug;58(8):721–728. doi: 10.1001/archpsyc.58.8.721. [DOI] [PubMed] [Google Scholar]
  • 9.Kilbourne AM, Justice AC, Rollman BL, et al. Clinical importance of HIV and depressive symptoms among veterans with HIV infection. J Gen Intern Med. 2002 Jul;17(7):512–520. doi: 10.1046/j.1525-1497.2002.10803.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Sternhell PS, Corr MJ. Psychiatric morbidity and adherence to antiretroviral medication in patients with HIV/AIDS. Aust N Z J Psychiatry. 2002 Aug;36(4):528–533. doi: 10.1046/j.1440-1614.2002.00999.x. [DOI] [PubMed] [Google Scholar]
  • 11.Arnsten JH, Demas PA, Grant RW, et al. Impact of active drug use on antiretroviral therapy adherence and viral suppression in HIV-infected drug users. J Gen Intern Med. 2002 May;17(5):377–381. doi: 10.1046/j.1525-1497.2002.10644.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.DiMatteo MR, Lepper HS, Croghan TW. Depression is a risk factor for noncompliance with medical treatment: meta-analysis of the effects of anxiety and depression on patient adherence. Arch Intern Med. 2000 Jul 24;160(4):2101–2107. doi: 10.1001/archinte.160.14.2101. [DOI] [PubMed] [Google Scholar]
  • 13.Singh N, Squier C, Sivek C, Nguyen MH, Wagener M, Yu VL. Determinants of nontraditional therapy use in patients with HIV infection. A prospective study. Arch Intern Med. 1996 Jan 22;156(2):197–201. [PubMed] [Google Scholar]
  • 14.Starace F, Ammassari A, Trotta MP, et al. Depression is a risk factor for suboptimal adherence to highly active antiretroviral therapy. J Acquir Immune Defic Syndr. 2002 Dec 15;31(Suppl 3):S136–S139. doi: 10.1097/00126334-200212153-00010. [DOI] [PubMed] [Google Scholar]
  • 15.Kumar V, Encinosa W. Effects of antidepressant treatment on antiretroviral regimen adherence among depressed HIV-infected patients. Psychiatr Q. 2009 Sep;80(3):131–141. doi: 10.1007/s11126-009-9100-z. [DOI] [PubMed] [Google Scholar]
  • 16.Merikangas KR, Akiskal HS, Angst J, et al. Lifetime and 12-month prevalence of bipolar spectrum disorder in the National Comorbidity Survey replication. Arch Gen Psychiatry. 2007 May;64(5):543–552. doi: 10.1001/archpsyc.64.5.543. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Cournos F, McKinnon K. HIV seroprevalence among people with severe mental illness in the United States: a critical review. Clin Psychol Rev. 1997;17(3):259–269. doi: 10.1016/s0272-7358(97)00018-4. [DOI] [PubMed] [Google Scholar]
  • 18.Walkup J, Crystal S, Sambamoorthi U. Schizophrenia and major affective disorder among Medicaid recipients with HIV/AIDS in New Jersey. Am J Public Health. 1999 Jul;89(7):1101–1103. doi: 10.2105/ajph.89.7.1101. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Evans DL, Charney DS. Mood disorders and medical illness: a major public health problem. Biol Psychiatry. 2003 Aug 1;54(3):177–180. doi: 10.1016/s0006-3223(03)00639-5. [DOI] [PubMed] [Google Scholar]
  • 20.Beyer J, Kuchibhatla M, Gersing K, Krishnan KR. Medical comorbidity in a bipolar outpatient clinical population. Neuropsychopharmacology. 2005 Feb;30(2):401–404. doi: 10.1038/sj.npp.1300608. [DOI] [PubMed] [Google Scholar]
  • 21.Beyer JL, Taylor L, Gersing KR, Krishnan KR. Prevalence of HIV infection in a general psychiatric outpatient population. Psychosomatics. 2007 Jan-Feb;48(1):31–37. doi: 10.1176/appi.psy.48.1.31. [DOI] [PubMed] [Google Scholar]
  • 22.Atkinson J, Young C, Pham T, et al., editors. Prioritizing Adherence Intervention Based on Self Assessment; Enhancing Adherence: A State of the Science Meeting on Intervention Research to Improve Anti-Retroviral Adherence; New Haven CT. 2005. [Google Scholar]
  • 23.Druss BG, Wang PS, Sampson NA, et al. Understanding mental health treatment in persons without mental diagnoses: results from the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2007 Oct;64(10):1196–1203. doi: 10.1001/archpsyc.64.10.1196. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Robins LN, Wing J, Wittchen HU, et al. The Composite International Diagnostic Interview. An epidemiologic Instrument suitable for use in conjunction with different diagnostic systems and in different cultures. Arch Gen Psychiatry. 1988 Dec;45(12):1069–1077. doi: 10.1001/archpsyc.1988.01800360017003. [DOI] [PubMed] [Google Scholar]
  • 25.Walkup J, Blank MB, Gonzalez JS, et al. The impact of mental health and substance abuse factors on HIV prevention and treatment. J Acquir Immune Defic Syndr. 2008 Mar 1;47(Suppl 1):S15–S19. doi: 10.1097/QAI.0b013e3181605b26. [DOI] [PubMed] [Google Scholar]
  • 26.Scott J, Pope M. Nonadherence with mood stabilizers: prevalence and predictors. J Clin Psychiatry. 2002 May;63(5):384–390. doi: 10.4088/jcp.v63n0502. [DOI] [PubMed] [Google Scholar]
  • 27.Li J, McCombs JS, Stimmel GL. Costs of treating bipolar disorder in the California Medicaid (Medi-Cal) program. J Affect Disord. 2002;71(1–3):131–139. doi: 10.1016/s0165-0327(01)00394-9. [DOI] [PubMed] [Google Scholar]
  • 28.Berk M, Berk L, Castle D. A collaborative approach to the treatment alliance in bipolar disorder. Bipolar Disord. 2004 Dec;6(6):504–518. doi: 10.1111/j.1399-5618.2004.00154.x. [DOI] [PubMed] [Google Scholar]
  • 29.Keller MB. Improving the course of illness and promoting continuation of treatment of bipolar disorder. J Clin Psychiatry. 2004;65(Suppl 15):10–14. [PubMed] [Google Scholar]
  • 30.Wagner GJ, Kanouse DE, Koegel P, Sullivan G. Adherence to HIV antiretrovirals among persons with serious mental illness. AIDS Patient Care STDS. 2003 Apr;17(4):179–186. doi: 10.1089/108729103321619782. [DOI] [PubMed] [Google Scholar]
  • 31.Wagner GJ, Kanouse DE, Koegel P, Sullivan G. Correlates of HIV antiretroviral adherence in persons with serious mental illness. AIDS Care. 2004 May;16(4):501–506. doi: 10.1080/09540120410001683420. [DOI] [PubMed] [Google Scholar]
  • 32.Himelhoch S, Brown CH, Walkup J, et al. HIV patients with psychiatric disorders are less likely to discontinue HAART. AIDS. 2009 Aug 24;23(13):1735–1742. doi: 10.1097/QAD.0b013e32832b428f. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Walkup J, Wei W, Sambamoorthi U, Crystal S. Antidepressant treatment and adherence to combination antiretroviral therapy among patients with AIDS and diagnosed depression. Psychiatr Q. 2008 Mar;79(1):43–53. doi: 10.1007/s11126-007-9055-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Walkup JT, Sambamoorthi U, Crystal S. Use of newer antiretroviral treatments among HIV-infected medicaid beneficiaries with serious mental illness. J Clin Psychiatry. 2004 Sep;65(9):1180–1189. doi: 10.4088/jcp.v65n0905. [DOI] [PubMed] [Google Scholar]
  • 35.Kemppainen JK, Levine R, Buffum M, Holzemer W, Finley P, Jensen P. Antiretroviral adherence in persons with HIV/AIDS and severe mental illness. J Nerv Ment Dis. 2004 Jun;192(6):395–404. doi: 10.1097/01.nmd.0000130132.55146.04. [DOI] [PubMed] [Google Scholar]
  • 36.Young RC, Biggs J, Ziegler V, Meyer D. A rating scale for mania: Reliability, validity, and sensitivity. British Journal of Psychiatry. 1978;133:429–435. doi: 10.1192/bjp.133.5.429. [DOI] [PubMed] [Google Scholar]
  • 37.Beck AT, Steer RA, Brown G. BDI-II Manual. San Antonio, TX: The Psychological Corp.; 1996. [Google Scholar]
  • 38.Hall RC. Global assessment of functioning. A modified scale. Psychosomatics. 1995 May-Jun;36(3):267–275. doi: 10.1016/S0033-3182(95)71666-8. [DOI] [PubMed] [Google Scholar]
  • 39.Vriesendorp R, Cohen A, Kristanto P, et al. Adherence to HAART therapy measured by electronic monitoring in newly diagnosed HIV patients in Botswana. Eur J Clin Pharmacol. 2007 Dec;63(12):1115–1121. doi: 10.1007/s00228-007-0369-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Twagirumukiza M, Kayumba PC, Kips JG, et al. Evaluation of medication adherence methods in the treatment of malaria in Rwandan infants. Malar J. 2010;9:206. doi: 10.1186/1475-2875-9-206. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Liu H, Golin CE, Miller LG, et al. A comparison study of multiple measures of adherence to HIV protease inhibitors. Ann Intern Med. 2001 May 15;134(10):968–977. doi: 10.7326/0003-4819-134-10-200105150-00011. [DOI] [PubMed] [Google Scholar]
  • 42.Applebaum AJ, Reilly LC, Gonzalez JS, Richardson MA, Leveroni CL, Safren SA. The impact of neuropsychological functioning on adherence to HAART in HIV-infected substance abuse patients. AIDS Patient Care STDS. 2009 Jun;23(6):455–462. doi: 10.1089/apc.2008.0181. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.McAllister-Williams RH. Relapse prevention in bipolar disorder: a critical review of current guidelines. J Psychopharmacol. 2006 Mar;20(2 Suppl):12–16. doi: 10.1177/1359786806063071. [DOI] [PubMed] [Google Scholar]
  • 44.Lu M, Rogers WH, Laws MB, et al. Covered Time (CT) vs. Percent Doses Taken (PDT): Comparing Two MEMS Summary Adherence Measures Using Viral Loads (VL) as Reference. Third International Conference of HIV Treatment Adherence; March 17, 18; Jersey City, New Jersey. 2008. [Google Scholar]
  • 45.Singh N, Berman SM, Swindells S, et al. Adherence of human immunodeficiency virus-infected patients to antiretroviral therapy. Clin Infect Dis. 1999 Oct;29(4):824–830. doi: 10.1086/520443. [DOI] [PubMed] [Google Scholar]
  • 46.Ellis RJ, Childers ME, Zimmerman JD, Simon DWF, Deutsch R, McCutchan JA. Human Immunodeficiency Virus-1 RNA Levels in Cerebrospinal Fluid Exhibit a Set Point in Clinically Stable Patients Not Receiving Antiretroviral Therapy. The Journal of Infectious Diseases. 2003;187(11):1818–1821. doi: 10.1086/375152. [DOI] [PubMed] [Google Scholar]
  • 47.R Development Core Team. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2010. [Google Scholar]
  • 48.SAS Institute Inc. JMP® 8 User Guide, Second Edition. Cary, NC: SAS Institute Inc.; 2009. [Google Scholar]
  • 49.The Depression and Bipolar Support Alliance. Bipolar Disorder Statistics. 2000 From: http://www.dbsalliance.org/site/PageServer?pagename=about_statistics_bipolar.
  • 50.Wendel CS, Mohler MJ, Kroesen K, Ampel NM, Gifford AL, Coons SJ. Barriers to use of electronic adherence monitoring in an HIV clinic. Ann Pharmacother. 2001 Sep;35(9):1010–1015. doi: 10.1345/aph.10349. [DOI] [PubMed] [Google Scholar]
  • 51.Levine AJ, Hinkin CH, Castellon SA, et al. Variations in patterns of highly active antiretroviral therapy (HAART) adherence. AIDS Behav. 2005 Sep;9(3):355–362. doi: 10.1007/s10461-005-9009-y. [DOI] [PubMed] [Google Scholar]

RESOURCES