Abstract
Background:
Depression is one of the most common complications of childbirth, and is experienced by approximately 17% of pregnant women and 13% of postpartum women. An estimated 85% of these women go untreated – an alarming statistic given the serious consequences for the mother, her child, other family members, and society. Professional societies (the American College of Obstetricians and Gynecologists and American Academy of Pediatrics) have recommended improvements in screening and treatment. Meta-analyses indicate that Cognitive Behavioral Therapy eHealth interventions are efficacious for depression, generally, and for perinatal depression, specifically. Earlier controlled trials have established the effectiveness and acceptability of MomMoodBooster (including an Australian version, MumMoodBooster), an eHealth program for ameliorating postpartum depression.
Objective:
To evaluate the effectiveness of a perinatal version of MomMoodBooster encompassing both prenatal and postpartum content in a healthcare delivery setting already providing universal screening and referral of at-risk patients as part of routine care.
Study Design:
A practical effectiveness study randomly assigned 95 pregnant and 96 postpartum women screened as depressed and satisfying eligibility criteria to experimental groups: the healthcare organization’s perinatal depression care program (routine care group) and routine care plus MomMoodBooster2 program (eHealth group). Eligibility criteria included: pregnant or <1 year postpartum, ≥18 years of age, no active suicidal ideation, access to broadband internet via desktop/laptop, tablet or smartphone, and English language proficiency.
Results:
Intent-to-treat analyses of group effects used fixed effects growth models to assess 12-week posttest change in outcomes. Results showed both groups significantly decreased depression severity, anxiety, stress, and automatic thoughts, and increased behavioral activation and self-efficacy. Relative to routine care, the eHealth group displayed significantly greater decreases in depression severity and stress. These group comparisons were not moderated by depression severity (screening or baseline), anxiety, stress and pregnant/postpartum status. Almost all (93%, n=89) of women in the eHealth group visited their program of whom 99% visited program sessions (M sessions visited=4.3 ± 2.0; M total session duration=73.0 minutes ± 70.2; 49% viewed all 6 sessions). Among confirmed eHealth program users who provided ratings: 96% (79/82) rated their program as easy to use, 83% rated it helpful, and 93% (76/82) indicated they would recommend it.
Conclusion:
Results support the effectiveness of using MomMoodBooster as a treatment option for perinatal depressed women – especially when combined with universal depression screening and referral. As such, the eHealth program shows promise as a tool to increase the reach of treatment delivery and to potentially reduce the number of untreated depressed perinatal women.
Keywords: perinatal depression, perinatal anxiety, postnatal depression, postpartum depression, prenatal depression, pregnancy depression, prenatal anxiety, pregnancy anxiety, postnatal anxiety, postpartum anxiety, cognitive behavioral therapy
Condensation
Adding an eHealth program to a healthcare organization’s universal screening and referral program results in significantly greater improvement in depression symptoms among perinatal patients.
INTRODUCTION
Perinatal Depression Typically Undertreated
Approximately 17% of pregnant women and 13% of postpartum women experience significant personal, social and economic costs from perinatal depression,1–3 defined as any major or subsyndromal depressive episode from pregnancy through the year following delivery.2,4 Depression in pregnancy is associated with delayed fetal development, infant prematurity, and low birth weight2,5 whereas postpartum depression is associated with subsequent hospitalization, compromised cognitive/psychosocial development, and diminished maternal-infant relationship.3,6 Yet significant gaps in identifying and treating perinatal depression have been documented.2 For example, a comprehensive meta-analysis7 concluded that approximately 85% of women with perinatal depression do not receive effective treatment. Fewer than half of perinatal depressed women seek help8 even though medical and preventive health organizations have called for increased screening and treatment opportunities.9–15 Internet-based (eHealth) treatments for perinatal depression can help overcome barriers experienced by patients (e.g., stigma, time availability) and by medical care providers (e.g., treatment cost and time availability of busy physicians and medical staff),16,17 but establishing their effectiveness via pragmatic application is critically important.
Cognitive Behavioral Therapy (CBT) Treatment Approaches
CBT for depression has shown considerable promise and widespread use in internet interventions18–20 and treatments targeting perinatal depression.16,21,22 For example, a series of studies have replicated the effectiveness and acceptability of MomMoodBooster for postpartum depression, across settings and counterfactual comparison groups.23–28
Moreover, the effects of eHealth interventions – especially when supported by coach calls – can be similar to those obtained in face-to-face psychotherapy,29 as demonstrated in our recent collaborative publication.26
Aims of Research
The current study was designed to test the perinatal version of the MomMoodBooster eHealth program (MMB2; optimized for – but not limited to – smartphone use and incorporating text messages in a healthcare delivery setting) with pregnant women (targeted by MMB2 for the first time) and postpartum women (well-established in previous studies). Primary outcome measures examined the extent to which MMB2 ameliorated the severity of depressive symptoms and anxiety. Secondary measures examined the extent to which MMB2 was used and rated as being usable and helpful. Study aims focused on replicating and extending results of randomized controlled trials showing that MomMoodBooster ameliorates depressive symptoms among postpartum women23–27 while establishing that pregnant women could experience similar benefit from using MMB2.
MATERIALS and METHODS
Participant Recruitment, Enrollment, and Randomization
Depressed perinatal women were recruited from NorthShore University HealthSystem – a Chicago-based healthcare system incorporating 6 hospitals, over 3,000 primary care physicians and specialists. Study recruitment followed a step-wise process embedded within NorthShore’s Perinatal Depression Program (PDP)30 beginning with universal screening approximately 26-28 weeks gestation and 6 weeks postpartum using the Edinburgh Postnatal Depression Scale (EPDS; score > 12).31,32 PDP social workers contacted women with positive depression screens to tailor recommended treatment, including community mental health referrals. Based on their clinical judgment of each patient’s presenting level of care and safety considerations as well as their knowledge of the Mom Mood Study’s eligibility criteria, social workers provided a brief description of the study in the call.
Interested women were referred to the study coordinator who provided a more complete study description and determined final eligibility using the following criteria: pregnant or <1 year postpartum, ≥18 years of age, no active suicidal ideation, access to broadband internet via desktop/laptop, tablet or smartphone, and English language proficiency. Women with affirmative answers to the EPDS self-harm item were included in the study if social work assessment deemed them low-risk for suicide. Patients with active suicidal ideation were excluded.
REDCap (Version 8.10.5)33 was used to accomplish all subsequent onboarding steps, including informed consent, randomization to group, online assessments, tailored emails, and data management.
Perinatal Depression Program (PDP)
NorthShore’s well-established PDP includes universal perinatal outpatient depression screening with centralized scoring and outreach, a referral network of community mental health providers, a 24/7 crisis hotline to respond to urgent/emergent patient needs, and relevant curriculum for obstetricians and nurse midwives.34–37
MomMoodBooster2 + Perinatal Depression Program (MMB2+PDP)
Women in the MMB2+PDP group could use MMB2 and the PDP. MMB2 recommends increasing pleasant activities to regain life balance, interrupting negative thoughts and increasing positive thoughts, seeking support from others, and tracking mood. MMB2 included videos, audios, animations, and editable lists in a browser-based Web app that responsively adapted to each user’s smartphone, tablet, laptop, or desktop device (Figure 1).38,39 During the 12-week active treatment phase, each of the six MMB2 sessions became available sequentially according to a weekly schedule. Thereafter, users could continue visiting MMB2 for 7 additional months. The study coordinator enrolled women to MMB2 using the program’s administrative website. Each pregnant woman’s due date was used by MMB2 to ask the participant if they had delivered their baby in order to change from antepartum to postpartum program content. Two team outreach calls were made by a NorthShore team member not trained in mental health treatment. Call #1, 2-4 weeks following randomization, focused on resolving any difficulties signing into MMB2. Call #2, scheduled after the posttest, collected open-ended feedback about the program.
Measures and Assessments
Participant characteristics were assessed at baseline and outcomes were assessed at baseline and the 12-week posttest. Participants who completed all assessments received a $100 e-gift card.
In terms of primary outcomes, the Patient Health Questionnaire (PHQ-9) was used to assess the severity of depressive symptoms has been well-validated,40–42 found reliable and sensitive,43 and widely used with perinatal depressed women44,45 – including in our prior MMB research,23,26,27 and other studies in large healthcare systems.10,13,42 The minimal clinically important difference (MCID) was used to evaluate the clinical significance of the intervention effects. Based on Lowe et al.,43 the MCID for the PHQ-9 was defined as a baseline to posttest PHQ-9 reduction of at least 5 points.
We used the Depression Anxiety Stress Scale (DASS-21)46–48 to assess anxiety symptom severity because perinatal anxiety is commonly comorbid with depression and has been related to adverse perinatal outcomes.49
For secondary outcomes, the DASS-21 stress scale46–48 was used to assess stress severity, the Behavioral Activation for Depression Scale (BADS-Short Form)50,51 to measure behavioral activation, the short-form version of the Automatic Thoughts Questionnaire (ATQ-SF)52,53 to measure negative thoughts associated with depression, and a measure of behavioral self-efficacy to assess use of MMB2 strategies to manage activities, positive/negative thinking, support, relaxation, and goal setting.
Additional measures included MMB2’s continuous and unobtrusive tracking of each user’s MMB2 visits, session visits (date, number, duration), and activities (e.g., number/duration of videos and animations viewed, personal list updates). In addition, women who visited MMB2 (confirmed by unobtrusive engagement metrics) were asked on the posttest to rate MMB2’s usability and helpfulness, the helpfulness of team outreach calls, and whether they would recommend MMB2. Participants in both groups were asked whether they used other mood management products or programs while in the study.
Data Analysis
Preliminary analyses examined distributional properties of measures, baseline equivalency, and missing data. Intent-to-treat analyses of group effects were performed using fixed effects growth models fit (SAS PROC MIXED; Version 9.4) and estimated with maximum likelihood. Individual variability in outcomes from baseline to posttest were predicted by a two-level dummy coded group variable (coded 0 for PDP and 1 for MMB2+PDP), a time variable (coded in months elapsed between baseline and posttest), and a group × time interaction. The Group x Time parameter estimate tests for differential change in outcomes in MMB2+PDP group relative to the PDP group. Effect sizes for the Group × Time interaction are equivalent to Cohen’s d.54 Moderation of Group × Time effects for the primary depression outcome (PHQ-9) were evaluated by adding in separate models, the main effects of baseline perinatal status (pregnant/postpartum), the EPDS primary screen, baseline PHQ-9, and all two- and three-way higher order interactions with group and time.
RESULTS
Participant Flow and EPDS Screen Scores
Figure 2 describes the flow of participants through the study from the EPDS screen, referrals to the study, and randomization to group. Of the 11,201 women screened using the EPDS during the study period, 2.5% (280/11,201) were referred to the study coordinator for possible inclusion in the study and 1.7% (191/11,201) satisfied eligibility criteria, consented, completed baseline and were randomized to condition. EPDS primary screen scores (MMB2+PDP: M=15.0 ± 2.9; PDP: M=15.2 ± 3.2) did not significantly differ by group. M of 20.3 days ± 20.7 (range=1-133 days) elapsed between the EPDS screen and baseline. Of randomized study participants, 93% (178/191) completed the posttest – a retention rate that did not differ by group.
Preliminary Analyses
Participant baseline demographic characteristics are presented in Table 1. Overall, the sample averaged 32 years of age and was predominately non-Hispanic (84%), White (67%), married or in a long-term relationship (94%). Thirty-three percent had a bachelor’s degree while 42% had an advanced degree. All scale scores approximated normal distributions and randomization was confirmed by non-significant group differences of characteristics and outcome results.
Table 1.
PDPa | MMB2+PDPb | |
---|---|---|
Perinatal status (%) | ||
Pregnant | 49.5 | 50.0 |
Postpartum | 50.5 | 50.0 |
Gravida (including current pregnancy) (%) | ||
1 | 41.9 | 45.8 |
2 | 28.0 | 25.0 |
3 | 14.0 | 22.9 |
4 or more | 16.1 | 6.3 |
Number of children (%) | ||
0 | 23.7 | 29.2 |
1 | 49.5 | 36.5 |
2 | 18.3 | 22.9 |
3 or more | 8.6 | 11.5 |
Age [M, (SD)] | 31.7 (5.2) | 32.1 (5.4) |
Ethnicity (% Hispanic) | 12.1 | 12.6 |
Race (%) | ||
American Indian or Alaskan Native | 0.0 | 1.1 |
Asian | 18.3 | 10.6 |
Black or African American | 7.5 | 8.5 |
White | 64.5 | 70.2 |
More than one race | 8.6 | 8.5 |
Unknown | 1.1 | 1.1 |
Married or in long-term relationship (% Yes) | 93.7 | 94.7 |
Level of education (%) | ||
Less than high school | 1.1 | 1.0 |
High school graduate | 12.6 | 17.7 |
GED | 1.1 | 2.1 |
Associate’s degree or Trade School | 7.4 | 7.3 |
Bachelor’s degree | 35.8 | 30.2 |
Master’s or other graduate degree | 35.8 | 32.3 |
Doctoral or postgraduate degree | 6.3 | 9.4 |
Income | ||
Up to $20,000 | 6.3 | 9.4 |
$20,001 to $40,000 | 9.5 | 7.3 |
$40,001 to $60,000 | 9.5 | 12.5 |
$60,001 to $80,000 | 12.6 | 8.3 |
Greater than $80,000 | 52.6 | 47.9 |
Prefer not to answer | 9.5 | 14.6 |
PDP=Perinatal Depression Program (routine care: universal screening and referral of at-risk patients);
MMB2+PDP=MomMoodBooster2 plus Perinatal Depression Program
Primary and Secondary Outcomes
Table 2 shows a descriptive summary of continuous outcomes by group controlling for both perinatal status (pregnant or postpartum) at baseline and assessment time (baseline and posttest). Figure 3 shows the trajectory of change calculated for PHQ-9 stress severity scores by group from baseline to posttest. Table 3 shows the results, including statistical significance, from the fixed effects from growth models for each a priori outcome measures by group. The Intercept parameter indicates model-implied outcome score at baseline for PDP. The Group parameter indicates difference in estimated baseline outcome scores for MMB2+PDP relative to PDP. Significant Time parameter estimates showed there were significant baseline to posttest decreases in depression severity, anxiety, stress, and automatic thoughts as well as increases in behavioral activation and self-efficacy. Absence of significant Group estimates for the outcomes indicates that the groups (MMB2+PDP vs PDP) did not differ on outcome scores at baseline, and is a test of the effectiveness of randomization to produce initially equivalent groups. However, significant Group × Time interactions demonstrated that, compared to PDP, the MMB2+PDP group achieved significantly greater baseline to posttest decreases in depression severity and stress. Other Group × Time interactions for anxiety, behavioral activation, automatic thoughts, and self-efficacy favored MMB2+PDP but they were not significant. In addition, no significant moderating effects of primary EPDS screen, baseline PHQ-9 score, and baseline perinatal status (pregnant/postpartum) were obtained for depression severity Group x Time interactions. All P-values were >.308.
Table 2.
PDPa | MMB2+PDPb | |||||||
---|---|---|---|---|---|---|---|---|
|
||||||||
Outcome c | Baseline | Posttest | Baseline | Posttest | ||||
| ||||||||
M | SD | M | SD | M | SD | M | SD | |
Depression Severity d | 10.24 | 5.57 | 7.48 | 5.67 | 10.68 | 4.96 | 5.78 | 4.42 |
Anxiety e | 5.96 | 4.43 | 4.12 | 3.83 | 5.23 | 3.90 | 2.81 | 2.82 |
Stress e | 10.36 | 4.53 | 8.10 | 4.65 | 9.61 | 4.05 | 5.79 | 3.47 |
Behavioral Activation f | 24.46 | 8.66 | 29.37 | 9.41 | 25.15 | 7.99 | 31.15 | 8.41 |
Automatic Thoughts g | 1.24 | 0.66 | 0.90 | 7.31 | 1.23 | 0.65 | 0.72 | 0.57 |
Self-efficacy h | 2.77 | 0.73 | 3.05 | 0.78 | 2.86 | 0.74 | 3.40 | 0.79 |
PDP= Perinatal Depression Program Routine Care (universal screening and referral of at-risk patients);
MMB2+PDP=MomMoodBooster2 plus Perinatal Depression Program;
Outcomes displayed in bold formatting were found to be significant in analyses summarized in Table 3;
PHQ-9: [Patient Health Questionnaire]: 9 items, 4-point scale (0=Not at all, 3=Nearly every day), item scores summed (maximum=27), higher scores indicated greater severity;
DASS-2 [Depression Anxiety Stress Scale both Anxiety and Stress subscales]: 7 items, 4-point scale (0=Did not apply to me at all – Never, 3=Applied to me very much, or most of the time), item scores summed (maximum=21), higher scores indicated greater anxiety;
BADS-SF [Behavioral Activation for Depression Scale Short Form]: 9-items, 7-point scale (0=Not at all, 6=Completely), item scores summed (maximum=54), higher scores indicated decrease in avoidance and increase in activation;
ATQ-SF [Automatic Thoughts Questionnaire Short Form]: 8-items, 4-point scale (1=Not at all, 4=A11 the time), item scores summed (maximum=32), higher scores indicated more frequent negative thoughts;
Self-efficacy: 5-point scale (1=Not at all confident, 5=Very confident), item scores averaged (maximum=5.0), higher scores indicated greater self-efficacy.
Table 3.
Outcome | Parameter | Estimate | SE | t-value | P-value | d |
---|---|---|---|---|---|---|
Primary | ||||||
Depression Severity a | Intercept | 10.24 | 0.53 | 19.32 | <.001 | |
Group | 0.44 | 0.75 | 0.58 | .561 | ||
Time | −0.89 | 0.18 | −5.08 | <.001 | −.47 | |
Group × Time | −0.76 | 0.25 | −3.03 | .003 | −.40 | |
Anxiety b | Intercept | 5.96 | 0.39 | 15.29 | <.001 | |
Group | −0.73 | 0.55 | −1.33 | .186 | ||
Time | −0.60 | 0.12 | −5.11 | <.001 | −.43 | |
Group × Time | −0.18 | 0.17 | −1.08 | .283 | −.13 | |
Secondary | ||||||
Stress b | Intercept | 10.36 | 0.43 | 24.11 | <.001 | |
Group | −0.74 | 0.61 | −1.23 | .221 | ||
Time | −0.73 | 0.15 | −4.84 | <.001 | −.51 | |
Group × Time | −0.51 | 0.22 | −2.37 | .019 | −.36 | |
Behavioral Activation c | Intercept | 24.46 | 0.76 | 32.26 | <.001 | |
Group | 0.68 | 1.07 | 0.64 | .524 | ||
Time | 1.62 | 0.31 | 5.22 | <.001 | .58 | |
Group × Time | 0.34 | 0.44 | 0.77 | .440 | .12 | |
Automatic Thoughts d | Intercept | 1.24 | 0.07 | 18.48 | <.001 | |
Group | −0.01 | 0.09 | −0.06 | .954 | ||
Time | −0.11 | 0.02 | −5.30 | <.001 | −.52 | |
Group × Time | −0.06 | 0.03 | −1.94 | .055 | −.27 | |
Self-Efficacy | Intercept | 2.77 | 0.08 | 35.67 | <.001 | |
Group | 0.10 | 0.11 | 0.89 | .376 | ||
Time | 0.09 | 0.03 | 3.42 | .001 | .38 | |
Group × Time | 0.08 | 0.04 | 2.06 | .041 | .32 |
PHQ-9;
DASS-21;
BADS-SF;
ATQ-SF
Additional analyses using the MCID for clinical significance of changes (baseline - posttest PHQ-9 scores) revealed that 43% (37/86) of MMB2+PDP participants showed a clinically significant decrease in PHQ-9 scores (OR = 2.12, 95% CI = 1.16-3.90, P-value = .015) compared to 26% (24/92) of PDP participants.
MMB2 Program Use and Ratings
Among the 96 MMB2+PDP women, almost all (93%, n=89) visited the MMB2 program (M visits=10.3 ± 8.7; M duration=93.8 minutes ± 84.2). Most women visited the program soon after receiving the automated email invitation (within M=3.3 days ± 5.2) and almost all women then continued to visit MMB2 multiple distinct days between their first and last visit (M=49.4 distinct days ± 30.2). Program visitors were also sent program-related text messages over the first 12-weeks of the program (M=37.8 SMS text messages ± 7.2).
Almost all program visitors (99%; 88/89) visited MMB2 program sessions (M sessions visited=4.3 ± .0; M=73.0 minutes total session duration ± 70.2) and 49% (43/88) viewed all 6 MMB2 sessions), 41% (n=36) viewed only prenatal sessions, 49% (n=43) viewed only postpartum sessions, and 10% (n=9) viewed both prenatal and postpartum sessions because they were pregnant at baseline and delivered during program participation. A total of 62% (55/89) visited the MMB2 library (M library visits=4.0 ± 3.8). A technical problem limited analysis of device type to 29 MMB2 visitors, of whom 48% used only a smartphone, 10% used both a smartphone and desktop/laptop, 31% used only a desktop/laptop, and 10% used a device that could not be categorized.
Among the 92% (82/89) of MMB2 program visitors who also provided ratings, 96% (79/82) rated MMB2 “somewhat” to “extremely” easy to use and 83% (68/82) rated MMB2 “somewhat” to “extremely” helpful. Eighty percent (59/78) of program visitors who rated MMB2 text messages reported that they were “somewhat” to “extremely” helpful. Of the 84 MMB2+PDP women who provided ratings, 76 (90%) participated in research team outreach Call #1, 65 (77%) in Call #2, and 57 women (68%) in both calls. Seventy-seven percent (65/84) of women who received calls and provided ratings described MMB2 outreach calls as “somewhat” to “very” helpful. 93% (76/82) of MMB2 program visitors who provided ratings indicated they would recommend the program.
Women in both groups reported using M=2.0 ± 1.4 of the 10 other products/programs (Table 4; N=178) with 16% (n=28) reporting use of no other products/programs, 21% (n=37) one product/program, 31% (n=55) reported two products/programs, 20% (n=36) reported three products/programs, 12% (n=22) four or more products/programs. Groups did not differ on the number of products/programs used (t[176] = 0.50, p-value=.627). The most common products/programs used were 46% (n=81) taking medication for depression, anxiety, or another mood issue, 38% (n=67) participating in individual face-to-face counseling, and 33% (n=58) receiving physician advice.
Table 4.
Other products and programs useda | PDPb (n=92) | MMB2+PDP c (n=86) | Both Groups (N=178) | |||
---|---|---|---|---|---|---|
| ||||||
n | % | n | % | n | % | |
Took medication for depression anxiety, or another mood issue | 39 | 42.4 | 42 | 48.8 | 81 | 45.5 |
Individual face-to-face counseling | 37 | 40.2 | 30 | 34.9 | 67 | 37.6 |
Saw my doctor who gave me advice | 30 | 32.6 | 28 | 32.6 | 58 | 32.6 |
Read self-help books | 22 | 23.9 | 22 | 25.6 | 44 | 24.7 |
Some “other” program | 15 | 16.3 | 13 | 15.1 | 28 | 15.7 |
Individual treatment program | 11 | 12.0 | 11 | 12.8 | 22 | 12.4 |
Saw a nurse/pediatrician who gave me advice | 13 | 14.1 | 9 | 10.5 | 22 | 12.4 |
Group treatment program | 6 | 6.5 | 11 | 12.8 | 17 | 9.6 |
Another internet-based treatment program | 4 | 4.3 | 4 | 4.7 | 8 | 4.5 |
Used hypnosis or acupuncture | 1 | 1.1 | 5 | 5.8 | 6 | 3.4 |
Stem question was “Since you enrolled in the NorthShore Mom Mood Study program 12 weeks ago, which of the following products or programs have you used to manage your mood?” and were given a list of 10 options;
PDP= Perinatal Depression Program (routine care: universal screening and referral of at-risk patients);
MMB2+PDP=MomMoodBooster2 plus Perinatal Depression Program
DISCUSSION
Principal Findings
The current study examined changes in pregnant and postpartum patients of a healthcare organization who screened positive for depression and consented to being randomly assigned to an established inhouse Perinatal Depression Program (PDP) or PDP plus the MMB2 eHealth program. While both groups significantly improved outcomes (depression severity, anxiety, stress, automatic thoughts, behavioral activation, and self-efficacy), the MMB2+PDP group significantly outperformed the PDP group in reducing depression severity and stress and improving self-efficacy. Absence of significant moderation findings indicated use of MMB2+PDP, relative to PDP, was equally effective across the observed range of screening scores, baseline depression severity, and perinatal (pregnant or postpartum) status. MMB2 users reported the program was easy to use, helpful, and that they would recommend it. The lack of significant group effects for anxiety symptoms was not consistent with results of our two prior trials26,27 that had higher baseline anxiety scores. In addition, group effects on anxiety may not have emerged because approximately 40% of the PDP group reported receiving medication and engaging in individual counseling.
Practice Implications
MMB2 significantly improved the impact of a routine perinatal depression care program in a clinical setting when used (a) with 2 or fewer team staff outreach calls (fewer calls than in prior MMB research26,27,55) and (b) without face-to-face clinical visits. These findings are consistent with those from a recent MMB study by Milgrom and colleagues26 as well as with positive outcomes in a meta-analysis of a broader group of CBT-based eHealth programs19 and they provide additional support for implementing MMB2 as an evidence-based adjunct for in-person treatment.
In contrast to previous MMB randomized controlled trials,26,27 many current study participants reported lower severity levels of depression at baseline, a finding related to our use of pragmatic study design in which women screened positive on the PDP EPDS assessment before being enrolled in the study and completing the baseline assessment (mean lag of approximately 3 weeks). During that interval, study participants may have interacted with an experienced PDP social worker who encouraged them to use programs/products intended to improve their mood. Additional benefits from treatment might result were delays between screening and the offer of treatment reduced.56
When used in a largely self-directed approach, MMB2 could fill the gap when in-person treatment options are limited as well as for women whose circumstances (e.g., COVID) and/or concerns (e.g., stigma, costs) reduce the acceptability of in-person help. Furthermore, following a stepped-care approach, MMB2 could be used as a “treatment of first resort” and women who continue to suffer from mood and anxiety might subsequently be offered more intensive (higher cost) treatment options instead of or as an adjunct to MMB2. A stratified stepped care model57 might offer MMB2 to subthreshold or mild-moderate depressed women with more intensive clinical programs offered to women with major depression. Alternatively, making MMB2 more available as a self-directed option for use by all perinatal patients could have important indirect prevention benefits.
Research Implications
Research using MMB2 with larger and more diverse samples of depressed pregnant women is clearly warranted. Additional research could also examine the impact of MMB2 on anxiety given that current results were not consistent with earlier findings. Implementation research would help to identify strategies to encourage MMB2 adoption and sustainability by types of organizations including healthcare delivery, insurance, corporate employers, etc. Dissemination research would identify and examine possible strategies for increasing the penetration, reach, and scalability of MMB2. Cost-benefit analyses would also be helpful in this regard.
Strengths and Limitations
The current study had noteworthy strengths including the practical effectiveness trial was conducted within the context of a robust PDP counterfactual that was similar to a comparative effectiveness trial. In addition, the MMB2 group extensively used their program and many users rated it as helpful and easy to use. Study limitations included the absence of a long-term follow-up assessment didn’t permit evaluation for durability of treatment effects, and participants received remuneration for completing study assessments, a feature that might be unlikely in real world program delivery. An important additional limitation is that the study sample may not be representative of all US perinatal women. For example, compared to CDC data on mothers at time of birth during 2021,58 the maternal age of the study sample (M=32) was slightly older than the M=29 age of mothers at time of birth in national data. In addition, the racial composition of the study sample was more White (study sample=67% vs national birth data=51%) and Asian (14% vs. 6%), and less Black (8% vs. 14%). Moreover, only 1% of our sample had less than a high school education compared to 22% among mothers of births in 2019.59 Depressed perinatal women seeking assistance may well differ on a number of dimensions (including demographics) from the national population of women giving birth. For example, the study sample racial demographics closely approximated those obtained from NorthShore’s 3,313 deliveries during 2021 (e.g., White: study sample=67% vs NorthShore=73; Asian: 14% vs. 15%; Black: 8% vs. 11%). Although the current study was not designed to fully powered to examine race/ethnicity/SES as potential moderating factors, it is prudent to caution against generalizing study results to all perinatal women. It is also reasonable to assume that the MMB2 program may require further adaptation to be responsive to other cultural populations.
Conclusions
Results of the current study further strengthen and broaden the empirical foundation for the effectiveness of using MomMoodBooster as an evidence-based treatment option for perinatal depressed women. Using extant eHealth programs like MMB2 – especially combined with depression screening and referral – represents an approach that could potentially increase the reach and scale of treatment delivery thereby helping to reduce the estimated 85% of perinatal depressed women who go untreated.
Supplementary Material
AJOG at a Glance.
A. Why was this study conducted?
To compare the effectiveness of (a) healthcare organization’s routine perinatal depression care (universal screening and referral) and (b) routine care plus eHealth (MomMoodBooster2 program).
B. Key findings
Considering baseline to 12-week posttest change, both groups significantly decreased depression severity, anxiety, stress, and automatic thoughts, and increased behavioral activation and self-efficacy. The eHealth group reported significantly greater decreases in depression severity (including clinical significance of those changes) in addition to decreases in stress compared to routine care group. Group results were not moderated by depression severity (screening or baseline), anxiety, stress and pregnant/postpartum status. Women rated the eHealth program usable and helpful, and recommended it.
C. What does this add to what is known?
Patient use of eHealth program significantly improves benefits of an established routine perinatal depression care program in a healthcare delivery setting.
ACKNOWLEDGMENTS
Michael O’Hara PhD provided consultation to this project and he provided extremely valuable feedback to our initial plans for MMB2, especially regarding content added to expand the program’s use to include pregnant women.
Acknowledge key MMB2 design & asset providers: Seth Revoal at Revolution Design (Eugene, OR); Justin Felt, Laura Waters and colleagues at Emberex (Eugene, OR); Lou Swing at Digital Distillery Media (Eugene, OR), Melissa Lubofsky and Kim Lindquist at 33 Media Group (Rancho Palos Verdes, CA); Mary Maddux for guided meditations and related music by Richard Maddux of Meditation Oasis (Sonoma, CA).
Two NorthShore social workers introduced the study during the course of their clinical work, Suzanne Caulfield LSW and Teri McKean LCSW. Members of the NorthShore MOMS Line and Perinatal Family Support Center teams.
Role of the funding source
This work was supported by the National Institute of Mental Health (NIMH) SBIR Phase II Grant R44 MH109191 and SBIR Phase I Grant R43 MH109191. The funder had no input into the design and conduct of the study; the collection, management, analysis, and interpretation of the data; the preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication.
Date of registration: 06/24/2019
Date of initial participant enrollment: 01/09/2020
Clinical trial identification number: NCT03995316
URL of the registration site: https://clinicaltrials.gov/ct2/show/NCT03995316
Trial registration: Research protocol approved (# EH18-060) by the NorthShore University HealthSystem Institutional Review Board.
Abbreviations
- ATQ-SF
Automatic Thoughts Questionnaire Short Form
- BADS-SF
Behavioral Activation for Depression Scale Short Form
- CBT
Cognitive Behavioral Therapy
- CONSORT
Consolidated Standards of Reporting Trials
- DASS
21 Depression Anxiety Stress Scale
- EPDS
Edinburgh Postnatal Depression Scale
- MCID
Minimal Clinical Important Difference
- MMB2
Perinatal Version of MMB program
- MMB2+PDP
MomMoodBooster2 plus Perinatal Depression Program
- PDP
Perinatal Depression Program Routine Care (universal screening and referral of at-risk patients)
- PHQ-9
Patient Health Questionnaire
- REDCap
Research Electronic Data CAPture web application
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Disclosure of any potential interest
The authors report no conflict of interest.
REFERENCES
- 1.Underwood L, Waldie K, D’Souza S, Peterson ER, Morton S. A review of longitudinal studies on antenatal and postnatal depression. Arch Womens Ment Health 2016;19:711–720. 10.1007/s00737-016-0629-1 [DOI] [PubMed] [Google Scholar]
- 2.Dagher RK, Bruckheim HE, Colpe LJ, Edwards E, White DB. Perinatal depression: Challenges and opportunities. J Womens Health (Larchmt) 2021;30:154–159. 10.1089/jwh.2020.8862 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Dagher RK, McGovern PM, Dowd BE, Gjerdingen DK. Postpartum depression and health services expenditures among employed women. J Occup Environ Med 2012;54:210–215. 10.1097/JOM.0b013e31823fdf85 [DOI] [PubMed] [Google Scholar]
- 4.O’Hara MW, McCabe JE. Postpartum depression: Current status and future directions. Annu Rev Clin Psychol 2013;9:379–407. 10.1146/annurev-clinpsy-050212-185612 [DOI] [PubMed] [Google Scholar]
- 5.Field T. Prenatal depression risk factors, developmental effects and interventions: A review. J Pregnancy Child Health 2017;4. 10.4172/2376-127X.1000301 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Milgrom J, Ericksen J, McCarthy R, Gemmill AW. Stressful impact of depression on early mother-infant relations. Stress and Health 2006;22:229–238. 10.1002/smi.1101 [DOI] [Google Scholar]
- 7.Cox EQ, Sowa NA, Meltzer-Brody SE, Gaynes BN. The perinatal depression treatment cascade: Baby steps toward improving outcomes. J Clin Psychiatry 2016;77:1189–1200. 10.4088/JCP.15r10174 [DOI] [PubMed] [Google Scholar]
- 8.Kim JJ, La Porte LM, Corcoran M, Magasi S, Batza J, Silver RK. Barriers to mental health treatment among obstetric patients at risk for depression. Am J Obstet Gynecol 2010;202:312 e311–315. 10.1016/j.ajog.2010.01.004 [DOI] [PubMed] [Google Scholar]
- 9.Hoffman MC. Pushing beyond the silos: the obstetrician’s role in perinatal depression care. J Matern Fetal Neonatal Med 2021;34:3813–3819. 10.1080/14767058.2019.1691990 [DOI] [PubMed] [Google Scholar]
- 10.Sidebottom AC, Vacquier M, LaRusso E, Erickson D, Hardeman R. Perinatal depression screening practices in a large health system: Identifying current state and assessing opportunities to provide more equitable care. Arch Womens Ment Health 2021;24:133–144. 10.1007/s00737-020-01035-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Lomonaco-Haycraft KC, Hyer J, Tibbits B, et al. Integrated perinatal mental health care: A national model of perinatal primary care in vulnerable populations. Prim Health Care Res Dev 2018:1–8. 10.1017/S1463423618000348 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Avalos LA, Raine-Bennett T, Chen H, Adams AS, Flanagan T. Improved perinatal depression screening, treatment, and outcomes with a universal obstetric program. Obstet Gynecol 2016;127:917–925. 10.1097/AOG.0000000000001403 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Flanagan T, Avalos LA. Perinatal obstetric office depression screening and treatment: Implementation in a health care system. Obstet Gynecol 2016;127:911–915. 10.1097/AOG.0000000000001395 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Earls MF. Incorporating recognition and management of perinatal and postpartum depression into pediatric practice (Committee on Psychosocial Aspects of Child and Family Health, American Academy of Pediatrics). Pediatrics 2010;126:1032–1039. 10.1542/peds.2010-2348 [DOI] [PubMed] [Google Scholar]
- 15.ACOG. ACOG Committee Opinion No. 757: Screening for perinatal depression. Obstet Gynecol 2018;132:e208–e212. 10.1097/AOG.0000000000002927 [DOI] [PubMed] [Google Scholar]
- 16.Loughnan SA, Joubert AE, Grierson A, Andrews G, Newby JM. Internet-delivered psychological interventions for clinical anxiety and depression in perinatal women: A systematic review and meta-analysis. Arch Womens Ment Health 2019;22:737–750. 10.1007/s00737-019-00961-9 [DOI] [PubMed] [Google Scholar]
- 17.Byatt N, Biebel K, Lundquist RS, et al. Patient, provider, and system-level barriers and facilitators to addressing perinatal depression. J Reprod Infant Psychol 2012;30:436–449. 10.1080/02646838.2012.743000 [DOI] [Google Scholar]
- 18.Gloaguen V, Cottraux J, Cucherat M, Blackburn IM. A meta-analysis of the effects of cognitive therapy in depressed patients. J Affect Disord 1998;49:59–72. 10.1016/s0165-0327(97)00199-7 [DOI] [PubMed] [Google Scholar]
- 19.Karyotaki E, Efthimiou O, Miguel C, et al. Internet-based Cognitive Behavioral Therapy for depression: A systematic review and individual patient data network meta-analysis. JAMA Psychiatry 2021;78:361–371. 10.1001/jamapsychiatry.2020.4364 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Etzelmueller A, Vis C, Karyotaki E, et al. Effects of internet-based Cognitive Behavioral Therapy in routine care for adults in treatment for depression and anxiety: Systematic review and meta-analysis. J Med Internet Res 2020;22:e18100. 10.2196/18100 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Cuijpers P, Karyotaki E. The effects of psychological treatment of perinatal depression: An overview. Arch Womens Ment Health 2021;24:801–806. 10.1007/s00737-021-01159-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Loughnan SA, Butler C, Sie AA, et al. A randomised controlled trial of 'MUMentum postnatal': Internet-delivered cognitive behavioural therapy for anxiety and depression in postpartum women. Behav Res Ther 2019;116:94–103. 10.1016/j.brat.2019.03.001 [DOI] [PubMed] [Google Scholar]
- 23.Danaher BG, Milgrom J, Seeley JR, et al. MomMoodBooster web-based intervention for postpartum depression: Feasibility trial results. J Med Internet Res 2013;15:e242. 10.2196/jmir.2876 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Solness CL, Kroska EB, Holdefer PJ, O’Hara MW. Treating postpartum depression in rural veterans using internet delivered CBT: Program evaluation of MomMoodBooster. J Behav Med 2020. 10.1007/s10865-020-00188-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Danaher BG, Milgrom J, Seeley JR, et al. Web-based intervention for postpartum depression: Formative research and design of the MomMoodBooster program. JMIR Res Protoc 2012;1:e18. 10.2196/resprot.2329 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Milgrom J, Danaher BG, Seeley JR, et al. Internet and face-to-face Cognitive Behavioral Therapy for postnatal depression compared with treatment as usual: Randomized controlled trial of MumMoodBooster. J Med Internet Res 2021;23:e17185. 10.2196/17185 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Milgrom J, Danaher BG, Gemmill AW, et al. Internet cognitive behavioral therapy for women with postnatal depression: A randomized controlled trial of MumMoodBooster. J Med Internet Res 2016;18:e54. 10.2196/jmir.4993 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Gemmill AW, Oliva JL, Ericksen J, Holt C, Holt CJ, Milgrom J. Web-based treatment for depression in pregnancy: A feasibility study of Mum2BMoodBooster. BMC Psychiatry 2022;22:476. 10.1186/s12888-022-04111-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Cuijpers P, Donker T, van Straten A, Li J, Andersson G. Is guided self-help as effective as face-to-face psychotherapy for depression and anxiety disorders? A systematic review and meta-analysis of comparative outcome studies. Psychol Med 2010;40:1943–1957. 10.1017/S0033291710000772 [DOI] [PubMed] [Google Scholar]
- 30.Cardone IA, Kim JJ, Gordon TE, Gordon SM, Silver RK. Psychosocial assessment by phone for high-scoring patients taking the Edinburgh Postnatal Depression Scale: Communication pathways and strategies. Arch Womens Ment Health 2006;9:87–94. 10.1007/s00737-005-0110-z [DOI] [PubMed] [Google Scholar]
- 31.Cox JL, Holden JM, Sagovsky R. Detection of postnatal depression. Development of the 10-item Edinburgh Postnatal Depression Scale. Br J Psychiatry 1987;150:782–786. 10.1192/bjp.150.6.782 [DOI] [PubMed] [Google Scholar]
- 32.Levis B, Negeri Z, Sun Y, Benedetti A, Thombs BD, Group DESDE. Accuracy of the Edinburgh Postnatal Depression Scale (EPDS) for screening to detect major depression among pregnant and postpartum women: systematic review and meta-analysis of individual participant data. BMJ 2020;371:m4022. 10.1136/bmj.m4022 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform 2019;95:103208. 10.1016/j.jbi.2019.103208 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.La Porte LM, Kim JJ, Adams M, Du H, Silver RK. The pattern of depression screening results across successive pregnancies. Am J Obstet Gynecol 2012;206:261 e261–264. 10.1016/j.ajog.2011.12.005 [DOI] [PubMed] [Google Scholar]
- 35.Kim JJ, Silver RK, Elue R, et al. The experience of depression, anxiety, and mania among perinatal women. Arch Womens Ment Health 2016;19:883–890. 10.1007/s00737-016-0632-6 [DOI] [PubMed] [Google Scholar]
- 36.Gordon TE, Cardone IA, Kim JJ, Gordon SM, Silver RK. Universal perinatal depression screening in an academic medical center. Obstet Gynecol 2006;107:342–347. 10.1097/01.AOG.0000194080.18261.92 [DOI] [PubMed] [Google Scholar]
- 37.Kim JJ, La Porte LM, Adams MG, Gordon TE, Kuendig JM, Silver RK. Obstetric care provider engagement in a perinatal depression screening program. Arch Womens Ment Health 2009;12:167–172. 10.1007/s00737-009-0057-6 [DOI] [PubMed] [Google Scholar]
- 38.Ater T. Building progressive web apps: Bringing the power of native to the browser. Sebastopol, CA: O’Reilly Media; 2017. [Google Scholar]
- 39.Danaher BG, Brendryen H, Seeley JR, Tyler MS, Woolley T. From black box to toolbox: Outlining device functionality, engagement activities, and the pervasive information architecture of mHealth interventions. Internet Interv 2015;2:91–101. 10.1016/j.invent.2015.01.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Kroenke K, Spitzer RL, Williams JB. The PHQ-9: Validity of a brief depression severity measure. J Gen Intern Med 2001;16:606–613. 10.1046/j.1525-1497.2001.016009606.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Spitzer RL, Kroenke K, Williams JB. Validation and utility of a self-report version of PRIME-MD: The PHQ primary care study. JAMA 1999;282:1737–1744. 10.1001/jama.282.18.1737 [DOI] [PubMed] [Google Scholar]
- 42.Sidebottom AC, Harrison PA, Godecker A, Kim H. Validation of the Patient Health Questionnaire (PHQ)-9 for prenatal depression screening. Arch Womens Ment Health 2012;15:367–374. 10.1007/s00737-012-0295-x [DOI] [PubMed] [Google Scholar]
- 43.Lowe B, Unutzer J, Callahan CM, Perkins AJ, Kroenke K. Monitoring depression treatment outcomes with the patient health questionnaire-9. Med Care 2004;42:1194–1201. 10.1097/00005650-200412000-00006 [DOI] [PubMed] [Google Scholar]
- 44.Grote NK, Katon WJ, Russo JE, et al. A randomized trial of collaborative care for perinatal depression in socioeconomically disadvantaged women: The impact of comorbid posttraumatic stress disorder. J Clin Psychiatry 2016;77:1527–1537. 10.4088/JCP.15m10477 [DOI] [PubMed] [Google Scholar]
- 45.Hirshler Y, Gemmill AW, Milgrom J. An Australian perspective on treating perinatal depression and anxiety: A brief review of efficacy and evidence-based practice in screening, psychosocial assessment and management. Ann Ist Super Sanita 2021;57:40–50. 10.4415/ANN_21_01_07 [DOI] [PubMed] [Google Scholar]
- 46.Anthony MM, Bieling PJ, Cox BJ, Enns MW, Swinson RP. Psychometric properties of the 42-item and 21-item versions of the Depression Anxiety Stress Scales in clinical groups and a community sample. Psychol Assess 1998;10:176–181. 10.1037/1040-3590.10.2.176 [DOI] [Google Scholar]
- 47.Lovibond SH, Lovibond PF. Manual for Depression Anxiety Stress Scales. 2nd. Sydney, Australia: Psychology Foundation of Australia; 1995. [Google Scholar]
- 48.Lovibond PF, Lovibond SH. The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behav Res Ther 1995;33:335–343. 10.1016/0005-7967(94)00075-u [DOI] [PubMed] [Google Scholar]
- 49.Grigoriadis S, Graves L, Peer M, et al. A systematic review and meta-analysis of the effects of antenatal anxiety on postpartum outcomes. Arch Womens Ment Health 2019;22:543–556. 10.1007/s00737-018-0930-2 [DOI] [PubMed] [Google Scholar]
- 50.Manos RC, Kanter JW, Luo W. The Behavioral Activation for Depression Scale-short form: Development and validation. Behav Ther 2011;42:726–739. 10.1016/j.beth.2011.04.004 [DOI] [PubMed] [Google Scholar]
- 51.Kanter JW, Mulick PS, Busch AM, Berlin KS, Martell CR. Behavioral Activation for Depression Scale (BADS) (Long and Short Form): Measurement Instrument Database for the Social Science (MIDSS), 2012. [Google Scholar]
- 52.Hollon SD, Kendall PC. Cognitive Self-Statements in Depression - Development of an Automatic Thoughts Questionnaire. Cognit Ther Res 1980;4:383–395. 10.1007/Bf01178214 [DOI] [Google Scholar]
- 53.Netemeyer RG, Williamson DA, Burton S, et al. Psychometric properties of shortened versions of the Automatic Thoughts Questionnare. Educ Psychol Meas 2002;62:111–129. 10.1177/0013164402062001008 [DOI] [Google Scholar]
- 54.Feingold A. Effect sizes for growth-modeling analysis for controlled clinical trials in the same metric as for classical analysis. Psychol Methods 2009;14:43–53. 10.1037/a0014699 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Danaher BG, Severson HH, Andrews JA, et al. Randomized controlled trial of MyLastDip: a Web-based smokeless tobacco cessation program for chewers ages 14-25. Nicotine Tob Res 2013;15:1502–1510. 10.1093/ntr/ntt006 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Cuijpers P. Indirect prevention and treatment of depression: An emerging paradigm? Clin Psychol Eur 2021;3:e6847. 10.32872/cpe.6847 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Delgadillo J, Ali S, Fleck K, et al. Stratified care vs stepped care for depression: A cluster randomized clinical trial. JAMA Psychiatry 2022;79:101–108. 10.1001/jamapsychiatry.2021.3539 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Osterman MJK, Hamilton BE, Martin JA, Driscoll AK, Valenzuela CP. Births: Final Data for 2020. Natl Vital Stat Rep 2022;70:17; February 7, 2022. US Department of Health and Human Services, Centers for Disease Control and Prevention, National Vital Statistics System (NVSS) http://dx.doi.org/Ihttps://www.cdc.gov/nchs/data/nvsr/nvsr70/nvsr70-17.pdf [PubMed] [Google Scholar]
- 59.Hamilton BE. Total fertility rates, by maternal educational attainment and race and Hispanic origin: United States, 2019. Natl Vital Stat Rep 2021;70:5; May 12, 2021. US Department of Health and Human Services, Centers for Disease Control and Prevention, National Vital Statistics System (NVSS) http://dx.doi.org/Ihttps://www.cdc.gov/nchs/data/nvsr/nvsr70/nvsr70-05-508.pdf [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.