Abstract
Objective
Improving use of effective contraception to prevent unintended pregnancy is a global priority, but misperceptions and concerns about contraception are common. Our objective was to evaluate an interactive website to aid informed choice of contraception.
Methods
The Contraception Choices website is an interactive digital intervention which offers tailored advice to aid contraception decision-making (www.contraceptionchoices.org). In a parallel single-blind trial, we randomised 927 women aged 15–30 years from six clinic settings to access the intervention website (n = 464) or to a waiting-list control group (n = 463). The study was initially a feasibility trial, evolving into an evaluation of efficacy, with two primary outcomes at six months: long-acting reversible contraception (LARC) use, and satisfaction with contraceptive method. Secondary outcomes included self-reported pregnancy and sexually transmitted infection diagnoses. Free-text comments on the 3 and 6 month outcome surveys were analysed thematically.
Findings
There was no significant difference between intervention and control groups in the proportion of women using LARC [30.4% intervention versus 31.0% control; adjusted odds ratio 0.87 (95% confidence interval 0.60 to 1.28)]; satisfaction with contraceptive method [82.6% versus 82.1%; adjusted ordinal odds ratio 0.93 (95% CI 0.69 to 1.25)]; self-reported pregnancy [3.3% versus 4.1%; adjusted odds ratio 0.90 (95% CI 0.45 to 1.79)] nor sexually transmitted infection [5.3% versus 4.7%; adjusted odds ratio 0.72 (95% CI 0.55 to 2.36)]. Highly positive free-text comments from intervention participants indicated that the website facilitates contraception choice and can help women feel better prepared before consultation with healthcare providers.
Interpretation
The Contraception Choices website was popular for its design, trustworthy information and decision aids but it was not associated with significant differences in use of LARC or satisfaction with contraceptive method. An interactive website can aid contraception choice, but interventions that address factors beyond women’s control, such as access to services, and partner, family or community influences are needed to complement this approach.
Research in context
Preventing unintended pregnancy through effective use of contraception is essential for women’s health, but choosing between different contraceptive methods can be challenging, and the opportunity for adequate discussion during routine consultations is often constrained.
Evidence before this study
We conducted two systematic literature reviews: 1) Factors influencing contraception choice, uptake and use: a meta-synthesis of systematic reviews; and 2) Effectiveness of interactive digital interventions (IDI) for contraception choice, uptake and use. For the first review we searched PubMed, CDSR, Epistemonikos, DoPHER, DARE, NHS Economic Evaluation Database, Campbell Library, NIHR Health Technology Assessment, and Health Evidence Canada databases for systematic reviews which addressed contraceptive choice, uptake or use, from 2000 to 2017. PROSPERO registration number: CRD42017081521 https://www.crd.york.ac.uk/PROSPERO/display_record.php?RecordID=81521. We synthesised the findings of 18 systematic reviews of mostly moderate or high quality. They highlighted the importance of women’s knowledge, beliefs, perceptions of side effects and health risks, as well as relationship status, social network, economic and healthcare factors on contraception choice and use. For the second review, we searched 23 electronic databases, trials registers and reference lists for randomised controlled trials of IDI for contraception, including CENTRAL, MEDLINE, EMBASE, CINAHL, ERIC, ASSIA and PsycINFO, from start date to June 2017. PROSPERO registration number: CRD42017081636. We found only five randomised trials of IDI, all from the USA. Risk of bias prevented synthesis of results. www.crd.york.ac.uk/PROSPERO/display_record.php?RecordID=81636.
Added value of this study
Women’s common concerns about contraception – fear of hormones, weight gain, cancer, infertility, mood changes, breaks from contraception and changes in bleeding patterns – underpinned development of a new interactive website (www.contraceptionchoices.org). Contraception Choices addresses women’s concerns through succinct text; Q and A format (Frequently Asked Questions, Did you Know?; videos of women and health professionals); an effectiveness infographic, and an interactive decision aid (What’s right for me?).
In an online randomised trial with 927 women attending clinics, we found no association of the Contraception Choices intervention with the primary outcomes – satisfaction with contraceptive method and uptake of long-acting reversible methods at 6 months. Nor did we find an association with secondary adverse outcomes – sexually transmitted infections or pregnancy. Comments from women indicated that the website can meet young women’s need for information on the benefits and drawbacks of contraception, help them to make informed decisions, and feel better prepared before healthcare consultations. Contraception Choices is now available on the NHS website: www.nhs.uk/conditions/contraception/which-method-suits-me
Implications of all the available evidence
Interactive digital interventions (websites) can aid contraception choice, but other intervention research is needed to address wider influences on unintended pregnancy, including partner views, friends, family, the media, wider society and experiences with healthcare professionals. Future research could examine the impact of the website in different settings, e.g. schools or different countries. We hypothesise that use of the website during contraceptive consultations might improve the efficiency or quality of consultation, for both patients and healthcare providers. Appropriate methodology and time-scale for evaluating digital health interventions remains a key question.
Keywords: Contraception, contraceptive methods, interactive digital intervention, digital health, randomised controlled trial
Introduction
Control of fertility, and feeling satisfied with a chosen method of contraception are crucial to the health and wellbeing of women, but unintended pregnancy remains common and costly for individuals and for health services. Globally, about 40% of pregnancies are estimated to be unplanned.1 In Britain, around 45% of pregnancies are unplanned or ambivalent2 despite a range of freely available effective contraceptive methods, and abortion rates in England and Wales have changed little since 2011.3
Preventing unintended pregnancy involves many steps, including timely education, awareness and socially influenced behaviours to seek, choose and use contraception consistently and correctly.4 Health services have a key role to play by supporting people to choose and use an appropriate method that best meets their needs, but many people are not aware of the range of different methods available to them.5,6 The contraceptive pill and condoms are well known and widely used, but are not the most effective contraceptive methods. Long-acting reversible contraception (LARC), which includes intrauterine devices, intrauterine systems, implants and injections, are at least 20 times more effective than oral contraceptive pills and condoms7,8 but these methods are less well known, and not all services have the capacity to fit them.9
Increasingly women turn to online sources of information on sexual health,10 but information is of variable quality and accuracy11 and misperceptions about contraception are common.12 Hormonal contraception methods have many potential benefits apart from control of fertility, including treatment of acne, reduced period pain, lighter periods or no withdrawal bleeds, and reduction in premenstrual symptoms,13 but women may be more aware of risks and side effects than benefits of contraception.12
Interactive (tailored) digital interventions are effective for increasing contraception knowledge14 uptake of more effective contraceptive methods and contraception adherence,15–17 and decreasing unplanned pregnancy.18,19 Digital interventions offer the advantages of intervention content accuracy and fidelity, and the potential to reach large audiences with relatively low dissemination costs.20 We therefore developed an interactive website to aid informed choice of contraceptive method and then conducted a randomised controlled trial to evaluate its impact in clinic populations.
Aim
To assess the efficacy of the Contraception Choices website in comparison with control (waiting list) on uptake of long-acting contraceptive methods, and satisfaction with method choice in young women.
Methods
We conducted an individually randomised, parallel group-controlled trial that started as a feasibility trial and ended as an efficacy (clinical) trial. Approval was given by London Camden & Kings Cross Research Ethics Committee (Reference 17/LO/0112).
Summary of intervention development
We conducted two systematic reviews of the literature to generate the evidence base for the website: a review of reviews of factors influencing contraception choice and use and a review of interactive digital interventions for contraception. To gain the views of contraceptive users, we recruited women from sites in London that represent the settings in which the great majority of contraceptive care occurs in the UK:21 a general practice, two sexual health centres, an abortion clinic, a community pharmacy and an antenatal clinic. Eligibility criteria were women aged 15–30 years, ability to give informed consent, and interest in taking part in contraceptive research. We conducted focus groups and individual interviews to explore the views of 74 young women relating to contraception (access, acceptability, barriers, concerns, benefits and personal decisions around choices) and their views on website design and content. Working iteratively with the young women and a commercial software company (Moore Wilson), we synthesised the findings from the systematic reviews and qualitative research with women’s views to develop a trial-ready, self-guided website offering tailored advice.
Intervention
The Contraception Choices website offers tailored information to help users to decide which method of contraception might suit them best, to facilitate informed choices, satisfaction with choice, and uptake of more effective methods (Logic Model, supplementary material). Contraception Choices provides information about contraception: videos of women and health professionals discussing contraceptive experiences, concerns and misperceptions; an infographic representing contraception effectiveness; Did you Know? and Contraception FAQ sections which address common concerns, and an interactive decision tool What’s right for me? (See Supplementary file: Screenshots). The What’s right for me? tool (Figure 2) elicits seven individual priorities concerning contraception attributes. For example, selecting ‘Regular periods’ highlights methods compatible with regular periods (simultaneously fading out those that can alter the menstrual cycle) and the algorithm scores the highlighted methods more highly. Three methods most consistent with the individual’s preferences are displayed and compared side-by-side, and the user can export their results by email or text message. www.contraceptionchoices.org (Figure 1a and Figure 1b).
Figure 2.
CONSORT diagram.
Figure 1a.
Contraception Choices website: home page.
The website content and design is underpinned by a number of different theoretical principles including the following:
Human-centred design and collaboration with target users– we involved target users in content and design decisions, to ensure that content met young women’s needs, priorities and preferences.20
Tailoring – the What’s right for me? decision tool offers tailored information on contraception method choices to increase relevance for more effective learning, engagement, and behaviour change.22
Health belief model – the Effectiveness infographic addresses perceptions of risk of pregnancy, showing the relative effectiveness of different contraception methods.
Social cognitive theory – Contraception Choices videos draw on the influence of peers by featuring young women discussing their experiences of contraception including the potential benefits of different methods.
The COM B model: capability, opportunity, motivation-behaviour model23 – this takes into account factors which are on pathways to behaviour change.
A social determinants of health framework24 underpins the design of two infographics which convey barriers to contraception use across different domains: individual women, partners, family, peers, community, health services and wider society.
Participants
Inclusion criteria: women aged 15–30 years with a current or future need for contraception, attending one of the study sites, able to read English, with an active email account and access to the internet and willing to be followed up for 6 months. Exclusion criteria: unable to provide informed consent (e.g. severe learning difficulties) or need for a language advocate (because the intervention was intended to be accessed in private). At the sites described above, women waiting for their appointment (or in the pharmacy) were approached by a researcher with a ‘tablet’ computer and invited to take part in the trial. Women recruited via the online booking system were sent a text message inviting them to view the Contraception Choices website before their booked appointment. Those who expressed interest were recruited online via the tablet computer using software designed specifically for this trial, to confirm eligibility and register informed consent.
Procedures
Participants were asked to complete a short questionnaire at baseline which included demographic data (age, ethnicity, highest completed level of education) and whether English was their first language; current use of contraception, or reasons for non-use (including being pregnant); contraception method; from where the method was obtained, and whether it was free or paid for; satisfaction with current contraception (very satisfied, satisfied, neither satisfied or dissatisfied, dissatisfied and very dissatisfied); ever used contraceptive methods (same list as current use); and self-reported sexually transmitted infections in the last 3 months. Automated, computerised randomisation occurred immediately after baseline data collection.
Randomisation and masking
A randomisation list was generated by a random number based algorithm in the computer software Stata25 and incorporated into the trial software programme to allocate all participants to either the intervention or control group. The randomisation list was stratified by setting and used varying block sizes. Allocation was immediate (online) and concealed. Those randomised online to the intervention group gained access to the Contraception Choices website immediately; women randomised to the control group could access the website at the end of data collection, at which point we emailed them with a link to the website.
At 3 and 6 month follow-up, participants were emailed a short online survey asking what method of contraception they were using (including none) and how satisfied they were with the method; whether they had had a pregnancy and, if so, the outcome of the pregnancy (ongoing, gave birth, miscarried, terminated, or prefer not to say); and self-reported sexually transmitted infection in the previous 3 months.
All participants were asked whether they had visited the Contraception Choices website (control participants were asked in order to assess ‘contamination’) and asked a free-text question: “Has being in the study had any good or bad effects on your life?” Intervention group participants only were asked further questions about the website: how helpful it was in terms of ‘getting useful information about contraception’ and ‘finding a method of contraception that is right for you’ (five response options from very helpful to very unhelpful); whether they had discussed the website with anyone (including a doctor or nurse, pharmacist, partner, family or friends) and “What did you like or dislike about the website?” (as a free-text question).
Participants were sent electronic vouchers for completing follow-up surveys (£5 for the 3 month survey, £15 for the 6 month survey). All follow-up emails included a link to enable participants to withdraw from the study, including one sent immediately after enrolment.
The analysis of the primary outcomes was conducted blinded to allocation.
Outcomes – original feasibility trial
The primary outcome for the initial feasibility trial was follow-up rate at 6 months. Secondary outcomes at 6 months were: effectiveness of contraceptive method used, grouped from least to most effective as follows: no method, withdrawal or natural method, condoms or diaphragm, pill, patch or ring, LARC or sterilisation; change in effectiveness of method between baseline and 6 months; pregnancy and sexually transmitted infection.
Outcomes – efficacy trial
As we were developing Contraception Choices, its popularity with young women became very evident. Moreover, our presentations of the website to colleagues were met with demand to use it immediately in clinical practice. Future funding for a definitive trial was highly uncertain and withholding the website for a number of years, in order that a definitive trial with a control group of women who had not seen the website might be funded and completed, felt unpalatable. Fortunately, the opportunity arose to increase recruitment at minimal cost (see below) and so we were able to expand the feasibility trial into an efficacy trial by substituting the primary outcome (follow-up rate at 6 months) for two primary clinical outcomes that were originally secondary outcomes – use of LARC at 6 months and satisfaction with contraceptive method at 6 months. (We pre-specified that we would consider the trial to have demonstrated superiority of the intervention if a statistically significant benefit were observed for one or both outcomes without clear evidence of harm for either). Secondary outcomes were effectiveness of contraceptive method at 6 months; change in method from baseline to 6 months; pregnancy by 6 months and diagnosed sexually transmitted infection reported at 3 or 6 months.
To increase recruitment for the efficacy trial, we took advantage of an online booking system for appointments at one site (sexual and reproductive health clinic) by adding a hyperlink about the trial into the text message that women received to confirm their contraception clinic appointment. Clicking on the hyperlink took them directly to the trial website for recruitment and randomisation.
Data analysis
The original target sample size for the feasibility trial was 80 participants per setting, based on estimating a follow-up rate at 6 months of 70% to within 10% precision (95% CI 60% to 80%) for each setting. Changing the design to a clinical efficacy trial resulted in a total sample size of 930 participants, based on the power to assess the effect of the intervention on the revised primary outcome, use of LARC at 6 month follow-up. Specifically, assuming a follow-up rate of 70%, this sample size provides at 82% power to detect as significant (at the 5% level) an increase from 35% (control group prevalence) to 47% in LARC use in the intervention group. No formal sample size calculation was made for the other primary outcome. The standard 5% significance level was taken because although there are two primary outcomes we use results from both to assess whether the intervention is beneficial.
The primary analysis was by modified intention-to-treat, basing analysis on those who completed at least one follow-up outcome questionnaire. For each primary and secondary outcome listed earlier we present the percentage of participants if the outcome is binary (e.g. use of LARC) or ordinal (e.g. effectiveness of method, satisfaction with method) together with a 95% confidence interval. These percentages and means are reported separately by intervention and standard care arm. To formally assess differences between arms we used logistic regression (for binary outcomes), or ordinal logistic regression (for ordinal outcomes), reporting adjusted odds ratios with 95% confidence intervals.
The primary outcome of LARC use at 6 months was analysed among women in need of contraception (i.e. not pregnant or currently trying to become pregnant) and the primary outcome of satisfaction with method was analysed among women who were using a method at 6 months. The primary outcome of LARC use at 6 months was analysed stratified by LARC use at baseline, leading to three intervention effects: the effect in baseline LARC users, the effect in baseline non-users, and the overall effect adjusted for baseline LARC use. We pre-specified that if fewer than 90% of baseline LARC users in the control arm are using a LARC method at 6 months, then the primary effect measure would be the overall adjusted intervention effect, and otherwise (due to limited scope for increase in baseline LARC users) the primary effect measure would be the effect in baseline non-users only. Besides adjustment for baseline LARC use, analysis of both primary outcomes was also adjusted for satisfaction with method at baseline and by setting. A further subgroup analysis, based on testing an interaction term, was conducted for both primary outcomes to assess whether the effect of the intervention varies by setting (specifically between online and in-person recruitment). Comparisons for the primary outcomes between arms were based on multiple imputation where the primary outcomes at 6 months were imputed based on the outcomes at 3 months for participants who completed the 3 month outcome questionnaire but failed to complete the questionnaire at 6 months. Imputation was conducted using the chained equations approach and implemented using the mi impute function,10 twenty imputed datasets were generated. Considering our secondary outcomes, analysis of contraceptive effectiveness was restricted to women in need of contraception at 6 months, reporting of a change in method was restricted to women in need of contraception at both baseline and 6 months, and analysis of pregnancy at 6 months was restricted to women who were not pregnant or trying for a baby at baseline. All analysis was conducted in Stata 15 software.
A post hoc decision was made to conduct a ‘per-protocol’ analysis for the primary outcomes based on a comparison of intervention arm participants who reported seeing the Contraception Choices website with all control arm participants.
All free-text comments from the 3 month and 6 month trial follow up surveys were imported into NVIVO software, and coded. We used thematic analysis to identify patterns and links across the data set. Two researchers (AG and JAS) independently coded the data, categorised data by theme, and identified relationships between concepts to develop a coding frame. Coding decisions were reviewed by a third researcher (JB).
Role of the funding source
The funding source had no role in the writing of the manuscript or the decision to submit it for publication.
Results
Recruitment and follow-up
The first participant was randomised on 4 July 2017 and the last on 22 December 2017. The first recruitment through the online booking service was on 31 October and the last on 22 December 2017. Recruitment online was much faster than in the clinics. It took approximately 6 months to recruit 419 women from the clinic sites, and just over seven weeks to recruit 508 women via the online booking system (Table 1).
Table 1.
Baseline characteristics by study arm.
| Characteristic, % (n) | Intervention, N = 464 | Control, N = 463 |
|---|---|---|
| Site of recruitment | ||
| Sexual and reproductive health clinic | 8.6 (40) | 8.4 (39) |
| General practice | 7.3 (34) | 8.2 (38) |
| Abortion service | 4.5 (21) | 4.8 (22) |
| Maternity service | 8.4 (39) | 8.6 (40) |
| Community pharmacy | 6.9 (32) | 7.1 (33) |
| Sexual Health clinic for young people | 8.8 (41) | 8.6 (40) |
| Direct online booking (for the SRH clinic above) | 55.4 (257) | 54.2 (251) |
| Demographic factors | ||
| Age, median (IQR) | 24 (21–27) | 24 (21–27) |
| Ethnicity | ||
| White | 67.0 (306) | 71.0 (326) |
| Mixed | 11.6 (53) | 10.5 (48) |
| Asian | 10.3 (47) | 8.1 (37) |
| Black | 9.0 (41) | 8.3 (38) |
| Other | 2.2 (10) | 2.2 (10) |
| First language | ||
| English | 80.8 (375) | 84.2 (390) |
| Not English | 19.2 (89) | 15.8 (73) |
| Highest completed level of education | ||
| Degree | 51.3 (238) | 49.9 (231) |
| Diploma in higher education | 10.6 (49) | 9.7 (45) |
| A/AS levels | 21.3 (99) | 23.5 (109) |
| O levels / GCSE | 9.7 (45) | 7.8 (36) |
| Other | 5.4 (25) | 5.8 (27) |
| None | 1.7 (8) | 3.2 (15) |
| Contraception factors | ||
| Need for contraception | ||
| No – Pregnant | 9.7 (45) | 10.6 (49) |
| No – Trying for baby | 0.2 (1) | 0.2 (1) |
| Yes – Neither | 90.1 (418) | 89.2 (413) |
| Method used, if needed, ordered by efficacy | ||
| None | 17.7 (74) | 18.6 (77) |
| Unclear | 0.2 (1) | 0.5 (2) |
| Withdrawal/natural | 0 (0) | 1.2 (5) |
| Condom/diaphragm | 10.5 (44) | 11.1 (46) |
| Pill/patch/ring | 42.6 (178) | 39.0 (161) |
| LARC/sterilisation | 29.0 (121) | 29.5 (122) |
| Satisfaction with method, if using a method | ||
| Very dissatisfied | 4.3 (15) | 3.3 (11) |
| Dissatisfied | 11.6 (40) | 12.2 (41) |
| Neutral | 16.2 (56) | 17.2 (58) |
| Satisfied | 36.4 (126) | 32.6 (110) |
| Very satisfied | 31.5 (109) | 34.7 (117) |
LARC: long-acting reversible contraception
The last follow-up survey was completed on 16 August 2018. The CONSORT diagram (Figure 2) details the flow of participants through the trial. In total, 927 women were randomised to the website (n = 464) or to control group (n = 463) of whom 739 (80%) provided follow-up data at 6 months, and 786 women (86%) provided data at 3 and/or 6 months for analysis of primary outcomes with imputation.
Follow-up rates were similar across all sites (data not shown) except for the abortion service, where the follow-up rate was only 50%. The quality of the follow up survey data collected was very high, with all respondents providing the primary outcome data. Eighteen (2%) women (11 in the intervention group and 7 in the control group) withdrew from the trial without offering reasons but they did not request that their data be withdrawn from analysis.
The proportion of women who reported that they had seen the Contraception Choices website at any time during the trial was 86% in the intervention group and 7% in the control group, indicating good exposure in the intervention group and little ‘contamination’ in the control group.
Baseline data
Just over two-thirds of participants were from White ethnic groups, half were educated to degree level and four-fifths reported English as their first language (Table 1). Ten percent were pregnant at enrolment, while 90% indicated a current need for contraception to avoid unintended pregnancy. The most common method reported at baseline was the oral contraceptive pill at 39.5% (n = 167) in the intervention group and 34.6% (n = 146) in the control group, followed by LARC methods (Table 1). Around two-thirds of women were satisfied with their current method at baseline. (Only one woman reported being sterilised at baseline; she is not included in subsequent analysis because she did not complete a follow-up.)
Primary outcomes
There were no significant difference between intervention and control groups in the proportion of women using LARC at 6 months [30.4% intervention versus 31.0% control; adjusted odds ratio after imputation 0.87 (95% confidence interval 0.60–1.28)], or in level of satisfaction with contraceptive method [proportion being ‘satisfied’ or ‘very satisfied’ 82.6% intervention versus 82.1% control; adjusted ordinal odds ratio after imputation 0.93 (95% CI 0.69–1.25) based on the five ordered responses].
Around half of the participants in each group changed their method of contraception between baseline and 6 months; the most common change was to a more effective method (24% intervention group; 21% control) but 19% in the intervention group and 16% in the control group changed to a less effective method (Table 2). Among women who were using LARC at baseline, the proportion using LARC at 6 months was significantly higher in the control group than the intervention group (Table 2). Among participants not using LARC at baseline, there was a non-significantly higher proportion using LARC at 6 months in the intervention group. Across both study arms, satisfaction with method of contraception improved from around two-thirds at baseline to four-fifths at follow-up. There was no difference between groups in the proportion of women who were pregnant at 6 months (among women who were neither pregnant nor trying for a baby at baseline) or the proportion who reported a diagnosed sexually transmitted infection (STI) at 3 or 6 months (Table 2).
Table 2.
Comparison of outcomes between arms.
| Outcome measures, % (n) | Intervention | Control | OR (CI), Intervention vs. control | Adjusted OR (CI) |
|---|---|---|---|---|
| Primary outcomes | ||||
| LARC method in use at 6 months, overall N = 7131 | 30.4 (106) | 31.0 (113) |
p = 0.74 0.95 (0.69–1.30) |
p = 0.48 0.87 (0.60–1.28)6 |
| If using LARC at baseline, N = 201 | 58.8 (57) | 70.2 (73) |
p = 0.12 0.63 (0.35–1.12) |
p = 0.024 0.46 (0.23–0.90)7 |
| If not using LARC at baseline, N = 512 | 19.4 (49) | 15.4 (40) |
p = 0.40 1.22 (0.77–1.92) |
p = 0.49 1.18 (0.74–1.90)7 |
| Satisfaction with method used at 6 months, N = 6242 | ||||
| Very dissatisfied | 1.9 (6) | 1.6 (5) | ||
| Dissatisfied | 4.2 (13) | 5.8 (18) | p = 0.54 | p = 0.62 |
| Neutral | 11.3 (35) | 10.5 (33) | 0.91 (0.68–1.22) | 0.93 (0.69–1.25)6 |
| Satisfied | 39.9 (124) | 35.5 (111) | ||
| Very satisfied | 42.8 (133) | 46.7 (146) | ||
| Secondary outcomes | ||||
| Effectiveness of contraceptive method at 6 months, N = 7131 | ||||
| None | 10.9 (38) | 14.0 (51) | ||
| Withdrawal/natural | 1.7 (6) | 1.4 (5) | p = 0.63 | p = 0.33 |
| Condom/diaphragm | 12.6 (44) | 12.1 (44) | 1.07 (0.82–1.40) | 1.15 (0.87–1.52)8 |
| Pill/patch/ring | 44.4 (155) | 41.5 (151) | ||
| LARC/sterilisation | 30.4 (106) | 31.0 (113) | ||
| Change in method from baseline to 6 months, N = 6513 | ||||
| Change to more effective | 23.8 (76) | 20.5 (68) | ||
| Change to similarly effective | 11.3 (36) | 12.7 (42) | N/A | N/A |
| No change | 46.3 (148) | 51.1 (169) | ||
| Change to less effective | 18.8 (60) | 15.7 (52) | ||
| Pregnancy by 6 months, N = 6704 | 3.3 (11) | 4.1 (14) |
p = 0.66 0.86 (0.43–1.69) |
p = 0.76 0.90 (0.45–1.79)9 |
| STI diagnosis reported at 3 or 6 months, N = 6245 | 5.3 (16) | 4.7 (15) |
p = 0.76 1.12 (0.55–2.31) |
p = 0.72 1.14 (0.55–2.36)9 |
LARC: long-acting reversible contraception
Analysis restricted to the following subgroups as indicated
1. Not pregnant or trying for baby at 6 months
2. Using a method at 6 months
3. Not pregnant or trying for baby at baseline or 6 months, clear reporting of method at both time points
4. Not pregnant or trying for baby at baseline
5. Completed 3 and 6 month questionnaire items
Adjusted for the following baseline factors as indicated
6. LARC use, satisfaction with method, and setting
7. satisfaction with method and setting
8. effectiveness of method, satisfaction with method, and setting
9. effectiveness of method
The effects of the intervention on the primary outcomes did not vary significantly between online and in-person recruitment (data not shown). Post hoc, per-protocol analysis of the primary outcomes was not appreciably different to the modified intention-to-treat analysis (Table 3).
Table 3.
Post hoc per-protocol analysis of primary outcomes.
| Outcome measures, % (n) | Intervention (seen website) | Control (all) | OR (CI), Intervention vs. control | Adjusted OR (CI)* |
|---|---|---|---|---|
| LARC method in use at 6 months, N = 6601 | 31.4 (93) | 31.0 (113) |
p = 0.86 1.03 (0.74–1.44) |
p = 0.98 0.99 (0.66–1.49) |
| Satisfaction with method used at 6 months, N = 5782 | ||||
| Very dissatisfied | 1.1 (3) | 1.6 (5) | ||
| Dissatisfied | 4.2 (11) | 5.8 (18) | p = 0.87 | p = 0.95 |
| Neutral | 10.9 (29) | 10.5 (33) | 0.97 (0.72–1.33) | 1.01 (0.74–1.39) |
| Satisfied | 39.3 (104) | 35.5 (111) | ||
| Very satisfied | 44.5 (118) | 46.7 (146) |
LARC: long-acting reversible contraception.
Analysis restricted to the following subgroups as indicated.
1. Not pregnant or trying for baby at 6 months.
2. Using a method at 6 months.
* Adjusted for LARC use at baseline, satisfaction with method at baseline, and setting.
Women’s views of the Contraception Choices website
Of the 364 intervention participants with 6-month follow-up data, 309 (85%) reported seeing the Contraception Choices website. Of those, 97% found it helpful or very helpful for “getting useful information about contraception” and 87% responded that it was helpful or very helpful for “finding a method of contraception that is right for you.”
Over 91% (423/464) of intervention participants provided free-text comments about the website in follow-up surveys. Comments were strikingly positive, with praise for attractive website design, and the clarity of information presented. Analysis of the free-text comments indicated that the Contraception Choices website helped to increase participant’s knowledge about contraceptive methods and address their concerns, helped with thinking about changing to a different method, and feeling better prepared before clinic appointments.
Knowledge about different methods
Women liked information which helped them to weigh up the advantages and disadvantages of different methods:
“Gives you the ups and downs about each choice and also helps advise which one to choose.”
“I feel more clued up about potential contraception choices which is great! I think far too many think that the pill is the only way forward which is wrong!”
Tailored information
Women valued tailored feedback such as the What’s right for me? decision aid, to help them to choose contraceptive methods to suit their priorities.
“It’s what I’ve always looked for, a clear way to compare methods of contraception and find the best for you… It may seem crazy, but it’s really hard to find reliable and objective facts on contraception online.”
“I liked how easy it was to tailor a contraception to you and that it considered things like not wanting a period.”
Concerns and misperceptions
Many women appreciated information which addressed concerns and misperceptions, which was not necessarily easy to find elsewhere:
“I was particularly interested in reading that you don’t need a break from hormonal contraception.”
“It’s cleared up some of my doubts and things I worried about (probably unconsciously!) about hormonal contraception.”
“Really useful, accessible information covering concerns that you wouldn’t normally see on a medical website, like… specifically stopping periods.”
Prompting changes of contraceptive method
Several participants discussed their intention to change or consider swapping to a new method as a result of what they had learned or seen on the website. Of those who commented, all were thinking of switching to a more effective long-acting reversible contraceptive method.
“I think it’s (the website) got me thinking more about which contraception I should use. I’m quite happy with my pill and currently not sexually active, but I do think I would like to switch to a LARC if I am in a relationship again.”
Feeling empowered to speak to health professionals and more prepared before clinic appointments
Many participants reported feeling more empowered to speak to healthcare professionals about contraception and feeling better prepared for appointments to discuss contraception:
“It has made me feel more confident. Prior I didn’t really have anyone to speak to about contraception and I didn’t feel comfortable discussing it with my doctors so this bridged the gap.”
“It has led to an increased conversation with my GP practice regarding suitable alternative methods to the combined pill.”
“I think it’s really good to go in [to an appointment] prepared with what you’ve looked at and have an idea in your head before you make a decision about it.”
Barriers to accessing chosen contraception methods
Although some participants wanted to change to a more effective method of contraception, there were a number of reported barriers to accessing contraception.
“I wanted the coil but I found it difficult to find someone to fit it in London.”
“Long waiting times. GP did not offer the services to get implant fitted. Lack of sexual health clinics in my area means very long waiting times.”
Neutral or negative comments
There were only 12 neutral or negative comments, often from women who gave positive comments too, such as:
“Positive: easy to navigate, clear concise bullet points for advantages/disadvantages of each contraceptive method. Negative: no information relating to contraception as a treatment for endometriosis.”
Five negative comments were from women who wanted more specific information about particular contraceptive methods, and one person sought information about handling difficult conversations with general practitioners.
Discussion
In this randomised trial of the Contraception Choices website, we did not find significant differences between intervention and control groups in use of LARC or satisfaction with contraceptive method by 6 months. There were no significant differences in reported adverse effects, including pregnancy and STI diagnoses. The Contraception Choices website was very positively evaluated by young women, who indicated that it helped them to learn about contraception, to think about changing to a different method, and to feel better prepared before clinic appointments. However, difficulty accessing health services is an important barrier to accessing chosen methods of contraception.
Globally, there are an estimated 1.7 billion women of reproductive age (15–49 years).26 Ensuring access to accurate information about contraception to facilitate informed decisions about choice and use of contraception is an essential but challenging step towards prevention of unintended pregnancy.27,28 This paper describes the evaluation of Contraception Choices, an interactive website to aid informed choice of contraception. The evidence base for the website came from extensive systematic review of published literature and empirical qualitative research with young women, and the intervention is underpinned by behaviour change theory. The evaluation method was unusually rigorous for a website – according to recent NICE guidance, a randomised trial is the standard reserved for digital health technologies that aim to prevent and manage disease.29
The women who took part in the trial broadly reflect the ethnic diversity of London,30 the proportion of graduates31 and the proportion of people whose main language is English in the UK.32 At the outset, we did not expect to complete an efficacy (Phase III) trial of the website, but the demand for Contraception Choices from service providers, combined with the opportunity to rapidly expand recruitment and enlarge the trial, led to the transition from a feasibility to an efficacy trial. Aside from delays due to the lengthy process of obtaining all research permissions, the study procedures overall worked well, the online trial processes were highly efficient and the follow-up rate was good, with 86% of participants providing primary outcome data at 3 or 6 months. Recruitment in person was completed within the anticipated 6 months, but recruiting online (via the online booking system) was much more efficient, being faster and at no additional cost. Just over half of all participants were recruited this way in less than eight weeks.
Our study underscores other evidence that online trials are an efficient and acceptable way to conduct clinical trials of low-risk interventions.33 However, choice of methodology and appropriate time-scale for evaluating the impact of digital health interventions remains a key question; guidance on evaluation of complex interventions from the UK Medical Research Council,34 for example, with its meticulous but slow progression through development, feasibility, evaluation and implementation, seems out of step with the rapid pace of change in digital health.
Digital interventions such as Contraception Choices can meet a need for convenient, trustworthy online information and support for contraception decision-making.35,36 Our findings clearly show that the website was popular and well received by users and healthcare providers. Given the strikingly positive feedback about the website from women, the high level of intervention engagement (over four-fifths viewed the website) and low ‘contamination’ in the control group (only 7% of the control group reported seeing the website), the lack of difference in primary outcomes between groups was surprising.
Possible reasons for the observed lack of impact relate to the many influences on contraceptive use which are beyond an individual woman’s control,4 for example, the opinions of partners, peers, religious leaders and the wider community,4,37 and barriers to accessing services for a desired method, including difficulties in getting appointments, long waiting times and a lack of services that can fit LARC methods. Other possible explanations include needing longer follow-up for intentions to translate into action; measurement reactivity (i.e. the possible impact of asking the control group about their contraception use); the limitations of a broad outcome measure like ‘satisfaction with method’; the possibility that both intervention and control group were receiving high quality clinical care; and nearly a third were already using LARC methods at baseline so that the website could not show an additional impact. With hindsight, we might have seen significant differences between groups had we included intermediate outcomes, such as ‘feeling better informed about choice of methods’ or ‘feeling confident about discussing different contraceptive methods with a health professional’, but ultimately, clinically important outcomes are the number of women using an effective contraceptive method that they are happy with and the prevention of unwanted pregnancy.
In terms of implementing digital health interventions in NHS services, we found that directing patients to the Contraception Choices website via an automated text message to confirm a clinic appointment was simple and effective, without additional cost. Offering the link at the time of booking an appointment facilitates access to the website well before a contraceptive consultation and some women commented that they felt better prepared for their consultation as a result. The Contraception Choices website was offered for self-directed use before appointments (in clinic settings or at home), but the website could also be jointly accessed during consultations. For example, the three tailored contraception options generated by the What’s right for me? feature can be discussed with clinicians, and the website can also be displayed on the health professional’s computer screen during consultations. In this study, we did not aim to assess such a development in clinical practice, but an area for future research is to examine whether it could lead to more efficient or satisfying consultations, for both patients and clinicians, with potentially better clinical or health service outcomes.38 Electronic interventions for contraception which offer tailored material, increased frequency of intervention administration, and/or structured follow-up with a healthcare provider appear important for longer-term impact:35 high quality evidence is needed on the effectiveness of different intervention designs, and cost-effectiveness of different models of intervention delivery.
Contraception Choices is currently attracting over 15,000 visits a month and is promoted via a link on the NHS website39 which receives around 11 million visits per year. The availability of the website on the internet offers many possibilities for further use and evaluation. In our systematic review, we found remarkable consistency across the globe in factors influencing contraception choice, uptake and use. With appropriate adaptations, Contraception Choices may be of benefit to populations in different settings (e.g. schools) or different countries; we are currently exploring its utility in Botswana, a country with advanced e-health capacity and high HIV prevalence. We see a real opportunity to use Contraception Choices with vulnerable populations, including those where information and education to support effective use of contraception are scarce.
In conclusion, the Contraception Choices website was very popular with young women for its attractive design, engaging presentation of trustworthy information, and guidance in choosing a method tailored to individual preferences. However, we did not find any significant difference in use of LARC or satisfaction with contraceptive method at 6 months. Our systematic reviews confirmed multiple factors affecting women’s choice and use of contraception which go beyond informed choice, such as the influence of others including partners, friends, family, school, religion, wider culture and health services. The lack of effect on clinical outcomes in this trial, despite highly positive feedback from participants, highlights a gap between improving delivery of personalised information and impact on contraceptive use. An interactive website can address individual barriers to contraception choice and use such as lack of knowledge, concerns and misunderstandings, but interventions at other levels are needed to complement this approach.
Supplementary Material
Acknowledgements
We thank all the women who took part in the study.
We thank the staff at the study sites:
Margaret Pyke Centre, London.
Brook Euston, London.
British Pregnancy Advisory Service (bpas) London East.
Green Light Pharmacy, Shepherd’s Bush, London.
Ashford & St Peter’s Hospitals NHS Foundation Trust, Surrey.
Clerkenwell Medical Practice, London.
We thank Moore Wilson digital agency for its contribution to the design and development of the Contraception Choices website, Sealed Envelope for contribution to the trial online recruitment software and the PRIMENT Clinical Trials Unit and Trial Steering Committee.
Contraception Choices Study Group:
Patricia Lohr, British Pregnancy Advisory Service (bpas)
Jonathan Poon, Greenlight Pharmacy
Chris Wilkinson, Margaret Pyke Centre, Central North West London NHS Foundation Trust
Mandy McWhan, Brook, Euston, London
Vas James, UCL Elizabeth Garrett Anderson Institute for Women’s Health
Sarah Bourdin, UCL Elizabeth Garrett Anderson Institute for Women’s Health
Natalia Lago, Research Department of Primary Care and Population Health, UCL
Lisa Walton, Imperial College Healthcare NHS Trust
Caitlin Bawn, UCL Elizabeth Garrett Anderson Institute for Women’s Health
Conflict of interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Contributorship
All authors contributed to the design, development and/or evaluation of the Contraception Choices website, commented on successive drafts of the paper and approved the final version.
In particular, Julia Bailey led on the website development; Judith Stephenson led on the trial.
Ana Gubijev conducted the interviews and focus groups, reviewed YouTube videos, led the in-person trial recruitment, coordinated the trial follow-up, analysed the qualitative data and contributed to analysis of the trial data. Andrew Copas, trial statistician, led on the trial design and statistical analysis. Ann Blandford led on the expert input to the website design.
Data sharing
After de-identification, the trial data will be available, specifically all of the individual participant data collected during the trial. The trial protocol, statistical analysis plan, and informed consent form are also available. The data are available immediately following publication, with no end date. Data are available to researchers who provide a methodologically sound proposal, for the purposes of achieving the aims of the approved proposal. Proposals should be directed to Judith Stephenson (author for correspondence) and Andrew Copas (senior statistician). To gain access, data requestors will need to sign a data access agreement.
Ethical approval
Ethical approval was given by London Camden & Kings Cross Research Ethics Committee (Reference 17/LO/0112).
Funding
National Institute for Health Research Health Technology Assessment Programme Commissioned call to increase the uptake of long-acting contraception in young women. Study registration ISRCTN 13247829.
Guarantor
JS (corresponding author)
ORCID iDs
Judith Stephenson https://orcid.org/0000-0002-8852-0881
Julia V Bailey https://orcid.org/0000-0002-5001-0122
Ann Blandford https://orcid.org/0000-0002-3198-7122
Supplemental material
Supplemental material for this article is available online.
References
- 1.Sedgh G, Singh S, Hussain R. Intended and unintended pregnancies worldwide in 2012 and recent trends. Stud Fam Plann 2014; 45(3): 301–314. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Wellings K, Jones KG, Mercer CH, et al. The prevalence of unplanned pregnancy and associated factors in Britain: findings from the third National Survey of Sexual Attitudes and Lifestyles (Natsal-3). Lancet. 2013; 382(9907): 1807–1816. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.GOV.UK. Abortion Statistics, England Wales, 2017: Department of Health & Social Care 2017, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/763174/2017-abortion-statistics-for-england-and-wales-revised.pdf (2017, accessed 17 June 2020).
- 4.Pratt R, Stephenson J, Mann S. What influences contraceptive behaviour in women who experience unintended pregnancy? A systematic review of qualitative research. J Obstet Gynaecol 2014; 34(8): 693–699. [DOI] [PubMed] [Google Scholar]
- 5.Nettleman M, Brewer J, Ayoola A. Reasons for unprotected intercourse in adult women: a qualitative study. J Midwifery Women’s Health 2007; 52(2): 148–152. [DOI] [PubMed] [Google Scholar]
- 6.Black K, Lotke P, Buhling KJ, et al. A review of barriers and myths preventing the more widespread use of intrauterine contraception in nulliparous women. Eur J Contracep Repr 2012; 17(5): 340–348. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Winner B, Peipert JF, Zhao Q, et al. Effectiveness of long-acting reversible contraception. N Engl J Med 2012; 366(21): 1998–2007. [DOI] [PubMed] [Google Scholar]
- 8.Stoddard A, McNicholas C, Peipert JF. Efficacy and safety of long-acting reversible contraception. Drugs. 2011; 71(8): 969–980 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Williamson LM, Parkes A, Wight D, et al. Limits to modern contraceptive use among young women in developing countries: a systematic review of qualitative research. Reprod Health 2009; 6(3). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.McCarthy O, Carswell K, Murray E, et al. What young people want from a sexual health website: design and development of Sexunzipped. J Med Internet Res 2012; 14(5): e127. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Mangone ER, Lebrun V, Muessig KE. Mobile phone apps for the prevention of unintended pregnancy: a systematic review and content analysis. JMIR Mhealth Uhealth 2016; 4(1): e6–e. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Baxter S, Blank L, Guillaume L, et al. Views regarding the use of contraception amongst young people in the UK: a systematic review and thematic synthesis. Eur J Contracept Reprod Health Care 2011; 16(3): 149–160. [DOI] [PubMed] [Google Scholar]
- 13.Guillebaud JM. Contraception: Your Questions Answered. 7th ed London: Churchill Livingstone, 2017:p.424. [Google Scholar]
- 14.Schwarz EB, Gerbert B, Gonzales R. Computer-assisted provision of emergency contraception a randomized controlled trial. J Gen Intern Med 2008; 23(6): 794–799. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Garbers S, Meserve A, Kottke M, et al. Tailored health messaging improves contraceptive continuation and adherence: results from a randomized controlled trial. Contraception 2012; 86(5): 536–542. [DOI] [PubMed] [Google Scholar]
- 16.Garbers S, Meserve A, Kottke M, et al. Randomized controlled trial of a computer-based module to improve contraceptive method choice. Contraception 2012; 86(4): 383–390. [DOI] [PubMed] [Google Scholar]
- 17.Schwarz EB, Burch EJ, Parisi SM, et al. Computer-assisted provision of hormonal contraception in acute care settings. Contraception 2013; 87(2): 242–250. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Antonishak J, Kaye K, Swiader L. Impact of an online birth control support network on unintended pregnancy. Soc Marketing Q 2015; 21(1): 23–36. [Google Scholar]
- 19.Schwarz EB, Gerbert B, Gonzales R. Computer-assisted provision of emergency contraception: a randomized controlled trial. J Gen Intern Med 2008; 23(6): 794–799. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Bailey JV, Mann S, Wayal S, et al. Sexual health promotion for young people delivered via digital media: a scoping review 2015. Report No.: 3(13) Contract No.: 13, Southampton (UK): NIHR Journals Library; 2015. [PubMed]
- 21.French RS, Geary R, Jones K, et al. Where do women and men in Britain obtain contraception? Findings from the third National Survey of Sexual Attitudes and Lifestyles (Natsal-3). BMJ Sex Reprod Health 2018; 44(1): 16–26. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Lustria MLA, Noar SM, Cortese J, et al. A meta-analysis of web-delivered tailored health behavior change interventions. J Health Commun 2013; 18(9): 1039–1069. [DOI] [PubMed] [Google Scholar]
- 23.Michie S, Atkins L, West R. The Behaviour Change Wheel: A Guide To Designing Interventions. Surrey: Silverback, 2014. [Google Scholar]
- 24.Dahlgren G, Whitehead M. Tackling inequalities in health: what can we learn from what has been tried? King’s Fund, https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.kingsfund.org.uk%2Fprojects%2Ftime-think-differently%2Ftrends-broader-determinants-health&data=02%7C01%7C%7C40ccdc4d275148112dc108d8130d757c%7C1faf88fea9984c5b93c9210a11d9a5c2%7C0%7C0%7C637280295810513868&sdata=tclKax1dmlp4L6JjXLBb%2FVuz9wsw7Z1QBO5y3vrirsM%3D&reserved=0 https://www.kingsfund.org.uk/projects/time-think-differently/trends-broader-determinants-health (1993, accessed 17 June 2020).
- 25.StataCorp. Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC. 2017.
- 26.United Nations DoEaSA, Population Division World Population Prospects: The 2017 Revision, custom data acquired via website 2017, https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpopulation.un.org%2Fwpp%2FPublications%2FFiles%2FWPP2017_DataBooklet.pdf&data=02%7C01%7C%7C40ccdc4d275148112dc108d8130d757c%7C1faf88fea9984c5b93c9210a11d9a5c2%7C0%7C0%7C637280295810523873&sdata=tfPFswkLUXk%2B%2Ff5pAaui5kgR2V%2BKAsivYtxq%2BLMrbRA%3D&reserved=0 https://population.un.org/wpp/Publications/Files/WPP2017_DataBooklet.pdf (2017, accessed 17 June 2020).
- 27.Oringanje C, Meremikwu MM, Eko H, et al. Interventions for preventing unintended pregnancies among adolescents. Cochrane Database of Syst Rev 2009(4). [DOI] [PubMed] [Google Scholar]
- 28.Lopez LM, Grey TW, Chen M, et al. Strategies for improving postpartum contraceptive use: evidence from non‐randomized studies. Cochrane Database of Syst Rev 2014(9). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.National Institute for Health and Care Excellence. Evidence Standards Framework for Digital Health Technologies, https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.nice.org.uk%2Fabout%2Fwhat-we-do%2Four-programmes%2Fevidence-standards-framework-for-digital-health-technologies&data=02%7C01%7C%7C40ccdc4d275148112dc108d8130d757c%7C1faf88fea9984c5b93c9210a11d9a5c2%7C0%7C0%7C637280295810523873&sdata=fD1SfwLrbUOtyPI3ffuP%2BBDCLfWhkc2meCRMOn5nakE%3D&reserved=0 https://www.nice.org.uk/about/what-we-do/our-programmes/evidence-standards-framework-for-digital-health-technologies (2018, accessed 17 June 2020). [DOI] [PMC free article] [PubMed]
- 30.Ethnicity and National Identity in England and Wales, https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ons.gov.uk%2Fpeoplepopulationandcommunity%2Fculturalidentity%2Fethnicity%2Farticles%2Fethnicityandnationalidentityinenglandandwales%2F2012-12-11&data=02%7C01%7C%7C40ccdc4d275148112dc108d8130d757c%7C1faf88fea9984c5b93c9210a11d9a5c2%7C0%7C0%7C637280295810533861&sdata=DgktnR%2BBlFy%2B%2B18AB3enBysY7ttFDu38h6lA34Qk6DA%3D&reserved=0 https://www.ons.gov.uk/peoplepopulationandcommunity/culturalidentity/ethnicity/articles/ethnicityandnationalidentityinenglandandwales/2012-12-11 (2011, accessed 17 June 2020).
- 31.Graduates in the UK labour market, https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdata.gov.uk%2Fdataset%2F70e7d873-8d4a-4815-9e19-6b61b2c6557a%2Fgraduates-in-the-labour-market&data=02%7C01%7C%7C40ccdc4d275148112dc108d8130d757c%7C1faf88fea9984c5b93c9210a11d9a5c2%7C0%7C0%7C637280295810533861&sdata=%2Fws1tSYkBM27eQYe9NoFhTrPstj8LnjpTAZUxmqc7QM%3D&reserved=0 https://data.gov.uk/dataset/70e7d873-8d4a-4815-9e19-6b61b2c6557a/graduates-in-the-labour-market (2011, accessed 17 June 2020).
- 32.Team DPFaC. National Schedule of Reference Costs 2013-14. 2014 11/28/2014.
- 33.Bailey JV, Pavlou M, Copas A, et al. The Sexunzipped trial: optimizing the design of online randomized controlled trials. J Med Internet Res 2013; 15(12): e278. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Craig P, Dieppe P, Macintyre S, Michie S, Irwin N, Petticrew M et al. Developing and evaluating complex interventions: new guidance. BMJ 2008; 337: a1655. [DOI] [PMC free article] [PubMed]
- 35.Dewart CM, Serpico J, Steiner MJ, et al. Electronic interventions for changing knowledge, attitudes or practices regarding contraception: a systematic review. Contraception 2019; 100(1): 10–25. [DOI] [PubMed] [Google Scholar]
- 36.French RS, Wellings K, Cowan FM. How can we help people to choose a method of contraception? The case for contraceptive decision aids. J Fam Plann Reprod Health Care 2009; 35(4): 219–220. [DOI] [PubMed] [Google Scholar]
- 37.Blackstone SR, Nwaozuru U, Iwelunmor J. Factors influencing contraceptive use in Sub-Saharan Africa: A systematic review. Int Q Community Health Educ 2017; 37(2): 79–91. [DOI] [PubMed] [Google Scholar]
- 38.Dehlendorf C, Levy K, Kelley A, et al. Women’s preferences for contraceptive counseling and decision making. Contraception 2013; 88(2): 250–256. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Committee JF. British National Formulary. 60th ed London: British Medical Association and Royal Pharmaceutical Society, 2015. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.