The effect of programme testimonials on registrations for an online cognitive behaviour therapy intervention: a randomised trial

Abstract

Objective

Suboptimal visit-to-registration rates limit the reach and potential impact of online health interventions. In this study, we examined whether testimonials presented during a new-user registration process could increase the uptake rates of MoodGYM, an open-access automated intervention designed to reduce the symptoms of depression.

Methods

We conducted a three-armed parallel randomised controlled trial on the MoodGYM website involving variations of the first page of the registration process. Spontaneous visitors saw either the current pre-registration content, the current content supplemented by a first-person past user testimonial or the current content accompanied by a testimonial from a third-party health professional. The primary outcome measure was the proportion of visitors seeing each version who completed registration.

Results

A total of 14,267 visits were recorded across the three conditions. The past user, health professional and control condition registration rates were 45.6%, 45.5% and 45.5% respectively, indicating that there was no effect of registration content on registration rate. (p > 0.05)

Conclusions

There may be limited use for testimonials in well-established interventions that have programme benefits clearly stated and presented in other forms. However, we cannot exclude the possibility that testimonial effects might be moderated by positioning or other elements within the registration flow or by visitor source. Further research is required to examine these factors and, more broadly, assess whether modifications to other aspects of the registration process can improve visit-to-registration rates.

Introduction

Open access online cognitive behaviour therapy (CBT) interventions have been shown to be effective in decreasing symptoms of depression or anxiety in people living in the community.^1,2 These sites allow visitors to register for a programme that they complete at their own pace, enabling widespread and timely access to care that might not otherwise be available offline.

However, dropout and non-adherence is high in many web-based programmes.^3,4 This is of substantial concern given that engagement has been linked with greater efficacy.^5,6 In response, a rapidly growing body of research has focused on programme components associated with registrant engagement or retention. This body of work has identified factors that appear to be associated with increased usage across a range of health-related domains including tunnelling (e.g. modular, stepped content), update frequency and tailoring.^3,7–10

Despite this progress, few studies have examined the possible effects of pre-registration components on visitor retention through to registration. Yet, for many open-access programmes, only a small proportion of website visitors progress past this critical point. For instance, one study of an intervention promoting heart-healthy behaviours found that of 285,146 visitors over a 36-month period, only 81,574 (29%) completed the registration procedure.¹¹ Low visit-to-registration rates may represent a missed opportunity to maximise the reach of internet interventions. Even small rate uplifts have the potential to generate substantial increases in the number of people commencing programmes and thus being exposed to components designed to help them improve their health.

Factors affecting user engagement on first site visit

Research examining the cognitive deliberations of visitors to intervention websites suggests technical implementation factors and informational appeal guide the decision of whether to stay.¹²

Technical factors include the presence of a registration procedure along with the site’s design and navigation structure. The requirement to register before accessing content is a barrier to continuing.¹² Registration burden reduction mechanisms and navigational flow may therefore be productive areas for research aiming to improve visit-to-registration rates. However, it is unlikely that many health interventions could discard the registration process entirely. Certainly, for programmes involving user diaries or monitoring and reporting of symptom change over time, registration is an essential component that ensures progress can be stored while user information and enrolment remains confidential.

With regard to informational appeal, panels of marketing and intervention design experts,¹³ and internet user focus groups¹² suggest a number of factors may be targeted to increase visitor time on site and subsequent registration. These include: (a) provision of clear programme aims and ease of understanding (e.g. language employed); (b) perceived reliability and credibility of information; (c) perceived usefulness and achievability of behaviour change advice; (d) appealing tone of voice for site content; and (e) clear communication that the site can be used free of charge and allows for self-pacing.

The ways in which these factors might be manipulated will vary by intervention. Nevertheless, one communication device with the potential for widespread application that could incorporate a number of the factors outlined above is ‘the provision of previous users’ experiences or success stories’ via testimonials or narratives.¹²

Testimonials and health interventions

Only one prior published study has tested the effect of narratives on registrations for an online health intervention. Specifically, Buller et al.¹⁴ sent postal letters containing endorsements by local opinion leaders or testimonials by teachers, or both, for a web-based smoking prevention programme to nearly 400 schools in Colorado using a randomised controlled design. A range of outcomes were measured, including website registrations from each school as indicated by user self-identification from a drop-down list. The study found an increased proportion of registrations by teachers from schools receiving the opinion leader endorsement, but no increase in student use and no effect for teacher testimonials. Thus, results were mixed, with some limited evidence that offline endorsements could improve online registrations.

Testimonials have also been examined in the context of their effect on engagement by registered intervention users or, more broadly, treatment choices in clinical settings. Specifically, a 2008 randomised controlled study of a smoking cessation intervention examined testimonials as a post-registration component aimed at improving user engagement. It found an increase in programme engagement and smoking abstinence from highly tailored testimonials delivered to ‘ready to quit’ smokers.^6,15 However, a later trial involving a broader sample of smokers covering all pre-action stages of readiness failed to replicate these findings.¹⁶

Moreover, a recent systematic review from the decision-making literature examined the effect of stories outlining other patients’ experiences on people’s hypothetical or real-world healthcare judgements when using a decision aid. It found that the stories generally led to different decisions and preferences, and that there was higher recall for decision-aid information when it was presented with narratives than without. However, there was insufficient evidence to determine whether narratives improved decision making or instead, biased choices.¹⁷ An earlier systematic review presented very similar findings.¹⁸

Thus, the current body of research suggests that testimonials can influence people’s choices in a healthcare context, but that an effect may not be achieved in all settings or populations. The mixed results to date may be due to differences in context, presentation format, and content employed by studies. Furthermore, most studies have not explored how or why narratives affected decision making or engagement, or their ‘active’ components.

The effect of testimonial source is an exception to the general lack of clear information regarding the active components of narratives; many past studies in the decision-making literature have used narratives from either third parties (e.g. experts) or those from first-hand prior users or patients. The authors of a systematic review concluded that studies incorporating first-hand stories were more likely to report an effect compared with those using third-person narratives.¹⁸ Specifically, seven of the seventeen studies in the review used first-hand stories, with three of those reporting an effect. By contrast, only two of the ten studies using third-person narratives reported an effect.¹⁸

Other evidence on the effectiveness of testimonials and their active components: Marketing industry studies

Although few published peer-reviewed studies have examined the effects of testimonials on web-based intervention registrations, testimonials are used extensively in commercial settings and there is a body of non-peer reviewed research on the effectiveness of these endorsements. A search of the grey literature using the Google search terms [+testimonial “split test” “ab test”], [+testimonial test optimisation] and [+testimonial random test] returned a number of pages discussing studies of testimonial effectiveness undertaken by marketing optimisation specialists. Of those that did not require a subscription to view, the pages reported studies by three firms: MecLabs (MarketingExperiments), Visual Website Optimizer, and Optimizely. We searched the websites of those organisations for reports of trials designed to evaluate the effect of online testimonials. Ten studies were identified, each of which involved randomised testing of one or more testimonial variants against a status-quo control.

Reported results from the studies suggested testimonials could significantly increase sales conversions,^19,20 software trial registrations, ²¹ and dating site registrations.²² (the last study trialled other elements concomitantly). They also indicated that the following elements may alter visitor time on site or response: (a) use of a visual element such as a logo or human photograph, with bespoke authentic images preferable to stock images;^23–25 (b) tailoring to attributes or features salient to the visitor;²¹ (c) placement on page (e.g. towards the top) or closer to the point of conversion;^26–28 and (d) a focus on user benefits or problem solving rather than simple praise for a service or product.²¹ The latter point was also mentioned in the discussion of other studies, but without specific reference to trial results.²⁷

There may be substantial publishing bias in these sources, since their aim was to showcase the effectiveness of marketing optimization services and they do not disclose whether, or how many, studies of a similar nature were conducted that found no effect. Moreover, although all the above studies were reported as randomised controlled trials, key methodological details were often omitted. Nevertheless, these commercial trials offer ideas for testimonial variants that may be formally tested in health interventions.

Examining the effect of testimonials on registrations for a CBT intervention

The broader evidence base suggests on-site testimonials have the potential to increase registration rates. There is also some evidence from the decision-making literature that first-person (i.e. user) testimonials may promote higher registrations than third-party (expert) endorsement. However, the effect of testimonials on registrations has not been directly tested in the context of online behaviour change interventions. We therefore investigated the impact of testimonials on spontaneous visitor registrations for MoodGYM, a well-established online intervention for depression.

Specifically, we aimed to assess whether benefit-focused testimonials by either a past user or a health professional and presented at the top of the first page of a new-user registration flow significantly improved registrations compared with a control containing no testimonial. Our hypotheses were that:

1. testimonials would significantly improve registration rates, and

2. a past user testimonial would generate higher rates than a health professional testimonial.

Methods

The MoodGYM intervention

MoodGYM (moodgym.anu.edu.au) is a free, interactive web programme originally developed and tested in 2001. Using interactive diagrams and online exercises, it teaches the principles of CBT, including cognitive restructuring, pleasant events scheduling and relaxation, and provides strategies to cope with interpersonal problems. Randomised trials evaluating MoodGYM have shown that using two or more modules is linked to significant reductions in depression and anxiety symptoms.^29,30 These benefits extend to at least 12 months post-intervention.³¹

Currently, upon arriving at the MoodGYM site main landing page, visitors may select one of two routes: view more information on the intervention (signalling they are potential new users) or login to an existing account (i.e. if they are a returning registrant).

At the time of this study, those entering the new user route were presented with a sequential flow consisting of three pages: (a) an ‘Information’ page conveying potential benefits to visitors in bullet-point format; (b) a ‘Terms of use’ page outlining some details of the programme, the site privacy policy, and information security measures; and (c) a page containing the registration form. Figures A1–A3 in Appendix 1 present screenshots of each page.

Study design

The study was a three-armed parallel randomised controlled trial involving modifications of the first page of the registration flow. All spontaneous MoodGYM visitors who selected the new-user route were randomised using an automated computer-generated random-number assignment to receive the current pre-registration content (i.e. the control) or the latter content preceded by either a first-person past-user testimonial or a testimonial from a third-party health professional. Figures A1, A4 and A5 in Appendix 1 present the three variants.

The primary outcome measure was the proportion of visitors to each version who completed the registration procedure. Hence, each unique visit to the new user registration flow was recorded, along with the experimental variant received and whether or not the visit resulted in a completed registration.

In the first 6 months of 2015 (prior to the study), 152,933 visitors clicked through to the ‘sign up’ section of MoodGYM, with 74,233 (49%) completing registrations. Based on this pre-experimental data we scheduled the experiment to run over approximately a fortnight, in order to record at least 3500 cases for each of the three experimental cells. A priori power analysis indicated that this sample size was sufficient to detect a difference in registration completion rate of ∼3.5% (absolute) between the control and either variant with >80% power and alpha of 0.05 assuming an estimated base visit-to-registration proportion of 0.5.

Approval for the study was granted by the Australian National University Human Research Ethics Committee (protocol 2015/537).

Testimonial sourcing, selection and design

Studies employing closed-access trials in other contexts have used fictitious testimonials. This was not appropriate in the current live intervention context.

We sourced the third-party testimonial from a medical health practitioner who was known to recommend MoodGYM to her clients. The statement was written by the health professional and permission was obtained to use her photograph and name alongside it in the trial. Figure A4 in Appendix 1 presents the statement as it appeared.

The past-user testimonial was sourced from feedback forms completed by MoodGYM users who had related benefits or outcomes they had experienced following the use of the programme. As it was necessary to preserve the privacy of these past intervention registrants, it was not possible to use statements containing any identifying information. An initial pool of eight testimonials was produced by examining forms for anonymous statements that: were no longer than three sentences; did not require detailed understanding of the intervention to understand; and conveyed a benefit or experience not covered by another in the pool.

The eight candidate testimonials were then presented via a Google Consumer Survey to a general panel of internet users over the age of 18 years. Panel members were sourced from the United Kingdom, the largest single user visitor source for MoodGYM. A total of 750 respondents selected the anonymous statement they thought was most likely to prompt new users to try MoodGYM. Of the top-rated statements, one was selected by the research team for inclusion in the trial based on a subjective judgement of its likely broad appeal. Figure A5 in Appendix 1 presents the testimonial as it appeared on the website.

Shaffer and Zikmund-Fisher³² have proposed a taxonomy of testimonials, categorising them by purpose (e.g. to inform, engage, model behaviour, persuade, or comfort), content (e.g. covering the process of decision making or the decision experience) and evaluative valence (expression of satisfaction or dissatisfaction). Under that schema, the testimonials we tested can be classified as having a persuasive purpose, content that focuses on outcomes and experiences, and a positive evaluative valence.

Where possible, our testimonial design incorporated the potentially active elements noted earlier from industry studies. This included a photo or visual element next to the text and placement above the scroll line of the screen. We chose to place the testimonials on page one of the registration flow. Our assumption was that this was the first point at which potential new users would decide whether to continue with registration and therefore the point at which a testimonial might have the greatest effect.

Analyses

All analyses were performed using R version 3.2.2 (R Foundation for Statistical Computing, Vienna, Austria).³³ and the RStudio development environment version 0.99.489 (RStudio, Inc., Boston, MA, USA).³⁴ A Chi-square test was performed to assess effect across the three variants on the main outcome variable (registration) for the entire trial period. The alpha level was 0.05.

Results

The trial was undertaken between 6 and 25 November 2015. A total of 14,267 visits were recorded across the three groups. Table 1 shows the total visits and registrations for each condition. With registration rates of 45.6% (user testimonial) and 45.5% (professional testimonial), neither testimonial showed a significant increase in registration relative to the control rate of 45.5% (Chi-square p = 0.99).

Table 1.

Spontaneous visits by point in registration flow and experimental variant.

	Condition
Point in registration flow	Control	Past user	Health professional	Total
Information (first) page	4764	4757	4746	14,267
	100%	100%	100%	100%
Registration complete	2169	2171	2161	6501
	45.5%	45.6%	45.5%	45.6%

Discussion

The addition of testimonials did not increase registration rates for the MoodGYM CBT programme. To our knowledge, this is the first investigation of the effectiveness of online pre-registration testimonials in an open-access behaviour modification intervention.

As noted in the introduction, testimonials are in widespread commercial use and there is some empirical evidence for their effectiveness in increasing sales conversions, software trial registrations and dating site registrations. However, the current finding raises questions about the utility of pre-registration testimonials in evidence-based health interventions, particularly if the intervention is free, well established and has benefits clearly stated and presented in other forms. In such situations, testimonials may not provide sufficient additional information to alter visitor decision making for a substantial proportion of the site’s visitors.

Given these null findings, other means of modifying informational appeal or other pre-registration elements such as registration burden may be more productive or have broader application in a health intervention context.

Limitations and directions for future research

This study tested authentic testimonials incorporating elements likely, according to current knowledge, to maximise their effect. These were presented randomly to a large number of visitors to a well-established health intervention. Hence, the findings presented here are robust and might be expected to generalize to similar settings. However, there were some limitations of the study.

One important constraint was that data access restrictions precluded the analysis of registration rates by subgroups such as visitor source. This variable may moderate response to testimonials. For example, visitors from direct referrals such as from a physician or a mental health service website may be relatively committed to registering prior to arriving and therefore uninfluenced by positive narratives on the destination site in contrast to visitors with low levels of pre-commitment for whom a testimonial may have greater potential to influence registration. We were not able to explore this possibility in the absence of access to linkage log data containing referral sources and individual post-registration programme data for the spontaneous visitors in this study.

However, we note that based on aggregate unlinked self-report data for the period January to June 2015, 37% of registrants were referred to MoodGYM by a health professional. Hence, a substantial proportion of spontaneous visitors in the current study may have been highly pre-committed to engaging with the programme. An analysis restricted to potentially uncommitted visitors arriving at MoodGYM without prior information about the programme may have yielded different results from those reported in the current paper. Such an analysis would require the development of a reliable indicator of visitor pre-commitment, perhaps based on individual-level log data containing referral links. This would enable the pursuit of a potentially important new line of investigation that explores interactions between source, visitor pre-commitment, and testimonial manipulations in detail.

Indeed, general community familiarity with an intervention is likely to be positively correlated with visitor pre-commitment. Thus, testimonials may have an effect in newer, less established online interventions with lower baseline registration rates. For context, approximately 51% of MoodGYM visitors entering the registration route went on to register during the first half of 2015. Although industry benchmarking data does not exist for confirmation, it is likely this is a relatively high registration rate. Studies on websites relating to other interventions at earlier stages of development are therefore required.

It is also possible that elements between the Introduction page and registration completion eliminated a small effect from the testimonials at an earlier point in the registration flow. Ideally, an analysis of visits to the intermediary pages would have been undertaken to assess whether this occurred, but as noted earlier, we were unable to access to the web server log data necessary for such an investigation. Future studies examining testimonials should address this potential confound.

Related to this, one key element not tested here was the effect of testimonial position in the registration flow, on performance. It is possible that the statements would have been effective had they been presented later in the registration flow (e.g. on the registration form page) or earlier, on the main website landing page. Placement decisions for future studies should be informed by a pre-trial examination of points where visitors drop out of the registration flow, including time and point on page. Commercially, such an investigation is often termed a funnel analysis.

Another limitation related to the context of the current study is that past user testimonials in support of a public mental health intervention generally need to be anonymous, both to maintain user privacy and provide confidence to users that their information will remain secure. Yet, anonymity may diminish the effect of statements by reducing their perceived authenticity. We do not know whether this contributed to the lack of effect achieved by the first-person statement in our study.

More broadly, another factor that may limit effects from modifications to pre-registration material is the minimum constitutive attrition for a given intervention. For example, MoodGYM’s pre-registration-to-registration attrition rates were 49% during the first half of 2015. An unknown ‘constitutive’ proportion of that attrition will be due to visitors such as students, academics or other community members who arrive on the site out of passing interest and start the registration flow with no intention of commencing a programme, leaving very shortly thereafter. As an intervention approaches its constitutive rate, it will be difficult for new modifications to yield an effect on registration. At present, it is not clear what this minimum constitutive attrition rate may be for MoodGYM, nor how close the current level of attrition is to it. Future work might examine metrics (such as referral source, visitor country, site entry point or time spent on the landing page) with the potential to help distinguish highly transient visitors from others and provide a basis for realistically assessing the maximum possible effect of any proposed site modifications.

Finally, the fact we found no effect of testimonials in this study suggests investigations into other modifications of pre-registration material are warranted. These could include the introduction of interactive or multimedia information delivery, mechanisms for reducing the dense content of terms and conditions pages, or efforts to minimise registration form burden. Ideally, the form and focus of modifications would be guided by funnel analyses.

Conclusions

Very few published studies have examined mechanisms to increase visit-to-registration rates and thereby improve the reach of open-access online health interventions. This study is the first to experiment with on-site testimonials for this purpose. The results suggest past-user or health-professional narratives may not be effective for well-established interventions in which the programme benefits are clearly stated and presented in other forms. Many visitors to such interventions may already have high levels of familiarity and pre-commitment to registration, thereby reducing the overall effect of testimonials.

However, it is possible that testimonial effects may be moderated by positioning, other elements within the registration flow, or visitor source dynamics that were beyond the scope of our investigation. Further research is therefore required to examine these factors, as well as to assess whether modifications to other aspects of the registration process can generate substantive improvements to visit-to-registration rates.

Appendix 1

Figure A1.

Page one of the MoodGYM new user registration flow.

Figure A2.

Page two of the MoodGYM new user registration flow (compressed).

Figure A3.

Page three of the MoodGYM new user registration flow (compressed).

Figure A4.

Registration flow page one with health professional testimonial (snippet).

Note: The image and name have been blurred for publication but were not blurred in the experiment.

Figure A5.

Registration flow page one with past user testimonial (snippet).

Contributorship

BH and KG researched literature, conceived the study and developed the trial variants. KB was involved in protocol development, gaining ethical approval and coordinating protocol implementation and data access. BH conducted the analysis and wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: KB is a shareholder and director of e-hub Health Pty Ltd., which sells subscriptions to a commercial version of MoodGYM. KG serves as an honorary member of the Advisory Board to e-hub Health. She holds no shares in the company but, as an ANU co-founder of MoodGYM, may be entitled to a share in any future royalties paid by e-hub Health to ANU.

Ethical approval

The Australian National University Human Research Ethics Committee approved this study (protocol 2015/537).

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Guarantor

Benjamin Healey.

Peer review

This manuscript was reviewed by Niranjan Bidargaddi, Flinders University, Australia and one other individual who has chosen to remain anonymous.