Sequential multiple assignment randomized trial studies should report all key components: a systematic review

Abstract

Objective

Sequential Multiple Assignment Randomized Trial (SMART) designs allow multiple randomizations of participants; this allows assessment of stage-specific questions (individual randomizations) and adaptive interventions (i.e. treatment strategies). We assessed the quality of reporting of the information required to design SMART studies.

Study design and setting

We systematically searched four databases (PubMed, Ovid, Web of Science and Scopus) for all trial reports, protocols, reviews, and methodological papers which mentioned SMART designs up to June 15, 2020.

Results

Of the 157 selected records, 12 (7.64%) were trial reports, 24 (15.29%) were study protocols, 91 (58%) were methodological papers, and 30 (19.1%) were review papers. All these trials were powered using stage-specific aims. Only four (33.33%) of these trials reported parameters required for sample size calculations. A small number of the trials (16.67 %) were interested in determining the best embedded adaptive interventions. Most of the trials did not report information about multiple testing adjustment.

Furthermore, most of records reported designs that were mainly focused on stage-specific aims.

Conclusions

Some features of SMART designs are seldomly reported and/or used. Furthermore, studies using this design tend to not adequately report information about all the design parameters, limiting their transparency and interpretability.

1. Introduction

An adaptive intervention (AI), sometimes referred to as dynamic treatment regimens (DTRs), is a treatment strategy that formalizes the personalization of treatment through established decision rules that recommend when and how the treatment should change. This strategy would account for the patient’s treatment history and response to those treatments [1–4]. Often there will be a huge number of potential AIs that could be used, and it is difficult to collect robust information on which perform best.

An AI consists of four key elements: 1) critical decision point(s), comprising: the intervention to begin with; when and how to measure signs of response/nonresponse; how to maintain the success of the initial intervention; and what interventions may be used for non-responders. 2) intervention component(s), a set of intervention/treatment options at each critical decision point. 3) tailoring variable(s), an early indicator of the overall outcome (success or failure of the intervention). 4) decision rule(s), linking the tailoring variable(s) to the intervention components, at each critical decision point.

The sequential multiple assignment randomization trial (SMART) is design that involves multiple stages of randomization, and is used to develop and refine effective AIs, also known as dynamic treatment regimens (DTR), adaptive treatment strategies (ATS), multistage treatment strategies (MTS), proportionate interventions, or treatment policies [1,5–8].

AIs are said to be embedded in a SMART study. Each patient is randomly assigned to initial treatment and subsequent treatments are based on the intermediate outcome and patient’s characteristics. Each stage in a SMART design corresponds to one of the critical decisions involved in the AI. Investigators can build more complex AIs (more deeply tailored AIs) using extra information collected on potential moderators (e.g., baseline characteristics of the individual and/or context, adherence to and/or side effects from prior treatment stages) [5,9]. These tailored AIs are also called deeply embedded AIs.

SMART designs are a generalization of conventional randomized controlled trials (RCT) that allow participants to be randomized more than once, depending on intermediate outcomes. RCTs are designed to test efficacy or effectiveness of an intervention compared to a control condition, while SMARTs allow identifying treatment strategies that would otherwise take multiple RCTs to evaluate. Despite the use of the word ‘adaptive’ in AIs, SMARTs differ from adaptive designs in which each stage involves different participants (between-participants adaptations) [10]. In SMART design, adaptation is made at participant level (within-participant adaptations) [6,11]. Compared to crossover trials (CT), SMART designs are used to develop AIs, while CT are used to contrast the effects of stand-alone treatments. In a CT, patients receive the assigned sequence irrespective of their intermediate outcomes.

Like a traditional RCT, a SMART study must be appropriately designed to meet clearly defined objectives. The CONSORT statement [12] and the International Conference on Harmonization E9 (ICH E9) [13] provide guidelines on reporting and justifications on sample size determination for RCTs. Candlish et al. (2019) [8] reviewed how trials of multi-component and multi-stage interventions (i.e., stepped care, AI, DTR) were designed in practice, the type of statistical designs and analysis methods used in trials involving staged AIs. In their review, they also studied how AIs were analyzed compared to trials of non-proportional interventions. Miller (2019) [14] conducted a review to describe the unique characteristics of three types of adaptive interventions (i.e., Just In Time Adaptive Interventions, SMART, and stepped care) in behavioral interventions. To our knowledge, no paper has explored the reporting of trials using SMART designs. To address this knowledge gap, we conducted a systematic review.

In this systematic review, we aimed at assessing the quality of reporting the information required to design SMART studies in published reports. More specifically, our review questions were grouped into basic trials characteristics (research area, sample size calculation information), aims considerations (stage-specific aims, AIs related aims) and analyses considerations (use of multiple testing adjustments, analysis methods, software for sample size calculations and analysis).

2. Material and methods

This systematic review adheres to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [15]f. A protocol and a completed PRISMA checklist are given in Appendices A and B.

2.1. Literature search strategy

PubMed (including Medline), Ovid (including Embase, Medline, APA PsycINFO), Web of Science (Web of Science core collection, KCI-Korean journal database, Medline), and Scopus were searched between 12^th May and 15^th June 2020. Moreover, some reference lists of the studies included in the final analysis were checked, along with grey literature search, i.e., the methodology centre web-page at Pennsylvania University was searched too [16]. Search strategy is given in Appendix C.

2.2. Inclusion criteria

We included studies of any design, including qualitative, quantitative studies reporting data, review papers, protocols and methodological papers. We excluded feasibility or pilot studies, book chapters, abstracts and conference papers, studies reporting data from observational studies, proof of concept papers. Systematic reviews and meta-analyses papers were excluded, but they were checked for any eligible paper.

2.3. Paper screening and data extraction

All papers identified through the search were imported into EndNoteX9 and duplicates deleted. The screening was undertaken using Rayyan [17] in 3 stages. In the first stage, titles of all the papers were doubly screened (T.B. and G.U.) and any paper failing to meet inclusion criteria was excluded. In the second stage, abstracts were screened and reasons for exclusion recorded (T.B. and G.U.). Lastly, full texts of the remaining papers were screened for inclusion. T.B. and G.U. independently checked whether to include a study, and consensus was sought in cases there was a discrepancy. A data extraction tool was developed for this review in an Excel spreadsheet. The basic study characteristics, study aims, and analyses considerations were recorded wherever possible. More details about recorded items are given in the Appendix C.

2.4. Quality control

We did not undertake a quality assessment of the identified studies as the aim of this systematic review was to assess the quality of reporting the information required to design SMARTs studies in published reports.

3. Results

3.1. Search results

Figure 1 presents the process of study selection in this systematic review. Four databases were searched, identifying a total of 10,055 records. An additional 21 records were identified from grey literature search, i.e., the Pennsylvania State Methodology Centre, webpage [16]. After removing 3,940 duplicate records, 6,136 unique records were considered for title and abstract screening.

Figure 1. PRISMA study flow diagram. Number of records identified, included and excluded during the literature search.

A total of 285 records were considered for inclusion. 128 records did not meet the inclusion criteria: secondary analyses (n = 11), pilot studies (n = 15), conference papers (n = 26), not related to SMART design (n = 67), observational studies (n = 5), proof of concept papers (n = 2), and feasibility studies (n = 2).

A total of 157 records were included in the systematic review: 24 (15.3%) protocols, 91 (58%) methodological papers, 12 (7.6%) trials, and 30 (19.1%) review papers.

Table 1 summarizes the research areas in which SMART designs have been used, the countries in which the studies were carried out, and sources of funding. Most of the studies were conducted in health and social related research. This may be partly a consequence of the databases used in our search. However, within the health-related research, there was an extremely broad selection of domains, including for instance cancer [18–20], toxicology [21], mental health [22,23], HIV [24], weight loss [3].

Table 1. Characteristics of included studies.

Characteristic	N (%)
Area
Education	3 (1.91)
Health:
Psychiatry	30 (19.11)
Mood disorders	20 (12.74)
Cancer	23 (14.65)
HIV/AIDS	9(5.73)
Sport and exercises	8 (5.10)
Other areas	20 (12.74)
Social	12 (7.64)
Not specific	32 (20.38)
Country
Australia	1 (0.64)
Belgium	1 (0.64)
Brazil	2 (1.27)
Canada	10 (6.37)
China	1 (0.64)
German	1 (0.64)
Kenya	2 (1.27)
Singapore	1 (0.64)
South Africa	1 (0.64)
Spain	1 (0.64)
United Kingdom	1 (0.64)
United Kingdom-France	1 (0.64)
USA	134 (85.35)
Funding
Other	21 (13.37)
Private	13 (8.28)
Public	123 (78.35)

Other: this category comprises studies for which no funding information was given, and studies without clear funding. Other areas: this group comprises health related areas such as: surgery, cardiology, rheumatology, nutrition, addiction, dentistry.

Thirty-two (20.38%) records were not specific to any research area since we included methodological papers treating research questions that can be applied to any area. Three (1.91%) of the records considered research from educational trials [25–27]. Most of the studies were mainly conducted in the USA 134 (85.35%) and Canada 10 (6.37%).

A large majority of the studies were supported by public funds 123 (78.35%). Twenty-one (13.37%) of the records did not provide enough information about the source of funding.

3.2. Trials review

The characteristics of the 12 (7.6%) trials found in the systematic review are summarized in Table 2. A list of selected trials is given in Appendix D.

Table 2. Trials and protocols characteristics.

Characteristic	Trials (N = 12)	Protocols (N = 24)
	N (%)	N (%)
Area
Education	1 (8.33)
Health	11 (91.67)	21 (87.50)
Social		3 (12.5)
Number of initial Treatments
1	1 (8.33)	2 (8.33)
2	8 (66.67)	18 (75)
3	1 (8.33)	4 (16.67)
4	1 (8.33)
5	1 (8.33)
Multicentre trial
No	4 (33.33)	13 (54.17)
Yes	8 (66.67)	11 (45.83)
Sample calculation size reporting
No	8 (66.67)	2 (8.33)
Yes	4 (33.33)	22 (91.67)
Stage-specific aim
First stage	7 (58.33)	12 (50)
Second stage	5 (41.67)	12 (50)
Non-responders comparison
No	3 (25)	10 (41.67)
Yes	9 (75)	14 (58.33)
Embedded AIs comparison
No	9 (75)	18 (75)
Yes	3 (25)	6 (25)
Selecting interesting AIs
No	10 (83.33)	22 (91.67)
Yes	2 (16.67)	2 (8.33)
Selecting deeply embedded AIs
No	12 (100)	21 (87.50)
Yes		3 (12.50)
Finding optimal AIs
No	11 (91.67)	21 (87.50)
Yes	1 (8.33)	3 (12.50)
Use of multiple testing
No	9 (75)	19 (79.17)
Yes	3 (25)	5 (20.83)
Analysis method
ANOVA	1 (8.33)
Cox PH	3 (25.00)
GEE	1 (8.33)
LMM	6 (50.00)
Negative binomial regression	1 (8.33)
Embedded AIs as secondary aim
No	9 (75)	19 (79.17)
Yes	3 (25)	5 (20.83)
Sample size software
Not reported	11 (91.67)	20 (83.33)
R	1 (8.33)	1 (4.17)
PS		2 (8.33)
STPLAN		1 (4.17)
Analysis software
Not reported	7 (58.33)
R	1 (8.33)
SAS	1 (8.33)
R+SAS	1 (8.33)
SPLUS	1 (8.33)
SPSS	1 (8.33)
Country
Australia		1 (4.17)
Brazil	1 (8.33)	1 (4.17)
German		1 (4.17)
Kenya		2 (8.33)
South Africa		1 (4.17)
USA	11 (91.67)	18 (75)
Funding
Private	3 (25)	2 (8.33)
Public	8 (66.67)	22 (91.67)
Other	1 (8.33)

ANOVA: Analysis of Variance; Cox PH: Cox Proportional Hazard; GEE: Generalized Estimating Equations; LMM: Linear Mixed Model; PS: Power and Sample size calculation; SAS: Statistical Analysis System; STPLAN: Study Planning calculations; SPSS: Statistical Package for the Social Sciences.

^*Sums may not add due to rounding errors.

Basic trial characteristics

Most of the trials, 11 (91.67%), were conducted in health-related area (cancer [18,19], mental health [22,23,28], toxicology [21]).

One of the 12 trials started with one treatment option for all patients [29]. The remaining 11 (91.67%) started with at least two treatment options. Five (41.67%) of the 12 trials were conducted in children and adolescent population. Eight (66.67%) of the 12 trials were conducted in multicentre settings. Seven (58.33%) trials had continuous primary endpoints, while 3 (25%) trials used time-to-event primary endpoints. The remaining 2 trials (16.67%) had a binary and count primary outcome respectively. It was observed that only 4 (33.33%) of the trials reported all the parameters required for a priori sample size calculation (i.e., such that we were able to replicate their results using provided information). The smallest trial recruited 83 patients [22], while the largest recruited 2,000 patients [29]. The median sample size was 171 patients. Most of the trials (66.67%) were publicly funded and were conducted in the USA (91.67%).

Trial aims considerations

All the trials considered stage-specific aims in their analyses. All 12 trials (100%) considered the first stage aim as their primary aim. Moreover, 5 (41.67%) of the 12 trials also included second stage aims within their primary aims. One key component to SMART design is to re-randomise all or a certain group of patients, mostly non-responders to the initial treatment option. In this review, in 9 (75%) of the trials, there were treatment comparisons among non-responder to first stage treatment options. Another advantage of the SMART design compared to standard RCT, is the possibility to compare embedded AIs. Although none of the trials did consider this comparison as one of the primary aims, some of them considered this comparison as a secondary aim. Three (25%) of the trials did perform some comparisons among embedded AIs. One of the reasons to compare may be to identify/test some pre-specified AIs. This type of analysis was performed in 2 (16.67%) of the 12 trials [23,30]. Another type of analysis performed, if there is interest in AIs, is to build some AIs which optimizes an outcome of interest using method such as Q-learning [9,31]. This type of analysis was performed in one (8.33%) of the 12 trials [32].

Analysis considerations

Once the trial aims are clearly defined, the next point to consider is the type of analyses to be performed to answer the research questions. The statistical methods used to answer the primary research questions were Cox proportional hazard model (for the time-to-event) used in 3 (25%) trials; the negative binomial model, generalized estimating equations and analysis of variance, each used in one (8.33%) trial. In the remaining six (50%) trials, linear mixed-effects models were used. Only one trial (8.33%) reported information about the software that was used for sample size calculations. In the case of multiple primary aims, it may be appropriate to apply a multiple testing adjustment to control the overall chance of making a type one error. The planned method to adjust for multiple testing should generally be pre-specified and reported. In our review, 3 (25%) of the 12 trials mentioned that they considered multiple testing adjustment, using closed test procedure and Bonferroni methods [33]. The last point, which was covered during this systematic review, was about the software used for the final analyses. Five (41.67%) trials reported information about the software. Each of the following software packages was used once (8.33%): R [34], SAS, R + SAS, SPLUS and SPSS.

3.3. Protocol review

We noted that some of the information could not be collected since the protocols were published before completion of the studies. In total, we retrieved 24 protocols. Their characteristics are displayed in Table 2. A list of selected protocols is given in Appendix D.

Basic protocol characteristics

Most of the protocols, 21 (87.50%), concerned studies that were conducted in health-related area (cancer [35], chronic pain [36], mental health [37,38], HIV [39–41], tobacco cessation [42,43]). Eleven (45.83%) of the studies were conducted in multicentre settings. Most (18 (75%)of 24) of the protocols described studies that started with two initial treatment options. Fourteen (58.33%) of the protocols described studies whose primary endpoints were continuous outcomes. Nineteen (79.17%)of 24, were mainly conducted in the adult population. It was observed that 22 (91.67%) protocols provided all the parameters required for sample size calculations. Twenty-two studies (91.67%) were publicly funded and 18 (75%) were conducted in the USA.

Trial aims considerations

Twenty-three (95.83%) of the studies were powered to answer their primary research questions using stage-specific aims. One study compared an AI against a control treatment [37]. Thirteen (54.17%) had their primary aims based on the first stage goals while 12 (50%) had second stage goals within their primary aims. Two (8.33%) of the studies had both first and second stage-specific aims [43,44]. It was also observed that in 14 (58.33%) studies, it was planned to compare the outcome among patients who did not respond to their first stage treatment options. From Table 2, one can also see that few protocols (6, 25%) were interested in comparing embedded AIs in order to identify the best one; or in selecting some pre-specified AIs to be compared with the rest. A small number of studies, 3 (12.5%) out of 24, were interested in deeply embedded AIs, and in the estimation of optimal AIs [35,36,45]. Lastly, 5 (20.83%) studies considered the comparisons of embedded AIs as their secondary aims.

Analysis considerations

The reporting of the analysis considerations was different for protocols, compared to trials. For instance, the details about the analyses method used to answer the primary research question(s) and the software used were not generally specified at the time of writing the protocols. These details are mainly provided in the statistical analysis plan documents[46]. Five (20.83%) protocols mentioned that multiple comparisons will be adjusted for. However, only two of these mentioned which methods were going to be used [43, 47]. Most of the protocols, 20 (83.33%) out of 24, did not provide information about which software was used for sample size calculation. The Power and Sample size calculation software (PS) [48] was mentioned in two protocols, followed by R [34], and STPLAN [49] each mentioned once.

4. Discussion

In this systematic review, we identified 157 records, of which 36 (22.93%) contained most of the information needed to answer our research questions (12 trial reports and 24 protocols). We observed that the highest number of records were published in the USA. Most of the studies were conducted in multicentre settings and started with two treatment options in the first stage. Even though one can answer more research questions using a SMART design compared to standard RCT, it was observed that research mainly powered their studies using stage-specific aims. Statistically speaking, this may be explained by the fact that it is easier to power and analyze studies whose primary aims are based on the first stage. However, one must keep in mind that study design should not solely rely on ease of statistical methods. Most of the published trials did not provide parameters required for sample size calculations (we could not replicate their results using provided information), however this was not the case for the retrieved protocols. The problem of reporting on the parameters required for sample size calculations was also reported in the literature for RCTs [50].

One of the appealing features of the SMART design is the possibility to re-randomize all or a group of patients, mainly non-responders to the initial treatment, based on the response, patient’s characteristics, or behaviors observed during the previous treatment. This has advantages over more traditional designs such as a “one-stage-at-time” randomized trial (RT) and a RT in which a fully formed AI is compared to an appropriate control [51]. The first alternative treats each intervention stage as a different trial, hence requiring more participants compared to a SMART design. The use of tailoring variables allows answering more clinical and theoretical questions compared to the second alternative design.

It was observed that most of the published records had planned to make some comparisons among non-responders to first stage treatment options. The treatment options at each stage and the criteria on which randomizations are based determine a set of AIs said to be embedded in the SMART design. Researchers can power their study to address primary research questions concerning the AIs that are embedded in the design. Some publications used embedded AIs as their secondary aims. However, a low proportion of trials used analyses involving embedded AIs. Those analyses may involve some hypotheses testing aiming at identifying the most significant AI compared to a reference one or from a group of AIs. Another approach is to identify an AI leading to an optimal outcome. AIs comparisons may lead to many tests requiring some multiplicity adjustment. It was observed that not many publications did provide information about multiple testing procedures which were/or will be used in the analyses. We noted that most of the records did not provide information about statistical software which were used for sample size calculations and final analyses (especially for trials). Software tools for final analyses are not always available for potential users without a strong programming background. The Pennsylvania State Methodology Centre webpage [16] provides a wide range of software tools to compute sample size for SMART studies,

It was observed that most of the SMART studies were publicly funded. One explanation may be that fully powered SMART designs are large, expensive and generally multisite studies [52]. Another reason may be that SMARTs consider complex interventions or long-term treatment strategies whereas the pharmaceutical industry is focused on efficacy trials for new drug(s) for a particular stage of a condition. Pharmaceutical companies may be reluctant to fund studies which may combine or compare drugs in sequences with other companies’ drugs [53]. We also observed that publicly funded studies were larger compared to privately funded ones.

4.1. Reporting of SMART studies

Studies should clearly state their primary aim, one of: a) comparison of different interventions options at different stages of the interventions, b) comparisons of AIs that are embedded within the SMART design, c) construction of more deeply embedded AIs. Reports should show how the sample size calculation links to the primary aim. When the primary aim is to compare first-stage interventions, the standard sample size calculation formula can be used (significance level, power, effect size information should be provided). When the comparison concern second-stage intervention options, postulated information about overall initial (non)response rate is also required. If simulations are used for sample size calculations, all assumptions and scenario should be clearly reported. They should also clearly provide information about the analysis considerations: state whether multiple comparisons were performed, adjustment method used, primary analysis method, Software availability for primary analyses and sample size calculations. An example of good reporting, results for methodological and review papers are given in Appendix E.

Our review had some limitations. Due to resource limitations, it was not possible to supplement the database by checking all reference lists, or trial registries. Further work may include a supplementary search. We included only articles published in English. We may have missed studies published in journals not included in the databases we searched from or studies that did neither describe themselves as using a SMART design nor mention adaptive intervention nor other keywords. A good reporting example can be found in O’Keefe et al. (2019)[47]

5. Conclusions

Despite the increase in the use of SMART design, there still some appealing features of this design that are not widely used or reported well. Many methodological publications have been exploring different aspects of the SMART design. Trials implementing SMART design should report all key components wherever possible, to better aid the understanding and ongoing research.

What is new?

Key findings:

• Most of the trials using SMART designs did not provide parameters required for sample size calculations and were mainly powered using stagespecific aims.

• Very few studies considered analyses involving embedded adaptive interventions (AIs).

• Some attractive features of the SMART design were rarely used in reviewed studies.

What this adds to what is known?

• The CONSORT statement and the International Conference on Harmonisation E9 (ICH E9) provide guidelines on reporting and justifications on sample size calculation for randomised controlled trials (RCT).

• In a sequential multiple assignment randomised trial (SMART) design, some or all patients can be randomised more than once, especially nonresponders to the first stage treatment option. One can answer more research questions using a SMART design compared to a classical RCT design.

• While there are some reviews which summarised how multistage interventions were designed in practice, none of them covered the reporting of information required to design SMART studies.

What is the implication, what should change now?

• Researchers designing studies using SMART design should report all parameters required for sample size calculations and information related to multiple testing procedures.

• They should also consider aims involving embedded AIs rather than relying only on stage-specific aims. Increased availability of software tools to perform advanced analyses would also make SMART designs more appealing.