ASPECT-R—A Tool to Rate the Pragmatic and Explanatory Characteristics of a Clinical Trial Design

Abstract

Objective: Clinical and observational trials can be broadly categorized into having explanatory or pragmatic approaches with specific trial designs located somewhere along this spectrum. Two 10-domain instruments, the PRECIS and Pragmascope, have been developed to facilitate clinical trial design within a framework that is either more explanatory or pragmatic.

Design: We have adapted the PRECIS and Pragmascope instruments to permit both design support and post-hoc evaluation of clinical trials and to improve consistency of use and interpretation across raters. This adapted instrument, A Study Pragmatic-Explanatory Characterization Tool-Rating—or ASPECT-R—is described.

Results: Adaption of the PRECIS and Pragmascope instruments included reducing the 10 original domains to six. Each of the six ASPECT-R domains has a definition of domain terminology and detailed descriptive anchors. The domains are rated from 0 to 6, where 0 is considered extremely explanatory and 6 extremely pragmatic. Using an Excel®-based file with cover page cells for entry of the study objective(s) and study population of interest, the ASPECT-R instrument has individual domain-related worksheets where the user rates each of the six domains. Each of the six domain worksheets has a section provided for the user to summarize and record the rationale for their domain scoring. Each domain worksheet page also contains a radar graph that auto-populates each of the domain ratings as the user completes these ratings.

Conclusion: This new tool, ASPECT-R, should provide a reliable, objective way to rate studies along the explanatory-pragmatic spectrum that will better support trial design and facilitate interpretation of completed trials.

The complete ASPECT-R tool and guide materials can be accessed online by clicking or visiting this link: http://innovationscns.com/aspect-r-tool-and-training-materials/.

INTRODUCTION

Clinical trial designs differ considerably with respect to whether they are conducted under highly controlled and defined conditions (explanatory) or more broadly reflect real world circumstances (pragmatic). These different approaches to trial design are generally aimed at addressing different questions with varying strengths and limitations and differences in extent of generalizability. These differences in trial design may substantially impact results even when similar interventions or treatments are compared. Failure to consider the differences in trial design may lead to misinterpretation of individual studies and when studies are analyzed as a group (as in meta-analyses). Because specific trials are designed on a spectrum of “explanatoriness” or “pragmaticness,” methodologists have begun to develop instruments that facilitate identification of key explanatory and pragmatic elements in clinical trial designs.^1,2 By standardizing the characterization of study designs, they can support researchers, clinicians, healthcare providers, and policymakers in understanding and interpreting clinical trial results, particularly with respect to their generalizability to real world clinical practice.

The PRECIS (Pragmatic-Explanatory Continuum Indicator Summary) tool has been described by Thorpe et al as an instrument developed to assist researchers when designing trials along the explanatory-pragmatic spectrum.¹ This tool captures information along 10 study design domains: Practitioner Expertise (experimental); Flexibility of the Experimental Intervention; Eligibility Criteria; Primary Analysis; Practitioner Adherence; Participant Compliance; Outcomes; Follow-up Intensity; Practitioner Expertise (comparison); and Flexibility of the Comparison Intervention.¹ The PRECIS tool has been adapted as the Pragmascope by Tosh et al with the addition of a 6-point (0 to 5, very explanatory to very pragmatic) visual analog rating for each domain.²

ASPECT-R (A Study Pragmatic-Explanatory Characterization Tool-Rating; ©2014 Janssen Pharmaceuticals, Inc.) has adapted ideas from these instruments to build a still more versatile instrument. The number of domains has been reduced to six that are specifically related to the explanatory-pragmatic spectrum. Domains identified as redundant and domains focused on measures of study quality have been eliminated. In addition, detailed definitions of terms and descriptive anchors have been developed to facilitate greater reliability across raters. The objective of this brief report is to describe the development of ASPECT-R and highlight some of its strengths and limitations.

ASPECT-R DOMAINS AND RATINGS

ASPECT-R consists of six domains deemed important for characterizing the explanatory to pragmatic spectrum of study designs. These six domains and their respective definitions are summarized in Table 1.

TABLE 1.

ASPECT-R domains and definitions cf domain terminology^*

DOMAIN	DEFINITION OF DOMAIN TERMINOLOGY
Participant eligibility criteria	Considerations include the intended treatment population of interest that has been identified by the study’s authors (e.g., in the study title, objectives, discussion, conclusions)
Intervention flexibility	Considerations include posology, dose, dosing interval, windows allowed for dosing; permitted concomitant treatments. The domain is rated separately for experimental and comparisons treatment interventions
Medical practice setting/practitioner expertise	Considerations include experience, skills and resources of the practitioner and the treatment team; the healthcare delivery system; standards of care at the site, and local cultural practices that may influence medical delivery or outcomes. The domain is rated separately for experimental and comparisons treatment interventions.
Follow-up intensity and duration	Considerations include frequency and length of visits and the number and the scope of the assessments.
Outcome(s)	Considerations include evaluation of measure(s) by which the interventions’ effects are assessed and how well they reflect outcomes that are used and considered important to real world practice.
Participant adherence	Considerations include the degree to which the subjects are encouraged and tracked for adherence to study-related procedures.

^*Original domains, descriptors, and concept from Thorpe et al 2009 (see reference ¹)

The ratings for each of the six domains range from 0 to 6, where, in general, 0=extremely explanatory; 1=very explanatory; 2=explanatory; 3=elements of both designs; 4=pragmatic; 5=very pragmatic; and 6=extremely pragmatic. Specific, detailed descriptive anchors for each rating have been developed to guide the rater. The domains of Intervention Flexibility and Medical Practice Setting/Practitioner Expertise include separate ratings for the experimental and comparator intervention to account for differences in the relative pragmatic approach that may sometimes exist in these design elements. The experimental and comparison ratings of each of these two domains are averaged with the score plotted on the corresponding radar graph.

ASPECT-R EXCEL® FILE AND VISUALIZATION OF THE RATINGS

ASPECT-R is available as an Excel®-based file that contains nine worksheets (i.e., a cover worksheet page [Figure 1, Appendix 1]) and, as illustrated in Figures 2 through 9 (Appendix 1), one worksheet for each of the six domains (with two worksheets provided for the domains of Intervention Flexibility-Experimental and -Comparison; and two provided for Medical Practice Setting/Practitioner Expertise-Experimental and -Comparison]). The ASPECT-R rating for each of the domains is auto-populated on a radar graph that is embedded in each of the domain worksheet pages with the graph building as the rater completes each domain rating.

ASPECT-R RATERS EXPERTISE AND EXPERIENCE

Ideally, persons who use the ASPECT-R tool to rate studies have knowledge of and experience with designing clinical research studies. An advanced degree is not necessarily required to rate study domains; however, scoring of individual domains does require particular expertise regarding the study’s population of interest, including its epidemiology and the clinical characteristics of the underlying illness/disorder, the course of the illness/disorder, general treatment regimens and modalities, as well as anticipated responses. Ideally, the rater has a broad understanding of treatment practices for the therapeutic area being examined.

The rater’s ability to complete the tool in an accurate, reliable, and timely manner is also dependent upon their training and experience with respect to ASPECT-R, its appropriate application, and an understanding of the definitions of terms used. At a minimum the rater should review and understand the training materials that have been developed for the tool prior to completing an ASPECT-R rating. These materials include a training slide set, and the Rating Considerations and Rating Anchors embedded in the instrument itself.

SOURCE DOCUMENTS AND TIME REQUIREMENTS

Domain ratings for ASPECT-Rare based upon the rater’s review of the published manuscript and any additional information that reflects the study design and trial methodology. Other source documents that may be available and provide insight and information for the domain ratings include the study protocol and study report. These and other documents may be available on the clinical trials registration website (clinicaltrials.gov) or through other publicly available means. In some cases information may be best obtained through contact with the study authors and/or investigators themselves.

The cover page of the Excel® file for ASPECT-R provides cells for the rater to document the source documents utilized in determining the domain ratings, the particular study’s objective, and the study population of interest. Completing these elements is a critical first step when conducting an ASPECT-R analysis, as these considerations will significantly impact the subsequent ratings assigned to each domain.

The individual ASPECT-R domain worksheets also contain a free-text cell for the rater to summarize the rationale for their score. This section should be used to document information from the study design that contributes to their ratings. The rater may also use this section to add personal knowledge of the study design. It is valuable to indicate here whether the study was part of a regulatory submission, as such trials require a high degree of rigor that likely contributes to the ‘pragmaticness’ of its design. Overall, documentation of the rationale for these ratings will facilitate better validity and reliability.

ASPECT-R generally takes approximately 30 to 60 minutes to complete for a given study; however, this may vary based upon the quality and clarity of the source documents, the rater’s knowledge of the disease state and its management, and their familiarity with ASPECT-R.

DISCUSSION

Both explanatory and pragmatic study design approaches have value in the assessment of clinical interventions. Neither is considered intrinsically superior to the other. Indeed, many studies have both explanatory and pragmatic design characteristics. However, the increasing importance of real-world data for the provision of healthcare has driven the need for an increased understanding of the explanatory and pragmatic characteristics of clinical trials. For example, healthcare providers want to know if an intervention works in patients with concomitant medications and/or comorbid conditions (factors that often result in these individuals being excluded from explanatory studies). Such questions are often difficult to answer with traditional explanatory studies that are designed to establish intervention or treatment efficacy under highly controlled scenarios and with selected patients.

A limitation to ASPECT-R at its current stage of development is that its use has largely been limited to the authors. To address its broader applicability, an inter-rater reliability validation study has been completed and accepted for publication.³ In addition, completion of ASPECT-R can sometimes be limited by poorly documented or unavailable information regarding important features of study design such as site training and resources. The optimal application of ASPECT-R requires that users have considerable clinical trial expertise regarding the population of interest (e.g., schizophrenia) and its treatment (e.g., antipsychotic agents, psychotherapy). Another limitation to the findings from ASPECT-R is that it does not consider the quality of the study’s design and conduct, or the validity of the interpretation of study’s considering the development of a complementary ASPECT-type tool that will assess the clarity of the research question as well as the appropriateness of the study’s design, procedures, conduct, analytical methods, and result interpretation.

In conclusion, ASPECT-R provides an improved descriptive approach for raters to consistently identify where a study’s key design domains lie along the pragmatic to explanatory continuum. As such, ASPECT-R ratings should support the development of better pragmatic trial designs. They should also facilitate better understanding of a completed study’s generalizability to real-world circumstances.

The complete ASPECT-R tool and guide materials can be accessed online by clicking or visiting this link: http://innovationscns.com/aspect-r-tool-and-training-materials/.

APPENDIX 1. Figures 1–9: ASPECT-R worksheets

FIGURE 1.

ASPECT-R—Defining the study objective and population of interest

FIGURE 2.

Domain and domain descriptions and ratings for participant eligibility (Domain 1)

FIGURE 3.

Domain and domain descriptions and ratings for intervention flexibility-experimental (Domain 2a)

FIGURE 4.

Domain and domain descriptions and ratings for intervention flexibility-comparison (Domain 2b)

FIGURE 5.

Domain and domain descriptions and ratings for medical practice setting/practitioner expertise-experimental (Domain 3a)

FIGURE 6.

Domain and domain descriptions and ratings for medical practice setting/practitioner expertise-comparison (Domain 3b)

FIGURE 7.

Domain and domain descriptions and ratings for follow-up intensity/duration (Domain 4)

FIGURE 8.

Domain and domain descriptions and ratings for primary trial outcomes (Domain 5)

FIGURE 9.

Domain and domain descriptions and ratings for participant compliance (Domain 6)