Reporting of Protocol Rationale and Content Validity in Randomized Clinical Trials of T'ai Chi: A Systematic Evaluation

Abstract

Background: Mind–body exercise interventions are typically multimodal, complex, and pluralistic, and few have been developed with the goal of therapeutically targeting a specific medical population. It is thus important that clinical trials evaluating mind–body interventions provide some justification for the use of the specific protocol being evaluated.

Objectives: This article reports the results of a systematic review of the quality of reporting of protocol rationale and content validity for using a specific t'ai chi protocol in a randomized controlled trial (RCT).

Methods: Electronic literature searches were conducted using PubMed/MEDLINE, EBSCOhost, and the Cochrane Library from inception through June 2015. Search terms were Tai Chi, Taiji, Tai Chi Chuan; searches were limited to English-language RCTs. Inclusion and exclusion of trials were reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The authors assessed the level of reporting with respect to t'ai chi protocol validation using a 5-point grading system based on whether (1) a specific protocol was mentioned, (2) rationale for the specific protocol was stated and supported, and (3) formal evaluation for content validity was conducted.

Results: There was significant heterogeneity in the quality of reporting related to protocol rationale and content validity. A total of 171 publications were identified. Studies met between 0 and 4 validity criteria (of possible 5, more indicating better quality), with a mean of 2.52 (±SD 1.2) and median of 3. Twenty (12%) trials did not mention a specific t'ai chi protocol, 10 (6%) trials met 0 of 5 criteria, and 47 (31%) studies met 4 of 5 criteria. Formal validity assessments were employed in only one trial.

Conclusions: The poor quality of protocol rationale and content validity reporting limits our ability to accurately evaluate the evidence of t'ai chi as a therapeutic intervention. The development of formal guidelines for developing and reporting intervention validity for multimodal mind–body exercises like t'ai chi may improve the quality and interpretability of research.

Introduction

Compared with efficacy trials evaluating single-agent pharmacological therapies, trials evaluating the effectiveness and safety of multimodal nonpharmacological interventions, including mind–body exercise programs, pose unique methodological challenges. Multimodal mind–body interventions are complex and include many potential therapeutic components or “ingredients,” all of which can vary in intensity, duration, frequency, and mode of delivery within a given protocol.^1–5 How these protocol parameters are chosen and delivered could significantly impact therapeutic effects. Moreover, how they are reported impacts the inferences drawn from individual and collective studies. This complexity is further compounded by the pluralism of many mind–body therapies, with practices like t'ai chi and yoga having many different styles and training regimens, each varying in composition and relative emphasis of therapeutic components.^2,6

The development and preliminary evaluation of traditional multimodal mind–body interventions typically follow a very different trajectory compared with pharmacological agents, further complicating evaluations of effectiveness. Pharmacological research often begins with a well-defined mechanistic model, in which a specific compound targets a known physiological pathway.⁷ These agents are often first evaluated in vitro, followed by animal and preclinical studies, before being evaluated in Phase II and III clinical trials.

In contrast, many mind–body interventions are rooted in folk traditions and have been passed down over generations through oral transmission and interpersonal training, with little formal systematic evaluation. Moreover, it is not uncommon for the first clinical evaluation of a mind–body protocol to be a Phase II trial, often bypassing basic physiological mechanisms research, dosing studies, or even case series establishing protocol feasibility and preliminary estimates of safety and effectiveness. For these reasons, it is critical for publications evaluating mind–body interventions to provide context regarding the rationale and content validity of the specific protocol being evaluated.

Evaluation of content validity, more commonly discussed in the context of survey psychometric testing and instrument development,⁸ has also been employed in the development of nonpharmacological interventions.^9,10 In this article, the authors refer to content validity as the extent to which the intervention employed in the study is appropriate and specific for the intended study question, population, and outcomes. Expert panels have been used to guide the systematic development and optimization of the content and delivery of web-based interventions to enhance behavioral coping for school-age children,¹¹ and for informing feasibility, acceptability, and relevance of behavioral interventions to support family caregivers of stroke survivors.¹²

Not including an evaluation and/or refinement phase addressing the content validity of nonpharmacological interventions, before formal testing in a clinical trial, could misinform conclusions regarding general efficacy or safety. For example, in a study of exercise for enhancing functional rehabilitation in early-stage stroke survivors, the content validity of a thoughtfully developed stroke-specific graded walking protocol is likely to be different than a generic high-intensity jogging program. A trial evaluating the latter program is unlikely to support its safety and rehabilitation benefits, not because exercise is not helpful to stroke survivors; but rather because the specific protocol was not well-matched to the study population and outcome goals. In addition to leading to misinformed conclusions, lack of a priori protocol justification or evaluation of content validity is likely to lead to nonoptimal use of research resources.

The extension of Consolidated Standards of Reporting Trials (CONSORT) guidelines to nonpharmacological trials, including complementary and alternative medical (CAM) therapies, has brought to the fore the importance of including a clear description of the different components of interventions, the procedure for tailoring interventions to individual participants, justification for using a specific therapeutic protocol, and reference to any previous validity studies.^13,14 Modifications of CONSORT guidelines specific to acupuncture trials, for example, have included items addressing the rationale for specific treatments based on historical context, literature sources, and consensus methods.¹⁴ Similarly, recent guidelines for comparative effectiveness trials of complex CAM interventions have also recommended including justification for all protocol components.¹⁵ However, formal guidelines for reporting of multimodal mind–body exercises have not been developed, and little research has systematically assessed the quality of reporting specifically related to the rationale and content validity of mind–body protocols chosen for evaluation in clinical trials.

T'ai chi is a low-impact, mind–body exercise that originated in China and has become increasingly popular in the West. T'ai chi integrates musculoskeletal training (e.g., strength, flexibility, and coordination), breath training, and a variety of cognitive skills (e.g., body awareness, focused mental attention, and relaxation).^1,2 Studies have shown that t'ai chi can improve a number of health conditions including cardiorespiratory health,^16–22 musculoskeletal conditions, balance, and fall risk,^23–28 cognition,^29,30 and emotional well-being.^31,32 Critical analyses of this literature have already highlighted concerns about reporting of safety, dosage, choice of control groups, and analysis and reporting of results.^1,2,33,34 However, to our knowledge, there has yet to be a systematic evaluation of reporting of the justification for use and content validity of specific t'ai chi protocols employed in randomized controlled trials (RCTs).

In this article, the authors report the results of a systematic review of the quality of reporting related to three broad questions that reflect increasing degrees of validity for using a t'ai chi protocol in a trial: (1) Was a specific t'ai chi protocol mentioned as opposed to t'ai chi in general? (2) Was a rationale for choosing a specific t'ai chi training protocol explicitly stated and supported with relevant references?;(3) Was the specific t'ai chi protocol formally evaluated for content validity?

Materials and Methods

Literature search

Electronic literature searches were conducted using PubMed/MEDLINE, EBSCOhost, and the Cochrane Library from inception through June 2015. Search terms were Tai Chi, Taiji, Tai Chi Chuan; searches were limited to English-language RCTs. Hand searches were performed of retrieved articles for additional references.

Eligibility criteria

All RCTs that were published in English and used t'ai chi as an intervention were included, with the exception of study designs that evaluated multiple pragmatically delivered community-based protocols within a given trial. Trials that used t'ai chi as a control group were also excluded; otherwise control groups were not restricted. Populations were not restricted. Inclusion and exclusion of trials were reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Inclusion following screening was determined by two independent investigators (D.L. and D.B.). Discrepancies were resolved through discussion and use of a third arbiter when needed.

Data extraction and synthesis

The following data were systematically extracted from each eligible study by at least two independent investigators (D.L., D.B., and E.G.): journal citation; country of trial conduct; population; and details of t'ai chi intervention including general style, specific protocol or form used, instructor qualifications, details of training regimen including dose, frequency, and duration. Outlined in Figure 1, using an author-developed 5-point grading system, we assessed the quality and level of reporting based on our three broad questions that reflect increasing degrees of validity for using a t'ai chi protocol in a trial. One point was given for each of the individual criterion met.

FIG. 1.

Scoring criteria flowchart. Specific t'ai chi protocol: was a specific t'ai chi protocol mentioned as opposed to t'ai chi in general? For example, Yang style 24. Therapeutic principles/components of specific t'ai chi protocol: were therapeutic components of the specific t'ai chi protocol mentioned or described? For example, breathing, mindfulness, meditation, relaxation, imagery and visualization, biomechanical integration, and physical exercise. Population/outcomes matched to the specific t'ai chi protocol: was there explicit mention of the relevancy or appropriateness of the specific t'ai chi protocol and/or its therapeutic principles to the study population or study outcomes? References: were references provided that support the use of the specific t'ai chi protocol? Formal content validity testing: was there a formal evaluation of the content validity of the t'ai chi protocol? For example, a Delphi process.

Criterion 1 addresses whether a specific t'ai chi protocol was reported. Studies received 1 point if a specific t'ai chi protocol (e.g., Yang style 108 movement form, 56 Chen style competition form) was mentioned as opposed to t'ai chi in general. Criteria 2 to 4 address whether a rationale for the specific t'ai chi protocol was reported. Studies received 1 point for fulfilling each criterion.

Criterion 2 requires mention of therapeutic principles or components (e.g., body awareness, breathing, meditation, relaxation, imagery, and visualization) of the specific t'ai chi protocol (e.g., reporting use of Yang style 24 short form that emphasizes self-awareness, deep breathing, and gentle stretches).

Criterion 3 requires explicit mention of the relevancy or appropriateness of the specific t'ai chi protocol and/or its therapeutic principles to the targeted study population or study outcomes (e.g., simplified Yang style protocol used in a pulmonary disease population included meditative breathing exercises to improve gas exchange, or use of shorter stances in Sung style t'ai chi for arthritis to minimize knee strain in knee osteoarthritis study).

Criterion 4 requires references cited that discuss or support the therapeutic use of the specific t'ai chi protocol in previous studies or publications.

Criterion 5 directly requires formal content validity testing of the specific t'ai chi protocol described. The study received 1 point if there was report of a formal evaluation and/or refinement phase addressing the validity of the specific intervention before testing in a clinical trial (e.g., a Delphi process with an expert panel addressing population match, feasibility, acceptability, relevance of specific components, and exercises). Discrepancies related to data extraction and study scoring were resolved through discussion and use of a third arbiter when needed. Analytic methods were limited to descriptive statistics calculated using Microsoft Excel. Risk of bias assessment was not performed as the goal of this review was not to make statements about study findings and intervention efficacy.

Results

Study selection

The initial database search yielded 2,514 distinct citations. After screening, 171 publications met the eligibility criteria and were included in the review. See PRISMA flow diagram (Fig. 2) for a summary of our literature search and article selection process.

FIG. 2.

PRISMA flow diagram. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; RCT, randomized controlled trial.

General characteristics of t'ai chi interventions

T'ai chi interventions varied greatly in style, specific forms or routines, protocol content, class frequency and duration, and length of training period (Supplementary Appendix). The majority of studies (n = 100; 58%) employed interventions described as “simplified” research protocols, that is, components or isolated elements of traditional t'ai chi forms taught using t'ai chi principles; 32 (19%) employed complete traditional t'ai chi sets (e.g., Chen 18 form, Yang style 24 form); 15 (9%) included a subset of movement sequences derived from traditional t'ai chi choreographed sets (e.g., first 18 moves of Yang 108); 6 (4%) included elements or principles of t'ai chi integrated into other exercises (including cognitive or physical exercises); and 18 (11%) studies did not provide sufficient t'ai chi intervention details (Fig. 3a).

FIG. 3.

Characteristics of t'ai chi protocols used in RCTs. (a) Percentage of studies employing each of the five different types of t'ai chi protocols. (b) Percentage of studies meeting specific validation criteria. RCT, randomized controlled trial.

Individual t'ai chi sessions varied in duration from 5 to 120 min. The frequency of sessions varied from one to seven times per week, with overall training programs lasting 1 weekend to 2 years. Qualifications of instructors were mentioned in 113 of 171 studies.

Quality of content validity reporting

Validity scores based on our five criteria for the 171 eligible trials ranged from 0 to 4 (maximum possible score of 5). The average score was 2.52 (±SD 1.24); the median score was 3. Twenty (12%) trials did not mention a specific t'ai chi protocol. Of the 151 trials that did mention a specific t'ai chi protocol (Criterion 1), 47 (31%) studies received a total of four points for additionally mentioning Criteria 2, 3, and 4: therapeutic principles/components of the specific protocol, match to population/outcomes, and references. Formal validity assessment (Criterion 5) was employed in only one trial,³⁵ although only limited detail on the process of validity assessment was provided (Fig. 3b).

Discussion

The main finding of this systematic review is that the quality of reporting of training protocol rationale and content validity in RCTs of t'ai chi is generally poor. Only one study employed formal validity testing and almost half the studies (44%) met no more than two of the five criteria related to the justified use of a specific training protocol for the population or outcome targeted in their study. The implications of these findings are compounded by both the heterogeneity of the t'ai chi interventions themselves (including protocol content, and training frequency and duration), the heterogeneity of the populations invested, and the fact that nearly 60% of the protocols evaluated represent simplified contemporary training regimens only recently developed for research and clinical applications (i.e., not well-established traditional or complete choreographed forms). These findings point to a critical gap that limits our ability to interpret the results of individual trials, and to generalize findings across multiple studies. Our findings also support the need for guidelines to improve the reporting of t'ai chi and related mind–body exercise interventions delivered in clinical trials.

Poor reporting of intervention content and targeted purpose has been reported in other reviews of nonpharmacological trials. For example, Dijkers et al.³⁶ summarized quality of reporting in 171 studies of medical rehabilitation, the majority of which were nonpharmacological. They reported that 62% of the 344 interventions included in their review were inadequately characterized. They noted that in addition to limiting interpretation of clinical effectiveness, lack of well-defined interventions decreases the ability of experienced researchers or clinicians to replicate procedures, and even within a study, to assure fidelity of delivery across clinicians and patients.

Chen et al.³⁷ published an excellent model of formal protocol validity testing for a t'ai chi intervention (Simplified Tai-Chi Exercise Program [STEP]); however, this study was not included in our review as it was not a randomized trial. The study team first used a patient focus group composed of frail elders residing in long-term care facilities, who had been exposed to a short t'ai chi program based on a traditional t'ai chi form, to inform the development of a prototype STEP. Then, a modified Delphi process was employed, through a mail survey, to solicit and integrate feedback from a team of 10 experts with knowledge of geriatrics, t'ai chi, and/or kinematics. Results from this first phase of feedback were synthesized into a pilot training video and training manual. These materials were then sent back to the expert panel along with a structured survey soliciting input on usability, clarity, and suitability of each movement. Feedback from these surveys was then used to finalize the STEP training protocol that was used in follow-up clinical studies.³⁷ Similar approaches were employed by Wiedemann in the development of a qigong training regimen for chronic neck pain.³⁸

The inclusion of a stage of formal validity testing of t'ai chi and related mind–body interventions, especially for protocols that deviate from widely used, previously tested, or traditional regimens, parallels calls by the National Center for Complementary and Integrative Health at the National Institutes for Health³⁹ for a more graded, incremental approach to clinical trials that includes thoughtful, prepilot protocol development. This validity testing can include evaluation of protocol content, and parameters related to optimal delivery (frequency, duration, and location of classes), qualifications and training of instructors, and supportive training materials (e.g., printed manuals, DVDs, and web-based modules). This important early work can minimize investment in protocols with high likelihood of failure right from the start. Although the addition of this validity testing stage may require time and modest resources, ultimately, this strategy promises to be a much more efficient use of the limited resources available to research in mind–body therapies overall.

There are a number of limitations of this study. First, the analysis was limited to English-language studies. Including studies in other languages may have better represented the evidence and improved generalizability to other cultures. Second, our inclusion of literature was limited to June 2015, and thus does not reflect the most current state of evidence. A subsequent formal PubMed review using identical search criteria as employed in this report spanning from June 2015 to June 2018 identified 52 additional eligible trials. A review of these articles identified only one new study that used a validated intervention known as the STEP, developed by Chen et al.,³⁷ and already discussed above as a model for formal protocol validity testing. Otherwise, no new studies that used formal validity testing were identified; thus, formal inclusion of these studies would not substantially alter our conclusions.

Conclusion

The relatively poor quality of protocol rationale and content validity reporting limits our ability to evaluate the evidence of t'ai chi as a therapeutic intervention. At a minimum, standards for reporting t'ai chi protocol content should be developed and employed. In addition, and especially for Phase II and III trials, guidelines for evaluating and reporting intervention validity may improve aspects of the quality, interpretability, and return on investment in research.

Author Disclosure Statement

P.M.W. is the founder and sole owner of the Tree of Life Tai Chi Center. P.M.W.'s interests were reviewed and managed by the Brigham and Women's Hospital and Partner's HealthCare in accordance with their conflict of interest policies. D.L., G.Y.Y., E.G., and D.B. have no competing financial interests.

Supplementary Material

Supplementary Appendix