So much research, so little application: Barriers to dissemination and practical implementation of Tai Ji Quan

Abstract

Despite the large number of articles published in the medical literature advocating the use of Tai Ji Quan for a wide variety of health-related outcomes, there has been little systematic broad-scale implementation of these programs. It may be argued that the lack of funding from organizations capable of implementing and overseeing large-scale programs, such as governmental health agencies or national non-governmental organizations concerned with healthcare for older adults, is to blame. However, the evidence these organizations need to justify underwriting such programs is in short supply because of conflicting priorities and standards related to determining the efficacy and effectiveness of Tai Ji Quan. Establishing efficacy through acceptable designs such as randomized controlled trials involves strict protocols to ensure meaningful internal validity but different approaches are needed to demonstrate meaningful effectiveness (external validity) outside the study setting. By examining the quality, quantity, and relative proportions of the randomized controlled trials, systematic reviews, and dissemination studies reported in the medical literature, this paper highlights the disparity in emphasis between efficacy and effectiveness research that has impeded the development of a cohesive literature on Tai Ji Quan and concludes that until more researchers develop a systematic, long-range commitment to investigating its health-related benefits, the research related will remain fractured and sporadic, limiting the incentive of large funding agencies to support its wide-spread use.

1. Introduction

Tai Ji Quan is a form of physical and mental training derived from the Chinese martial art of WuShu¹ that has become increasingly popular among older adults outside of China over the past 40 years because of its purported beneficial effects on physical and mental well-being. While its signature slow, rhythmic, no-impact movement characteristics have obvious appeal to this demographic, whether practicing Tai Ji Quan can actually enhance health by improving physical function, diminishing the risk of disease or curing illness is still unclear for the majority of conditions to which it has been applied.² Although there is a growing body of research supporting the value of Tai Ji Quan practice in maintaining or improving health-related quality of life, overall the evidence is limited, fractured, and poorly reported.³ Because widespread implementation of any health intervention requires the support of state or national agencies, whether governmental or non-governmental, it may be argued that the lack of a cohesive and compelling literature on Tai Ji Quan has impeded its use as a population-level health intervention even as more studies examining the health value of Tai Ji Quan across an increasing range of conditions appear in the literature. The apparent disconnect between the growing interest in Tai Ji Quan by medical researchers, clinicians and the general public and the reluctance of funding agencies to promote Tai Ji Quan programs on a large scale may be found in the fact that primarily two different types of evidence are need by funding agencies⁴ before they are willing to underwrite a public health initiative – evidence of efficacy and evidence of effectiveness, including cost-effectiveness - but the vast majority of researchers have only focused on establishing efficacy, and often not very well, in “one off” studies.

2. Efficacy vs. effectiveness

Although efficacy and effectiveness are often used interchangeably, it is useful to parse their difference to understand the problems facing Tai Ji Quan as a population-level health promotion program. Efficacy is considered the capacity of an intervention to produce a beneficial change under ideal circumstances, such as in a research setting, but effectiveness is determined by how well the intervention works in the real world.⁵ From a research design standpoint, establishing efficacy is primarily concerned with internal validity while the key issue for effectiveness is external validity; therefore each has different research design considerations and standards by which to establish their value. While there are several research designs that can be used to determine efficacy, well-designed randomized controlled trials (RCTs) are considered the gold standard in this regard because of their methodological framework within which strict controls allow the intervention to show its causal relationship to an outcome, if such a relationship exists, and remove alternative explanations for any change(s) identified. Unfortunately, there is no universal gold standard to determine effectiveness although the RE-AIM model⁶ has found increasing favor among researchers for its theoretical grounding and the appropriateness of the evaluation dimensions of the model as they relate to the feasibility of introducing an intervention into a community setting. The five parts of RE-AIM cover individual-level aspects (Reach and Efficacy), organization-level considerations (Adoption and Implementation) and shared (individual and organization) responsibilities (Maintenance). Finally, potential health promotion organizations need evidence that funding a program will be fiscally responsible, that is, even if a program is shown to be efficacious and effective, it has to have significant return-on-investment capacity in terms of health gained to money spent to be worthwhile implementing. Cost-effectiveness analysis typically involves measures such as dollars per quality adjusted life year (QALY) gained or disability adjusted life year (DALY) prevented to quantify the financial burden against health gain.^7,8

Given this background, the preponderance of efficacy studies in TJQ research is understandable. If efficacy cannot be demonstrated it makes little sense to try to implement a program outside the research setting (i.e., determine effectiveness) and less sense to consider any potential cost-benefits issues as there are no discernable benefits. Alternatively, the ideal process should begin with well-designed RCT that drive the development of programs that can be implemented in community settings and that maximize health improvement per dollar spent compared to existing interventions.^9,10 Unfortunately, this logical sequence has been stalled at the first step by the questionable quality of many RCTs and the paucity of coordinated research programs on Tai Ji Quan that go beyond one-time, isolated efficacy studies. The problem is highlighted by the relative numbers of different types of reports in the medical and healthcare literature. Table 1 summarizes the results of a search on PubMed/Medline, the largest medical literature resource in the world, conducted in September 2013 using all variations of the term Tai Ji Quan. A total of 710 articles were returned, of which only nine were published prior to 1990 (including two RCTs). Given that Tai Ji Quan was only introduced into the American mainstream in the 1970s, the lack of scientific interest before 1990 in what may have been considered, at best, a type of complementary and alternative medicine or, at worst, a fringe activity or quackery is not surprising. However, Tai Ji Quan continued to spread to the point that according to a National Health Interview Survey report published in 2007, almost 2.5 million Americans indicated they had practiced Ti Ji Quan in the previous year.¹¹ Paralleling participant growth was interest from the scientific and medical communities, spurred by the pioneering work of Wolf,^12–14 as seen in the number of Tai Ji Quan-related publications since 1990, including 173 RCTs and 162 reviews.

Table 1.

Number and types of articles on Tai Ji Quan indexed in Medline/PubMed

Era	RCT	Reviews	Dissemination studies	Cost-effectiveness	Other articles	Total
< 1990	2	0	0	0	7	9
1990–2003	25	24	0	0	46	105
2004–2008	76	49	1	0	108	234
2009–2013	72	89	3	3	195	362

However, it is instructive to note the significant increase in the number of review papers, especially over the past 5 years, compared to the growth of RCT. Systematic and critical reviews, often involving meta-analyses, are designed to summarize the current state of knowledge in a field and to impose some order on disparate findings. It is clear from the conclusions of the most recent reviews that the field of Tai Ji Quan research is very active but in disarray.

For example, the 27 reviews involving Tai Ji Quan (either as the primary intervention for a particular condition or as one of several interventions investigated to determine their effect on a particular condition) published between January 2012 and October 2013 covered 15 different conditions, including fall prevention,^15–18 rheumatology,^18–23 Parkinson’s disease,^24–26 fibromyalgia,^27,28 psychological conditions,^29–32 cardiovascular conditions,^33–36 peripheral neuropathy³⁷, and chronic pain.³⁸ Despite the range of conditions being investigated, the consensus of the reviews’ authors is that the research literature, including that involving RCT, leaves much to be desired. Too few studies involving specific conditions, small sample sizes, poor protocols, inadequate controls, inconsistent intervention characteristics (e.g., dosage, format), and widely varying project lengths and outcome variables (even for the same condition) were determined to undermine the value of the findings from these studies and/or make it difficult to discern a clear relationship between Tai Ji Quan and conditions of interest.

Not surprisingly, the most consistent recommendation from these reviews was for more, better designed and conducted RCT to avoid the problems that detract from the credibility of the results. There was no recommendation to increase research on the effectiveness of Tai Ji Quan. Given that evidence of efficacy is the foundation on which any subsequent work must be based, this position is quite logical. If the intervention cannot demonstrate an effect under the “ideal” conditions involved in RCT, there is little to argue for implementing it in an environment such as a community setting that is less conducive to its effects becoming evident.

3. The importance of a co-ordinated approach to research

However, this appeal for better efficacy evidence by increasing the number of RCT may be misplaced. If the majority of RCT, even if well conducted, are not part of a coordinated research program then the literature will remain a collection of disparate results that makes clearly identifying the utility of Tai Ji Quan as a prevention program or health-promoting activity problematic because of random and significant variations in intervention characteristics, protocols, sample populations and outcome variables. For example, of the 175 Tai Ji Quan RCTs indexed by Medline/PubMed, 112 involved one-time only research groups; 11 principal investigators (PIs) conducted two Tai Ji Quan RCT, seven groups had three RCTs and two PIs had four each. While a lack of cohesion in the study details of the one-time only groups is to be expected, it is instructive to note that of the multiple RCT researchers only Li and his colleagues^39–45 had specifically developed and systematically implemented a Tai Ji Quan protocol across all of their studies. Appropriately this line of research produced two of the four dissemination studies, with a pilot developmental phase⁴⁴ and a full implementation project,⁴⁵ in accordance with the framework discussed previously (three of the dissemination studies involved fall prevention^44–46 and one involved end-stage renal disease.⁴⁷) The remaining two dissemination studies,^46–47 as well as one large RCT investigating fall prevention that was implemented in community settings,⁴⁸ were not built on any specific precedent efficacy research and constituted a form of pragmatic or practical clinical trial.⁴ Nonetheless, two of the implementation projects for fall prevention^45,46 aptly used the RE-AIM model to measure the effectiveness of their intervention. The results, if applied appropriately, can provide a meaningful foundation for the feasibility of large-scale community implementation and future cost-effectiveness analysis.

With fall prevention being the most common application of Tai Ji Quan health-related research, the fact that the only cost-effectiveness studies related to Tai Ji Quan available to date^49–51 all focus on fall prevention is not unreasonable. However, all three involved statistical modeling that did not use data from specific RCT or implementation studies but rather secondary analyses based on systematic reviews and meta-analytic techniques. Although they are important first steps in building a critical mass of evidence that can be used by policy-makers to determine how to best promote population health, data from actual implementation studies are needed to ensure an accurate understanding of Tai Ji Quan fall prevention cost-benefits for various programs. Additionally, as noted by Frick and colleagues,⁵¹ not only does the cost-effectiveness of individual fall prevention programs need to be established but the relative cost-effectiveness of different programs is critical to identifying best practices and ensuring integrated healthcare systems allocate resources in the most fiscally prudent way. For example, of the three Tai Ji Quan programs^14,44,48 recommended by the U.S. Centers for Disease Control and Prevention (CDC) as fall prevention interventions,⁵² only one⁴⁴ has been funded by the CDC and specifically translated into a community-based program, formally tested for its effectiveness, and implemented in multiple states across the country.⁹ Having a program like this, with proven efficacy, translated into a format that meets the recommendations to be a covered service under multiple sections of the Affordable Care Act^10,53 (the U.S. government mandate that requires both government and private insurers to provide coverage for prevention services without co-pays or cost-sharing) opens a significant door to broad dissemination. However, without additional programs against which to measure the real-world impact of this one program the potential to identify the Tai Ji Quan fall prevention framework that will have the greatest influence on the health of the population will be unrealized.

4. Conclusion

There is little doubt that Tai Ji Quan has gained a substantial following within the healthcare community as researchers and clinicians look for new and more powerful interventions to prevent, treat or cure illness and injuries associated with modern life. The depth of its impact in the consciousness of medical researchers may be seen by the fact that in reviews for two disparate conditions (phantom limb pain⁵⁴ and stress incontinence in women⁵⁵), the authors specifically comment on the absence of Tai Ji Quan studies related to these conditions. However, it is more clearly evident in the increasing number of studies using Tai Ji Quan to address a growing array of conditions ranging from age-related decline in physical function to psychological well-being and metabolic pathologies. Unfortunately, very little of the work done to date is part of a deliberate and well-planned strategy to systematically address efficacy, effectiveness and cost-effectiveness issues related to Tai Ji Quan as an intervention to prevent health declines or to enhance physical and psychological function. Despite the high quality of some individual research projects, methodological weaknesses in Tai Ji Quan efficacy studies and the overall disjointed and inconsistent research approach to examining Tai Ji Quan does not provide sufficient grounds for governmental or non-governmental organizations to underwrite comprehensive implementation of Tai Ji Quan for the good of the public’s health.^10,53 Concerted, systematic and coordinated long-term research programs by individual researchers or collaborative groups are critical if the full potential of Tai Ji Quan to enhance health-related quality of life is to be realized on a broad scale.