Real world evidence: experience and lessons from China

Abstract

Xin Sun and colleagues discuss the development of real world evidence in Chinese healthcare and propose strategies to improve its quality and usefulness

Worldwide, real world evidence has become a topic of broad interest in healthcare. Its definition, however, has not achieved wide consensus.¹ As an umbrella term, real world evidence comes from a spectrum of studies that apply various epidemiological methods to data collected from real world settings.² Real world data can be derived from a wide range of sources, such as routine healthcare (eg, electronic medical records), traditional epidemiological studies (eg, classical cohort studies), surveillance (eg, spontaneous adverse drug events monitoring), administrative databases (eg, death registers, medical claims), or personal devices (eg, regular blood pressure measured with mobile devices). Study designs are generally classified into three categories: pragmatic clinical trials, which may or may not be randomised; observational studies involving prospective collection of data; and observational studies using retrospective administrative databases (box 1).

Box 1. Classification and data sources of real world studies.

Pragmatic clinical trials

• Varying, may include multiple sources as below

Observational studies involving prospective data collection

• Disease registries

• Patient surveys

• Traditional cohort studies

• Data collected from mobile devices

Observational studies using existing administrative data

• Electronic medical records

• Medical claims data

• Birth or death registries

• Surveillance databases

• Spontaneous adverse drug events databases

Real world evidence can be used for developing medical products and informing healthcare practice and policy making. Examples of its uses include support for identification of unmet medical needs,³ design of registered clinical trials,⁴ post-approval drug safety assessment and pharmacovigilance,⁵ payment and coverage decisions,⁶ healthcare quality improvement,⁷ new indications of medical products,⁸ assessment of healthcare technologies,⁹ and clinical practice guideline development.¹⁰ In addition, the abundance and diversity of data allows exploration of clinical research questions other than healthcare interventions, such as disease burdens, prognoses, and clinical predictions.

A common misunderstanding is that traditional randomised controlled trials do not reflect the real world setting, and that all observational studies are real world.¹ In fact, randomised controlled trials may include components of real world settings (eg, broad eligibility criteria and pragmatic trials)¹¹, and real world studies may have elements that are not part of regular care (eg, intensified follow-up). Instead of a dichotomy there is a continuum in the study features of traditional randomised controlled trials and real world studies, with external validity increasing as more real world features are included in the design.

Some also argue that observational studies have advantages over randomised controlled trials in assessing the “real world” treatment effects of healthcare interventions.¹² However, randomised controlled trials that have practical features continue to provide the best available evidence in the assessment of benefits and common harms, given the importance of randomisation in balancing known and unknown prognostic factors. Observational studies can provide important evidence when we want to assess harms of very uncommon events, answer harm questions that involve interactions between multiple interventions, or generate hypotheses for further testing.

Chinese experience with real world evidence

In China, the concept of real world evidence arose from awareness of the limitations of traditional clinical trials, and the need for additional evidence to inform healthcare practice and policy decisions.¹³ As early as 2002, the Chinese Ministry of Labour and Social Security hosted an academic forum on the use of insurance claims data for drug formulary decision and pharmacoeconomic evaluation.¹⁴

The term “real world evidence” was not explicitly used until 2010, when researchers from traditional Chinese medicine carried out a real world study to evaluate traditional Chinese medicine interventions, mainly to accommodate the complexities of such interventions.¹³ Since then, the Chinese research community has started to embrace the concept, and has adopted the same definition as the international research community, although some terms, such as outcomes research and comparative effectiveness research, which share overlapping concepts, have also sometimes been used.^{15 16} For instance, the Chinese Medical Doctor Association began to promote outcomes research in 2012, and the use of observational studies to assess the effects of healthcare interventions. In 2016, the Chinese Evidence-based Medicine Centre organised a national workshop on real world evidence to explain the concept and introduce methods of real world studies to the Chinese audience. In 2017, the centre hosted a national academic forum on the use of real world evidence in healthcare decision making.

In fact, efforts to generate real world evidence started far earlier than the official introduction of the concept. The first registry in China was the Shanghai Tumour Registry, which collected data from children aged under 15 years between 1973 and 1977.¹⁷ Since then, more disease registries have been established. Our search of PubMed found that 338 articles on registries have been published since 1983. Cancer and cerebrovascular and cardiovascular diseases are the most common topics, accounting for 75% of the publications. More registries were started after the introduction of the concept of real world evidence (fig 1, appendix 1). Some of these registries were developed at the national level. Both public and private sponsors have supported the studies, and some of the studies served simply to monitor disease.

Fig 1.

Number of published peer reviewed articles based on registries and retrospective databases in China. More details are given in appendices 1-3 on bmj.com

The first use of retrospective administrative data to produce real world evidence was in 1993, when a case-control study used medical records data of 2691 participants to examine the association between cholecystectomy and the risk of colorectal cancer.¹⁸ More recently, the interest in using electronic medical records data has increased, largely because of the growth of information technologies, and adoption of electronic medical records systems in large hospitals across the country since 2005.¹⁹

The earliest study that used electronic medical records was reported in 2006; it investigated phenotypes and clinical patterns of Crohn’s disease.²⁰ In certain disease areas (eg, cancer), disease specific databases were also created that used a standardised process and data structure to collect data from electronic medical records across hospitals, many of which were funded by private organisations. Such databases captured data during patient hospitalisation, and patients may not be followed up if transferring to other healthcare institutions. In some regions (eg, Xiamen city), databases of electronic health records are supported and guided by local health authorities. These databases link information from community hospitals and larger hospitals, health insurance databases, and birth and death registries, making longitudinal follow-up of patients possible. In China, 523 peer reviewed articles have been cited in PubMed since 1993 using administrative databases (fig 2, appendix 2). Again, cancer and cardiovascular and cerebrovascular diseases are the most common topics of studies using retrospective databases.

In the past decade, efforts have focused on observational studies. Few pragmatic clinical trials have been conducted. Our search of both PubMed and the Chinese Biomedical Literature Database found only 16 trials explicitly labelled as pragmatic (appendix 3), all of which were published in the past decade, and nine were on traditional Chinese medicine treatments. Most trials had moderate sample sizes; only three had 700 or more participants (median 280, range 50 to 28 130). The trial with the longest follow-up investigated the feasibility and effectiveness of a comprehensive intervention package for cardiovascular disease care for three years.²¹ These trials were mainly funded by government grants. In addition, a few published review articles discussed the concept and methods of such designs.^{22 23}

A number of challenges remain in the production of real world evidence. Our experience and observations have shown several problems in the use of retrospective data. The most important one is data accessibility and sharing. Concerns exist about confidentiality and the extent to which researchers may use the data (ie, data ownership). Explicit policies for data access, consent, and privacy are lacking.²⁴

Another problem is the accuracy, completeness, and consistency of data. Established and systematic technical guidance for data collection or quality control is not available. Ethical issues are a further challenge. In many cases, the ethical approval process for retrospective database studies has proved lengthy and difficult, and ethics review boards often question the potential benefits of such studies. Lack of rigorous methods when using these retrospective databases is common—for example, failure to set eligibility criteria or include a control group. Pragmatic clinical trials and registries are less common in China, mainly because of the lack of funding, especially from public agencies, to support prospective studies. When conducting such studies, maintaining adequate follow-up in the real world setting is an important concern, mainly because patients can freely choose healthcare institutions and doctors without referral, and healthcare data are not connected across healthcare institutions.

A difficulty applicable to all types of real world evidence is the wide gap between the limited research capacity and the great demand. Researchers well trained in the methods are lacking, especially leading scientists who have an in-depth understanding of evolving frameworks and methodologies, research data, and healthcare settings.

Awareness and use of real world evidence among doctors, regulators, health insurers, and other health policy makers have increased in the past few years. The main uses are for post-approval monitoring and assessment of medical products, and payment and coverage decisions. However, opinions on the value of real world evidence differ between policy makers. Misunderstandings about real world evidence, similar to those that have occurred in the international community, are common.^{25 26} Tools to support the appropriate interpretation of real world evidence and to inform decisions are urgently needed.

Although the production of real world evidence is at an early stage, a few important governmental initiatives have begun in China. One such programme is the China Hospital Pharmacovigilance System, which was started in 2015 by the National Centre for Adverse Drug Reactions Monitoring at the Chinese State Food and Drug Administration. This nationwide programme was designed to identify safety signals and determine the association between drug exposure and adverse events, particularly those that occur during hospital admission. This system complements the current passive drug safety monitoring system. The system includes electronic medical records data from 300 hospitals and healthcare claims data and will be used to generate real world drug safety evidence through the use of established drug risk analytics tools, which will enable policy makers to make more informed decisions.

The China Health Policy Evaluation and Technology Assessment Network is another important government initiative through which routine or rapid assessment of healthcare technologies is commissioned by the Chinese health authorities. To assess innovative or costly healthcare technologies, retrospective data and public surveillance data are used to generate evidence on disease burdens, clinical benefits, and harms. Retrospective data are often preferred to inform rapid policy decisions when clinical evidence is lacking or locally relevant evidence is unavailable.

In the future, we expect that the need for real world evidence will continue to increase. One particular development is the use of real world evidence from the Chinese population to develop evidence based guidelines. This work is ongoing in cardiovascular diseases:²⁷ the China Patient-centred Evaluative Assessment of Cardiac Events Prospective Study is being conducted to inform the prospective Chinese guideline on percutaneous coronary interventions.²⁷

Improving real world evidence in China

We propose the following strategies to improve the quality and usefulness of real world evidence in China. Firstly, government bodies such as the National Health and Family Planning Commission of China and the Chinese State Food and Drug Administration should reinforce regulations to deal with concerns about data sources, such as confidentiality, privacy protection, data sharing, and ethical approval. The establishment of an explicit policy framework and regulatory documents will greatly improve the development of China’s real world evidence initiatives.

Secondly, a research community to foster collaboration across institutions and stakeholders must be developed urgently. This is needed to cultivate an environment of mutual understanding and cooperation to develop shared goals and agendas. We call for public and private collaborations such as the Observational Medical Outcomes Partnership in the US. Recently, a national real world evidence consortium (China REal world evidence ALliance, ChinaREAL) has been established, which brings together researchers, doctors, journal editors, and policy makers across the country. The main aims include support for the production of high quality real world evidence and translation of evidence into practice and policy decisions in the Chinese setting. One of the efforts is to develop consistent and coherent technical guidance to improve data quality, and standardise diagnosis, procedures, and drug coding systems.

Thirdly, an important step is for academic institutions to develop rigorous academic postgraduate degree and fellowship programmes to foster the next generation of researchers and leaders. Educational opportunities should also be accessible to other stakeholders, including doctors, policy makers, and industry, to improve understanding and use of real world evidence.

Lastly, we call for public and private research foundations to support research that can improve data infrastructure and develop new technologies and methods. We believe such investments will greatly improve the production and use of real world evidence in China. Although the movement started later in China than in developed countries, we are confident that, with endeavours from all sides, real world evidence will further boost evidence based healthcare practice and policy decisions.

Key messages.

• Real world evidence has gained wide attention in China in the past few years

• Disease registries and retrospective databases are the two main types of real world studies in the China; limited resources are available for pragmatic clinical trials

• Use of real world evidence for healthcare practice and policy decisions is limited at present, although there are a few important governmental initiatives

• To advance the real world evidence movement, China must develop an explicit policy framework; reinforce consistent and coherent technical standards; strengthen research capacity, and foster collaborative efforts across institutions and stakeholders

Web Extra.

Extra material supplied by the author

Appendix 1:

Appendix 2:

Appendix 3: