Innovative research methods for studying treatments for rare diseases: methodological review

Abstract

Objective To examine methods for generating evidence on health outcomes in patients with rare diseases.

Design Methodological review of existing literature.

Setting PubMed, Embase, and Academic Search Premier searched for articles describing innovative approaches to randomized trial design and analysis methods and methods for conducting observational research in patients with rare diseases.

Main outcome measures We assessed information related to the proposed methods, the specific rare disease being studied, and outcomes from the application of the methods. We summarize methods with respect to their advantages in studying health outcomes in rare diseases and provide examples of their application.

Results We identified 46 articles that proposed or described methods for studying patient health outcomes in rare diseases. Articles covered a wide range of rare diseases and most (72%) were published in 2008 or later. We identified 16 research strategies for studying rare disease. Innovative clinical trial methods minimize sample size requirements (n=4) and maximize the proportion of patients who receive active treatment (n=2), strategies crucial to studying small populations of patients with limited treatment choices. No studies describing unique methods for conducting observational studies in patients with rare diseases were identified.

Conclusions Though numerous studies apply unique clinical trial designs and considerations to assess patient health outcomes in rare diseases, less attention has been paid to innovative methods for studying rare diseases using observational data.

Introduction

Though an individual rare disease is by definition uncommon, according to the statutory definitions set in the United States (prevalence <200 000 people each year; equating to a prevalence of approximately <64 per 100 000 people) and European Union (<50 per 100 000 people), more than 6800 different conditions qualify as rare diseases and 6-8% of the population is affected.^{1 2 3} This translates to about 60 million people in the United States and EU alone. Rare diseases comprise a heterogeneous set of conditions that afflict various organ systems, have wide ranging prognoses, and even vary along a gradient of rareness.

Many barriers exist to advancing knowledge of and treatment options for rare diseases.⁴ The small patient populations can dampen commercial interest in development of treatments. Yet even for those rare conditions where funding is plentiful and manufacturers of therapeutics are engaged, methodological and data constraints limit the ability to generate evidence on patient health outcomes. The most obvious challenge to conducting rigorous research is the small number of eligible participants for a given study. In addition, geographic dispersion of patients, lack of knowledge about the clinical course of disease, and lack of appropriate comparator treatments further hinder the generation of evidence.⁵ As a result relatively little is known about the clinical course of many rare diseases and few treatment options exist.

However, there may be pathways for collectively advancing the study of rare diseases. Although rare diseases may present unique clinical problems, the methodological challenges to studying health outcomes are often communal. In recent years, innovative epidemiological and clinical trial methods have been developed that offer promise for promoting more efficient and effective research. Because rare diseases are so clinically dissimilar, clinicians, scientists, and other stakeholders working in one medical specialty may not be familiar with methods being applied in other disciplines. Thus, we conducted a methodological review to catalogue and describe innovative approaches to studying health outcomes in patients with rare diseases. Our goal was to identify innovative approaches to research that have been, or can be, applied to overcome the methodological challenges inherent in the study of rare diseases.

Methods

Search strategy

We searched PubMed, Embase, and Academic Search Premier from their commencement through December 2012 for English language articles that included the following terms: “rare diseases”, “orphan drug”, “comparative effectiveness”, “evidence-based medicine”, “health technology assessment”, “outcome assessment”, “methods”, “epidemiology”, and “registries”. The supplementary file provides details of the search strategies.

We also conducted ad hoc searches of the three reference databases as well as general internet searches in Google using search terms specific to individual rare diseases (for example, progeria) and names of methods (for example, response adaptive randomization) identified in the database searches. Finally, we mined the reference lists of qualifying articles to supplement our search.

Article selection

We combined the results of each search strategy and removed duplicates. One author (LT) screened titles and abstracts to exclude those articles that were clearly not relevant. Another author (JJG) conducted a second stage screening of those articles that passed the title and abstract screens. We included articles covering randomized trial design and analysis methods and methods for conducting observational research. Articles relating to other facets of rare diseases and their treatments (for example, those related to clinical practice or policy) were excluded.

Data extraction

We extracted descriptive information about each article, including information on the authors, title, and publication. If the article focused on a specific rare disease, we extracted the name of the condition. We then summarized the unique methods proposed or used in each article to study patient health outcomes in rare diseases. If the article presented an empirical application of an innovative method, we extracted the study’s objective, the number of participants, the description of the method, and the description of the outcome.

For the qualitative synthesis, we classified novel research methods relating to the study of rare diseases into two broad categories: advances in clinical trial design for patients with rare diseases, and methods for observational studies of health outcomes in rare diseases. In each category we highlighted the most innovative research methodologies, and, where possible, provided examples of their applications.

Results

We identified 5346 records through our search process. After removing duplicates and performing an initial title screening to exclude those that were clearly irrelevant to our review, we identified 442 potentially relevant articles and, after the subsequent two stage screening process, we obtained full text versions of 55 articles. Of these, 46 proposed or employed methods for studying patient health outcomes in rare diseases (figure). Articles covered a wide range of rare diseases, from amyotrophic lateral sclerosis to multiple myeloma to uveal melanoma. Of the 13 articles that involved an empirical application, the number of participants ranged from 23 to 4980. Most of the articles (33/46, 70%) were published between 2008 and 2012. Table 1 presents a summary of the research methods we identified and their advantages in the setting of research into rare diseases.

PRISMA flow diagram

Table 1.

 Summary of research strategies for studying rare diseases and their advantages

Strategy	Description	Address small Nos of patients and outcomes					Promote recruitment and retention
		Minimize No of required participants	Make use of conventionally underpowered studies	Maximize outcome information among participants	Facilitate confounding adjustment with sparse data		Maximize No of participants who receive treatment	Expand access to studies and participants
Study design options:
Factorial designs	Two or more treatments can be simultaneously compared in a single group of study participants	X	—	—	—		—	—
Response-adaptive randomization	Increases participants’ probability of being exposed to more effective treatment and reduces total sample size	X	—	—	—		X	—
Sequential designs	Can identify differences in treatments before the end of planned enrollment	X	—	—	—		—	—
Crossover, n-of-1, alternating designs	Using patients as their own controls both guarantees treatment and increases statistical efficiency	X	—	—	X		X	—
Use continuous outcome	As compared with binary outcome, continuous measures increase statistical efficiency	—	—	X	—		—	—
Use surrogate outcome	Can be measured before patients are lost to follow-up for hard clinical endpoints	—	—	X	—		—	—
Use composite outcome	Combining multiple outcomes into a single endpoint increases number of events	—	—	X	—		—	—
Use repeated measure outcome	Allowing patients to contribute more than one event can increase total number of events	—	—	X	—		—	—
Increase duration of follow-up	Longer studies permit capture of more outcome events among participants	—	—	X	—		—	—
Case-control sampling	Reduces study size by sampling a portion of patients who do not experience an outcome	—	—	—	—		—	—
Recruitment and enrollment strategies:
Focus on high risk patients	Outcomes are more likely to occur in high risk patients	—	—	X	—		—	—
Trial networks and distributed data networks	Infrastructure for multicenter studies can permit recruitment of larger and geographically diverse groups of patients	—	—	—	—		—	X
Statistical options:
Increase α	Small patient populations may preclude sample sizes with sufficient power to detect effects using conventional thresholds	—	X	—	—		—	—
Propensity scores	Can permit adjustment for more potential confounders than outcome regression modeling	—	—	—	X		—	—
Incorporation into larger evidence context:
Conduct study as part of prospectively planned meta-analysis	Individual small studies may not provide definitive evidence about a question, but can be combined to yield sufficient power	—	X	—	—		—	—
Incorporate study into bayesian framework	Small studies can help increase the certainty around a clinical question	—	X	—	—		—	—

Clinical trial designs used in patients with rare diseases

Conventional parallel group randomized controlled trials, which randomly allocate participants to one of two or more treatment groups, are not always feasible in rare conditions.⁶ We found 19 articles proposing or employing novel clinical trial methods for studying therapeutic interventions in rare diseases. These approaches were classified into two groups: designs that minimize the total number of participants, and designs that maximize the number of on-treatment participants.

Minimizing trial sample size

Investigators studying rare diseases have tried to deal with the small pool of potential trial participants. Some proposed or made adjustments to traditional randomized trials. For example, when considering the treatment period, choosing a longer trial duration can reduce sample size requirements by capturing more events among the trial participants.⁷ Focusing on high risk patients can reduce sample size and study duration,⁸ and using genetic testing can reduce variability between individuals and allow inclusion of patients before they experience symptoms.⁹ Finally, some investigators have sought to reduce sample size by tackling multiple treatment options in a factorial study, in which two (or more) treatment comparisons are carried out simultaneously.¹⁰ Factorial designs provide answers to multiple questions within the same study population. This reduces the total number of patients required to answer all of the questions of interest but does not reduce the number of patients required to answer each individual question.

Another way to reduce sample size requirements in rare disease studies is through selection of the outcome measure using a continuous outcome variable, a surrogate marker, a composite endpoint, or repeated measure outcome. Identifying a continuous outcome variable, rather than a binary measure, can enhance statistical efficiency.⁷ For example, percentage reduction in a continuous measurement imparts greater statistical power in an analysis than an outcome measurement based on the proportion of patients who attain some threshold in reduction of the measure, provided that the continuous outcome variable has a small variance. Surrogate endpoints, such as biomarkers, that predict whether patients will experience clinical outcomes of interest may also be useful, but validating biomarkers as good surrogates of the clinical outcome of interest can be difficult. They can further enhance statistical power since a potentially small number of patients in a study experience the hard endpoint of interest, whereas nearly all patients have measured values of the biomarker.^{7 11} When hard clinical endpoints are preferred, combining multiple outcomes into a single composite outcome measure can increase the number of observed events and thus the statistical power.¹² Repeated outcome measurements permit patients to contribute more than one outcome event or measurement, which also increases study power, allowing more precise estimation of variance between patients while permitting estimation of the variance within patients.¹²

A third approach to the sample size problem is to build networks to allow broader access to trials. Development of clinical trial networks for rare diseases can facilitate the conduct of multicenter and even multinational randomized trials.¹³ Trial networks facilitate the recruitment of larger and more geographically diverse patient populations than may be permitted by single center studies.¹⁴ The existence of such networks can also decrease the time required to complete a trial.¹⁴ For example, Goss and colleagues provide a comprehensive overview of clinical trial networks for rare diseases in the context of the Cystic Fibrosis Therapeutics Development Network.¹⁴

Finally, we found investigators who proposed and used novel trial design strategies to account for small pools of patients with rare diseases. Trials featuring an “adaptive design” allow modification of some aspects of the trial based on prospectively planned interim data analyses. The two basic types of adaptive designs are adaptive randomization and sequential trials. In trials using adaptive randomization, the probability of being randomized to an intervention changes during the enrollment period. The goal of adaptive randomization may be to minimize imbalance in baseline covariates among treatment groups (covariate-adaptive randomization) or to increase the proportion of patients assigned to the seemingly more effective treatment while reducing overall trial enrollment (response-adaptive randomization). By contrast, in sequential trials, data are analyzed intermittently to guide decisions on termination when safety concerns, futility, efficacy, or a combination of these factors is demonstrated. Trials that are stopped early because of important interim results require fewer patients. However, to control for multiple testing, trials that are not stopped early generally require larger sample sizes compared with similarly designed non-sequential trials. Chow and colleagues, Gupta and colleagues, and Cornu and colleagues have all summarized adaptive and sequential design methods in clinical trials and provide examples of applications to rare diseases.^{15 16 17} Gupta and colleagues also provide a framework for selecting among these approaches for studies of rare diseases.

Many variants of adaptive randomization and sequential designs are applicable to studying rare diseases because they can reduce the sample size required for conventional trials. In addition, certain adaptive designs can also increase participants’ probability of receiving the most effective treatment, which can encourage enrollment in a trial.^{7 11 12 15 16 18} The decision about whether to use an adaptive design involves considering whether a set sample size can be reasonably recruited, the number of therapeutic options to be compared, and whether preliminary data suggest one treatment is superior.¹⁶ Cornu and colleagues proposed an algorithm for choosing an experimental design for small randomized clinical trials that also involves judging whether the outcome is reversible, whether the treatment response is likely to be rapid, and whether investigators seek to minimize the time participants are receiving placebo.¹⁷

Even if investigators use one of these innovative designs or adaptations of traditional trials in studying a rare disease treatment, individual trials of patient health outcomes may not be capable of attaining sufficient power to reject the null hypothesis using a conventional frequentist threshold (α=0.05). One solution is to increase α, as was done in the alternating design trial of itraconazole by Gallin and colleagues.¹⁹ Another solution is to conduct the underpowered study and incorporate the results into a prospectively planned meta-analysis.^{18 20 21} A third option is to incorporate the results into a bayesian framework. Lilford and colleagues recommend the third approach for trials in rare diseases in which the individual trials are unlikely to result in a definitive answer but each can change the level of certainty around the clinical question.²² The bayesian approach uses all available data—from the trial and other sources—to calculate probabilities that a particular treatment is effective. These probabilities can then be applied to clinical practice. Bayesian methods can also be useful in individual studies (randomized controlled trials and observational) of health outcomes in rare diseases.¹¹

Tan and colleagues described a bayesian approach to combining previous data with data from a new randomized controlled trial by creating scores that are then used to weight the pieces of evidence according to their pertinence, validity, and precision.²³ The validity scores enable investigators to down-weight evidence based on studies with flaws or other concerns, such as confounding in non-randomized trials. Pertinence scores are based on how closely the information from each source relates to the information to be gained in the trial. In theory, pertinence scores could also be based on the degree to which the evidence streams are relevant to patients’ decision making and could therefore support patient centered decision making. The authors make the case that such a bayesian approach can increase the robustness of information from small trials and can be used to help design and provide justification for such trials. However, bayesian approaches require appropriate specification of a prior distribution, which may be subjective or based on limited information.

Maximizing on-treatment participants

Trials that guarantee participants receive an intervention can enhance recruitment for patients with rare diseases who have limited treatment options. Some of these designs can also reduce recruitment requirements compared with alternative conventional parallel group randomized controlled trials. For example, crossover trials involve randomizing patients to treatment at one time (or several times) and to no treatment (or treatment with a comparator) at another time (or other times).^{10 12 13 16 23 24} In addition to guaranteeing treatment, crossover designs are more statistically efficient than their parallel group randomized controlled trial counterparts. Crossover trials are particularly well suited to studying treatments for chronic conditions in which the treatments provide immediate relief of symptoms. But crossover trials generally cannot be used to study treatments that have curative effects or conditions that are rapidly changing. Many rare diseases are chronic conditions that progress over time. Changes in the disease over time that are unrelated to the treatment under study can cause bias in crossover trials. Crossover trials also require a transient treatment effect to minimize carryover effects into the subsequent treatment periods.

In the most basic crossover design involving two treatments, patients are randomly assigned to one treatment, followed by a washout period, and then receive a different treatment. Other patients are randomized to the reverse ordering. More complex crossover studies include so called alternating designs, in which patients are randomly assigned to each treatment at multiple time points.²⁵ Gallin and colleagues conducted a randomized crossover trial to examine itraconazole for fungal infections in patients with chronic granulomatous disease.¹⁹ Given the rarity of this disease, it took 10 years to enroll only 39 patients. The investigators randomly assigned patients to receive itraconazole or placebo for one year and then to alternate annually between itraconazole and placebo. While this approach could not provide much information on the long term safety of itraconazole treatment, the multiple observations that each patient contributed made it possible to achieve sufficient statistical power (defined as a two sided type I error probability of 0.10) with only 39 participants.²⁵

An n-of-1 study is a special type of crossover design in which the trial comprises one patient.^{10 11 12 13 16 23 24} Within clinical practice settings, healthcare providers administer a treatment and a control at randomly determined times and observe subsequent outcomes. These trials require the same general assumptions as crossover trials. While statistical inference cannot be made based on a single n-of-1 trial, results of multiple such studies can be aggregated in case series or even meta-analyzed quantitatively.²⁶ Investigators in the Netherlands are developing an n-of-1 trial service integrated in the Dutch healthcare system to generate evidence on the efficacy of treatments for rare neuromuscular diseases.²⁷ It will involve testing treatments that are available on the market but not necessarily approved for the neuromuscular indications. The project will create protocols for each n-of-1 trial and will collect the data in an electronic registry system. Less common variants of crossover designs include the Latin square design, the stepped wedge design, and the randomized withdrawal design.¹⁷ Cornu and colleagues and Gupta and colleagues provide more detailed descriptions of the application of these clinical trial designs to studying treatments in rare diseases.^{16 17}

Methods for observational studies of health outcomes in rare diseases

In addition to the often small samples, studies using observational data to assess patient health outcomes in rare diseases face important challenges. For example, there is often no appropriate comparison group against which to compare outcome frequencies in patients with rare diseases and even when there is, controlling for confounding can be difficult because the risk factors of those outcomes are usually not well understood. Table 2 summarizes methods that have been proposed or used to analyze health outcomes in patients with rare disease in observational data. These methods can be generally classified into four categories: advanced methods to tackle confounding, self controlled observational study designs, approaches for case-control studies, and prospective inception cohorts.

Table 2.

 Selected observational studies of health outcomes in patients with rare diseases

Lead author, reference No	Rare condition	Study objective	No of patients	Outcome	Methodological approach
Nakamura42	Amyotrophic lateral sclerosis (ALS)	Compare clinicians’ and patients’ perspectives on symptomatic treatment	4375	Concordance between patient and clinician perceptions	Used patient reported perceptions of drug effectiveness for ALS symptoms and compared this with survey based physician perceptions
Wicks43	ALS	Investigate whether off-label treatment with lithium slows disease progression of disease	447	Change in the revised ALS functional rating scale	Matched lithium users to non-users and followed for improvements in disease progression
Barash44	Creutzfeldt-Jakob disease	Determine incidence of Creutzfeldt-Jakob disease in Veterans Health Administration and describe clinical features	115	Incidence and clinical features of Creutzfeldt-Jakob disease	Used linked electronic information system to study outcomes in patients with rare diseases
Schick45	Endometrial stromal cancers	Assess rates, time, and sites of recurrence for patients with endometrial stromal sarcomas	66	Overall survival, disease-free survival, locoregional control	Used descriptive statistics, Kaplan-Meier plot, and multivariable time to event model
Cole28	Gaucher disease	Compare odds of splenectomy in patients with and without avascular necrosis	4980	Splenectomy	Proposed risk-set matching to reduce selection bias in case-control studies
Pugnet46	Giant cell arteritis	Examine incidence and predictors of corticosteroid withdrawal in giant cell arteritis	103	Corticosteroid withdrawal	Proposed use of claims data for observational studies in rare diseases
McCann47	Juvenile dermatomyositis	Identify epidemiological, clinical, and laboratory characteristics of juvenile dermatomyositis	122	Patient characteristics, clinical features over time, and drug use over time	Prospectively measured disease activity and patient reported health outcomes
Ozsahin48	Olfactory neuroblastoma	Assess outcomes in patients with olfactory neuroblastoma	77	Overall survival, disease-free survival, locoregional and local control	Used descriptive statistics, Kaplan-Meier plot, and multivariable time to event model
Fasnacht49	Pulmonary arterial hypertension	Describe characteristics of patients in Swiss pulmonary arterial hypertension registry	23	Patient characteristics, clinical features over time, and drug use over time	Described clinical course with respect to changes in clinical features and treatments
Sun50	Subependymal giant cell astrocytomas	Compare prevalence of clinical conditions related to disease of interest before and after surgery among patients with tuberous sclerosis complex	47	Change in prevalence of 20 different conditions	Used pre-observational/post-observational design
Sun51	Subependymal giant cell astrocytomas	Examine outcomes after resection of disease of interest among patients with tuberous sclerosis complex	47	Diagnosis, repeated surgeries, surgical complications	Described outcomes in patients after surgery

Advanced methods to deal with confounding

Some authors have suggested the use of certain advanced methods to tackle confounding in studies of rare disease health outcomes, such as propensity scores.^{28 29} When comparing patients being treated for a particular rare disease to patients with the same disease but who are not being treated, confounding will occur if the determinants of one patient’s receipt of treatment over another are also risk factors for the outcome of interest. Often, many such confounders can be present. Propensity scores reduce the dimensionality of confounding in observational studies by summarizing all potential confounders into a single scalar score.³⁰ This tool is particularly useful in studies in which there are few outcome events relative to the number of confounders, which is a defining characteristic of rare diseases.³¹ In a study of a dose-response effect of enzyme replacement therapy in patients with Gaucher disease type 1, Grabowski and colleagues created propensity scores to summarize multiple confounders and then used the scores to match patients who received different doses of enzyme therapy.³² Though propensity scores can facilitate adjustment for many potential confounders by modeling the exposure rather than the outcome, neither propensity scores nor traditional outcome regression modeling can overcome confounding due to unmeasured variables.

Self controlled observational study designs

Self controlled observational designs may be useful in the rare disease setting. These approaches are observational analogues to the randomized crossover trials described above in which patients act as their own controls. These studies can be indexed by outcome, such as in case-crossover designs,³³ in which the frequency of exposure is compared during different time points among those who develop the outcome. They can also be indexed by exposure, such as the self controlled case series,³⁴ in which the frequency of outcome is compared during different time points among those exposed to the intervention of interest. Notable for patients with rare diseases, these approaches are immune to confounding by factors that do not change over time because of the within person comparisons. Similar to randomized self controlled trial designs, self controlled observational methods enhance statistical power and therefore reduce sample size requirements. Self controlled observational methods are subject to the same limitations as randomized self controlled trial designs but can also be susceptible to time varying confounding, such as when worsening of disease, which may be a risk factor for the outcome of interest, may also prompt treatment.

Case-control designs

Several observational studies of rare diseases have used a case-control design, which is particularly useful in settings in which outcomes are rare and require primary data collection methods. Case-control studies involve sampling from an underlying cohort of patients rather than utilizing information on all cohort patients, which can be resource prohibitive. Schmidt-Pokrzywniak and colleagues conducted an institutional based case-control study to examine risk factors for uveal melanoma.³⁵ Rather than using a full cohort approach, the authors recruited cases from a referral center for eye tumors and sampled controls from among the cases’ siblings and from local ophthalmologists’ case loads. The case-control design yields an estimate of the same effect estimate as if the entire underlying cohort were used, but with slightly less precision given the sampling. In addition to reducing sample size requirements by identifying all cases and sampling controls, the case-control design allows investigators to easily examine multiple risk factors related to the outcome of interest. In other articles, Schmidt-Pokrzywniak and colleagues have examined the relations between uveal melanoma and mobile phone use, occupational cooking, and ultraviolet radiation.^{36 37 38}

Cole and colleagues conducted a case-control study using the International Collaborative Gaucher Group registry.²⁸ The authors compared the odds of splenectomy in patients with avascular necrosis (cases) with the odds in patients without avascular necrosis (controls). The authors used risk set matching, which can reduce bias in case-control studies relative to other sampling strategies. In risk set matching, controls are sampled from sets of patients at risk for the outcome at the time of the corresponding case event. These sets are usually defined by calendar time but can be defined by other variables as well, such as age and sex.

Prospective inception cohorts

A fourth group of studies employed prospective inception cohort designs, which are also sometimes referred to as “new user” designs when cohort inception is defined by the start of some medical treatment.^{39 40} Inception cohorts permit investigators to establish clear temporality among study variables (that is, baseline confounders, exposures, and outcomes) and capture outcome events that occur shortly after entry to the cohort. This approach is particularly important for outcomes related to medical interventions that may be immediately affected by those interventions. While inception cohorts increase validity of observational studies, they can be difficult to implement for rare diseases because they require restricting the already small patient population to those with an observable start of the exposure, risk factor, or disease of interest. Identifying patients at the onset of a rare disease can be challenging because there can be a long lag time associated with making accurate diagnoses for rare diseases. Thus, patients enrolling in registries and other data sources may have had the underlying condition and subsequent treatment for some time. In addition, identifying “new users” of medical treatments for rare diseases outside of clinical trials can be limited if a large proportion of patients with the disease participated in the trial and were exposed to the treatment. Bernard and colleagues described the design and implementation of institution based prospective inception cohort studies in pediatric thrombosis and stroke research.⁴¹

Discussion

In this review of methods that have been proposed for and used to study health outcomes in rare diseases, we identified a wide variety of non-traditional approaches. The majority of the identified articles were published in 2008 or later, highlighting the increasing interest in this area. Most articles also focused on innovations in methods for clinical trials intended to minimize the number of participants needed to meet the study goals or to maximize the proportion of participants who receive active treatment to encourage enrollment.

Implications for randomized trials

Advances in clinical trial design relevant to rare diseases are well developed, having been discussed in several technical articles and applied in many clinical scenarios. Cornu and colleagues provide examples of studies that have used each of 12 different randomized designs in the setting of rare diseases.¹⁷ They and Gupta and colleagues have also proposed frameworks to aid selection of randomized clinical trial methods for studying health outcomes in rare diseases.^{16 17} Both algorithms pose similar questions to address whether the assumptions of crossover and n-of-1 trials are likely to hold, such as whether the intervention of interest has only a short term effect on the outcome. Gupta and colleagues’ algorithm asks about whether sufficient numbers of patients are likely to be recruited for a given design and offers alternatives when this is not the case. Cornu and colleagues’ algorithm explicitly asks about whether objectives of the study include minimizing the time patients are receiving placebo or ensuring that patients receive active treatment by the end of the trial. Until a unified framework is developed, both algorithms can be used to help decide the most appropriate design to study health outcomes in patients with rare diseases.

Implications for observational studies

In addition to dealing with considerations about general design and analysis (for example, outcome selection, incorporation of evidence into larger context), our methodological review is the first to go beyond randomized trial methods for studying rare diseases. This is important because in small sample sizes, randomization will not always achieve its goal of balancing patient characteristics between treatment groups. In contrast with the body of literature on clinical trial methods in rare diseases, however, the literature on observational methods is considerably less mature. Several observational studies presented only descriptive frequencies of outcomes after a treatment and often with no comparison group, limiting the inferences that can be drawn about the treatment and subsequent outcomes. In general, observational studies of rare diseases used the same methods that are used to study health outcomes in more common conditions. However, several advanced observational methods that are used to study outcomes in common conditions—including propensity scores and self controlled designs—are particularly well suited for tackling confounding in the setting of rare events. Propensity scores deal with confounding in between person comparisons, whereas self controlled designs implicitly tackle time invariant confounding by making within person comparisons. It is important to note, however, that statistically controlling for confounding may not always be possible, even with propensity scores in studies with few participants.

In addition to the often small samples, studies of patient health outcomes in rare diseases using observational data face other important challenges. For example, there is often no appropriate group against which to compare outcome frequencies in patients with rare diseases and, even when there is, controlling for confounding can be difficult because the risk factors of those outcomes are usually not well understood. Yet, little work has been done to develop or apply methods to directly deal with these challenges. We did not identify any novel observational methods that have been developed to study outcomes in rare diseases. As observational data on rare diseases become more ubiquitous, greater attention is needed on methods to analyze these data to validly evaluate health outcomes in patients with rare diseases.

Limitations of this study

This survey of research methods for rare diseases has several limitations. Firstly, our literature search was focused on articles that mentioned “rare disease” in a searchable field. Because of the large number of unique rare diseases, we were not able to search for applications of innovative methods related to each specific disease. In addition, our review was intended to provide a general overview of non-traditional methods that have been proposed or applied to studying rare diseases. If other non-traditional methods exist that might be applicable to rare diseases but have not yet been discussed in a publication in the databases we searched, we may not have identified them. Moreover, our review is intended to enhance awareness of the availability and use of innovative methods for studying health outcomes in rare diseases and is not intended to provide a technical review of these methods, which can be found in the cited references. Finally, while we searched three databases, two of which include biomedical journals and a third that covers disciplines including psychology, physics, and engineering, it is possible that we missed relevant methods that have been used in other specialties, such as the social sciences.

Conclusions and future directions

Despite these limitations, we found several promising strategies that may contribute substantial advances to the study of health outcomes in patients with rare diseases. Some of these methods (for example, crossover designs and propensity scores) are already used in studies of common conditions. Awareness of the armamentarium of research tools available will help investigators design studies in patients with specific rare diseases and will help clinicians interpret the results of these studies when treating patients with these conditions. Observational studies are an important approach for studying health outcomes in rare diseases, particularly as patient registries and electronic healthcare databases continue to grow and offer richer clinical information. However, greater attention to innovative methods for using observational data to study rare disease health outcomes is needed.

What is already known on this topic

• Many barriers exist to advancing knowledge of and treatment options for rare diseases

• Because rare diseases are clinically dissimilar, clinicians, scientists, and other stakeholders working in one medical specialty may not be familiar with methods being applied in other disciplines

What this study adds

• Several promising strategies that may contribute substantial advances to the study of health outcomes in patients with rare diseases have been proposed, particularly for randomized trials

• Greater attention to innovative methods for using observational data to study rare disease health outcomes is needed

Cite this as: BMJ 2014;349:g6802

Web Extra. Extra material supplied by the author

Details of search strategy