Comparing genome sequencing technologies to improve rare disease diagnostics: a protocol for the evaluation of a pilot project, Genome-wide Sequencing Ontario

Abstract

Background:

Genome-wide sequencing has emerged as a promising strategy for the timely diagnosis of rare diseases, but it is not yet available as a clinical test performed in Canadian diagnostic laboratories. We describe the protocol for evaluating a 2-year pilot project, Genome-wide Sequencing Ontario, to offer high-quality clinical genome-wide sequencing in Ontario, Canada.

Methods:

The Genome-wide Sequencing Ontario protocol was codesigned by the Ontario Ministry of Health, the Hospital for Sick Children in Toronto and the Children’s Hospital of Eastern Ontario in Ottawa. Enrolment of a prospective cohort of patients began on Apr. 1, 2021. Eligible cases with blood samples available for the index case and both parents (i.e., trios) are randomized to receive exome sequencing or genome sequencing. We will collect patient-level data and ascertain costs associated with the laboratory workflow for exome sequencing and genome sequencing. We will compare point estimates for the diagnostic utility and timeliness of exome sequencing and genome sequencing, and we will determine an incremental cost-effectiveness ratio (expressed as the incremental cost of genome sequencing versus exome sequencing per additional patient with a causal variant detected).

Interpretation:

Findings from this work will provide robust evidence for the diagnostic utility, cost-effectiveness and timeliness of exome sequencing and genome sequencing, and will be disseminated via academic publications and policy briefs. Findings will inform provincial and cross-provincial policy related to the long-term organization, delivery and reimbursement of clinical-grade genome diagnostics for rare disease.

An accurate diagnosis is an essential component of care for patients with rare diseases, enabling tailored patient management, cascade family testing, family planning and peer support.¹ Exome sequencing (ES) has emerged as a diagnostic test in many jurisdictions to facilitate diagnoses for patients with a suspected rare genetic disease.^2–4 Extending the capabilities of ES, genome sequencing (GS) offers coverage of the coding and noncoding regions of the genome, and improved detection of copy number and structural variants.⁵ The term genome-wide sequencing (GWS) has emerged to refer to both ES and GS. In addition to its diagnostic capabilities, GWS can also be used to identify secondary findings — variants that are unrelated to the indication for testing but are associated with medically actionable, often presymptomatic health risks.^6,7

Despite the enhanced capabilities of GS compared to ES, a recent meta-analysis found that the diagnostic utility (i.e., rate of identifying causative genotypes in known disease genes) of GS and ES was not significantly different.⁸ However, for both ES and GS, the likelihood of diagnosis was significantly higher when sequencing was performed on the index case plus 2 biological relatives (usually parents; referred to as “trios”) compared to the index case alone (referred to as “singletons”).⁸ Although studies of the cost-effectiveness of ES and GS are emerging for a range of conditions,^9–11 a recent systematic review of 36 studies concluded that the economic evidence base to support the use of ES and GS in the clinic is very limited and that robust comparative studies are urgently needed to support their translation into clinical practice.¹¹

Alongside this growing evidence base, the Ontario Ministry of Health, which is responsible for administering the health care system for Ontario’s population of almost 15 million,¹² has been instrumental in enabling access to clinical ES in Ontario since 2014 through its Out-of-Country/Out-of-Province (OOC/OOP) Prior Approval Program.¹³ This program is an exceptional access mechanism that allows the use of services outside the province or country when they are not available in the province and if specific criteria are met. These criteria are informed by input from relevant experts and stakeholders, and by professional best practice statements.

In Ontario, criteria for access to ES were based on a report for the Ontario Ministry of Health’s Genetic Testing Advisory Committee;¹⁴ this report was informed by the Canadian College of Medical Geneticists position statement on the clinical implementation of GWS.¹⁵ In 2020, in response to demand for local ES services, Ontario Health (an agency of the Ontario Ministry of Health) undertook a comprehensive health technology assessment of ES and GS for people with unexplained developmental disabilities or multiple congenital anomalies.¹⁶ Based on the evidence reviewed in that assessment, the Ontario Health Technology Advisory Committee recommended public funding for ES as a second-tier test for these individuals.^9,16

As a result of policy efforts and the evolution in recent years of exceptional access programs in Ontario and other Canadian provinces, approximately 1500 Canadian patients with rare diseases receive access to clinical ES performed outside the country each year.¹⁷ Although the approach to date has enabled access to high-quality ES in an important subset of eligible patients, establishing local infrastructure is essential to enable the development of integrated laboratory services, a knowledge base that is locally representative and a performance measurement system that will guide technical and policy decisions related to the use of this technology over time.

We developed the Genome-wide Sequencing Ontario (GSO) pilot project, the objective of which is to evaluate the clinical performance and cost-effectiveness of ES and GS in Ontario. The project is intended to provide a high-quality, timely, cost-effective, equitable and sustainable service for residents of the province. GSO will provide evidence to inform longer-term decisions about technology adoption in Ontario and serve as a model for evidence development and clinical translation in other jurisdictions where clinical GWS policy is a priority. We describe a protocol that aims to evaluate the performance of this 2-year pilot project in Ontario.

Methods

Study design and setting

Informed by the strategies that have guided implementation efforts for ES and GS internationally,^18,19 we have developed a 2-year, prospective, hybrid type 2 implementation-effectiveness study design, tailored to Ontario’s needs and resources.²⁰ With a design that blends components of clinical effectiveness and implementation research, this approach enables more rapid translational gains, more effective implementation strategies and more useful information for decisionmakers.²⁰ Although this project is not designed as a hypothesis-driven effectiveness trial, participants are randomized to receive ES or GS to mitigate bias in determining and comparing performance and implementation outcomes for these sequencing strategies.

GSO is cohosted by the genome diagnostics laboratories and divisions of clinical genetics at The Hospital for Sick Children (SickKids) in Toronto and the Children’s Hospital of Eastern Ontario (CHEO) in Ottawa. Although the project was developed to meet the real-world evidence needs of our end users at the Laboratory and Diagnostics Branch, Ontario Ministry of Health, these policy partners did not influence the design or the dissemination plan for the work.

We will use the Standards for Quality Improvement Reporting Excellence (SQUIRE) reporting guidelines checklist.²¹

Intervention

Routine (i.e., nonrapid) genome-wide sequencing — ES and GS — are performed on the Novaseq6000 sequencing platform at SickKids, followed by genome alignment and variant calling. Resulting variants undergo annotation, filtration and interpretation. A cloud-based platform facilitates data flow between laboratories, analysis and delivery, storage, and sharing (Figure 1). Harmonized interpretation and reporting protocols were developed by the SickKids and CHEO laboratories based on technical specifications from an expert working group established by the Ministry of Health Laboratories and Diagnostics Branch.²²

Figure 1:

Genome-wide Sequencing Ontario harmonized, multi-institutional model for delivering genome-wide sequencing (GWS). CHEO = The Children’s Hospital of Eastern Ontario, GS = genome sequencing, ES = exome sequencing, SickKids = The Hospital for Sick Children.

Participants

The target population includes adults and children and aligns with the eligibility criteria for GWS that are in use by Ontario’s OOC/OOP Prior Approval Program, as established in the Genetic Testing Advisory Committee report¹⁴ and the Canadian College of Medical Geneticists position statement.¹⁵ Specifically, eligible patients include those for whom (a) a baseline genetics evaluation has been completed (e.g., phenotyping, family history, pretest genetic counselling and consent, and, where indicated, chromosome microarray and targeted testing, including biochemical testing); (b) a genetic etiology is the most likely explanation for the phenotype, as supported by a clinical presentation that includes any one of the following: moderate to severe developmental or functional impairment, multisystem involvement, a progressive clinical course that cannot be explained by another cause, or a differential diagnosis that includes 2 or more conditions that would require evaluation by separate gene panels; and (c) blood samples are available from the index patient and both parents (i.e., trios).

Ineligible patients include those for whom (a) the clinical presentation is limited to isolated mild intellectual disability or learning disabilities, nonsyndromic autism, isolated neurobehavioural disabilities (e.g., attention-deficit/hyperactivity disorder) or isolated neuropsychiatric conditions (e.g., schizophrenia); or (b) the phenotype is highly specific to a known genetic condition for which optimized genetic testing (e.g., a multigene panel) exists or is suggestive of an aneuploidy, a methylation defect or a trinucleotide repeat disorder.

As an implementation project, the sample size (n = 650) is based on OOC/OOP Program annual testing volumes for SickKids and CHEO.

Procedure

Patients are enrolled prospectively and complete a clinical consent form that is harmonized across sites to enable data-sharing between the SickKids and CHEO laboratories, optimizing the consistency of variant interpretation. After a genetics consultation during which ES or GS is deemed to be indicated, the requisition and sample collection are completed. All requisitions are reviewed by a genetic counsellor; if the patient meets the eligibility criteria, DNA is extracted and sent to the SickKids laboratory for sequencing.

Requisitions for testing of cases with uncertain eligibility are reviewed by a joint committee of clinical and laboratory experts. If eligibility criteria are not met, testing does not proceed, and a rejection report is issued to the ordering provider. For cases that are excluded, a physician can choose to appeal, whereupon the joint committee process would be used to evaluate any substantive additional clinical information or published evidence that is provided to support the case for testing. If preferred, ordering providers can submit a request for pre-approval before sample collection.

Randomization and matching

Patients are randomly assigned to ES or GS using a 1:1 ratio and an unblinded, stratified, permuted block randomization design. Cases are stratified by clinical site, phenotype and molecular testing history. Depending on the expected recruitment in each stratum, block sizes (i.e., the number of patients in each stratum) are 2, 4 or 6.

Outcomes

Three core dimensions on quality care will be ascertained: timeliness, diagnostic utility and cost-effectiveness.²³ Timeliness represents the core implementation outcome, and diagnostic utility and cost-effectiveness represent core effectiveness outcomes. In addition to patient-level clinical characteristics (i.e., enrolment site, age, physical features or symptoms, ethnicity, consanguinity, molecular testing history and test urgency) and patient preferences related to research recontact and receipt of secondary findings, patient-level details are collected to facilitate the assessment of implementation and effectiveness outcomes.

Patient-level details relevant to the assessment of timeliness include the following: the date samples are received or accessioned; the date eligibility for ES or GS is confirmed; the date ES or GS is initiated and completed; the date the ES or GS result is reported; and the date segregation analysis is initiated and reported. Timeliness will be defined from a laboratory perspective and measured as the number of weeks elapsed from sample accessioning to laboratory reporting. It will be reported as the proportion of cases for whom laboratory turnaround time is less than 12 weeks.

Patient-level details relevant to the assessment of effectiveness in the form of diagnostic utility include the following: the number and laboratory interpretation of primary ES or GS variants identified (i.e., pathogenic variant, likely pathogenic variant, variant of uncertain clinical significance); the clinical interpretation of primary ES or GS variants identified (i.e., diagnostic, partially diagnostic, potentially diagnostic, nondiagnostic); and the number and interpretation of medically actionable secondary variants identified. Diagnostic utility will be defined as the rate of causative, pathogenic or likely pathogenic genotypes in known disease genes. It will be reported as the proportion of cases for whom diagnostic, partially diagnostic and medically actionable secondary variants are identified.

To facilitate the assessment of cost-effectiveness, costs for ES and GS will be determined by microcosting the laboratory workflow components for each sequencing approach.^9,16,24,25 Data on the volume of resource use and unit prices for each workflow input will be obtained from laboratory protocols and managers. Cost-effectiveness will be reported as the incremental costs of local GS compared to ES for each additional patient with a molecular diagnosis achieved.

Data sources

Patient characteristics and preferences related to research recontact and receipt of secondary findings are ascertained from ES or GS requisition forms that are submitted to the laboratory. Laboratory process measures (i.e., dates) are ascertained from the SickKids and CHEO laboratory information systems, and the results of the ES or GS analyses are obtained from the harmonized, cloud-based genome analysis platform.

All data are entered into a centralized REDCap database. Data collection is facilitated by members of the study team and monitored monthly by a data audit team for accuracy and completion. Missing data identified through monthly quality checks will be obtained by the laboratory genetic counsellor from the ordering provider, technologist, genome analyst or director assigned to the case.

Statistical analysis

We will analyze our data using descriptive statistics (mean, median, range and standard deviation). Point estimates for diagnostic utility and timeliness will be compared statistically for ES and GS. We will examine the relationship between clinical characteristics (e.g., age, age of onset, sex, phenotype, prior testing history) and diagnostic yield using parametric or nonparametric univariate statistics as appropriate. If indicated, we will investigate explanatory variables of diagnostic utility using a regression model.

To facilitate the cost-effectiveness analysis, we will determine a cost per trio ES and cost per trio GS. Costs for each input related to the ES and GS laboratory workflows will be calculated by multiplying resource volume by unit price. For labour, the time in minutes for each task will be multiplied by wage rates. We will use the most recently available price units (2021). Otherwise, the prices as reported in previous microcosting studies^24,25 will be assumed to be stable based on consultation with laboratory managers. The costs of laboratory workflow inputs for each sequencing approach will be summed to determine a per-sample cost. We will then undertake a cost-effectiveness analysis from an institutional payer perspective, in which the difference in costs between GS and ES will be calculated and divided by the difference in diagnostic yield to calculate an incremental cost-effectiveness ratio, expressed as the incremental cost of GS versus ES per additional patient detected with a pathogenic variant.

As an implementation project, this cost-effectiveness analysis is limited to laboratory costs and outcomes. Given the complexity of diagnostic outcomes and the heterogeneity of the patient sample, it will not be possible to model patients’ health states and the range of treatment options for a given diagnostic result to generate health benefits over a lifetime (such as quality-adjusted life-years). All costs will be reported in 2021 Canadian dollars.

Ethics approval

Individuals who undergo GWS as part of this protocol provide clinical consent for GWS and research consent through Clinical Trials Ontario [CTO-1577], the provincial body responsible for approving clinical trials and observational studies that involve 2 or more academic or health care institutions in Ontario.

Interpretation

As a robust comparison of diagnostic utility, cost-effectiveness and timeliness for clinical grade ES and GS, this study will be instrumental in guiding provincial policy and funding decisions related to their ongoing use in clinical care. We will also assess the relationship between key clinical characteristics (e.g., age, age of symptom onset, phenotype, prior testing history) and these performance outcomes to understand the relative performance of these technologies in different patient groups. In particular, our data will inform the development of diagnostic pathway guidelines that provide recommendations on the use of ES and GS for different clinical indications and when the pathway should include ES versus GS. Finally, having established a centralized sequencing–distributed interpretation model between 2 sites, data collected for this project will inform the feasibility and costs of scaling this model across the province.

In addition to disseminating our findings through academic publications and presentations, we will generate a report for the provincial policy partners who have been engaged in the design and implementation of this work. We will also develop educational materials for an expanding network of ordering providers to improve genomic literacy among non–genetics providers and generate lay summaries of our findings for patients with rare disease.

All patients are provided with the option to participate in future research as part of their clinical consent process. Patients who consent to recontact can enrol in additional research ethics board–approved studies linked to GSO, including the clinical reanalysis of genomic data and the patient and system impacts of medically actionable secondary findings. Findings from these studies will inform best practices and guidelines for clinical reanalysis and for managing secondary findings in a single-payer health care system. Participants in GSO will also be able to enrol in other rare disease research initiatives focused on gene discovery, characterizing natural history and trialling precision therapies.²⁶

Moving beyond Ontario, we will engage our clinical, research, policy and funding partners to understand province-specific barriers and facilitators of ES and GS implementation. We will also strategize (as a community of practice invested in the equitable and sustainable delivery of high-quality genomic medicine) to develop concrete implementation structures and processes for Canadians.

Limitations

The primary limitations of this work relate to the inclusion of cases from only 2 clinical sites in Ontario, and to the restricted inclusion criteria. With an emphasis on pediatric patients with neurodevelopmental phenotypes, findings related to the performance of ES and GS may be limited in their generalizability to adult patients with rare disease and to those with non-neurodevelopmental phenotypes.

Conclusion

This protocol lays the foundation for the ongoing development and implementation of clinical ES and GS technologies. Going forward, our evaluation framework will not only serve to monitor performance and inform continuous system improvement, but also guide our evaluation of emerging-omic technologies and applications in the years to come.