Mixed-methods evaluation of the NHS Genomic Medicine Service for paediatric rare diseases: study protocol

Celine Lewis; James Buchanan; Angus Clarke; Emma Clement; Bettina Friedrich; Jillian Hastings-Ward; Melissa Hill; Ruth Horn; Anneke M Lucassen; Chris Patch; Alexandra Pickard; Lauren Roberts; Saskia C Sanderson; Sarah L Wynn; Cecilia Vindrola-Padros; Monica Lakhanpaul

doi:10.3310/nihropenres.13236.2

. 2022 Feb 23;1:23. Originally published 2021 Nov 22. [Version 2] doi: 10.3310/nihropenres.13236.2

Mixed-methods evaluation of the NHS Genomic Medicine Service for paediatric rare diseases: study protocol

Celine Lewis ^1,^2,^a, James Buchanan ^3,⁴, Angus Clarke ⁵, Emma Clement ⁶, Bettina Friedrich ¹, Jillian Hastings-Ward ⁷, Melissa Hill ^2,⁸, Ruth Horn ⁹, Anneke M Lucassen ^10,¹¹, Chris Patch ^12,¹³, Alexandra Pickard ¹⁴, Lauren Roberts ¹⁵, Saskia C Sanderson ¹⁶, Sarah L Wynn ¹⁷, Cecilia Vindrola-Padros ¹⁸, Monica Lakhanpaul ¹

¹Population, Policy and Practice, UCL GOS Institute of Child Health, London, UK

²London North Genomic Laboratory Hub, Great Ormond Street Hospital for Children NHS Foundation Trust, London, UK

³Health Economics Research Centre, Nuffield Department of Population Health, University of Oxford,, Oxford, UK

⁴NIHR Oxford Biomedical Research Centre, Oxford, UK

⁵Division of Cancer and Genetics, Cardiff University School of Medicine, Cardiff, UK

⁶Clinical Genetics and Genomic Medicine, Great Ormond Street Hospital for Children NHS Foundation Trust, London, UK

⁷Independent Chair, Participant Panel at Genomics England, London, UK

⁸Genetics and Genomic Medicine, UCL GOS Institute of Child Health, London, UK

⁹The Ethox Centre and the Wellcome Centre for Ethics and Humanities, Department of Population Health, University of Oxford, Oxford, UK

¹⁰Clinical Ethics and Law, Faculty of Medicine, University of Southampton, Southampton, UK

¹¹Centre for Personalised Medicine, Wellcome Centre for Human Genetics, University of Oxford, Oxford, USA

¹²Genomics England, Queen Mary University of London, London, UK

¹³Counselling, Society and Ethics Research, Wellcome Genome Campus, Cambridge, UK

¹⁴Genomics Unit Specialised Commissioning, NHS England, London, UK

¹⁵Genetic Alliance UK, London, UK

¹⁶Our Future Health, London, UK

¹⁷Unique – the Rare Chromosome Disorder Support Group, Oxted, UK

¹⁸Department of Targeted Intervention and Rapid Research Evaluation and Appraisal Lab (RREAL),, University College London, London, UK

Email: celine.lewis@ucl.ac.uk

Competing interests: James Buchanan has received travel expense reimbursement from Illumina to attend meetings. Jillian Hastings Ward is the independent Chair of the Participant Panel at Genomic England and also sits on the National Genomics Board and the NHS GMS People and Communities Forum. Christine Patch has been on a secondment with Genomics England as Clinical Lead for Genetic Counselling since October 2016. Alexandra Pickard is Deputy Director of Genomics at NHS England. Saskia Sanderson is Chief Behavioural Scientist at Our Future Health. Sarah Wynn is a representative on the Genomics Clinical Reference Group and sits on the NHS GMS People and Communities Forum. No other competing interests were disclosed.

Roles

Celine Lewis: Conceptualization, Funding Acquisition, Methodology, Project Administration, Supervision, Visualization, Writing – Original Draft Preparation

James Buchanan: Conceptualization, Methodology, Writing – Review & Editing

Angus Clarke: Conceptualization, Funding Acquisition, Methodology, Writing – Review & Editing

Emma Clement: Conceptualization, Methodology, Writing – Review & Editing

Bettina Friedrich: Methodology, Writing – Review & Editing

Jillian Hastings-Ward: Conceptualization, Methodology, Writing – Review & Editing

Melissa Hill: Conceptualization, Methodology, Writing – Review & Editing

Ruth Horn: Conceptualization, Methodology, Writing – Review & Editing

Anneke M Lucassen: Conceptualization, Funding Acquisition, Methodology, Writing – Review & Editing

Chris Patch: Conceptualization, Methodology, Writing – Review & Editing

Alexandra Pickard: Conceptualization, Methodology, Writing – Review & Editing

Lauren Roberts: Conceptualization, Methodology, Writing – Review & Editing

Saskia C Sanderson: Conceptualization, Funding Acquisition, Methodology, Writing – Review & Editing

Sarah L Wynn: Conceptualization, Methodology, Writing – Review & Editing

Cecilia Vindrola-Padros: Conceptualization, Methodology, Writing – Review & Editing

Monica Lakhanpaul: Conceptualization, Funding Acquisition, Methodology, Writing – Review & Editing

PMCID: PMC7612282 EMSID: EMS140786 PMID: 35098132

Version Changes

Revised. Amendments from Version 1

We have amended the subheadings to the different studies to lead with the objective rather than the method. We have clarified how we will use the CFIR framework – it will provide a structured approach to interview questions and analysis, but interpretation and recommendations will be developed in conjunction with our advisory team with decisions documented. We have included a number of the suggested questions raised by Reviewer 1 in the topic guides. We have clarified the recruitment procedures for Studies 1 and 4 to address concerns around risk of bias. We have also clarified the role that PIs will have in identifying potential participants. We have amended the description of Study 2, highlighting that it is an exploratory study to gain a snapshot of the different ways that appointments take place, rather than a study to make generalisation across sites, condition types and/or participant characteristics. We have acknowledged that we will reflect on the limitations of the observational method in any report or papers. We have expanded on the description of the consent process, and said that verbal consent will be recorded electronically and signed/dated by the researcher with a copy of the electronic record sent to the participant. We have clarified that to quantify the changes and impact made following dissemination of findings, we will add an agenda point at each advisory team meeting to ask members involved in setting policy and practice how they have used the emerging findings. We have clarified that the summary tables will not only focus on barriers and facilitators, they will also capture the process of implementation, perceptions etc. We have added further detail around who will be invited to take part in the workshop as well as what will happen during the workshop.

Abstract

Background

A new nationally commissioned NHS England Genomic Medicine Service (GMS) was recently established to deliver genomic testing with equity of access for patients affected by rare diseases and cancer. The overarching aim of this research is to evaluate the implementation of the GMS during its early years, identify barriers and enablers to successful implementation, and provide recommendations for practice. The focus will be on the use of genomic testing for paediatric rare diseases.

Methods

This will be a 4-year mixed-methods research programme using clinic observations, interviews and surveys. Study 1 consists of qualitative interviews with designers/implementers of the GMS in Year 1 of the research programme, along with documentary analysis to understand the intended outcomes for the Service. These will be revisited in Year 4 to compare intended outcomes with what happened in practice, and to identify barriers and facilitators that were encountered along the way. Study 2 consists of clinic observations (pre-test counselling and results disclosure) to examine the interaction between health professionals and parents, along with follow-up interviews with both after each observation. Study 3 consists of a longitudinal survey with parents at two timepoints (time of testing and 12 months post-results) along with follow-up interviews, to examine parent-reported experiences and outcomes. Study 4 consists of qualitative interviews and a cross-sectional survey with medical specialists to identify preparedness, facilitators and challenges to mainstreaming genomic testing. The use of theory-based and pre-specified constructs will help generalise the findings and enable integration across the various sub-studies.

Dissemination

We will disseminate our results to policymakers as findings emerge, so any suggested changes to service provision can be considered in a timely manner. A workshop with key stakeholders will be held in Year 4 to develop and agree a set of recommendations for practice.

Keywords: genomics, genomic medicine service, rare disease, paediatric, protocol, mixed methods

Plain English summary

Background and aims: Genome sequencing (where a person’s entire genetic code is mapped) is set to dramatically transform patient care and medical outcomes. Recently, genome sequencing was introduced as part of routine clinical care in the NHS, through the Genomic Medicine Service (GMS). The aim of this research is to understand how genome sequencing is being delivered in the first few years of the Service, in particular what the barriers and enablers are to successful delivery. The focus of the study will be the use of genome sequencing for children with undiagnosed conditions.

Study design: This is a four-year study in which we will conduct: observations of clinic appointments; interviews with policy makers and health professionals designing and implementing the new service; and surveys/interviews with parents of patients undergoing genomic testing. By the end of this study we will have:

- a better understanding of the intended v actual outcomes of the GMS,
- insights into what happens during clinical encounters,
- understand what the entire testing process is like for parents from being offered genomic testing to receiving their results and beyond, including the clinical as well as emotional and practical outcomes, and
- understand how healthcare professionals feel about delivering the GMS, particularly those that are non-genetic specialists, including how prepared they feel to deliver genomic testing.

Patient and public involvement: Parents of children who have been through the testing process have helped us design this study. They have inputted into surveys and topic guides, and will be involved throughout the study as members of the advisory team so that we can ensure the findings are used to improve the quality of care patients and families receive.

Dissemination: The findings from this research will be shared with organisations such as NHS England and NHS Improvement so that recommendations can be implemented swiftly.

Introduction

The Genomic Medicine Service

In October 2018, a new nationally commissioned Genomic Medicine Service (GMS) was established by NHS England. This service, built around seven Genomic Laboratory Hubs (GLHs), aims to deliver consolidated, state of the art, high throughput and high-quality genomic testing (including both genome and exome sequencing) with equity of access for patients affected by rare diseases and cancer ¹. The GMS capitalises on the infrastructure and learning from the 100,000 Genomes Project, a world-leading initiative set up in England in 2015 with the explicit aim of embedding genomic medicine into clinical care to improve diagnosis and management of patients affected by selected rare and inherited diseases and cancer ². The NHS will be the first national healthcare system in the world to offer whole genome sequencing as part of routine care.

The overall goal of the GMS is that from 2020, and by 2025, genomic medicine will be embedded in multiple clinical pathways in routine care, where appropriate, and linked to a broader NHS long term plan of sequencing 500,000 whole genomes for patients with certain rare diseases and cancers, incorporating the latest genomics advances into routine healthcare to improve diagnosis, stratification and treatment of illness, and supporting research and innovation ³. Ultimately, the aim is that by 2025, genomic technologies will be a fundamental component of medical training, and there will be a new taxonomy of medicine based on the underlying drivers of disease ⁴.

Mainstreaming genomics for rare disease diagnosis

Genome sequencing will be available as a first-line test for some rare and undiagnosed diseases, for example individuals with ultra-rare disorders or atypical manifestations of recognised monogenic disorders. In addition, certain tests specified in the new NHS England Genetic Test Directory can be ordered by medically qualified individuals specialised in a sub-discipline other than genetics (referred to hereon in as ‘medical specialists’), in both primary and secondary care, thus ‘mainstreaming’ genomics ¹. For example, in primary care, a general practitioner could order a cystic fibrosis carrier test, and in secondary care a neurologist or paediatrician could order genome sequencing for a patient with intellectual disability.

Preparation for genomic testing in the NHS GMS

Over the past few years, several initiatives have been implemented to prepare the workforce for genomic testing. In 2014, Health Education England (HEE) launched a four-year £20 million Genomics Education Programme (GEP) to ensure that the NHS workforce has the knowledge, skills and experience to keep the United Kingdom (UK) at the heart of the genomics revolution in healthcare ³. Other initiatives include a Masters in Genomic Medicine delivered by seven leading higher educational institutions; ‘genomics roadshows’ where genetic specialists have visited a wide range of clinical disciplines in hospitals to highlight genomics and how it can improve patient care; and a genomics toolkit developed by the Royal College of General Practitioners in partnership with the GEP to explain how genomic medicine impacts primary care ^3,
5,
6. However, the reach and utility of these resources have yet to be examined, and the informatics infrastructure including sample collection pathways and results delivery processes have yet to be finalised and tested.

To ensure patients and families are fully prepared for genomic testing in the NHS GMS, NHS England have prepared a range of patient-facing and online resources ^7–
9, as well as a ‘record of discussion’ form which will be used in the clinical pathway to record parents’ (of children unable to consent themselves) and patients’ test and research decisions ¹⁰. Genomics England has developed information specifically to support people making decisions about participating in research which will be done on a voluntary basis and consented separately from genomic testing for clinical care ¹¹. However, we do not yet know what patients’ and parents’ attitudes, understanding and experiences of genomic testing within a purely clinical context will be, whether they feel they have made an informed decision to undergo sequencing, what proportion will consent to donating their (or their child’s) data for research purposes (and if not, why not), and whether they are satisfied with the process overall.

Focus of current study

The first few years of the NHS GMS is an ideal opportunity for which to evaluate the implementation, service and patient outcomes of genomic testing in a clinical setting. It will enable us to make comparisons with the hybrid research-clinical context of the 100,000 Genomes Project where much research has already taken place ^12–
18. Implementation science, the systematic study of methods that support the application of research findings and other evidence-based knowledge into policy and practice, is increasingly being seen as playing a critical role health services research ^19,
20. Previous research on new interventions has highlighted that as well as assessing outcomes, it is valuable to look at the process of the intervention as this can shed light on the mechanisms responsible for whether and how successful it is ²¹. This formative work can also enable researchers to suggest ways to answer questions about how interventions might be adapted and respond to change in order to produce positive outcomes ²².

Background

How will the NHS GMS impact parents and children?

The NHS GMS is set to have a profound impact on the management and diagnosis of children with rare diseases in the NHS. The majority (50–75%) of rare diseases affect children ²³ and in the past it has taken on average six years for a rare disease to be diagnosed, during which time patients are likely to have undergone extensive medical testing ^24,
25. Genomic sequencing has the potential to reduce this ‘diagnostic odyssey’ for some patients with rare diseases and their families. The diagnostic yield of genomic sequencing in previously unsolved paediatric cases is already around 40–50% and may increase as knowledge grows ²⁶. For children with a rare condition, a diagnosis can enable access to disease specific screening or treatments, provide a clearer prognosis and information about recurrence risk, enable parents to make contact with other parents, and facilitate access to social and educational support ²⁷. Psychological benefits for parents can include relief from guilt, understanding the origin of the child’s condition, validation in terms of offering legitimacy for the child’s behaviour and/or appearance, and ability to connect with others through support groups ²⁸. Previous research on patients and parents experiences of genomic testing, conducted during the 100,000 Genomes Project, highlighted that the majority were satisfied with the consenting process, felt they had made an informed decision to take part, and had largely positive attitudes towards sequencing, although concerns existed around data sharing and access, and the potential emotional impact of the results ¹³. Whilst participants generally understood what is involved in genome sequencing, the purpose and the benefits, there were misunderstandings around the limitations and associated uncertainties ¹⁸. For example, only around 70% of participants correctly understood that they may not receive any informative results about their child’s condition from whole genome sequencing ¹⁸. Reports of parents misinterpreting or overestimating the utility of findings from genomic testing have been cited elsewhere ^29,
30, and the importance of managing patient expectations to avoid disappointment or decisional regret has been raised by genetic specialists ^31,
32. Research focused on whether and how health professionals are managing parental expectations of genomic testing in the NHS GMS would therefore be of value.

Whilst evidence has begun to emerge about the clinical effectiveness of genomic testing (e.g. changes in clinical management, amended treatment plans) for patients from different condition groups ^33,
34, for example those having rapid genomic testing in the neonatal setting ³⁵ or those with developmental disorders ³⁶, we still have limited data on the psychosocial and behavioural impact of disclosing genomic results to parents, including whether and how the impact differs amongst different patient populations ³⁷. There is some evidence to suggest that parents of children with a known disease may be more prone to negative test-related psychological experiences following genomic testing than other population groups ³⁸. Results from the 100,000 Genomes Project indicated that some participants and in particular parents experienced distress and uncertainty following receipt of sequencing results. Similar findings have been reported elsewhere, with parents receiving exome sequencing results reporting feelings of frustration and isolation from the lack of available information about the condition ³⁹ as well as loss of hope for recovery ⁴⁰. However, this research is still in its infancy, and further research is essential to gain a more nuanced and complete understanding of the psychosocial and behavioural impact of genomic testing.

How will the NHS GMS impact health professionals?

The significant changes in the way testing is offered will impact across medical specialities and require the roles of both medical and genetic specialists to evolve ⁴¹. Widespread implementation and ‘mainstreaming’ of genomic medicine will depend on health professionals’ perceptions of the usability and value of the technology in day-to-day practice, however some of these professionals are sceptical of the positive impact genomic medicine will have on patient care ^12,
42. Studies have shown that many health professionals have limited genetics training and may be unprepared to conduct pre- and post-test counselling including interpreting test results and consenting/returning additional findings ^43,
44. Concerns also exist around lack of access to genetic professionals ⁴⁵ as well as the challenges of interpreting uncertain results and managing patients’ expectations about genome sequencing ⁴⁶. To ensure the successful transition of genomics from a specialist service to a mainstream service, thoughtful planning and procedures are required to prepare the workforce. This includes: training in genomics for healthcare professionals outside of clinical genetics including interpreting and returning genomic data back to patients; clear pathways for which tests to order for which indications; educational initiatives to ensure healthcare professionals taking consent feel equipped to do so; and increased interaction between genetic and medical specialists to support the delivery of testing outside the clinical genetics specialty ⁴¹.

Protocol

Research aims and objectives

This paper outlines a four-year mixed-methods research programme using observations, interviews and surveys to evaluate the NHS GMS for the diagnosis of paediatric rare diseases. Research will be conducted with key stakeholders designing and implementing the GMS, as well as health professionals (genetic and medical specialists) and parents of patients undergoing genomic testing to examine the intentions, experiences and outcomes of the new service ⁴⁷.

The aims are to:

1. Identify the resources, activities and intended and actual outcomes of the NHS GMS; identify any potential barriers to achieving the intended outcomes during the early years of the Service (2022–25);

2. Understand the processes and practices taking place by examining the interactions between health professionals and parents/patients during pre-test counselling and results delivery appointments;

3. Examine the experiences and outcomes of genomic testing that parents reported over time;

4. Identify the preparedness and experiences of medical specialists involved in delivering genomic medicine in mainstream NHS care in the first few years of the Service, and identify elements which make this easier or more difficult.

The findings from the research will be shared with NHS England and NHS Improvement contemporaneously to continue to drive improvements in the Service and develop recommendations for practice.

Methods and analysis

Research approach and conceptual framework. We will conduct a mixed-methods research programme, employing qualitative and quantitative approaches to provide a richer, deeper insight into the topic area, generating more knowledge, and increasing the validity of the findings ⁴⁸. We will work within a pragmatist paradigm in order to seek functional knowledge and produce positive change in clinical practice ⁴⁹. Pragmatism refers to a worldview that focuses on “what works” rather than what might be considered absolutely and objectively “true” or “real.” ⁵⁰.

We will use a theory-driven approach to understand how the NHS GMS is being implemented as well as to evaluate the outcomes from the Service. The Consolidated Framework for Implementation Research (CFIR) ²⁰ will be used as an explanatory framework to systematically assess the contextual factors including barriers and facilitators that influence implementation and adoption, and has been used previously to evaluate the implementation of genomic medicine ^51–
54. We also chose this framework as it is well-suited to guide the development of actionable findings as well as to rapid-cycle evaluation, which fits with our dissemination strategy whereby we update intervention implementers of our results as they emerge ⁵⁵. The framework provides a taxonomy of operationally defined constructs that are likely to influence implementation of complex programs, organised into five major domains: 1) Intervention Characteristics (features of the intervention itself which might influence implementation e.g. complexity); 2) Outer Setting (features of the implementation organisation e.g. leadership engagement); 3) Inner Setting (features of the external context or environment e.g. readiness); 4) Characteristics of Individuals (e.g. knowledge and beliefs of individuals); and 5) Process (strategies or tactics which might influence implantation e.g. planning). We will identify those domains and associated constructs which are most relevant to the study aims, and focus on those when developing the interview questions and analysing the interview data. We will evaluate implementation outcomes according to Proctor’s taxonomy, which comprises eight major domains - acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability ⁴⁷. In order to understand the patient/parent perspective, we will use a number of patient reported outcome measures including decisional conflict and regret, patient empowerment and satisfaction. The use of theory-based and pre-specified constructs will help to generalise the findings and enable integration across the various sub-studies, enabling us to build a stronger evidence base.

Study design overview. This study includes four parts which will each address different levels of implementation. The first part focuses on the programme level, where using an implementation science approach, we will conduct an interview study with key stakeholders from organisations tasked with designing and implementing the NHS GMS (e.g. Genomics England, NHS England, GMSA and GLH leads) along with desk-based documentary research to examine the initial programme theory (resources, activities and intended outcomes) underlying the NHS GMS. These will be compared with the actual outcomes in Year 4 in terms of effectiveness, adoption, fidelity, acceptability and uptake (Aim 1 – Study 1). We will also conduct interviews with genetic specialists tasked with delivering the NHS GMS to understand how they are experiencing the implementation (Aim 1 – Study 1). The second part focuses on the inter-personal level where we will conduct observations of clinical encounters to examine the interaction between healthcare professionals and parents, alongside interviews with the professionals and parents to understand the processes and practices taking place when consenting for and returning genomic test results (Aim 2 – Study 2). The third part will target the individual level focusing on parents’ and health professionals’ perspectives. We will conduct a longitudinal survey along with follow-up interviews to examine parent-reported experiences and outcomes from genomic testing, in particular comparing the outcomes of parents who receive a diagnostic result with those who do not (Aim 3 – Study 3). We will also conduct interviews followed by a cross-sectional survey with medical specialists in Year 4 to identify the preparedness, experiences and challenges to delivering genomic testing in the first few years of the NHS GMS (Aim 4 – Study 4). Findings from all parts will be integrated to identify overarching findings and recommendations for practice. See Figure 1 for overview of study design and timelines.

Patient and public involvement. The research programme has been co-designed with parents of children with rare diseases as well as patient advocates and key stakeholders. At the time of drafting the funding application, input was sought from Genetic Alliance UK, Rare Disease UK and SWAN UK (Syndromes without a name) to identify the key research questions and discuss study design, ensuring the design facilitated patient-orientated outcomes. CL also spoke with the Chair of the 100,000 Genomes Project Participant Panel as well as a parent of a child with a rare undiagnosed condition, to ensure the study design would capture what was important to parents.

Following approval for funding, an Advisory Team was set up which includes three parents of children with (previously) undiagnosed rare conditions and two patient advocates from the support groups SWAN UK and Unique: The Rare Chromosome and Gene Disorder Support Group. They have inputted into the study aims and objectives, reviewed and revised patient-facing documents including participant information sheets and topic guides (Studies 2 and 3), informed the selection of validated measures for a longitudinal survey study (Study 3) and commented on wording and answerability. Emergent findings will be shared with the advisory team throughout the study, and they will support the development of recommendations for policy and practice, ensuring that they are feasible and appropriate. Policy options will be evaluated using the APEASE framework (Acceptability, Practicability, Effectiveness, Affordability, Side-Effects, Equity) ⁵⁶. The APEASE criteria are a set of criteria used to make context-based decisions on intervention content and delivery. The advisory team will advise on plain language summaries and video abstracts to facilitate dissemination of the study findings to participants and relevant wider patient communities, and will be invited to co-author manuscripts. Parent participants will be reimbursed for their involvement in the project, in line with the NIHR Centre for Engagement and Dissemination’s payment policy ⁵⁷.

Study setting. Recruitment of participants will take place across seven NHS Trusts located across England. Sites have been selected to facilitate a diverse ethnic mix of participants as well as North v South and urban v rural settings. At each participating site, a health professional from the genetics department will act as local Principal Investigator (PI) for the study, however we will work closely with departments outside of clinical genetics who are delivering genomic testing to examine the issue of mainstreaming. Regular meetings will be held with the participating site and clinical staff to discuss any recruitment issues.

Participants. There will be two separate but parallel cohorts in this project; 1) parents of children (<16 years) with rare diseases undergoing genomic testing, who are making decisions on behalf of their children and may themselves be undergoing testing to help identify or interpret the results, and 2) health professionals (including genetic and other medical specialists), policy-makers, commissioners and organisational decision-makers who are delivering the GMS. By examining the way in which key stakeholders (parents, health professionals, policy-makers, decision-makers) perceive, experience and behave in the GMS, we will ensure that the findings and subsequent recommendations around the implementation of genomics into mainstream clinical practice are grounded in first-hand experiences.

For Study 2, In order to accommodate participants that do not speak/have limited English, we will translate the participant information sheet and consent form into those languages identified as being commonly spoken by parents attending genetic services (e.g. Gujarati, Bengali, Urdu, Polish, Punjabi). Regular dialogue between the research team and the PIs from participating sites will take place to monitor this. It will not be possible to translate the survey (Study 3) into these languages as the included measures have not been validated in these languages.

Detailed study plan

An overview of the study plan is provided in Figure 2. Data collection and analysis will be conducted primarily by Dr Celine Lewis (PI) and Dr Bettina Friedrich (research associate). Both are health psychologists who have extensive experience in research including qualitative interviews and data analysis. Celine also has extensive experience working in research related to genetic/genomic medicine. Bettina has a background in quantitative research. For Study 4 we will also be supported by an MSc student.

Study 1: Specify the programme theory of the GMS (Years 1 and 4)

Study design. In Year 1 of the study, we will collect documentary evidence (policy documents, journal articles and meeting presentations) as well as conduct qualitative interviews with key stakeholders (programme designers, implementers and providers). This will include people from organisations such as NHS England, Health Education England, Genomics England as well as ‘genomic champions’ ⁵⁸ from different clinical specialties. This will enable us to specify the initial programme theory underpinning the GMS and to identify the underlying assumptions about how the GMS is expected to work to achieve its expected outcomes. In addition, we will interview leads from each of the GMSAs/GLHs (clinical and scientific) tasked with embedding the Service as well as genetic specialists tasked with delivering the Service. These interviews will allow us to understand how health professionals are experiencing the implementation of the GMS during its first years including: readiness of the Service, the processes and procedures that have been put in place to deliver the Service, any individual and organisational adaptations that have been made, and any barriers and challenges they have faced.

In Year 1 a logic model (a visual representation of the theory) which describes the resources and activities (inputs) and intended outputs, outcomes and impact will be developed ⁵⁹ which will form the foundation for our understanding of what was intended for the GMS (the GMS ‘blueprint’). In Year 4, we will conduct further interviews and documentary analysis to understand if the programme theory has changed over time, if the programme as planned is different to the programme as performed (‘fidelity to the model’), changes in local practice over time and the factors that have acted as barriers and facilitators to implementation.

Interview guides will be informed by the study aims, the existing literature and the interests of the advisory team. We will identify which of the CFIR domains are most aligned with these particular topics of interest, and the individual constructs will be used to help determine the structure of the questions.

Data analysis. Data will be analysed using framework analysis ⁶⁰. This is an approach that facilitates identification of key themes as well as commonalities and differences in the data through comparison across as well as within cases. A codebook will be developed to facilitate team-based analysis which will be facilitated using NVivo software ⁶¹. The first step will consist of a deductive analysis, where data are coded according to the CFIR domains and constructs which have been prioritised as key. This will be followed by an inductive analysis, to allow for any new themes or unexpected findings. The same codebook will be applied to the analysis of both sets of interviews as well as the documentary evidence to enable cross-referencing and comparisons across the data.

Recruitment and sample size. GMS designers from each of the key organisations will be identified by the advisory team, and invited for interview. We will aim to recruit the key clinical and scientific rare disease lead from each of seven GMSAs/GLHs across England. To ensure we reach out to the most appropriate person, we will ask interviewees for suggestions of who to approach ⁶². Genetic specialists with an interest in rare disease will be recruited from across the seven participating hospital trusts. Informed consent along with participant information will be collected prior to interview. We anticipate that interviews will last approximately an hour, and will be conducted over video conferencing software. Interviews will continue until saturation is reached and, alongside documentary evidence, the initial programme theory has been identified. We anticipate this will be around 10 interviews with GMS designers, 14 interviews with people from the GMSAs/GLHs, and similarly around14 interviews with genetic specialists.

Study 2: Identify processes and practices when patients undergo genomic testing in the GMS (Years 2–3)

Study design. We will conduct direct observations (including audio and/or video-recordings) of clinical encounters (clinical pre-test counselling appointments as well as results delivery appointments) involving patients (children) and families undergoing genomic testing. A key benefit of observations is that they take place in natural settings that are the natural loci of activity ⁶³. This will be an exploratory study to understand what happens in a consent/return of results appointment. The aims are to understand consistencies and variations in the overarching structure of the appointments, evaluate the interactions between patient/parents and health professionals (including the information exchange and the questions and responses), and gain insight into the communication techniques that are employed by both parties. Our expectations are that the discussion will be consenter-led, that they will follow a relatively invariant pattern for discussing issues around consent, and that health professionals will want to maximise recruitment to research. In addition, the observations will offer insight into the various processes and practices required in order that health professionals can request genomic tests (patient choice forms, uploaded test requests etc). Observations of clinic appointment were previously conducted during the 100,000 Genomes Project and yielded valuable data ¹⁴. A structured observation guide using pre-determined categories identified through this previous work (e.g. checklist of particular topics, interactions between the professional and the [child] patient, notable non-verbal behaviours, paperwork and administrative aspects etc) will be used to standardise the observation and inform follow-up interview questions.

Following each observation there will be an immediate de-brief interview with the professional and an interview with parents 1–2 weeks later. The topic guides will focus on views and feedback related to the content of the appointment, their expectations and implicit goals from the interaction, and moderating factors that may have hampered or contributed to the success of the appointment. This will allow for comparison across the three data sources (interview recording, professional and parent interviews). Pairing observations/audio-recordings with interviews is valuable because this may reveal inconsistencies between participants’ responses to interview questions and what they actually do in practice ⁶⁴.

Data analysis. The analysis will be conducted from an interactionist perspective ⁶⁵ using concepts drawn from content analysis ⁶⁶ and thematic analysis ⁶⁷, facilitated using Nvivo ⁶¹. Data from the different sources will be given equal weighting and integrated at the data analysis stage, to explore the appointment from multiple perspectives.

Recruitment and sample size. Eligible patient/parent participants will be parents, carers or other family members of children undergoing genomic testing for rare disease diagnosis. Non-English speaking families will be eligible to participate provided the translator is able to translate the participant information sheet and consent form. Eligible health professional participants will be any health professionals tasked with consenting and/or delivering WGS results. Potential participants (health professionals and parents) will be purposively sampled to ensure variation in 1) condition type, 2) who is conducting the appointment (genetic or medical specialist) 3) result (diagnostic result, no-finding result and inconclusive result), and 4) site. By including seven sites from regionally diverse parts of the country, we hope to include parents who vary in terms of educational and socioeconomic status as well as ethnicity. People from minority ethnic groups may experience a higher incidence and prevalence of rare diseases than the general population for a variety of reasons, genetic and otherwise ⁶⁸. In addition, people from minority ethnic groups and other underserved populations are likely to experience even greater barriers to screening, diagnosis, and treatment of rare diseases than for common conditions due to a variety of cultural, socioeconomic, environmental and other factors ⁶⁸.

The local PI at each site will support identification of eligible health professional participants. They will be approached for participation and those that consent to take part in the study will be tasked with identifying suitable patient/parent participants. Observations of a given professional will take place no more than once to ensure maximum variation in participants. We will aim to observe ~20 consent appointments and similarly ~20 results return appointments (with different participants to those observed during the consent process) in line with previous research ¹⁴.

Study 3: Identify parent-reported experiences and outcomes from genomic testing in the GMS (Years 2–4)

Study design. Surveys and interviews will be conducted with parents of children with rare diseases to evaluate parent-reported experiences and outcomes from genomic testing in the GMS.

An online survey will be administered at two time-points; after pre-test counselling (T1) and approximately 12 months after results-disclosure (T2). Our primary outcome measure is decisional regret at T2, as measured on the validated Decisional Regret scale ⁶⁹. In particular, we will compare whether decisional regret differs between parents of patients who get a diagnostic result compared with those that get a no primary findings result. Whilst there is limited data on the psychological effects of disclosing genomic sequencing results to parents of paediatric patients, a number of studies have shown that a subset of parents may be likely to experience decisional regret ^37,
70 and that regret may be linked to parents interpretation of the child’s result as negative and of frustration with uncertain results ⁷⁰. Key research questions will be whether there is a difference in levels of decisional regret depending on result status, and whether there is a difference in levels of decisional regret depending on clinical indication.

Secondary outcomes include knowledge ⁷¹, attitudes ¹⁸ (adapted from previous research ⁷²), self-reported informed decision-making ¹⁸, decisional conflict ⁷³, generalised anxiety ⁷⁴ parental empowerment ⁷⁵, health-related quality of life of the child ⁷⁶, family impact ⁷⁷, psychological impact ⁷⁸ and satisfaction with appointment ⁷⁹ (see Table 1). We will explore whether parent characteristics e.g. education, ethnicity, and personality traits e.g. intolerance for uncertainty ⁸⁰ and resilience ⁸¹ are associated with particular psychological outcomes.

Table 1. Summary of survey measures.

Survey domain	Description	Time 1	Time 2
Participant characteristics and personality traits
Participant characteristics	Child age, parent/carer age, gender, education, number of children, ethnicity, religion and religiosity, income	✓	X
General anxiety	Generalised Anxiety Disorder Questionnaire (GAD-7). A seven-item measure for screening and severity measuring generalised anxiety disorder. Items are rated on a 4-point Likert scale ⁷⁴	✓	✓
Resilience	Brief resilience scale. A six-item measure for assessing the ability to bounce back or recover from stress. Items are rated on a 5-point Likert scale ⁸¹.	✓	X
Intolerance for Uncertainty	Short version of the Intolerance for Uncertainty scale. A 12-item measure for assessing intolerance for uncertainty. Items are rated on a 5-point Likert scale ⁸⁰.	✓	X
Attributes of informed decision-making
Knowledge	Nine-item knowledge of genome sequencing (KOGS) measure that is context-neutral and focuses on what is involved in having genome sequencing (including ‘what is a genome’), and the limitations and uncertainties of genome sequencing. Each statement is rated as either true, false or don’t know ⁷¹. In addition, we will include a number of knowledge items developed specifically for the this study which relate to the way that the Service is being offered.	✓	X
Attitude	Five-item scale examining general attitudes to genome sequencing e.g. harmful – beneficial, unimportant – important, measured on a five-point Likert scale ¹⁸.	✓	✓
Self-reported informed decision-making	Question used previously in survey on genome sequencing in the 100,000 Genomes Project ¹⁸.	✓	X
Decisional conflict	Sixteen-item measure with five-point Likert scale which assess decisional certainty or conflict about a healthcare decision ⁷³	✓	X
Decisional-regret	Five-item measure with five-point likert scale which assesses regret or remorse about a healthcare decision, with scores ranging from 0 to 100. DRS scores can be defined into three categories: no decision regret (DRS score 0), mild decision regret (DRS score 1–25), and moderate to high decision regret (DRS score >25) ⁶⁹.	X	✓
Test results	Study specific question to assess what result the patient received (a diagnostic result, a no- findings result or an uncertain result)	X	✓
Clinical, psychosocial and behavioural outcomes
Parental empowerment	Genomics Outcome Scale: six-item questionnaire with five-point likert scale which captures the theoretical construct of empowerment relating to genomic medicine ⁷⁵	✓	✓
Health-related quality of life (child)	EQ-5D-Y (ages 4–15): Comprises five dimensions: mobility, looking after myself, doing usual activities, having pain or discomfort and feeling worried, sad or unhappy. Each dimension has 3 levels: no problems, some problems and a lot of problems. The caregiver (the proxy) is asked to rate the child’s/ adolescent’s health-related quality of life in their (the proxy’s) opinion ⁷⁶.	✓	✓
Psychological impact	Adapted 12-item version of the Feelings About genomic Testing Results (FACToR) with five- point Likert scale which measures the specific impact of result disclosure after genomic testing ⁷⁸	X	✓
Family impact	PEDS-QL Family impact module: sixteen-item questionnaire with five-point Likert Scale which explores problems with communication, worry, daily activities, family relationships ⁷⁷	✓	✓
Clinical, social and behavioural impact of results	Study specific questions which explore: changes to clinical management, understanding the likely course of the condition, changes to child’s/parent’s lifestyle, connecting with specific rare disease support groups/other families, communication with medical professionals, reproductive decision-making and identification of other at-risk family members. Each item will have 5 levels (not at all – a great deal).	X	✓
Satisfaction with appointment	Seven-item patient-satisfaction measure for use in a clinical genetics setting ⁷⁹	✓	✓

Open in a new tab

To complement the quantitative results, a subset of survey responders will also be invited for a qualitative interview. Interviews will focus on parents’ expectations, experiences of and satisfaction with the consent appointment/return of results, perception of care received, clinical, behavioural, and psychosocial impact of the result, unexpected outcomes, and recommendations for service improvement.

Data analysis. This mixed methods study will use a concurrent design with quantitative and qualitative data collected in parallel and given equal status, the purpose being to seek a more complete understanding using complementary methods ⁸². Qualitative and quantitative data will be analysed separately and integrated at the point of interpretation. Each set of findings will be brought together into one explanatory framework ⁸².

For the quantitative data, frequencies, means and standard deviations will be calculated and descriptive statistics will be reported. Correlations and comparative analyses will be conducted to identify changes over time (between T1 and T2). We will conduct correlations and t-tests (normally distributed variables) or Spearman’s rank correlation and paired-Wilcoxon signed ranks tests (non-normally distributed variables) to examine bivariate associations between the primary dependent variables and participant characteristics (e.g. gender, age, employment, education, ethnicity, resilience etc). Analysis will be facilitated using SPSS software ⁸³.

Qualitative data will be analysed using codebook thematic analysis ⁸⁴. This is a flexible analytic method where a codebook with both deductive (guided by theory and/or previous literature) and inductive (emerging from the text) codes are used to guide data coding, and allows for multiple researchers to systematically code the text. Codes are then collated to form sub-themes and themes, patterns of meaning anchored by a shared idea or concept. Analysis will be facilitated using Nvivo ⁶¹.

Recruitment and sample size. Survey participants will be recruited from across the seven participating recruitment sites with the aim of recruiting participants from different geographical, ethnic and socio-economic backgrounds. We will recruit participants whose children have different clinical indications (e.g. neurological including intellectual disability, developmental delay and/or epilepsy, renal, cardiac) as well as those with single system (e.g. a heart defect) and multisystem (e.g. a kidney and heart defect) to facilitate exploratory comparisons across disease groups. Potential participants will be identified by either the health professional conducting the consent discussion or a research co-ordinator and sent an information sheet and copy of the survey (including an online link). Each survey will be given a unique identifier to track which site responders were recruited from as well as the clinical indication. Interview participants will be selectively sampled for maximum variation in terms of condition, result (diagnostic, negative or inconclusive) and socio-demographic factors. Where both parents attended the initial genomic testing appointment, only one parent per family (‘the main care-giver’) will be invited to complete the survey in order to avoid non-independence of results as family members may influence each other’s responses.

To compare decision regret between those parents of patients who received a diagnostic result and those who don’t, a minimum of 67 participants are required in both groups to achieve a medium effect size (0.5) with an 80% power level. As diagnostic rates using genomic testing are currently around 40% when trio-based analysis is performed ²⁶, a minimum of 168 participants is required. To account for drop-out between the T1 and T2 surveys, which was around 50% in previous research ¹⁸, we will aim to recruit around 400 participants at T1. We are aiming for around 60 completed surveys per site at T1 so that comparison across sites can be made.

Recruitment for interviews will continue until saturation is reached, however we aim to interview around 20–30 parents at both timepoints. This is in line with previous qualitative interview studies exploring parental experiences of genomic testing ¹³. Ideally, the same parents will take part in interviews across the two timepoints to examine the patient journey including parent expectations and outcomes.

Study 4: Obtain perspectives including readiness, process and outcomes of different medical specialists (Year 4)

Study design. Cross-sectional qualitative interviews will be conducted with non-genetic medical specialists to explore their experiences of current genomic practice. The topic guide will be informed through the CFIR and Proctor’s taxonomy, and include questions to assess their preparedness for delivering genomic medicine (consenting patients and delivering results), technical and infrastructure support available, how genomic medicine fits into their current practice including impact on care delivered, appropriateness of genomic test directory, outstanding education and training needs, interaction with genetic specialists, whether the nature of their clinical interactions with patients and families has changed over time, and to identify policy and/or service provider factors affecting ‘mainstream’ implementation of genomic medicine, including emergent enablers and barriers. The findings from the interviews will be used to inform the development of an anonymous cross-sectional online survey, which will also use validated measures to assess concepts such as acceptability ⁸⁵, feasibility ⁸⁵, implementation leadership support ⁸⁶ and organisational change expectations ⁸⁷. Survey data will provide evidence to policy makers about the effectiveness of mainstreaming.

Data analysis. Qualitative data analysis will be thematically coded ⁸⁴ using a codebook approach. The first step will consist of a deductive analysis, where data are mapped on to the CFIR and Proctor domains and constructs. This will be followed by an inductive analysis, where new themes or unexpected findings are elicited through coding and categorising. Quantitative data will be analysed using descriptive statistics.

Recruitment and sample size. Medical specialists from a chosen set of four to five specialties who are (expected to be) involved in the mainstreaming of genomic medicine (e.g. community paediatricians, paediatricians, paediatric neurologists, paediatric cardiologists) will be invited for interview. To reduce risk of bias in terms of approaching those known to the research team (who may be more likely to have positive views towards genomic medicine), we will recruit participants through societies, email distribution lists and social media. Interviews will be conducted until saturation is reached, but we expect to interview around 5–10 per speciality in line with previous qualitative research looking at health professionals’ experiences of offering genomic testing ¹².

We will use a multipronged recruitment strategy to distribute the survey to reduce the risk of bias. The online survey will be administered through GEL as well as with links circulated through health professional associations (e.g. Royal College of Paediatrics and Child Health), hospital newsletters, social media as well as across the seven participating sites. As this is a single topic community study, we will aim to recruit around 400 participants (around 100 for each medical speciality).

Data synthesis and interpretation – Study 5

The findings from the four studies will be analysed separately. However, at the end of the study we will integrate the data to draw overarching conclusions about service provision. Summary tables which capture the process of implementation, participants’ perceptions of the GMS including barriers and facilitators as well as the programme theory at the central and local levels will be developed. Proposed recommendations to address identified barriers will also be included in these tables. To enhance trustworthiness, qualitative data analysis will be conducted by multiple researchers and decisions around data synthesis will be documented. In addition, the advisory team, including the PPI group will support the interpretation of the data and ensure credibility of the data analysis. Decisions around how recommendations are developed will be documented for transparency and a consensus approach will be taken whereby a majority of team members need to be in agreement for a recommendation to be put forward. Further refinement of recommendations for practice will be developed at a workshop in Year 4 with key stakeholders (including clinical and laboratory staff from across the GMSAs and GLHs, policy makers from organisations such as NHSE and GEL and patient group representatives) who will be identified by the research and advisory team. During the workshop we will present the key findings from this body of research alongside the associated draft recommendations developed by the advisory team. Participants will be split into groups to discuss key findings and draft recommendations before reporting back to the plenary. Detailed notes recording the discussion around the refinement and prioritising of recommendations will be taken. These recommendations will be detailed in the final project report.

Ethics and data processing

The research will be conducted in accordance with the UK Policy Framework For Health and Social Care Research which sets out the principles of good practice in the management of research ⁸⁸. Ethical approval for the study was granted on the 16th July 2021 by the London-Bloomsbury Research Ethics Committee (21/PR/0678). Participants (patients, parents, health professionals and/or other key stakeholders) will be given a participant information sheet at the time of being invited to take part in the study. Prior to any observation or interview taking place, consent will be sought and recorded, either verbally (if the observation/interview is taking place virtually) or in written form (if the observation/interview is taking place face-to-face). In the case of verbal consent, each item on the consent form will be read aloud with replies from the participant confirming consent recorded digitally. In addition, an electronic copy of the consent form with the participants name will be signed and dated by the researcher with a copy sent to the participant. For studies 2 and 4, returning a completed survey will be considered implied consent to participate.

Interview data will be digitally recorded, transcribed by a professional transcription company with which a confidentiality agreement is in place. Transcripts will be de-identified and stored along with audio-recordings and de-identified survey responses in the UCL Data Safe Haven which is certified to the ISO27001 information security standard.

Dissemination – Study 6

As well as disseminating results through traditional academic forums such as peer-reviewed publications, we will engage directly with health professionals, policy makers, patients and the public. Crucially, we will disseminate our results to the intervention implementers (e.g. NHS England, Genomic Partnership Board), as findings emerge, so any suggested changes to service provision can be considered in a timely manner. Our results will be shared in the form of short reports and/or slide-sets. We will measure the impact of reporting our findings, i.e. any change that have been made as a result of these findings. We will also share regular study updates via the social media channels and newsletters of patient groups including SWAN UK, Genetic Alliance UK and Unique, who are on the advisory team. At the end of the study we will produce a series of video abstracts aimed at patients and the public to showcase the key findings from the research. We will reach out to those participants that took part in this programme of research, and send them links to these abstracts, so that they can understand the findings from this work.

Discussion

The NHS GMS will undoubtedly improve the diagnosis and management of patients and their families affected by rare genetic diseases, and provide emotional relief for parents who have been searching for answers. Whilst some of the potential issues (educational, logistical etc) have been identified and are being addressed prior to the start of the Service, there will inevitably be unanticipated barriers and challenges along the way. This research programme provides a unique opportunity to holistically evaluate the expectations and outcomes of the NHS GMS for paediatric rare disease diagnosis, and provide insights and recommendations to improve service delivery. It will also add to our understanding of the experience of parents undergoing and health professionals delivering genomic testing in routine clinical practice.

Our mixed-methods approach will provide rich, comprehensive insights into the facilitators, challenges and barriers of delivering the NHS GMS. Examining both parents’ and health professionals’ experiences will ensure that experiences and outcomes are explored from multiple perspectives. In designing this study, we have engaged with patients as well as other key stakeholders such as health professionals and policy makers at inception to ensure the research will provide important insights for service improvement and to increase the likelihood that the recommendations will be adopted by policy makers. Our advisory team also comprises a broad range of expertise across genomics including geneticists, genetic counsellors, clinical scientists, behavioural scientists, ethicists, health economists and policy makers who can provide critical insight into the study findings and ensure they are fed back to relevant parties in a timely manner. A key challenge for the project is that there are multiple sub-studies that require buy-in from health professionals across a range of specialties. Moreover, the covid pandemic has meant that many research projects are taking longer to get approved and there have been delays in getting the NHS GMS up and running.

In a recent strategy for genomics set out by the Department of Health, the Minister for Innovation wrote that “the biggest gains are being made through collaborations across a range of expertise from clinicians, engineers, social scientists, mathematicians, and data scientists.” ³. The NHS GMS provides an ideal opportunity to use approaches from social and behavioural science to examine implementation, experiences and outcomes of service providers, patients and other key stakeholders.

This work will provide important evidence for both the NHS and other countries implementing genomics into their national healthcare systems. The study materials (topic guides, surveys etc) can be used by other researchers examining the implementation and outcomes of offering genomic testing in routine clinical practice. Moreover, research findings (both qualitative and quantitative) from this body of work can be compared with those from other national genomic implementation programmes to understand where commonalities and differences exist. In this way, the research community can begin to build a coherent and comprehensive picture of the implementation of genomic medicine in clinical practice.

Data availability

No data are associated with this article. Anonymised data underlying the results will be hosted in the UCL Data Repository and a DOI will be referenced in research publications.

Reporting guidelines

The checklist for the protocol can be found at the following link:

Lewis, Celine (2021): SRQR Checklist for protocol paper. University College London. Figure. https://doi.org/10.5522/04/16847794

Data are available under the terms of the CC BY 4.0 which ensures that research is openly available whilst still ensuring that others give credit, in the form of a citation, should they use or refer to the research object. This license lets others distribute, remix, tweak, and build upon the work (even commercially) as long as they credit the original creation.

Award information

This research is funded through an NIHR Advanced Fellowship Grant (NIHR300099), awarded to Celine Lewis.

Acknowledgements

Heartfelt thanks to Dr Stephanie Best from Macquarie University, Sydney, Australia and Australian Genomics, for her valuable feedback on a draft of this protocol. Thanks also to Dr Melissa Martyn and Professor Clara Gaff from Melbourne Genomics, Australia, for sharing their views and experiences regarding which measures to include in the parent survey.

Funding Statement

This research is funded through an NIHR Advanced Fellowship Grant (NIHR300099), awarded to Celine Lewis. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

[version 2; peer review: 3 approved, 1 approved with reservations]

References

1. NHS England: National Genomic Test Directory.2019; [Accessed 5th August 2019]. Reference Source [Google Scholar]
2. Genomics England: The 100,000 Genomes Project Protocol.Genomics England Ltd.: London.2017. Reference Source [Google Scholar]
3. HM Government: Genome UK: the future of healthcare.2020. Reference Source [Google Scholar]
4. Robinson J: Everything you need to know about the NHS genomic medicine service.2020; [Accessed 25th October 2021]. Reference Source [Google Scholar]
5. Health Education England: Master's in Genomic Medicine.2020; [Accessed 28th September 2020]. Reference Source [Google Scholar]
6. Royal College of General Practitioners: Genomics toolkit.[Accessed 28th September 2020]. Reference Source [Google Scholar]
7. NHS England: Genomics resources. [Accessed 1st September 2021]. Reference Source [Google Scholar]
8. NHS England: Whole genome sequencing for rare disease: Information for patients and family members.NHS England,2020. Reference Source [Google Scholar]
9. NHS England: Whole genome sequencing for a rare disorder: easy read.NHS England.2020. Reference Source [Google Scholar]
10. NHS Genomic Medicine Service: Record of Discussion Regarding Genomic Testing.North Bristol NHS Trust, Editor. Reference Source [Google Scholar]
11. Genomics England: Taking part. [Accessed 14th June 2021]. Reference Source [Google Scholar]
12. Sanderson SC, Hill M, Patch C, et al. : Delivering genome sequencing in clinical practice: an interview study with healthcare professionals involved in the 100 000 Genomes Project. BMJ Open. 2019;9(11):e029699. 10.1136/bmjopen-2019-029699 [DOI] [PMC free article] [PubMed] [Google Scholar]
13. Lewis C, Sanderson S, Hill M, et al. : Parents' motivations, concerns and understanding of genome sequencing: a qualitative interview study. Eur J Hum Genet. 2020;28(7):874–884. 10.1038/s41431-020-0575-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
14. Sanderson SC, Lewis C, Patch C, et al. : Opening the "black box" of informed consent appointments for genome sequencing: a multisite observational study. Genet Med. 2019;21(5):1083–1091. 10.1038/s41436-018-0310-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
15. Dheensa S, Lucassen A, Fenwick A: Fostering trust in healthcare: Participants' experiences, views, and concerns about the 100,000 genomes project. Eur J Med Genet. 2019;62(5): 335–341. 10.1016/j.ejmg.2018.11.024 [DOI] [PubMed] [Google Scholar]
16. Ormondroyd E, Mackley MP, Blair M, et al. : "Not pathogenic until proven otherwise": perspectives of UK clinical genomics professionals toward secondary findings in context of a Genomic Medicine Multidisciplinary Team and the 100,000 Genomes Project. Genet Med. 2018;20(3):320–328. 10.1038/gim.2017.157 [DOI] [PMC free article] [PubMed] [Google Scholar]
17. Ballard LM, Horton RH, Dheensa S, et al. : Exploring broad consent in the context of the 100,000 Genomes Project: a mixed methods study. Eur J Hum Genet. 2020;28(6):732–741. 10.1038/s41431-019-0570-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
18. Sanderson SC, et al. : Decision-making, attitudes and understanding amongst patients and relatives invited to undergo genome sequencing in the 100,000 Genomes Project: a multi-site survey study. Genetics In Medicine. In press. [DOI] [PubMed] [Google Scholar]
19. Hussey PS, Anderson GF, Osborn R, et al. : How does the quality of care compare in five countries? Health Aff (Millwood). 2004;23(3):89–99. 10.1377/hlthaff.23.3.89 [DOI] [PubMed] [Google Scholar]
20. Damschroder LJ, Aron DC, Keith RE, et al. : Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. 10.1186/1748-5908-4-50 [DOI] [PMC free article] [PubMed] [Google Scholar]
21. Hulscher ME, Laurant MG, Grol RP: Process evaluation on quality improvement interventions. Qual Saf Health Care. 2003;12(1):40–6. 10.1136/qhc.12.1.40 [DOI] [PMC free article] [PubMed] [Google Scholar]
22. Stetler CB, Legro MW, Wallace CM, et al. : The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21 Suppl 2(Suppl 2):S1–S8. 10.1111/j.1525-1497.2006.00355.x [DOI] [PMC free article] [PubMed] [Google Scholar]
23. European Organisation for Rare Diseases: Rare Diseases: Understanding this Public Health Priority.2005; [cited Accessed 25 August 2021]. Reference Source [Google Scholar]
24. Blöß S, Klemann C, Rother AK, et al. : Diagnostic needs for rare diseases and shared prediagnostic phenomena: Results of a German-wide expert Delphi survey. PLoS One. 2017;12(2):e0172532. 10.1371/journal.pone.0172532 [DOI] [PMC free article] [PubMed] [Google Scholar]
25. Vissers LELM, van Nimwegen KJM, Schieving JH, et al. : A clinical utility study of exome sequencing versus conventional genetic testing in pediatric neurology. Genet Med. 2017;19(9):1055–1063. 10.1038/gim.2017.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
26. Wright CF, FitzPatrick DR, Firth HV: Paediatric genomics: diagnosing rare disease in children. Nat Rev Genet. 2018;19(5):253–268. 10.1038/nrg.2017.116 [DOI] [PubMed] [Google Scholar]
27. Griffin BH, Chitty LS, Bitner-Glindzicz M: The 100 000 Genomes Project: What it means for paediatrics. Arch Dis Child Educ Pract Ed. 2017;102(2):105–107. 10.1136/archdischild-2016-311029 [DOI] [PubMed] [Google Scholar]
28. Ashtiani S, Makela N, Carrion P, et al. : Parents' experiences of receiving their child's genetic diagnosis: a qualitative study to inform clinical genetics practice. Am J Med Genet A. 2014;164A(6):1496–502. 10.1002/ajmg.a.36525 [DOI] [PMC free article] [PubMed] [Google Scholar]
29. Werner-Lin A, Zaspel L, Carlson M, et al. : Gratitude, protective buffering, and cognitive dissonance: How families respond to pediatric whole exome sequencing in the absence of actionable results. Am J Med Genet A. 2018;176(3):578–588. 10.1002/ajmg.a.38613 [DOI] [PubMed] [Google Scholar]
30. Tolusso LK, Collins K, Zhang X, et al. : Pediatric Whole Exome Sequencing: an Assessment of Parents' Perceived and Actual Understanding. J Genet Couns. 2017;26(4):792–805. 10.1007/s10897-016-0052-9 [DOI] [PubMed] [Google Scholar]
31. Bernhardt BA, Roche MI, Perry DL, et al. : Experiences with obtaining informed consent for genomic sequencing. Am J Med Genet A. 2015;167A(11):2635–46. 10.1002/ajmg.a.37256 [DOI] [PMC free article] [PubMed] [Google Scholar]
32. Tomlinson AN, Skinner D, Perry DL, et al. : "Not Tied Up Neatly with a Bow": Professionals' Challenging Cases in Informed Consent for Genomic Sequencing. J Genet Couns. 2016;25(1):62–72. 10.1007/s10897-015-9842-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
33. Shickh S, Mighton C, Uleryk E, et al. : The clinical utility of exome and genome sequencing across clinical indications: a systematic review. Hum Genet. 2021;140(10):1403–1416. 10.1007/s00439-021-02331-x [DOI] [PubMed] [Google Scholar]
34. Smith HS, Swint JM, Lalani SR, et al. : Clinical Application of Genome and Exome Sequencing as a Diagnostic Tool for Pediatric Patients: a Scoping Review of the Literature. Genet Med. 2019;21(1):3–16. 10.1038/s41436-018-0024-6 [DOI] [PubMed] [Google Scholar]
35. Mestek-Boukhibar L, Clement E, Jones WD, et al. : Rapid Paediatric Sequencing (RaPS): comprehensive real-life workflow for rapid diagnosis of critically ill children. J Med Genet. 2018;55(11):721–728. 10.1136/jmedgenet-2018-105396 [DOI] [PMC free article] [PubMed] [Google Scholar]
36. Wright CF, Fitzgerald TW, Jones WD, et al. : Genetic diagnosis of developmental disorders in the DDD study: a scalable analysis of genome-wide research data. Lancet. 2015;385(9975):1305–14. 10.1016/S0140-6736(14)61705-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
37. Brett GR, Martyn M, Lynch F, et al. : Parental experiences of ultrarapid genomic testing for their critically unwell infants and children. Genet Med. 2020;22(12):1976–1985. 10.1038/s41436-020-0912-4 [DOI] [PubMed] [Google Scholar]
38. Robinson JO, Wynn J, Biesecker B, et al. : Psychological outcomes related to exome and genome sequencing result disclosure: a meta-analysis of seven Clinical Sequencing Exploratory Research (CSER) Consortium studies. Genet Med. 2019;21(12):2781–2790. 10.1038/s41436-019-0565-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
39. Rosell AM, Pena LD, Schoch K, et al. : Not the End of the Odyssey: Parental Perceptions of Whole Exome Sequencing (WES) in Pediatric Undiagnosed Disorders. J Genet Couns. 2016;25(5):1019–31. 10.1007/s10897-016-9933-1 [DOI] [PubMed] [Google Scholar]
40. Krabbenborg L, Vissers LE, Schieving J, et al. : Understanding the Psychosocial Effects of WES Test Results on Parents of Children with Rare Diseases. J Genet Couns. 2016;25(6):1207–1214. 10.1007/s10897-016-9958-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
41. Barwell J, Snape K, Wedderburn S: The new genomic medicine service and implications for patients. Clin Med (Lond). 2019;19(4):273–277. 10.7861/clinmedicine.19-4-273 [DOI] [PMC free article] [PubMed] [Google Scholar]
42. Weipert CM, Ryan KA, Everett JN, et al. : Physician Experiences and Understanding of Genomic Sequencing in Oncology. J Genet Couns. 2018;27(1):187–196. 10.1007/s10897-017-0134-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
43. Aiello LB: Genomics Education: Knowledge of Nurses Across the Profession and Integration Into Practice. Clin J Oncol Nurs. 2017;21(6):747–753. 10.1188/17.CJON.747-753 [DOI] [PubMed] [Google Scholar]
44. McClaren BJ, Crellin E, Janinski M, et al. : Preparing Medical Specialists for Genomic Medicine: Continuing Education Should Include Opportunities for Experiential Learning. Front Genet. 2020;11:151. 10.3389/fgene.2020.00151 [DOI] [PMC free article] [PubMed] [Google Scholar]
45. Salm M, Abbate K, Appelbaum P, et al. : Use of genetic tests among neurologists and psychiatrists: knowledge, attitudes, behaviors, and needs for training. J Genet Couns. 2014;23(2):156–63. 10.1007/s10897-013-9624-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
46. Wynn J, Lewis K, Amendola LM, et al. : Clinical providers' experiences with returning results from genomic sequencing: an interview study. BMC Med Genomics. 2018;11(1):45. 10.1186/s12920-018-0360-z [DOI] [PMC free article] [PubMed] [Google Scholar]
47. Proctor E, Silmere H, Raghavan R, et al. : Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76. 10.1007/s10488-010-0319-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
48. Moran-Ellis J, Alexander VD, Cronin A, et al. : Triangulation and integration: processes, claims and implications. Qual Res. 2006;6(1):45–59. 10.1177/1468794106058870 [DOI] [Google Scholar]
49. Creswell JW: Research design. Qualitative, quantitative, and mixed methods approaches.2nd ed. ed. Thousand Oaks, California: Sage,2003. Reference Source [Google Scholar]
50. Weaver K: Pragmatic Paradigm.In The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation.B.B. Frey, Editor. SAGE Publications, Inc.: Thousand Oaks,2018. 10.4135/9781506326139.n534 [DOI] [Google Scholar]
51. Orlando LA, Sperber NR, Voils C, et al. : Developing a common framework for evaluating the implementation of genomic medicine interventions in clinical care: the IGNITE Network's Common Measures Working Group. Genet Med. 2018;20(6):655–663. 10.1038/gim.2017.144 [DOI] [PMC free article] [PubMed] [Google Scholar]
52. Best S, Brown H, Lunke S, et al. : Learning from scaling up ultra-rapid genomic testing for critically ill children to a national level. NPJ Genom Med. 2021;6(1):5. 10.1038/s41525-020-00168-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
53. Levy KD, Blake K, Fletcher-Hoppe C, et al. : Opportunities to implement a sustainable genomic medicine program: lessons learned from the IGNITE Network. Genet Med. 2019;21(3):743–747. 10.1038/s41436-018-0080-y [DOI] [PMC free article] [PubMed] [Google Scholar]
54. Zebrowski AM, Ellis DE, Barg FK, et al. : Qualitative study of system-level factors related to genomic implementation. Genet Med. 2019;21(7):1534–1540. 10.1038/s41436-018-0378-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
55. Keith RE, Crosson JC, O'Malley AS, et al. : Using the Consolidated Framework for Implementation Research (CFIR) to produce actionable findings: a rapid-cycle evaluation approach to improving implementation. Implement Sci. 2017;12(1):15. 10.1186/s13012-017-0550-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
56. West R, Michie S, Atkins L, et al. : Achieving behaviour change: A guide for local government and partners.PHE publications: London,2019. Reference Source [Google Scholar]
57. National Institute for Health Research: Centre for Engagement and Dissemination - Recognition payments for public contributors.2016; Accessed 25th Otober 2021]. Reference Source [Google Scholar]
58. PHG Foundation: Genomics in Mainstream Medicine Working Group.2017; Accessed 16th July 2020]. Reference Source [Google Scholar]
59. W.K. Kellogg Foundation: Logic Model Development Guide.W.K. Kellogg Foundation: Battle Creek, Michigan,2004. Reference Source [Google Scholar]
60. Gale NK, Heath G, Cameron E, et al. : Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13:117. 10.1186/1471-2288-13-117 [DOI] [PMC free article] [PubMed] [Google Scholar]
61. QSR International Pty Ltd: Nvivo. (released in March 2020).2020. Reference Source [Google Scholar]
62. Sadler GR, Lee HC, Lim RS, et al. : Recruitment of hard-to-reach population subgroups via adaptations of the snowball sampling strategy. Nurs Health Sci. 2010;12(3):369–374. 10.1111/j.1442-2018.2010.00541.x [DOI] [PMC free article] [PubMed] [Google Scholar]
63. Angrosino M, Rosenberg J: Observations on observation.In Collecting and Interpreting Qualitative Materials. N.K. Denzin and Y.S. Lincoln, Editors. Sage: Thousand Oaks, CA,2012. Reference Source [Google Scholar]
64. Holtrop JS, Rabin BA, Glasgow RE: Qualitative approaches to use of the RE-AIM framework: rationale and methods. BMC Health Serv Res. 2018;18(1):177. 10.1186/s12913-018-2938-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
65. Atkinson P, Housley W: Interactionism.London: Sage,2003. Reference Source [Google Scholar]
66. Hsieh HF, Shannon SE: Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88. 10.1177/1049732305276687 [DOI] [PubMed] [Google Scholar]
67. Braun V, Clarke V: Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101. 10.1191/1478088706qp063oa [DOI] [Google Scholar]
68. National Centre for Advancing Translational Sciences: Annual Report on the Rare Diseases and Conditions Research Activities of the National Institutes of Health FY 2000.2005. Reference Source [Google Scholar]
69. O'Connor A: User Manual – Decision Regret Scale Ottawa: Ottawa Hospital Research Institute. (modified 2003)1996; [Accessed 16 July 2013]. Reference Source [Google Scholar]
70. Wynn J, Ottman R, Duong J, et al. : Diagnostic exome sequencing in children: A survey of parental understanding, experience and psychological impact. Clin Genet. 2018;93(5):1039–1048. 10.1111/cge.13200 [DOI] [PMC free article] [PubMed] [Google Scholar]
71. Sanderson SC, Loe BS, Freeman M, et al. : Development of the Knowledge of Genome Sequencing (KOGS) questionnaire. Patient Educ Couns. 2018;101(11):1966–1972. 10.1016/j.pec.2018.07.011 [DOI] [PubMed] [Google Scholar]
72. Marteau TM, Dormandy E, Michie S: A measure of informed choice. Health Expect. 2001;4(2):99–108. 10.1046/j.1369-6513.2001.00140.x [DOI] [PMC free article] [PubMed] [Google Scholar]
73. O'Connor A: User Manual – Decisional Conflict Scale.(16 item statement format).1993; (updated 2010) [Accesed 16 July 2013]. Reference Source [Google Scholar]
74. Spitzer RL, Kroenke K, Williams JB, et al. : A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006;166(10):1092–7. 10.1001/archinte.166.10.1092 [DOI] [PubMed] [Google Scholar]
75. Grant PE, Pampaka M, Payne K, et al. : Developing a short-form of the Genetic Counselling Outcome Scale: The Genomics Outcome Scale. Eur J Med Genet. 2019;62(5):324–334. 10.1016/j.ejmg.2018.11.015 [DOI] [PubMed] [Google Scholar]
76. Wille N, Badia X, Bonsel G, et al. : Development of the EQ-5D-Y: a child-friendly version of the EQ-5D. Qual Life Res. 2010;19(6):875–86. 10.1007/s11136-010-9648-y [DOI] [PMC free article] [PubMed] [Google Scholar]
77. Varni JW, Sherman SA, Burwinkle TM, et al. : The PedsQL Family Impact Module: preliminary reliability and validity. Health Qual Life Outcomes. 2004;2:55. 10.1186/1477-7525-2-55 [DOI] [PMC free article] [PubMed] [Google Scholar]
78. Li M, Bennette CS, Amendola LM, et al. : The Feelings About genomiC Testing Results (FACToR) Questionnaire: Development and Preliminary Validation. J Genet Couns. 2019;28(2):477–490. 10.1007/s10897-018-0286-9 [DOI] [PubMed] [Google Scholar]
79. Zellerino B, Milligan SA, Brooks R, et al. : Development, testing, and validation of a patient satisfaction questionnaire for use in the clinical genetics setting. Am J Med Genet C Semin Med Genet. 2009;151c(3):191–9. 10.1002/ajmg.c.30214 [DOI] [PubMed] [Google Scholar]
80. Carleton RN, Norton MA, Asmundson GJ: Fearing the unknown: a short version of the Intolerance of Uncertainty Scale. J Anxiety Disord. 2007;21(1):105–17. 10.1016/j.janxdis.2006.03.014 [DOI] [PubMed] [Google Scholar]
81. Smith BW, Dalen J, Wiggins K, et al. : The brief resilience scale: assessing the ability to bounce back. Int J Behav Med. 2008;15(3):194–200. 10.1080/10705500802222972 [DOI] [PubMed] [Google Scholar]
82. Greene JC, Caracelli VJ, Graham WF: Toward a Conceptual Framework for Mixed-Method Evaluation Designs. Educ Eval Policy Anal. 1989;11(3):255–274. 10.3102/01623737011003255 [DOI] [Google Scholar]
83. Corp I: IBM SPSS Statistics for Windows, Version 27.0.Armonk, NY: IBM Corp,2020. Reference Source [Google Scholar]
84. Braun V, Clarke V: Conceptual and Design Thinking for Thematic Analysis. Qual Psychol. 2021. 10.1037/qup0000196 [DOI] [Google Scholar]
85. Weiner BJ, Lewis CC, Stanick C, et al. : Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108. 10.1186/s13012-017-0635-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
86. Aarons GA, Ehrhart MG, Farahnak LR: The Implementation Leadership Scale (ILS): development of a brief measure of unit level implementation leadership. Implement Sci. 2014;9(1):45. 10.1186/1748-5908-9-45 [DOI] [PMC free article] [PubMed] [Google Scholar]
87. Armenakis AA, Bernerth JB, Pitts JP, et al. : Organizational change recipients' beliefs scale: Development of an assessment instrument. J Appl Behav Sci. 2007;43(4):481–505. 10.1177/0021886307303654 [DOI] [Google Scholar]
88. NHS Health Research Authority: UK policy framework for health and social care research.2017. Reference Source [Google Scholar]

NIHR Open Res. 2022 Feb 28. doi: 10.3310/nihropenres.14382.r28507

Reviewer response for version 2

Julian Barwell ¹, Maurice Dungey ²

We are happy to proceed with this project given the responses and revisions given.

Is the study design appropriate for the research question?

Partly

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Partly

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

Genomic Medicine

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

NIHR Open Res. 2022 Feb 24. doi: 10.3310/nihropenres.14382.r28509

Reviewer response for version 2

Zornitza Stark ¹

No further comments.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

Clinical genomics, health economics, implementation science, bioethics

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

NIHR Open Res. 2021 Dec 22. doi: 10.3310/nihropenres.14352.r28442

Reviewer response for version 1

Lotte Krabbenborg ¹, Karine Wendrich ²

General:

This article presents the study protocol of a mixed-methods study to evaluate the NHS Genomic Medicine Service for paediatric rare diseases. This is a highly relevant study and we like the mixed-methods approach (surveys, interviews, observations) with the inclusion of different stakeholder groups (designers/implementers, healthcare providers and patients). This way, a variety of important topics and perspectives are covered. The study protocol seems to be very well thought through. However, we would recommend some adaptations to how the protocol is currently presented in the article. According to us, the main points of improvement are a better substantiation of certain decisions and more attention for the broader relevance of this study, as further elaborated in the comments.

Major comments:

At the start and the end of the article, we would like this study to be placed in a broader context than the NHS Genomic Medicine Service alone, in order to show the relevance of this study beyond the NHS GMS. For instance, what could other healthcare programs or the field of implementation science gain from your study?

As an example, in the description of study 1, the authors rightly state:

“Implementation science, the systematic study of methods that support the application of research findings and other evidence-based knowledge into policy and practice, is increasingly being seen as playing a critical role health services research ^47,54. Previous research on new interventions has highlighted that as well as assessing outcomes, it is valuable to look at the process of the intervention as this can shed light on the mechanisms responsible for whether and how successful it is ⁵⁵. This formative work can also enable researchers to suggest ways to answer questions about how interventions might be adapted and respond to change in order to produce positive outcomes ⁵⁶.”

This kind of information should be presented upfront, already in the Introduction, to highlight the importance of the current study. Similarly, in the Discussion section at the end of the article, the authors only talk about the NHS GMS, whereas this would be a good place to discuss how this study can have a broader contribution, beyond the current study case.
The Introduction presents a lot of information. This information is concise and well written, but to us it feels like an overload of information without a clear focus. Related to our previous comment, we suggest the authors to first present some general information about the GM, followed by the relevance and the focus of the current study. Next, the authors could have a “background information” section, which discusses the previous research on topics that are relevant for the study.
As the authors mention, the GMS will be implemented for rare diseases and cancer. Why do the authors only focus on paediatric diseases in this study? We would say that it is highly interesting to compare the implementation of the GMS for (paediatric) rare diseases and cancer. So, why has the decision been made to focus on paediatric rare diseases?

Minor comments:

When presenting the research approach and conceptual framework, the authors mention that they use a theory-driven approach. We would like the authors to explain why they have chosen these theories. Why are these the most suitable, given the aim of this study? Moreover, the authors could consider to shortly explain the different domains of The Consolidated Framework for Implementation Research and Proctor’s taxonomy, as now it remains quite abstract.
The figures are really nice, providing a clear schematic overview of the study and the different sub-studies. In Figures 1 and 2, we recommend the authors to change the titles of studies 3 and 4. Based on the titles in these figures, it is not clear that study 3 is about parents and study 4 is about healthcare providers. As we see it, both studies are mixed methods with parents and healthcare providers respectively. So, our suggestion is to rename study 3 something like: “mixed methods parental perspective” and study 4: “mixed methods medical specialist perspective”.
Why are the stakeholder interviews and document analysis in study 1 only performed in year 1 and 4 of the study? We can imagine that adding another timepoint in the middle of the study period would allow for gaining more insight into the implementation dynamics.
Regarding the recruitment of patients and healthcare providers in studies 2, 3 and 4: how will you gain access to these patients and healthcare providers? Will this be done through the principal investigator of each of the seven study sites?
We really like the idea of a workshop with key stakeholders at the end of the study to further refine the recommendations for practice. Maybe you can include some additional information about this workshop? What kind of stakeholders will participate, how will they be recruited? Will this workshop also be recorded and analysed for research purposes?
In study 3, respondents seem to be exposed to a lot of surveys (nicely summarize in Figure 3), which can place a high burden on the respondents. It is our understanding that the main focus of study 3 is on decisional regret. The authors mention a lot of secondary outcomes. Are all these surveys and secondary outcomes really necessary? You should not just collect as much data as possible, “just because you can”, but only in case you really need this information.
The authors state: “Prior to any observation or interview taking place, consent will be sought and recorded, either verbally (if the observation/interview is taking place virtually) or in written form (if the observation/interview is taking place face-to-face).” - Our experience is that written consent can also be obtained in the case of a virtual observation/interview, for instance by sending the consent form to the participant by e-mail. So, we suggest the authors to also obtain written consent for the virtual observations and interviews.

When addressing these comments, we believe the authors will present a nice article about a highly interesting study protocol. We wish the authors the best of luck with the execution of the study protocol.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

Patient participation in research, whole genome / exome sequencing, new medical technologies, ehealth, responsible research and innovation

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.

NIHR Open Res. 2022 Feb 14.

Celine Lewis ¹

We thank both reviewers for their thoughtful and constructive comments on the study protocol. We hope that we have addressed these points to your satisfaction.

General:

Major comments:

At the start and the end of the article, we would like this study to be placed in a broader context than the NHS Genomic Medicine Service alone, in order to show the relevance of this study beyond the NHS GMS. For instance, what could other healthcare programs or the field of implementation science gain from your study?

As an example, in the description of study 1, the authors rightly state:

“Implementation science, the systematic study of methods that support the application of research findings and other evidence-based knowledge into policy and practice, is increasingly being seen as playing a critical role health services research47,54. Previous research on new interventions has highlighted that as well as assessing outcomes, it is valuable to look at the process of the intervention as this can shed light on the mechanisms responsible for whether and how successful it is55. This formative work can also enable researchers to suggest ways to answer questions about how interventions might be adapted and respond to change in order to produce positive outcomes56.”

This kind of information should be presented upfront, already in the Introduction, to highlight the importance of the current study. Similarly, in the Discussion section at the end of the article, the authors only talk about the NHS GMS, whereas this would be a good place to discuss how this study can have a broader contribution, beyond the current study case.

Thank you for these suggestions. We have moved the section on implementation science into the Intro, and added text around the broader contribution of the study beyond the current study case.

The Introduction presents a lot of information. This information is concise and well written, but to us it feels like an overload of information without a clear focus. Related to our previous comment, we suggest the authors to first present some general information about the GM, followed by the relevance and the focus of the current study. Next, the authors could have a “background information” section, which discusses the previous research on topics that are relevant for the study.

We have made the changes as suggested.

As the authors mention, the GMS will be implemented for rare diseases and cancer. Why do the authors only focus on paediatric diseases in this study? We would say that it is highly interesting to compare the implementation of the GMS for (paediatric) rare diseases and cancer. So, why has the decision been made to focus on paediatric rare diseases?

Yes, we agree it would be very interesting to compare the implementation of the GMS for rare diseases with cancer. Unfortunately, with our limited funding and resources (1 x PI – 4 years and 1 x research associate – 2 years) it isn’t possible to do this. As this is NIHR Advanced Fellowship, the PI focused on paediatric rare disease as this is her main area of interest and therefore builds on her previous work.

Minor comments:

When presenting the research approach and conceptual framework, the authors mention that they use a theory-driven approach. We would like the authors to explain why they have chosen these theories. Why are these the most suitable, given the aim of this study? Moreover, the authors could consider to shortly explain the different domains of The Consolidated Framework for Implementation Research and Proctor’s taxonomy, as now it remains quite abstract.

We have stated in the text that we chose the CFIR because it has been used previously to evaluate the implementation of genomic medicine. We have also added that it is suitable to guide development of actionable findings as well as to rapid-cycle evaluations. We have also added in descriptions of the domains.

The figures are really nice, providing a clear schematic overview of the study and the different sub-studies. In Figures 1 and 2, we recommend the authors to change the titles of studies 3 and 4. Based on the titles in these figures, it is not clear that study 3 is about parents and study 4 is about healthcare providers. As we see it, both studies are mixed methods with parents and healthcare providers respectively. So, our suggestion is to rename study 3 something like: “mixed methods parental perspective” and study 4: “mixed methods medical specialist perspective”.

These have been changed as per Reviewer Nina Sperber’s comments and now include the word parent (study 3) and medical specialists (study 4).

Why are the stakeholder interviews and document analysis in study 1 only performed in year 1 and 4 of the study? We can imagine that adding another timepoint in the middle of the study period would allow for gaining more insight into the implementation dynamics.

We agree this would be the ideal, however, it is not going to be possible due to resource limitations.

Regarding the recruitment of patients and healthcare providers in studies 2, 3 and 4: how will you gain access to these patients and healthcare providers? Will this be done through the principal investigator of each of the seven study sites?

The PIs at each recruitment site will support identification of health professionals for study 2 (who will then be tasked with identifying potential patient participants). For study 3, the health professional consenting the patient or the research co-ordinator at each site will identify and send out the survey to potential participants. For study 4, we will adopt a multi-pronged approach to reduce the risk of bias (recruiting through societies, email distribution lists and social media). These recruitment strategies have now been made clearer in the text.

We really like the idea of a workshop with key stakeholders at the end of the study to further refine the recommendations for practice. Maybe you can include some additional information about this workshop? What kind of stakeholders will participate, how will they be recruited? Will this workshop also be recorded and analysed for research purposes?

We have added further detail around who will be invited to take part in the workshop as well as what will happen during the workshop.

In study 3, respondents seem to be exposed to a lot of surveys (nicely summarize in Figure 3), which can place a high burden on the respondents. It is our understanding that the main focus of study 3 is on decisional regret. The authors mention a lot of secondary outcomes. Are all these surveys and secondary outcomes really necessary? You should not just collect as much data as possible, “just because you can”, but only in case you really need this information.

We are currently in the process of piloting the time 1 survey with parents recruited through a support group to ensure the included measures and questions are appropriate and to check the time required to answer the survey. The feedback has been that the survey is clear and answerable and was not considered too long. Many of the survey items were used in our previous study reporting decisions, attitudes and outcomes of patients in the 100,000 Genomes Project [see https://www.gimjournal.org/article/S1098-3600(21)01124-2/fulltext#secsectitle0010 ] where the response rate was over 50%.

We will make clear in the Participant Information Sheet the expected time required to complete the survey. All the items in the survey have been identified by our advisory team (including PPI team) as important constructs in our understanding of the experience, impact and outcomes of genomic testing for patients with rare disease. Many have also been used in other studies which will allow us to compare across projects and countries [see for example https://pubmed.ncbi.nlm.nih.gov/31189963/ ] We have also sought input from our Australian colleagues who are conducting a similar study with patients undergoing genomic testing as to their views on the importance of and experience using the included measures. As a gesture of appreciation for their time, we are also offering all participants a voucher worth £10.

The authors state: “Prior to any observation or interview taking place, consent will be sought and recorded, either verbally (if the observation/interview is taking place virtually) or in written form (if the observation/interview is taking place face-to-face).” - Our experience is that written consent can also be obtained in the case of a virtual observation/interview, for instance by sending the consent form to the participant by e-mail. So, we suggest the authors to also obtain written consent for the virtual observations and interviews.

We want to make the process of consent as easy as possible for the participant. The HRA does not require written consent for non-clinical trials, and where risk to the participant is minimal. In order to ensure that that consent is valid, each item on the consent form will be read aloud and the participant required to say ‘yes’ to each item. Verbal consent will be recorded digitally so there is a clear record of the consent process. In addition, it will be recorded electronically and signed/dated by the researcher and a copy of the electronic record sent to the participant as well as kept in the study file. We have added this additional detail in the manuscript.

NIHR Open Res. 2021 Dec 7. doi: 10.3310/nihropenres.14352.r28439

Reviewer response for version 1

Nina Sperber ^1,²

The authors have presented a mixed-methods protocol to evaluate implementation of genomic medicine with a focus on paediatric rare diseases. This project includes four parts which will each address different levels of implementation. The first part focuses on a program level, aiming to develop a logic model of the NHS GMS to evaluate concordance between intended and actual outcomes and identify barriers to outcomes by examining documents and interviewing key stakeholders involved in delivering services (including program designers, implementers, and providers). The second part focuses on an inter-personal level by observing providers and families (parents and child patients) conducting genomic testing in clinical settings, using a structured guide used in the 100,000 Genomes Project, to denote observations about appointment structure and provider/patient communication, shedding light on processes and practices in naturalistic environments. The third part will target an individual level, focusing on parents’ perspectives about genomic testing with a longitudinal survey and interview data to measure decisional regret, comparing parents who receive a diagnostic result with those who do not. The fourth part will also target an individual level to obtain perspectives of different non-genetic medical specialists about different aspects of implementation including readiness, process, and outcomes using a cross-sectional survey developed with validated measures (acceptability, feasibility, implementation leadership support, and organisational change expectations) and from prior thematic qualitative analysis. Findings from all parts will be integrated to identify overarching findings and recommendations and discussed with key stakeholders.

This paper clearly describes rationale for and objectives of the project. The authors describe a need to understand effects of the relatively newly established Genomic Medicine Service. While some prior work has studied what health care professionals need to integrate genomic medicine into routine care, such as specialized training and access to genomics specialists, gaps exists in our understanding of effective processes and procedures. The authors rationalize their focus on paediatric rare diseases by describing that children make up 50-75% of rare disease cases; they describe a benefit of genomic testing in reducing the “diagnostic odyssey” and need for more information on effects for parents. They outline four objectives commensurate with each of the four parts of the overall design.

The study design is appropriate for the research questions. Generally, they seek to delineate the process of GMS implementation and resultant outcomes, considering different contextual locations and perspectives. Their overall study design of four parts that include multiple types of methods (e.g., document review, individual interviews, participant observations) and mixing of methods (e.g., integrating qualitative and quantitative data about parent-reported experiences or using qualitative data to create a provider survey) will lead them to this end. They have organized the Detailed Study Plan section by research method or technique. For example, they describe the first part (Study 1) as “Implementation interviews with key stakeholders”. While the methods seem appropriate for the questions of each study part, the protocol may be strengthened by leading with objective rather than method. For example, the subheading for Study 1 could be “Specify Programme Theory of the GMS” (or something like that). The methods used to achieve that objective would then be described. This change would emphasize the multiple levels of analysis to capture different facets of program implementation and better follow the way it is outlined in the Background and Aims section.

Details of the analysis are clearly described; however, some questions remain:

There could be more specificity about how CFIR will be used: it is very large and typically requires discussion ahead of time to prioritize constructs for specific studies, especially for qualitative interviews.
More detail is also needed about interviews. How will interview questions be developed? There is a CFIR Guide available for free online. Will those questions be used to ask about CFIR constructs? How long will the interviews last and will they be conducted remotely or in-person? The length of interviews matter because it seems like they could be very long if using CFIR, Proctor, literature, etc. I think more detail in this section would be helpful, including stating if key constructs and outcomes will be selected, rather than asking about all of them.
One minor question is whether the sample size of parents in Study 2 for the qualitative interviews will be sufficient to stratify and compare those who receive a diagnostic result vs. those who do not. This comparison was described in the quan. part but not the qual.
In Study 3, will recruitment site be factored into modelling? The case was made that they offer diversity. It would be helpful to know how site is accounted for, or rationale for not (e.g., maybe education, ethnicity etc. are proxies?).
In Data Synthesis from All Studies section, please describe what you mean by this sentence: “Summary tables will be developed to identify context-specific barriers and facilitators (or suggested changes) to implementation.” It would be helpful to have a better sense of what the final product might look like. It seems that there was emphasis throughout the paper on using explanatory models and identifying mechanisms, and so the final synthesis might include more about process rather than summary of facilitators and barriers. It may be worthwhile to consider Intervention Mapping or something like that.

In general, this is a strong paper with a strong rationale and study design. Changes in formatting would help with clarity. Some additional detail would strengthen description of methods and analysis.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Not applicable

Reviewer Expertise:

Mixed-methods, qualitative methods, implementation science, health services research, genomic medicine implementation

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

NIHR Open Res. 2022 Feb 14.

Celine Lewis ¹

We thank the reviewer for their thoughtful and constructive comments on the study protocol. We hope that we have addressed these points to your satisfaction.

Thank you for this overview of the study. As suggested, we have changed the subheadings to lead with objective rather than method.

Details of the analysis are clearly described; however, some questions remain:

There could be more specificity about how CFIR will be used: it is very large and typically requires discussion ahead of time to prioritize constructs for specific studies, especially for qualitative interviews.

We agree that there is a need to prioritise domains and constructs and these will be identified with input from the advisory team, study aims and the literature. We are planning to use the five overarching domains as main codes to analyse our qualitative data as well as allow themes to emerge inductively. We have now specified this in the manuscript.

More detail is also needed about interviews. How will interview questions be developed? There is a CFIR Guide available for free online. Will those questions be used to ask about CFIR constructs? How long will the interviews last and will they be conducted remotely or in-person? The length of interviews matter because it seems like they could be very long if using CFIR, Proctor, literature, etc. I think more detail in this section would be helpful, including stating if key constructs and outcomes will be selected, rather than asking about all of them.

We will identify which of the CFIR constructs are most aligned to the aims of the study and interests of the advisory team. The online CFIR guide will be used to help think about the structure of the questions, for those domains/constructs of interest. They will be used to develop a priori codes for the analysis with subcodes emerging inductively in an iterative process. We are allowing 1 hour for the interviews (Study 1) and these will be conducted remotely using Office Teams. We have provided this additional detail in the manuscript.

One minor question is whether the sample size of parents in Study 2 for the qualitative interviews will be sufficient to stratify and compare those who receive a diagnostic result vs. those who do not. This comparison was described in the quan. part but not the qual.

I think this point is referring to Study 3 – we will continue to conduct interviews until saturation is reached.

In Study 3, will recruitment site be factored into modelling? The case was made that they offer diversity. It would be helpful to know how site is accounted for, or rationale for not (e.g., maybe education, ethnicity etc. are proxies?).

Each survey will be given a unique identifier so that we know which site the responder is from as well as the clinical indication. We are aiming for around 60 completed surveys per site so that comparison across sites can be made. This detail has been added.

In Data Synthesis from All Studies section, please describe what you mean by this sentence: “Summary tables will be developed to identify context-specific barriers and facilitators (or suggested changes) to implementation.” It would be helpful to have a better sense of what the final product might look like. It seems that there was emphasis throughout the paper on using explanatory models and identifying mechanisms, and so the final synthesis might include more about process rather than summary of facilitators and barriers. It may be worthwhile to consider Intervention Mapping or something like that.

The summary tables will not only focus on barriers and facilitators, they will also capture the process of implementation, perceptions etc. We will also be mapping the programme theory guiding the intervention at the central and local levels and this will also be included in the summary tables. We have added further details about this in the manuscript.

NIHR Open Res. 2021 Dec 7. doi: 10.3310/nihropenres.14352.r28440

Reviewer response for version 1

Zornitza Stark ¹

Thank you for the opportunity to review this study protocol.

This will be an important study that will describe the early implementation experience of mainstreaming genomic testing for paediatric rare disease in a public healthcare system. Strengths of the proposed study include the ability to compare against similar data generated while a hybrid research-clinical model was rolled out in the same healthcare system through the 100k genomes project. The study team are also to be commended on their mixed-methods, multi-disciplinary approach to evaluation, and the use of implementation theory frameworks and validated instruments, which will enable comparison with studies from other healthcare systems. Co-design with patients/families/support groups is also a key strength.

Minor reservations:

With regards to Study 1, if I understood correctly, the implementation outcome measures will be gathered through interviews with key designers/implementers of the service. While their insights will be invaluable, consideration should be given to collecting and reporting some objective measures such as uptake and variability of uptake based on geographical location or specialty for example.

Study 2: Clinic observations by a third party can potentially inhibit consultations. I wonder whether consultation recordings, particularly in the era of COVID restrictions and increasing use of telehealth may serve the purpose better?

Study 4: It would have been interesting to collect information on workforce preparedness prospectively rather than retrospectively, or at two time points in order to enable comparison and capture the role of interventions designed to increase preparedness.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

Clinical genomics, health economics, implementation science, bioethics

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

NIHR Open Res. 2022 Feb 14.

Celine Lewis ¹

We thank the reviewer for their thoughtful and constructive comments on the study protocol. We hope that we have addressed these points to your satisfaction.

Reviewer 2: Zornitza Stark

Thank you for the opportunity to review this study protocol.

Minor reservations:

We agree that it will be important to report uptake and variability of uptake across different trusts and specialties. NHS England will be routinely collecting this data, and we are in discussions with them as to how we might be able to access this data for analysis during the evaluation study, as this would require extra permissions for sharing and publishing.

This is something we have considered, although we might then potentially miss useful non-verbal data if we only have a recording of the consultation, particular in relation to the administrative elements that occur when patients are consented for genomic testing. Observations during clinical encounters are widely used as a data collection method. The presence of researchers will always change the dynamic of behaviour and this is the case in observations as well as other methods such as interviews. It is also hoped that by conducting follow-up interviews with the clinician and patient/parent, we can counteract this potential limitation through triangulating the findings. We will take a reflexive stance in the research and record cases where we felt that our presence changed the dynamic. We will reflect on this in the limitations section of any reports or papers.

We agree that it would be interesting to explore this amongst non-genetic specialists. One of our MSc students will be looking at this topic from the viewpoint of community paediatricians in 2022, so relatively early on in the roll-out of the new GMS, but unfortunately we do not currently have enough manpower to explore this from the viewpoint of other non-genetic specialists.

NIHR Open Res. 2021 Dec 6. doi: 10.3310/nihropenres.14352.r28441

Reviewer response for version 1

Julian Barwell ¹, Maurice Dungey ²

From Dr Julian Barwell

In order to make this review as insightful and impactful as possible I believe it would be helpful to use the staff review work package to ask a number of targeted questions:

Technical support: Technically do the clinicians and scientists feel that the appropriate infrastructure is in place to assist with consent, sample collection, logistically with the tracking of samples and delivery of genomic results through interpretation of variant pipelines, result placement in electronic patient records and governance of result interpretation and patient management taking into consideration variants and the presence or absence of family history/clear dysmorphological features?
Infrastructural support: Did the support of Clinical Research Network help when this became available and what barriers were felt locally when requesting managerial support given acute and social care strains on hospital finances? Has this support been maintained in between projects and new downstream investment funding?
Diagnostic decision making: Do the clinicians and scientists feel that the appropriate genetic and genomic tests are being ordered to ask the clinical question being asked with regards to sensitivity and specificity? For example taking into consideration read depth and other mechanisms of disease such as copy number variants, uniparental disorder or trinucleotide repeats.
Clinical Impact: Has an appropriate downstream cost-benefit impact analysis been carried out on appointments and investigations versus screening, personalised treatments and reproductive health decision making on the basis of the identification of additional diagnostic results?
What impact has the introduction of genomic medicine had on the care delivered by clinicians beyond the early adopters that have led or assisted with recruitment of patients to flagship implementation projects.
Views on next steps to break down barriers to implementation to mainstreaming and the use of personalised medicine more broadly such as the role of pharmacogenetics, tumour or microbial sequencing and polygenic risk scores?

With assurance that these areas will be covered I would be very happy to support this protocol.

From Dr Dungey

Thank you for the opportunity to review ‘Mixed-methods evaluation of the NHS Genomic Medicine Service for paediatric rare diseases: study protocol’. As the authors point out Whole-Genome sequencing is on the cusp of becoming increasingly available, utilised and mainstreamed as part of the NHS England Genomic Medicine Service (GMS). The enthusiasm, preparedness, impact and consequences of this on healthcare workers including genetic specialists, and particularly those working in medical specialties who may order tests in the mainstreamed setting, is currently unknown. For patients (and parents of affected children) it is currently uncertain what their attitude and understanding of genome sequencing will be; and therefore the patient experience and consequences of GMS is unclear.

This mixed-measures project encompasses four studies with four key aims (as laid out in the protocol):

Identify the resources, activities and intended and actual outcomes of the NHS GMS; identify any potential barriers to achieving the intended outcomes during the early years of the Service (2022–25);
Understand the processes and practices taking place by examining the interactions between health professionals and parents/patients during pre-test counselling and results delivery appointments;
Examine the experiences and outcomes of genomic testing that parents report over time;
Identify the preparedness and experiences of medical specialists involved in delivering genomic medicine in mainstream NHS care in the first few years of the Service, and identify elements which make this easier or more difficult.

There are a numerous strengths to these studies. The protocol is grounded in theory and utilises a number of different established frameworks to assess their aims. The multiple studies with different designs and involving varied stakeholders will give an interesting full picture of the impact of implementing GMS and, some insights of how to improve implementation for other services. Another big strength is the involvement of key stakeholders in the study planning and continued involvement throughout the studies and in dissemination of their findings. Studies 3 and 4 use mixed-measures approaches that will give valuable information on patient (or specifically parent) experiences and decisional regret, and the experiences of mainstreamed genomic testing.

There are a number of challenges in the study designs:

Recruitment to the study is crucial. The study uses purposive sampling, which is a valid qualitative approach but may have a number of weaknesses because the results are designed to be immediately clinically applicable, and not just theory generating. Potential concerns would be the a reliance on the researchers judgement on correct participants to approach, the risk of bias, and whether the results are less generalisable.
An explanation of how the researchers will involve the key stakeholders around their clinical responsibility has not been documented, and how they will account for bias if certain groups are under/over-represented in the study.
In observing clinical interactions, it is possible that observation will alter both the health professional and the parent/patient’s actions. This is a considerable weakness of observation and must be acknowledged in interpretation of results.
In the second study the authors will attempt to interview participants from a variety of settings (different sites, conditions, results, ethnicities, socioeconomic backgrounds, type of healthcare professional). This will likely make results harder to interpret and with only 40 interviews the data collected may be too heterogenous.
The authors plan to disseminate the findings of their work to the stakeholders throughout the study so that the service provision can be improved, it is unclear how this will be quantified in terms of changes made and the impact of these changes.
It is unclear from the protocol the details of who (and how many researchers) will be conducting interviews, observations, and data analysis. These details need to be clarified.
It is important there is transparency of how decisions are made in the data synthesis and interpretation from all studies and how recommendations are developed.

Overall, this appears to be a thorough research project that will give recommendations for other services. On the whole the methodology is sound, and we are happy to approve with some considerations to the above points.

Is the study design appropriate for the research question?

Partly

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Partly

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

Genomic Medicine

NIHR Open Res. 2022 Feb 14.

Celine Lewis ¹

We thank both reviewers for their thoughtful and constructive comments on the study protocol. We hope that we have addressed these points to your satisfaction.

Reviewer 1 Dr Julian Barwell and Dr Dungey

Dr Julian Barwell

This is a very welcome addition to the portfolio to assess the impact of genomic medicine. I am pleased to see a mixed model focus on the patient experience from expectation through to result delivery and beyond. The team assembled is very strong. In order to make this review as insightful and impactful as possible I believe it would be helpful to use the staff review work package to ask a number of targeted questions:

Technical support: Technically do the clinicians and scientists feel that the appropriate infrastructure is in place to assist with consent, sample collection, logistically with the tracking of samples and delivery of genomic results through interpretation of variant pipelines, result placement in electronic patient records and governance of result interpretation and patient management taking into consideration variants and the presence or absence of family history/clear dysmorphological features?

Infrastructural support: Did the support of Clinical Research Network help when this became available and what barriers were felt locally when requesting managerial support given acute and social care strains on hospital finances? Has this support been maintained in between projects and new downstream investment funding?

Diagnostic decision making: Do the clinicians and scientists feel that the appropriate genetic and genomic tests are being ordered to ask the clinical question being asked with regards to sensitivity and specificity? For example taking into consideration read depth and other mechanisms of disease such as copy number variants, uniparental disorder or trinucleotide repeats.

Clinical Impact: Has an appropriate downstream cost-benefit impact analysis been carried out on appointments and investigations versus screening, personalised treatments and reproductive health decision making on the basis of the identification of additional diagnostic results?

What impact has the introduction of genomic medicine had on the care delivered by clinicians beyond the early adopters that have led or assisted with recruitment of patients to flagship implementation projects.

Views on next steps to break down barriers to implementation to mainstreaming and the use of personalised medicine more broadly such as the role of pharmacogenetics, tumour or microbial sequencing and polygenic risk scores?

With assurance that these areas will be covered I would be very happy to support this protocol.

Thank you for raising these questions. We will ensure these topic areas are covered in our topic guides (for both studies 1 and 4).

Dr Dungey

This mixed-measures project encompasses four studies with four key aims (as laid out in the protocol):

Identify the resources, activities and intended and actual outcomes of the NHS GMS; identify any potential barriers to achieving the intended outcomes during the early years of the Service (2022–25);

Understand the processes and practices taking place by examining the interactions between health professionals and parents/patients during pre-test counselling and results delivery appointments;

Examine the experiences and outcomes of genomic testing that parents report over time;

Identify the preparedness and experiences of medical specialists involved in delivering genomic medicine in mainstream NHS care in the first few years of the Service, and identify elements which make this easier or more difficult.

There are a number of challenges in the study designs:

Recruitment to the study is crucial. The study uses purposive sampling, which is a valid qualitative approach but may have a number of weaknesses because the results are designed to be immediately clinically applicable, and not just theory generating. Potential concerns would be the a reliance on the researchers judgement on correct participants to approach, the risk of bias, and whether the results are less generalisable.

Thank you for this point. For Study 1 purposive sampling seems the most appropriate, given the relatively small number of GMS designers and implementers (particularly those with an interest in rare diseases). We hope that by approaching everyone who has been identified as being of relevance, we minimise the risk for bias. In addition, findings will be combined with those from the documentary analysis. For Study 1, the advisory team have identified the key people working within NHS England, Genomic England, Health Education England etc that would be able to speak about the experience of implementing the GMS for rare disease diagnosis and we have approached them for interviews. Similarly, at each GMSA/GLH we are approaching those leads (one clinical and one scientific) working within rare disease so that we target the most appropriate persons. These details have been added in the description of Study 1.

For Study 4, we agree that purposive sampling for interviews might result in bias e.g. If only those with positive views / known to be early adopters are approached. One way to reduce this risk would be to use an approach whereby we do not target participants but rather advertise the interview study more broadly e.g. through royal societies, email distribution lists, social media etc highlighting that we want a range of views and not just from those who have adapted genomic medicine in their clinical practice. We have revised the recruitment approach for Study 4 accordingly.

An explanation of how the researchers will involve the key stakeholders around their clinical responsibility has not been documented, and how they will account for bias if certain groups are under/over-represented in the study.

Ideally, in Study 4 we would like to conduct interviews with clinical specialists from around 5 different specialties who are offering genome sequencing to paediatric patients, and where we might expect there to be reasonable differences in terms of clinical impact of a diagnosis. This will also have to be balanced against where it is going to be practical to involve particular clinical groups (i.e. if reasonable relationships exist with those clinical specialties in the hospitals we are recruiting from). Similarly, for the survey, we will target those clinical specialties tasked with using genomic medicine. We will use a multipronged recruitment strategy to try and ensure a national sample of medical specialists across diverse specialties working in paediatric rare diseases. We will enlist the help of NHSE and GEL to advertise and circulate as well as advertise the survey through Royal Colleges and societies, hospital newsletters, member email distribution lists, social media. We will make it clear on the survey that is equally important to hear from those specialists who have used genomic medicine as those who haven’t. These details have been clarified in the protocol.

In observing clinical interactions, it is possible that observation will alter both the health professional and the parent/patient’s actions. This is a considerable weakness of observation and must be acknowledged in interpretation of results.

We acknowledge that this is an inherent limitation of the observation but felt that the benefits of observing appointments method (non-verbal interaction, administrative aspects) outweighed these potential limitations. We will take a reflexive stance in the research and record cases where we felt that our presence changed the dynamic. We will also reflect on this in the limitations section of any reports or papers.

In the second study the authors will attempt to interview participants from a variety of settings (different sites, conditions, results, ethnicities, socioeconomic backgrounds, type of healthcare professional). This will likely make results harder to interpret and with only 40 interviews the data collected may be too heterogenous.

Yes, we agree that on reflection it would be difficult to make comparisons and say anything specific about these participant groups. Moreover, education and ethnicity will be difficult to identify prior to approaching potential participants. We will therefore view Study 2 more as an exploratory study to build a picture of the different ways that appointments take place (e.g. is consenting done at the same clinic appointment or as a separate consent appointment), rather than aiming to make comparisons across these different groups.

The authors plan to disseminate the findings of their work to the stakeholders throughout the study so that the service provision can be improved, it is unclear how this will be quantified in terms of changes made and the impact of these changes.

Thank you for raising this important question. We will add an agenda point at each advisory team meeting to ask members involved in setting policy and practice i.e those members from NHS England and Genomics England), how they have used the emerging findings related to policy or practice from the study (presented at the previous meeting or in the interim period). In addition, we will ask whether there were any findings they chose not to use and the reason behind this. The responses will be recorded in the minutes of the meeting. We will also report the changes that were made to policy and practice in any reports or academic papers as part of the research findings.

It is unclear from the protocol the details of who (and how many researchers) will be conducting interviews, observations, and data analysis. These details need to be clarified.

We have added that this will mainly be performed by 2 researchers – Celine Lewis and Bettina Friedrich, but that where possible MSc students will also support the study.

It is important there is transparency of how decisions are made in the data synthesis and interpretation from all studies and how recommendations are developed.

Thank you for this comment. These decisions will be documented. The frameworks will provide a structured approach to analysis but interpretation and recommendations will be developed in conjunction with our advisory team with decisions documented. For the development of recommendations, a consensus approach will be taken whereby the majority of participants have to agree for a recommendations to be put forward. These details have been added to the manuscript.

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

No data are associated with this article. Anonymised data underlying the results will be hosted in the UCL Data Repository and a DOI will be referenced in research publications.

Reporting guidelines

The checklist for the protocol can be found at the following link:

Lewis, Celine (2021): SRQR Checklist for protocol paper. University College London. Figure. https://doi.org/10.5522/04/16847794

Award information

This research is funded through an NIHR Advanced Fellowship Grant (NIHR300099), awarded to Celine Lewis.

[ref-1] 1. NHS England: National Genomic Test Directory.2019; [Accessed 5th August 2019]. Reference Source [Google Scholar]

[ref-2] 2. Genomics England: The 100,000 Genomes Project Protocol.Genomics England Ltd.: London.2017. Reference Source [Google Scholar]

[ref-3] 3. HM Government: Genome UK: the future of healthcare.2020. Reference Source [Google Scholar]

[ref-4] 4. Robinson J: Everything you need to know about the NHS genomic medicine service.2020; [Accessed 25th October 2021]. Reference Source [Google Scholar]

[ref-5] 5. Health Education England: Master's in Genomic Medicine.2020; [Accessed 28th September 2020]. Reference Source [Google Scholar]

[ref-6] 6. Royal College of General Practitioners: Genomics toolkit.[Accessed 28th September 2020]. Reference Source [Google Scholar]

[ref-7] 7. NHS England: Genomics resources. [Accessed 1st September 2021]. Reference Source [Google Scholar]

[ref-8] 8. NHS England: Whole genome sequencing for rare disease: Information for patients and family members.NHS England,2020. Reference Source [Google Scholar]

[ref-9] 9. NHS England: Whole genome sequencing for a rare disorder: easy read.NHS England.2020. Reference Source [Google Scholar]

[ref-10] 10. NHS Genomic Medicine Service: Record of Discussion Regarding Genomic Testing.North Bristol NHS Trust, Editor. Reference Source [Google Scholar]

[ref-11] 11. Genomics England: Taking part. [Accessed 14th June 2021]. Reference Source [Google Scholar]

[ref-12] 12. Sanderson SC, Hill M, Patch C, et al. : Delivering genome sequencing in clinical practice: an interview study with healthcare professionals involved in the 100 000 Genomes Project. BMJ Open. 2019;9(11):e029699. 10.1136/bmjopen-2019-029699 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-13] 13. Lewis C, Sanderson S, Hill M, et al. : Parents' motivations, concerns and understanding of genome sequencing: a qualitative interview study. Eur J Hum Genet. 2020;28(7):874–884. 10.1038/s41431-020-0575-2 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-14] 14. Sanderson SC, Lewis C, Patch C, et al. : Opening the "black box" of informed consent appointments for genome sequencing: a multisite observational study. Genet Med. 2019;21(5):1083–1091. 10.1038/s41436-018-0310-3 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-15] 15. Dheensa S, Lucassen A, Fenwick A: Fostering trust in healthcare: Participants' experiences, views, and concerns about the 100,000 genomes project. Eur J Med Genet. 2019;62(5): 335–341. 10.1016/j.ejmg.2018.11.024 [DOI] [PubMed] [Google Scholar]

[ref-16] 16. Ormondroyd E, Mackley MP, Blair M, et al. : "Not pathogenic until proven otherwise": perspectives of UK clinical genomics professionals toward secondary findings in context of a Genomic Medicine Multidisciplinary Team and the 100,000 Genomes Project. Genet Med. 2018;20(3):320–328. 10.1038/gim.2017.157 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-17] 17. Ballard LM, Horton RH, Dheensa S, et al. : Exploring broad consent in the context of the 100,000 Genomes Project: a mixed methods study. Eur J Hum Genet. 2020;28(6):732–741. 10.1038/s41431-019-0570-7 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-18] 18. Sanderson SC, et al. : Decision-making, attitudes and understanding amongst patients and relatives invited to undergo genome sequencing in the 100,000 Genomes Project: a multi-site survey study. Genetics In Medicine. In press. [DOI] [PubMed] [Google Scholar]

[ref-19] 19. Hussey PS, Anderson GF, Osborn R, et al. : How does the quality of care compare in five countries? Health Aff (Millwood). 2004;23(3):89–99. 10.1377/hlthaff.23.3.89 [DOI] [PubMed] [Google Scholar]

[ref-20] 20. Damschroder LJ, Aron DC, Keith RE, et al. : Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. 10.1186/1748-5908-4-50 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-21] 21. Hulscher ME, Laurant MG, Grol RP: Process evaluation on quality improvement interventions. Qual Saf Health Care. 2003;12(1):40–6. 10.1136/qhc.12.1.40 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-22] 22. Stetler CB, Legro MW, Wallace CM, et al. : The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21 Suppl 2(Suppl 2):S1–S8. 10.1111/j.1525-1497.2006.00355.x [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-23] 23. European Organisation for Rare Diseases: Rare Diseases: Understanding this Public Health Priority.2005; [cited Accessed 25 August 2021]. Reference Source [Google Scholar]

[ref-24] 24. Blöß S, Klemann C, Rother AK, et al. : Diagnostic needs for rare diseases and shared prediagnostic phenomena: Results of a German-wide expert Delphi survey. PLoS One. 2017;12(2):e0172532. 10.1371/journal.pone.0172532 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-25] 25. Vissers LELM, van Nimwegen KJM, Schieving JH, et al. : A clinical utility study of exome sequencing versus conventional genetic testing in pediatric neurology. Genet Med. 2017;19(9):1055–1063. 10.1038/gim.2017.1 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-26] 26. Wright CF, FitzPatrick DR, Firth HV: Paediatric genomics: diagnosing rare disease in children. Nat Rev Genet. 2018;19(5):253–268. 10.1038/nrg.2017.116 [DOI] [PubMed] [Google Scholar]

[ref-27] 27. Griffin BH, Chitty LS, Bitner-Glindzicz M: The 100 000 Genomes Project: What it means for paediatrics. Arch Dis Child Educ Pract Ed. 2017;102(2):105–107. 10.1136/archdischild-2016-311029 [DOI] [PubMed] [Google Scholar]

[ref-28] 28. Ashtiani S, Makela N, Carrion P, et al. : Parents' experiences of receiving their child's genetic diagnosis: a qualitative study to inform clinical genetics practice. Am J Med Genet A. 2014;164A(6):1496–502. 10.1002/ajmg.a.36525 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-29] 29. Werner-Lin A, Zaspel L, Carlson M, et al. : Gratitude, protective buffering, and cognitive dissonance: How families respond to pediatric whole exome sequencing in the absence of actionable results. Am J Med Genet A. 2018;176(3):578–588. 10.1002/ajmg.a.38613 [DOI] [PubMed] [Google Scholar]

[ref-30] 30. Tolusso LK, Collins K, Zhang X, et al. : Pediatric Whole Exome Sequencing: an Assessment of Parents' Perceived and Actual Understanding. J Genet Couns. 2017;26(4):792–805. 10.1007/s10897-016-0052-9 [DOI] [PubMed] [Google Scholar]

[ref-31] 31. Bernhardt BA, Roche MI, Perry DL, et al. : Experiences with obtaining informed consent for genomic sequencing. Am J Med Genet A. 2015;167A(11):2635–46. 10.1002/ajmg.a.37256 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-32] 32. Tomlinson AN, Skinner D, Perry DL, et al. : "Not Tied Up Neatly with a Bow": Professionals' Challenging Cases in Informed Consent for Genomic Sequencing. J Genet Couns. 2016;25(1):62–72. 10.1007/s10897-015-9842-8 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-33] 33. Shickh S, Mighton C, Uleryk E, et al. : The clinical utility of exome and genome sequencing across clinical indications: a systematic review. Hum Genet. 2021;140(10):1403–1416. 10.1007/s00439-021-02331-x [DOI] [PubMed] [Google Scholar]

[ref-34] 34. Smith HS, Swint JM, Lalani SR, et al. : Clinical Application of Genome and Exome Sequencing as a Diagnostic Tool for Pediatric Patients: a Scoping Review of the Literature. Genet Med. 2019;21(1):3–16. 10.1038/s41436-018-0024-6 [DOI] [PubMed] [Google Scholar]

[ref-35] 35. Mestek-Boukhibar L, Clement E, Jones WD, et al. : Rapid Paediatric Sequencing (RaPS): comprehensive real-life workflow for rapid diagnosis of critically ill children. J Med Genet. 2018;55(11):721–728. 10.1136/jmedgenet-2018-105396 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-36] 36. Wright CF, Fitzgerald TW, Jones WD, et al. : Genetic diagnosis of developmental disorders in the DDD study: a scalable analysis of genome-wide research data. Lancet. 2015;385(9975):1305–14. 10.1016/S0140-6736(14)61705-0 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-37] 37. Brett GR, Martyn M, Lynch F, et al. : Parental experiences of ultrarapid genomic testing for their critically unwell infants and children. Genet Med. 2020;22(12):1976–1985. 10.1038/s41436-020-0912-4 [DOI] [PubMed] [Google Scholar]

[ref-38] 38. Robinson JO, Wynn J, Biesecker B, et al. : Psychological outcomes related to exome and genome sequencing result disclosure: a meta-analysis of seven Clinical Sequencing Exploratory Research (CSER) Consortium studies. Genet Med. 2019;21(12):2781–2790. 10.1038/s41436-019-0565-3 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-39] 39. Rosell AM, Pena LD, Schoch K, et al. : Not the End of the Odyssey: Parental Perceptions of Whole Exome Sequencing (WES) in Pediatric Undiagnosed Disorders. J Genet Couns. 2016;25(5):1019–31. 10.1007/s10897-016-9933-1 [DOI] [PubMed] [Google Scholar]

[ref-40] 40. Krabbenborg L, Vissers LE, Schieving J, et al. : Understanding the Psychosocial Effects of WES Test Results on Parents of Children with Rare Diseases. J Genet Couns. 2016;25(6):1207–1214. 10.1007/s10897-016-9958-5 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-41] 41. Barwell J, Snape K, Wedderburn S: The new genomic medicine service and implications for patients. Clin Med (Lond). 2019;19(4):273–277. 10.7861/clinmedicine.19-4-273 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-42] 42. Weipert CM, Ryan KA, Everett JN, et al. : Physician Experiences and Understanding of Genomic Sequencing in Oncology. J Genet Couns. 2018;27(1):187–196. 10.1007/s10897-017-0134-3 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-43] 43. Aiello LB: Genomics Education: Knowledge of Nurses Across the Profession and Integration Into Practice. Clin J Oncol Nurs. 2017;21(6):747–753. 10.1188/17.CJON.747-753 [DOI] [PubMed] [Google Scholar]

[ref-44] 44. McClaren BJ, Crellin E, Janinski M, et al. : Preparing Medical Specialists for Genomic Medicine: Continuing Education Should Include Opportunities for Experiential Learning. Front Genet. 2020;11:151. 10.3389/fgene.2020.00151 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-45] 45. Salm M, Abbate K, Appelbaum P, et al. : Use of genetic tests among neurologists and psychiatrists: knowledge, attitudes, behaviors, and needs for training. J Genet Couns. 2014;23(2):156–63. 10.1007/s10897-013-9624-0 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-46] 46. Wynn J, Lewis K, Amendola LM, et al. : Clinical providers' experiences with returning results from genomic sequencing: an interview study. BMC Med Genomics. 2018;11(1):45. 10.1186/s12920-018-0360-z [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-47] 47. Proctor E, Silmere H, Raghavan R, et al. : Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76. 10.1007/s10488-010-0319-7 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-48] 48. Moran-Ellis J, Alexander VD, Cronin A, et al. : Triangulation and integration: processes, claims and implications. Qual Res. 2006;6(1):45–59. 10.1177/1468794106058870 [DOI] [Google Scholar]

[ref-49] 49. Creswell JW: Research design. Qualitative, quantitative, and mixed methods approaches.2nd ed. ed. Thousand Oaks, California: Sage,2003. Reference Source [Google Scholar]

[ref-50] 50. Weaver K: Pragmatic Paradigm.In The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation.B.B. Frey, Editor. SAGE Publications, Inc.: Thousand Oaks,2018. 10.4135/9781506326139.n534 [DOI] [Google Scholar]

[ref-51] 51. Orlando LA, Sperber NR, Voils C, et al. : Developing a common framework for evaluating the implementation of genomic medicine interventions in clinical care: the IGNITE Network's Common Measures Working Group. Genet Med. 2018;20(6):655–663. 10.1038/gim.2017.144 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-52] 52. Best S, Brown H, Lunke S, et al. : Learning from scaling up ultra-rapid genomic testing for critically ill children to a national level. NPJ Genom Med. 2021;6(1):5. 10.1038/s41525-020-00168-3 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-53] 53. Levy KD, Blake K, Fletcher-Hoppe C, et al. : Opportunities to implement a sustainable genomic medicine program: lessons learned from the IGNITE Network. Genet Med. 2019;21(3):743–747. 10.1038/s41436-018-0080-y [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-54] 54. Zebrowski AM, Ellis DE, Barg FK, et al. : Qualitative study of system-level factors related to genomic implementation. Genet Med. 2019;21(7):1534–1540. 10.1038/s41436-018-0378-9 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-55] 55. Keith RE, Crosson JC, O'Malley AS, et al. : Using the Consolidated Framework for Implementation Research (CFIR) to produce actionable findings: a rapid-cycle evaluation approach to improving implementation. Implement Sci. 2017;12(1):15. 10.1186/s13012-017-0550-7 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-56] 56. West R, Michie S, Atkins L, et al. : Achieving behaviour change: A guide for local government and partners.PHE publications: London,2019. Reference Source [Google Scholar]

[ref-57] 57. National Institute for Health Research: Centre for Engagement and Dissemination - Recognition payments for public contributors.2016; Accessed 25th Otober 2021]. Reference Source [Google Scholar]

[ref-58] 58. PHG Foundation: Genomics in Mainstream Medicine Working Group.2017; Accessed 16th July 2020]. Reference Source [Google Scholar]

[ref-59] 59. W.K. Kellogg Foundation: Logic Model Development Guide.W.K. Kellogg Foundation: Battle Creek, Michigan,2004. Reference Source [Google Scholar]

[ref-60] 60. Gale NK, Heath G, Cameron E, et al. : Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13:117. 10.1186/1471-2288-13-117 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-61] 61. QSR International Pty Ltd: Nvivo. (released in March 2020).2020. Reference Source [Google Scholar]

[ref-62] 62. Sadler GR, Lee HC, Lim RS, et al. : Recruitment of hard-to-reach population subgroups via adaptations of the snowball sampling strategy. Nurs Health Sci. 2010;12(3):369–374. 10.1111/j.1442-2018.2010.00541.x [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-63] 63. Angrosino M, Rosenberg J: Observations on observation.In Collecting and Interpreting Qualitative Materials. N.K. Denzin and Y.S. Lincoln, Editors. Sage: Thousand Oaks, CA,2012. Reference Source [Google Scholar]

[ref-64] 64. Holtrop JS, Rabin BA, Glasgow RE: Qualitative approaches to use of the RE-AIM framework: rationale and methods. BMC Health Serv Res. 2018;18(1):177. 10.1186/s12913-018-2938-8 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-65] 65. Atkinson P, Housley W: Interactionism.London: Sage,2003. Reference Source [Google Scholar]

[ref-66] 66. Hsieh HF, Shannon SE: Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88. 10.1177/1049732305276687 [DOI] [PubMed] [Google Scholar]

[ref-67] 67. Braun V, Clarke V: Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101. 10.1191/1478088706qp063oa [DOI] [Google Scholar]

[ref-68] 68. National Centre for Advancing Translational Sciences: Annual Report on the Rare Diseases and Conditions Research Activities of the National Institutes of Health FY 2000.2005. Reference Source [Google Scholar]

[ref-69] 69. O'Connor A: User Manual – Decision Regret Scale Ottawa: Ottawa Hospital Research Institute. (modified 2003)1996; [Accessed 16 July 2013]. Reference Source [Google Scholar]

[ref-70] 70. Wynn J, Ottman R, Duong J, et al. : Diagnostic exome sequencing in children: A survey of parental understanding, experience and psychological impact. Clin Genet. 2018;93(5):1039–1048. 10.1111/cge.13200 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-71] 71. Sanderson SC, Loe BS, Freeman M, et al. : Development of the Knowledge of Genome Sequencing (KOGS) questionnaire. Patient Educ Couns. 2018;101(11):1966–1972. 10.1016/j.pec.2018.07.011 [DOI] [PubMed] [Google Scholar]

[ref-72] 72. Marteau TM, Dormandy E, Michie S: A measure of informed choice. Health Expect. 2001;4(2):99–108. 10.1046/j.1369-6513.2001.00140.x [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-73] 73. O'Connor A: User Manual – Decisional Conflict Scale.(16 item statement format).1993; (updated 2010) [Accesed 16 July 2013]. Reference Source [Google Scholar]

[ref-74] 74. Spitzer RL, Kroenke K, Williams JB, et al. : A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006;166(10):1092–7. 10.1001/archinte.166.10.1092 [DOI] [PubMed] [Google Scholar]

[ref-75] 75. Grant PE, Pampaka M, Payne K, et al. : Developing a short-form of the Genetic Counselling Outcome Scale: The Genomics Outcome Scale. Eur J Med Genet. 2019;62(5):324–334. 10.1016/j.ejmg.2018.11.015 [DOI] [PubMed] [Google Scholar]

[ref-76] 76. Wille N, Badia X, Bonsel G, et al. : Development of the EQ-5D-Y: a child-friendly version of the EQ-5D. Qual Life Res. 2010;19(6):875–86. 10.1007/s11136-010-9648-y [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-77] 77. Varni JW, Sherman SA, Burwinkle TM, et al. : The PedsQL Family Impact Module: preliminary reliability and validity. Health Qual Life Outcomes. 2004;2:55. 10.1186/1477-7525-2-55 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-78] 78. Li M, Bennette CS, Amendola LM, et al. : The Feelings About genomiC Testing Results (FACToR) Questionnaire: Development and Preliminary Validation. J Genet Couns. 2019;28(2):477–490. 10.1007/s10897-018-0286-9 [DOI] [PubMed] [Google Scholar]

[ref-79] 79. Zellerino B, Milligan SA, Brooks R, et al. : Development, testing, and validation of a patient satisfaction questionnaire for use in the clinical genetics setting. Am J Med Genet C Semin Med Genet. 2009;151c(3):191–9. 10.1002/ajmg.c.30214 [DOI] [PubMed] [Google Scholar]

[ref-80] 80. Carleton RN, Norton MA, Asmundson GJ: Fearing the unknown: a short version of the Intolerance of Uncertainty Scale. J Anxiety Disord. 2007;21(1):105–17. 10.1016/j.janxdis.2006.03.014 [DOI] [PubMed] [Google Scholar]

[ref-81] 81. Smith BW, Dalen J, Wiggins K, et al. : The brief resilience scale: assessing the ability to bounce back. Int J Behav Med. 2008;15(3):194–200. 10.1080/10705500802222972 [DOI] [PubMed] [Google Scholar]

[ref-82] 82. Greene JC, Caracelli VJ, Graham WF: Toward a Conceptual Framework for Mixed-Method Evaluation Designs. Educ Eval Policy Anal. 1989;11(3):255–274. 10.3102/01623737011003255 [DOI] [Google Scholar]

[ref-83] 83. Corp I: IBM SPSS Statistics for Windows, Version 27.0.Armonk, NY: IBM Corp,2020. Reference Source [Google Scholar]

[ref-84] 84. Braun V, Clarke V: Conceptual and Design Thinking for Thematic Analysis. Qual Psychol. 2021. 10.1037/qup0000196 [DOI] [Google Scholar]

[ref-85] 85. Weiner BJ, Lewis CC, Stanick C, et al. : Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108. 10.1186/s13012-017-0635-3 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-86] 86. Aarons GA, Ehrhart MG, Farahnak LR: The Implementation Leadership Scale (ILS): development of a brief measure of unit level implementation leadership. Implement Sci. 2014;9(1):45. 10.1186/1748-5908-9-45 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-87] 87. Armenakis AA, Bernerth JB, Pitts JP, et al. : Organizational change recipients' beliefs scale: Development of an assessment instrument. J Appl Behav Sci. 2007;43(4):481–505. 10.1177/0021886307303654 [DOI] [Google Scholar]

[ref-88] 88. NHS Health Research Authority: UK policy framework for health and social care research.2017. Reference Source [Google Scholar]

PERMALINK

Mixed-methods evaluation of the NHS Genomic Medicine Service for paediatric rare diseases: study protocol

Celine Lewis

James Buchanan

Angus Clarke

Emma Clement

Bettina Friedrich

Jillian Hastings-Ward

Melissa Hill

Ruth Horn

Anneke M Lucassen

Chris Patch

Alexandra Pickard

Lauren Roberts

Saskia C Sanderson

Sarah L Wynn

Cecilia Vindrola-Padros

Monica Lakhanpaul

Roles

Version Changes

Revised. Amendments from Version 1

Abstract

Background

Methods

Dissemination

Plain English summary

Introduction

The Genomic Medicine Service

Mainstreaming genomics for rare disease diagnosis

Preparation for genomic testing in the NHS GMS

Focus of current study

Background

How will the NHS GMS impact parents and children?

How will the NHS GMS impact health professionals?

Protocol

Research aims and objectives

Methods and analysis

Figure 1. Overview of study timelines.

Detailed study plan

Figure 2. Study design.

Study 1: Specify the programme theory of the GMS (Years 1 and 4)

Study 2: Identify processes and practices when patients undergo genomic testing in the GMS (Years 2–3)

Study 3: Identify parent-reported experiences and outcomes from genomic testing in the GMS (Years 2–4)

Table 1. Summary of survey measures.

Study 4: Obtain perspectives including readiness, process and outcomes of different medical specialists (Year 4)

Data synthesis and interpretation – Study 5

Ethics and data processing

Dissemination – Study 6

Discussion

Data availability

Reporting guidelines

Award information

Acknowledgements

Funding Statement

References

Reviewer response for version 2

Julian Barwell

Maurice Dungey

Roles

Reviewer response for version 2

Zornitza Stark

Roles

Reviewer response for version 1

Lotte Krabbenborg

Karine Wendrich

Roles

Celine Lewis

Reviewer response for version 1

Nina Sperber

Roles

Celine Lewis

Reviewer response for version 1

Zornitza Stark

Roles

Celine Lewis

Reviewer response for version 1

Julian Barwell

Maurice Dungey

Roles

Celine Lewis