The PERMIT guidelines for designing and implementing all stages of personalised medicine research

Abstract

Personalised medicine (PM) research programmes represent the modern paradigm of complex cross-disciplinary research, integrating innovative methodologies and technologies. Methodological research is required to ensure that these programmes generate robust and reproducible evidence. The PERMIT project developed methodological recommendations for each stage of the PM research pipeline. A common methodology was applied to develop the recommendations in collaboration with relevant stakeholders. Each stage was addressed by a dedicated working group, specializing in the subject matter. A series of scoping reviews that mapped the methods used in PM research and a gap analysis were followed by working sessions and workshops where field experts analyzed the gaps and developed recommendations. Through collaborative writing and consensus building exercises, the final recommendations were defined. They provide guidance for the design, implementation and evaluation of PM research, from patient and omics data collection and sample size calculation to the selection of the most appropriate stratification approach, including machine learning modeling, the development and application of reliable preclinical models, and the selection and implementation of the most appropriate clinical trial design. The dissemination and implementation of these recommendations by all stakeholders can improve the quality of PM research, enhance the robustness of evidence, and improve patient care.

Introduction

Over the past decade, medical research has undergone major changes due to the convergence of high throughput technologies with the digital revolution¹. This novel data-driven approach has resulted in a growing number of complex, cross-disciplinary projects requiring new methodologies, technologies and research infrastructure services. The personalised medicine (PM) research pipeline is a prime example of such complex research programmes, presenting major methodological and regulatory challenges. While investigators have developed and implemented innovative solutions, rigorous methodological research is required to ensure robust evidence is generated. The PERsonalised MedicIne Trials (PERMIT) project was therefore designed to develop a set of methodological recommendations to support investigators, evaluators, peer-reviewers and regulators involved in designing, funding, conducting, publishing and validating PM research.

Initially promoted as a single biomarker-driven treatment decision, PM now refers to a broader range of research paradigms^2,3. In particular, novel -omics technologies and AI approaches allow for patient stratification using complex multi-omics signatures and subsequent machine learning (ML) stratification, as well as conventional statistical methods. This technological push makes PM a growing priority in the medical research communities and health industry, as well as among policy makers and funding agencies⁴.

Although PM research programmes can vary in their structure, PERMIT considered the four steps illustrated in Fig. 1 to address the methodological challenges raised by the PM research pipeline.

Fig. 1.

The personalised medicine research pipeline: Stage 1 – Collection of patient and omics data: collection of patient data, including –omics data, through retrospective and/or prospective cohorts. Stage 2 – Machine learning (ML) stratification: Use of patient data to stratify the patient population into homogeneous clusters, through either conventional statistical or ML methods. Stage 3 - The translational development: Assessing the efficacy and safety of the treatment options in preclinical models is often required. Stage 4 - Clinical trials - therapeutic solutions are tested in the various patient clusters, and the effectiveness of the personalised vs. non-personalised approach can be assessed.

This article presents the key recommendations for the four steps, providing an ensemble view of the PM research pipeline and guidance from end-to-end.

An operational definition of PM research was applied throughout PERMIT: a set of comprehensive methods (methodology, statistics, validation, technology) to be applied in the different phases of the development of a personalised approach to treatment, diagnosis, prognosis or risk prediction. Ideally, robust and reproducible methods should cover all steps between generating the hypothesis (e.g., a given stratum of patients could better respond to a treatment), its validation and preclinical development, and up to the definition of its value in a clinical setting.

Methods

A common methodological framework was developed for the PERMIT project and followed to develop the recommendations, with some flexibility to account for the specificities of the four stages of the pipeline (Fig. 2). N-of-1 trials were not addressed in PERMIT. Each stage was addressed by a dedicated working group, led by a team specializing in the subject matter. Regular meetings of team leaders ensured alignment across the different sets of recommendations.

Fig. 2.

Common methodological framework of the PERMIT Project.

The main steps of the methodology were:

Background information: scoping reviews and gap analysis

Each working group carried out a scoping review, following a common protocol⁵ to map the methodologies applied and identify the existing gaps, from a technical, methodological, and regulatory perspective as well as the existing standardization and harmonization challenges. The results of each scoping review were discussed in a joint gap analysis workshop.

Development phase: working sessions and workshops

Each group organized online working sessions and workshops with project partners and field experts from key stakeholder groups to discuss the gaps and challenges and start defining the recommendations to address them. Furthermore, an interdisciplinary meeting was organized to discuss the regulatory gaps and challenges spanning from designing and building stratification cohorts to the translational step.

Finalization phase: collaborative writing and consensus

The outcomes of the working sessions and workshops were then translated by each group into the four subsets of recommendations. Through collaborative writing in smaller groups and formal consensus exercises, the final version of the recommendations was defined.

Results

The series of scoping reviews and subsequent workshops identified significant weaknesses in the design, conduct and reporting of PM research and the detailed findings are reported elsewhere^6–9. Recommendations were developed for each stage of the PM research pipeline and are the subject of detailed publications^10,11. The key recommendations are described in the following four chapters:

Collection of patient cohort and -omics data

Prospective or retrospective collection and consolidation of patient data is the foundation of the PM research pipeline. Typically achieved through observational studies, the collection of omics, imaging, clinical, lifestyle, exposome and other data provides the basis for the stratification of patients sharing similar biomarker profiles into homogenous clusters. These recommendations highlight key aspects that must be considered to ensure sufficient and adequate data is collected.

• 3.1.1Both retrospective and prospective creation of cohorts have advantages and disadvantages¹² and these must be thoroughly assessed when selecting an approach for the stratification cohort.
• 3.1.2When integrating data from multiple retrospective cohorts to increase sample size, an ex-ante harmonization approach should be favoured over an ex-post approach whenever possible. When integrating data from different cohorts prospectively, interoperable data standards should be adopted to ensure harmonization and seamless integration. In both cases, interoperable multimodal data management systems should be implemented.
• 3.1.3The most relevant data for the stratification analysis should align with study objectives. Other factors including the cost of data generation, data quality and accuracy, must be taken into consideration.
• 3.1.4Thorough sample size estimation should be included in every study design, and should demonstrate that the number of samples per study group provides sufficient statistical power to detect between-group differences at the expected level of sensitivity and specificity for a given minimum effect size. The sample size for individual discovery and validation studies must be estimated separately. In PM, no unique method is available to calculate adequate sample size(s) for stratification cohort(s). It should be estimated using dedicated statistical software tools, ideally using simulations with real-world data from a pilot study. If no pilot data is available, the most similar public dataset, or a simulation or parametric power estimation with a conservative variance estimate can be applied to avoid a rule-of-thumb approach. Alternatively prior assumptions on key data properties can be used to estimate the sample size.
• 3.1.5For most stratification projects, only one discovery and one validation cohort suffice, as long as they are both representative of the patient population addressed and the sample sizes are deemed sufficient according to the sample size calculation. Otherwise, including further cohorts should be considered.
• 3.1.6Data quality and monitoring requirements depend on the objective of the stratification. Whenever stratification drives treatment or healthcare decisions, monitoring should comply with the ISO 20916:2019 standard and omics/biomarker data generation should be conducted using CE marked in vitro diagnostic (IVD) devices.
• 3.1.7To select the most appropriate stratification approach, pilot data should be used to assess whether classical statistical hypothesis tests provide sufficiently robust and discriminative individual biomarker candidates for stratification, or whether a multi-variable ML approach for biomarker signature can account for a lack of discriminative power for individual markers and a complex correlation structure between multiple relevant markers.
• 3.1.8An external validation on a comparable series of patients must be designed when possible, in accordance with the study characteristics. The validation cohort should ideally be collected prospectively. Otherwise, biomarker robustness should be ensured by doing nested cross-validation. If validation with a different cohort is not possible, it is recommended to perform a three-group split of the internal data into a training set for model building and optimization, a validation set for performance assessment, and a test set for additional confirmation of the reproducibility of the performance estimates.
• 3.1.9A complete analysis of the risk of bias should be conducted and published. For personalized medicine studies, this analysis should go beyond standard tools for randomized controlled trials (e.g. Cochrane RoB 2) to address potential biases specific to stratification cohorts and biomarker studies. Key areas to assess include: 1)Selection bias in cohort recruitment and data collection; 2) Information bias in biomarker measurement and outcome assessment; 3) Confounding factors that may influence stratification or treatment effects; 4) Overfitting or lack of external validation in ML models; 5) Missing data and loss to follow-up; 6) Reporting bias of biomarker analyses.
• 3.1.10An infrastructure for management and integration of patient data compliant with local and international regulation is essential. Regulatory expertise in the research team is equally important. Although exploratory stratification research may aim to merely revisit disease taxonomy or have a better understanding of disease prognosis in patient subgroups, it is recommended that clinical regulation be heeded at an early stage and studies be designed with an outlook on the findings ultimately leading to the use of the stratification assay to drive patient care (see recommendation 3.1.6).

This first series of recommendations addresses gaps identified⁷ while covering a broader spectrum of issues to be considered while designing discovery and validation cohort studies supporting patient stratification and the identification of complex biomarker signatures.

Machine learning methods for patient stratification

Conventional biomarker discovery approaches for patient stratification have used statistical methods to identify single markers that discriminate between patient sub-groups with distinctive disease-relevant characteristics. However, for many complex and heterogeneous disorders, using more advanced ML analyses to identify multi-variable biomarker signatures can provide more robust and accurate stratification models.

The recommendations for ML stratification methods have been grouped chronologically by the main study phases (planning phase, discovery and modelling phase, validation phase) as follows:

Planning phase

• 3.2.1To avoid inadequate sample sizes and underpowered study designs, (see 3.1.7), a small-scale pilot study should be conducted before the main discovery study, in order to collect prior data as input for a statistical sample size estimation (for relevant methods see¹³). Researchers can also use algorithmic methods to optimise the matching and selection of biospecimens to be included in an experimental profiling study and integrate complementary public biological data into the analysis^14,15.
• 3.2.2Strong imbalances in the number of available biospecimens for the relevant study groups can complicate the identification of robust discriminative patterns by ML. To address this common limitation, a detailed plan for further subject recruitment combined with a prior sample size estimation (see 3.2.1) is recommended. Retrospectively, class imbalances can be addressed in the modelling phase, using dedicated sub-sampling or sample weighting approaches¹⁶.
• 3.2.3In longitudinal patient stratification studies, study participants may drop out due to reasons related or unrelated to the study (informative or uninformative censoring). Detailed planning for further subject recruitment is also instrumental here. Furthermore, dropouts require special consideration in the data analysis, by reviewing possible causes of censoring and associated biases, and applying dedicated imputation and censored data analysis methods^17,18.

Discovery and modelling phase

• 3.2.4To avoid inadequate data pre-processing, filtering and normalisation methods, comprehensive quality control analyses should be performed before and after data pre-processing^19,20. In particular, samples that fail multiple outlier and quality control checks should be filtered out, and distribution assumptions should be reviewed to select pre-processing techniques specifically tailored to the observed data distributions. It is also advisable to compare multiple pre-processing approaches by means of quality control or benchmark analyses.
• 3.2.5Without prior knowledge on which types of modelling approaches may best fit the available data, researchers may want to compare multiple representative types of modelling approaches (e.g. linear modelling approaches, nearest neighbour-based methods, neural networks, rule-based and decision-tree-based approaches) using a nested cross-validation with suitable parameter optimization approaches²¹. As an alternative to single ML algorithms, integrating multiple approaches via ensemble learning methods should be considered²².
• 3.2.6Avoiding overfitting or underfitting of a model to its training data, i.e. finding a suitable balance between the specificity of the model to the training data and its expected ability to generalise to other unseen data sets, is essential. Several methods have been proposed to detect and prevent over- and underfitting. Researchers can apply dedicated regularization methods and optimize model complexity parameters using nested cross-validation^21,23. Combining ML methods with prior feature selection and dimension reduction methods can help to avoid building overly complex overfiting models²⁴.

Validation phase

• 3.2.7ML models require appropriate validation using cross-validation and external testing analyses. To ensure the robustness of validation studies and the resulting model performance statistics, the sizes and composition of the training and test sets should be chosen adequately in the prior study design; and robust bootstrapping or cross-validation techniques combined with relevant performance metrics should be applied²⁴.
• 3.2.8While single-cohort analyses can provide initial performance estimates for ML stratification, external validation analyses on independent cohorts representing different ethnic and geographic backgrounds are required to ensure that the model is applicable and accurate across different populations. Meta-analyses of datasets from multiple cohorts can increase the robustness of feature selection and model building analyses, as can validations on distinct cohorts representing different populations.
• 3.2.9Explainability and interpretability of ML models for stratification can be a key requirement for assessing their trustworthiness and biological plausibility. In these settings, so-called “white-box” ML algorithms²⁵ should be used to ensure models are sufficiently explainable, both in terms of biological interpretability and the provision of comprehensive data-driven justifications for the model’s prediction. This is particularly important for regulators.

ML stratification appears as a new and promising instrument in PM. However, care should be taken to carefully plan, implement and validate ML models to ensure reliable evidence is generated.

Translational methods for personalised medicine

Undeniable progress has been made in developing innovative and complex preclinical model systems that hold promise for recapitulating patient clustering. Despite these efforts, to date, fundamental deficits in translational methods prevent the further development of PM. The scoping review identified and highlighted these significant gaps and challenges⁶.

To address them, the recommendations for the translational step of PM research focus on five areas: clinically relevant translational research; robust model development; transparency and education; revised regulation; and interaction with clinical research and patient engagement.

Clinically relevant translational research

Increased emphasis on using clinically relevant research models is imperative, and to achieve this there should be standards for model relevance. Selection of preclinical models must be evidence-based, and researchers should demonstrate awareness of the model’s limitations when interpreting results. Diverse biological mechanisms and patient heterogeneity usually cannot be represented by one single model system, therefore a combination of models should be considered for complex diseases.

Robust model development

Research models must be robust to be predictive, and there should be a common implementation framework for rigorous research, to provide reliable preclinical data for clinical trials. In addition, further efforts should be made to validate, qualify and adopt innovative preclinical methods. To facilitate systematic validation processes, key stakeholders must support and promote robust model development through specific funding and policies and encourage joint efforts between pharmaceutical industries and academic institutions. In particular for PM, further developing and validating the capacity of preclinical models such as cellular, organoid, patient-derived xenograft and in silico models to recapitulate patient strata will be paramount. Additionally, further resources should be made available for the maintenance of newly developed models.

Transparency, education and stakeholders’ engagement in preclinical research in PM

Transparency is essential, and reliable reporting and data sharing must be a requirement for both academia and industry. Open science is advocated for by many universities, organisations and policy makers^26,27. Pre-registering preclinical study protocols in open access databases, is another step to implement transparency. Pre-registration can support researchers during the study design phase, it can reduce the use of animals for unnecessary duplications, as well as facilitate the generation of systematic reviews and meta-analyses of preclinical studies, all contributing to increasing the quality of research²⁸. Education and training of researchers is needed to promote high quality and reproducible translational research, to facilitate a cultural change.

Revised regulation

Preclinical studies are subjected to regulation for good laboratory practice and laws for animal protection, however, these legislations reflect minimum requirements. Additional guidelines are needed, to ensure preclinical evidence is clinically relevant and encourages the adoption of patient-derived models. In addition, regulators and ethics committees reviewing and approving clinical trials should have harmonised guidelines and standards for evaluating the relevance and robustness of preclinical models. Regulatory frameworks should facilitate the incorporation of novel and innovative patient-based models in the existing drug-development pipeline.

Relationship with clinical research and patient engagement

Interdisciplinary interactions between clinical and preclinical stakeholders are vital to address the causes of translational failure and enhance efforts to develop robust research models. Patient engagement is a key aspect of PM, including preclinical research activities, and should be facilitated and incentivised by all stakeholders with the aim to co-create research relevant to its end-users. Targeted public funding is required to support the creation of collaborative and multidisciplinary translational ecosystems that connect resources and services, to translate scientific discoveries into benefits for patients.

Development and validation of robust and predictive preclinical models that capture clinical phenotypes and enable patient stratification for complex diseases, is fundamental for further development of personalised approaches. Implementing these recommendations is ambitious, and can only be achieved by united efforts from research institutions and universities, industry, regulatory agencies and funders.

Clinical trials for personalised medicine

As more complex designs are implemented in PM research, it is essential to ensure that trial designs truly correspond to the research questions, and lead to robust and reproducible evidence. Four types of recommendations (general, personalised vs. non-personalised strategies, design-specific, topic-specific) for PM clinical trials were developed.

General

These recommendations emphasise the need to adhere to high standards of evidence generation in clinical research. It is important to ensure the balance between early access to innovative treatments in case of unmet medical need, and larger randomised trials required by health technology assessment (HTA) agencies. The risks and costs of recommending poorly evaluated treatment strategies should not be underestimated. Conducting larger randomised controlled trials (RCT) on a wisely chosen target can reduce those risks, even if they are lengthier and costlier²⁹.

• 3.4.1Fundamental principles of clinical evidence generation equally apply to PM, which includes principles of good clinical practice, notably the quality by design/proportionality frameworks³⁰, and the statistical principles of clinical trials^31,32. Of particular importance is the need for a control group, with random allocation as much as possible and concurrent controls. Randomisation is fundamental to eliminate biases, and separate the prognostic value of a biomarker from its ability to predict the treatment effect, as single-arm enrichment trials with only biomarker-positive participants cannot overcome those issues^33,34.
• 3.4.2PM trials should be designed for demonstrating a differential treatment effect according to the phenotype or genotype of participants. They should aim at demonstrating that the effect of an intervention depends on how individuals are characterised.
• 3.4.3Biomarkers and algorithms used in PM trials should have demonstrated external validity. Techniques employed to measure a biomarker in a trial, or algorithms used for stratification must be validated and CE-marked in the context where they would be routinely used.
• 3.4.4Routine care settings should be captured in trials as much as possible. Since the population treated through routine care is often broader than the trials’ population, and trial protocols may also modify management in the control group, it is important to capture routine care settings in trials as much as possible, and increase generalisability³⁵.

Personalised vs. non-personalised medicine strategies

Trials comparing a PM to a non-PM strategy are relevant for many stakeholders, in particular HTA bodies. No specific design was recommended for these trials, acknowledging that the study design should depend on the study objective(s) and knowledge about the biomarker(s). Also, we recognize that in certain situations such trials would be difficult, if not impossible to conduct, such as very rare subpopulations, or when the standard of care (SOC) is already personalised.

• 3.4.5A PM strategy should be compared to the SOC (including a non-personalised one when relevant) using RCTs, as much as possible.
• 3.4.6Trials comparing a personalised to a non-personalised strategy should account for the whole PM pipeline, including sample collection and analysis. This is particularly important for evaluating cost-effectiveness.

Design-specific

These recommendations are directed to the so-called master protocols (basket, umbrella and platform trials). These trial designs, commonly associated with PM⁸, raise several challenges both in terms of designs and evaluation by health authorities^36,37. While many issues have been covered in previous reviews, these are not specific to PM.

• 3.4.7For basket and umbrella trials, randomisation should be preferred for confirmatory trials. It is also important to use clinically relevant and patient-important outcomes. Of note, most basket and umbrella trials conducted until recently were non-controlled trials³⁸. Given the benefits of a control arm and randomisation in eliminating biases, these should be favoured.
• 3.4.8For platform trials, an intervention should be compared to concurrent controls; adding a new stratum when a new arm is added, which may raise issues to maintaining blinding in the case of multiple dummies³⁶. Also, at least one research question for biomarker-negative patients should be considered.

Topic-specific

These recommendations match different trial designs to the typology of PM, acknowledging that the choice of a particular design should depend on prior knowledge on the biomarker, status of development of the intervention, the context of use and the intended purpose of personalisation. Therefore, four typologies of PM have been distinguished, and matched to adequate trial designs in Table 1:

Table 1.

Clinical trial designs that are best suited for each typology of personalised medicine and relevant references.

Type of personalised medicine	Most suitable designs
Targeted medicine / trials	Biomarker strategy designs, adaptive biomarker designs, tandem two stage design, basket trial, umbrella trial33,34,38,50–52
Stratified medicine / trials	Classical designs separating within- and between-patient variability, marker stratified designs, hybrid designs, adaptive signature designs, multi-arm multi-stage design, platform trial, umbrella trial50,52
Individualised treatment / medicine	Classical designs separating within-and between-patient variability53–55
Individualised dynamic regimes	Sequential Multiple Assignment Randomized Trial (SMART) designs56

• Targeted medicine (precision medicine): treatment is adapted to the understanding of the pathophysiology of the disease, or one of its aspects, generally by targeting a gene or protein;

• Stratified medicine: patients are stratified into different clusters based on the collection of data characterising their phenotype or genotype, but with less emphasis on a precise molecular matching of the treatment;

• Individualised medicine: treatment is tailored to each patient rather than a group of patients (e.g. chimeric antigen receptor [CAR] T-cell therapies);

• Individualised medicine with a dynamic regime: treatment is adjusted over time depending on the patient’s response in addition to their individual characteristics (e.g. dose adaptation over time).

While designs used in PM trials vary according to the research question, stringent methodological standards should ensure robustness and reproducibility of trials and reduce bias. In addition, adoption of the PM strategy requires medico-economic assessment of the PM pipeline as compared to the non-personalised SOC.

There have been recent proposals for innovative trial designs, including the use of external or hybrid control arms (the latter referring to augmenting the control arm with external controls), the use of in silico trials and even the numerical generation of artificial controls using artificial intelligence algorithms trained on existing data^39–44. While in silico trials, modelling and simulation can be useful to optimise the design of clinical trials⁴⁵, the role of such innovative approaches in replacing the designs we have considered in this article and their acceptability by regulators and HTA bodies are still unclear and a matter of controversy^46,47. Those approaches are also not specific to PM, although they may also be valuable for PM. Of note, one study investigated how a hybrid control arm could have improved the estimation of the treatment effect in an umbrella early phase trial⁴⁸. No specific recommendation on the use of those innovative approaches for PM were developed, but they may be considered, and their role in the development of PM would deserve further investigation.

No specific recommendations were developed for the use of real-world evidence (RWE) but the role it can play in all stages of the PM pipeline must be recognized. In clinical trials they can be used to find external controls, or to train algorithms to generate virtual controls, or to transpose or project the treatment effect as estimated in a RCT in the target population. RWE extracted from electronic health records, clinical databases, patient registries, and even from wearables or patient-reported outcomes can inform the very first stage of the PM research pipeline. RWE may be particularly valuable for biomarker discovery in large diverse populations, assessing effectiveness of personalized interventions in routine practice, enabling long-term follow-up, and informing trial designs. However, careful consideration must be given to data quality, potential biases, and appropriate analytical methods when using RWE for PM. Researchers should consult established frameworks, such as the NICE RWE Framework, when incorporating RWE into PM studies.

Conclusion

The PERMIT methodological recommendations cover the full PM research pipeline to help PM stakeholders when designing, funding, conducting, publishing and validating complex and multistep PM research programmes. As with any new scientific field, PM still requires methodological research for instance, to improve sample size evaluation or to define standards for HTA evaluation of PM strategies. Such methodological development should contribute to the dialogue with regulators to optimise the regulatory ecosystem. Similarly, regulatory science research must address these arising challenges to make appropriate decisions on the validation of AI methods, on authorisation of complex clinical trials, on market access and reimbursement, and more. Furthermore, the ethical implications and challenges regarding data protection of PM research participants should be addressed in depth⁴⁹. As the field continues to evolve, the PERMIT recommendations should be broadly disseminated among the involved stakeholder communities – public and private investigators, industry, sponsors, regulators, HTAs, ethics committees, patient associations, healthcare providers, payers, funding bodies and journal editors - to strengthen the robustness of evidence, the impact of PM research and the quality of clinical care.

Author contributions

The manuscript was written through contributions of all authors. PG and JD coordinated the PERMIT project. JD obtained project funding. VF, EO, EG, RP, CS, JMH and JSM coordinated the dedicated working groups for the design and revision of the recommendations for each of the four stages of the pipeline. RB and CG coordinated the methodology. PG prepared the original draft and revisions. All authors have read and approved the final version of the manuscript.

Funding

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 874825.

Data availability

The data that support the findings of this study are available on Zenodo (https://zenodo.org/communities/permit-project/?page=1&size=20) and further data can be made available by the corresponding author upon request.

Declarations

Competing interests

The authors declare no competing interests.