Skip to main content
BMJ Open logoLink to BMJ Open
. 2024 Sep 5;14(9):e084119. doi: 10.1136/bmjopen-2024-084119

Implementing a pharmacogenomic-driven algorithm to guide antiplatelet therapy among Caribbean Hispanics: a non-randomised clinical trial

Hector J Nuñez-Medina 1,0, Mariangeli Monero 2,0, Lorna M Torres 1, Enrique Leal 1, Lorena Gonzalez-Sepulveda 3, Ángel M Mayor 4, Jessicca Y Renta 5, Edgardo R González-García 1, Ariel González 1, Kyle Melin 6, Stuart A Scott 7, Gualberto Ruaño 8, Dagmar F Hernandez-Suarez 9, Jorge Duconge 10,
PMCID: PMC11381646  PMID: 39242160

Abstract

Objectives

To assess whether genotype-guided selection of oral antiplatelet drugs using a clinical decision support (CDS) algorithm reduces the rate of major adverse cardiovascular and cerebrovascular events (MACCEs) among Caribbean Hispanic patients, after 6 months.

Design

An open-label, multicentre, non-randomised clinical trial.

Setting

Eight secondary and tertiary care hospitals (public and private) in Puerto Rico.

Participants

300 Caribbean Hispanic patients on clopidogrel, both genders, underwent percutaneous coronary intervention (PCI) for acute coronary syndromes, stable ischaemic heart disease and documented extracardiac vascular diseases.

Interventions

Patients were separated into standard-of-care (SoC) and genotype-guided (pharmacogenetic (PGx)-CDS) groups (150 each) and stratified by risk scores. Risk scores were calculated based on a previously developed CDS risk prediction algorithm designed to make actionable treatment recommendations for each patient. Individual platelet function, genotypes, clinical and demographic data were included. Ticagrelor was recommended for patients with a high-risk score ≥2 in the PGx-CDS group only, the rest were kept or de-escalated to clopidogrel. The intervention took place within 3–5 days after PCI. Adherence medication score was also measured.

Primary and secondary outcomes

The occurrence rate of MACCEs (primary) and bleeding episodes (secondary). Statistical associations between patient time free of events and predictor variables (ie, treatment groups, risk scores) were tested using Kaplan-Meier survival analyses and Cox proportional-hazards regression models.

Results

The genotype-guided group had a clinically lower but not significantly different risk of MACCEs compared with the SoC group (8.7% vs 10.7%, p=0.56; HR=0.56). Among high-risk score patients, genotype-driven guidance of antiplatelet therapy showed superiority over SoC in reducing MACCE incidence 6 months postcoronary stenting (adjusted HR=0.104; p< 0.0001).

Conclusions

The potential benefit of implementing our PGx-CDS algorithm to significantly reduce the incidence rate of MACCEs in post-PCI Caribbean Hispanic patients on clopidogrel was observed exclusively among high-risk patients, with apparently no evident effect in other patient groups.

Trial registration number

NCT03419325.

Keywords: cardiovascular disease, genetics, adverse events, clinical decision-making, risk factors


STRENGTHS AND LIMITATIONS OF THIS STUDY.

  • This study fills a critical gap in the field by examining the incidence rates of major adverse cardiovascular and cerebrovascular events (primary endpoint) and bleedings (secondary endpoint) among Caribbean Hispanic cardiovascular patients undergoing percutaneous coronary intervention, thus offering pertinent and timely data to the research community.

  • This study also contributes to address under-representation of minority populations in the field of clinical pharmacogenomics by being the first to externally validate a novel clinical decision support app for tailoring antiplatelet therapy in a Caribbean Hispanic population.

  • Our experimental approach allows for testing the benefits of using a customised genotype-guided prescribing algorithm to individually tailor oral P2Y12 inhibitor therapy after stratifying the population by risk scores.

  • Due to the nature of our study, a matched non-concurrent subcohort of patients was used as standard-of-care controls (non-randomised trial).

  • Our study was not powered to detect significant differences either among patients with and without the non-functional CYP2C19*2 allele or for individual events.

Introduction

Coronary heart disease (CHD) is the most common type of heart disease according to the Centers for Disease Control and Prevention, killing more than 382 000 people in 2020.1 According to the American Heart Association (AHA), Puerto Rico has the highest age-adjusted prevalence of CHD (6.6%) among all ethno-geographical regions.2 In the USA, Hispanics >40 years old have a higher prevalence of cardiovascular risk factors for CHD such as high cholesterol, diabetes, high blood pressure, obesity and smoking.3 4

Clopidogrel, a thienopyridine prodrug, is commonly prescribed as an antiplatelet therapy to prevent adverse cardiovascular outcomes (eg, stent thrombosis) or reduce the risk of further ischaemic events among patients undergoing percutaneous coronary interventions (PCI) for acute coronary syndromes (ACS) with ST-segment elevation myocardial infarction (STEMI), non-STEMI and unstable angina; or among patients with cerebrovascular accident; stable coronary artery disease (CAD), currently redefined as chronic coronary syndrome, specifically stable ischaemic heart disease (SIHD) and peripheral arterial occlusive disease (PAD).5,7 The highly polymorphic CYP2C19-mediated pathway is the most important drug-metabolising enzyme involved in the biotransformation of clopidogrel into an active metabolite that binds irreversibly to the ADP receptor P2Y12 on the platelet membrane to inhibit platelet aggregation.8 9 Despite its demonstrated efficacy, some studies have shown significant interindividual variability in response due to CYP2C19 genetic polymorphisms.10 11 The reduced function CYP2C19*2 variant is strongly associated with a poor metaboliser phenotype, which produces less of the clopidogrel active metabolite and, hence, a poor clinical outcome.12 Nevertheless, it is widely accepted that the prevalence of common CYP2C19 alleles significantly differs among various ethnic groups, along with the presence of ethno-specific variants, allelic heterogeneity, and variations in allelic effect sizes within specific populations.13 Patients who are resistant to clopidogrel are classified as having high on-treatment platelet reactivity.14

According to the Updated Expert Consensus Statement on Platelet Function and Genetic Testing for Guiding P2Y12 Receptor Inhibitor Treatment in PCI, there is conflicting evidence on whether or not guiding treatment based on genetics is beneficiary.15 The POPular trial study reported a reduction of thrombotic events when using a genetic risk score-guided treatment in European PCI patients.16 However, Caribbean Hispanics are often under-represented in pharmacogenomic studies, exacerbating healthcare disparities in this admixed and historically medically underserved population. Consequently, the clinical utility of implementing a genotype-guided antiplatelet therapy remains unclear for this population. This lack of information contributes to poor care management of this population and an increase in poor health outcomes.

Minority populations of ancestrally diverse backgrounds, bearing a disproportionate burden of chronic conditions like heart disease, lack representation in genetic research.13 17 This disparity in access to precision medicine exacerbates healthcare inequities. Misguided recommendations worsen outcomes, hindering pharmacogenetics (PGx) preventive goals. Predictive power hinges on diverse genomic data. Generalising pharmacogenomics to global health warrants caution while the inclusion of diverse participants is crucial for equity. Genetic knowledge gaps in Caribbean Hispanics, due to under-representation, hinder cardiovascular disease risk and clopidogrel response understanding. Unique genomic architectures across ethnicities necessitate expanded Hispanic studies to mitigate health disparities. Our study addresses this gap, providing essential major adverse cardiovascular and cerebrovascular event (MACCE) occurrence data and exploring risk score utility in antiplatelet therapy optimisation for Caribbean Hispanics.

The purpose of this study was to demonstrate the benefit of a PGx-guided approach (clinical decision support (CDS)) over standard of care (SoC) in reducing the occurrence of MACCEs among Caribbean Hispanic patients on clopidogrel. To reduce the incidence rate of MACCEs among Caribbean Hispanics on clopidogrel and validate a CDS tool designed to make actionable recommendations about the optimal treatment option in each patient, we aimed to assess statistical associations between patient time free of clinical endpoints and predictor variables (ie, treatment groups, risk scores ≥2) in this under-represented population using Kaplan-Meier survival analyses and Cox proportional-hazards regression models.

Methods

Trial design

A full description of the study design has been published.18 This was an open-label, multicentre, non-randomised clinical trial of pharmacogenomic-guided antiplatelet treatment optimisation, with a prospective and longitudinal data collection, conducted in patients undergoing PCI for ACS or SIHD and with documented extracardiac vascular disease (ie, PAD) from the Commonwealth of Puerto Rico. The study was registered at ClinicalTrials.gov with a unique identifier number: NCT03419325. All participants signed a written informed consent prior to enrolment.

Participants

A total of 300 Caribbean Hispanics patients on clopidogrel were recruited at eight medical facilities across the island of Puerto Rico (ie, Cardiovascular Center of Puerto Rico and the Caribbean, San Francisco Hospital, Pavia Hospital and UPR-Hospital Dr. Federico Trilla) from 11 January 2018 to 9 June 2022 and follow-up over 6 months (final date of follow-up was 8 December 2022). It ensures a comprehensive representation of the population diversity and complex healthcare landscape on the island, reflecting the distinct demographics, healthcare access and medical practices across different regions. Online supplemental table S1 summarises the inclusion/exclusion criteria for the study. Online supplemental material section provides further details on patients’ enrolment.

Collection of specimens and data

Individual clinical and demographic data were retrieved from electronic health records. Up to two 3.0 mL 3.2% citrate tubes of whole blood were collected from each participant (online supplemental figure S1). For DNA isolation, a 200 µL of blood was processed in the QIAcube following the QIAamp DNA Blood Mini Kit Protocol (QIAGEN, USA) and DNA quantification was performed using the NanoDrop 2000 Spectrophotometer.

Platelet functional testing

Collected blood samples from each patient were used to individually determine residual on-treatment platelet reactivity with the VerifyNow P2Y12 platelet function assay following manufacturer instructions. Results were reported as P2Y12 platelet reactivity units (PRU). A PRU cut-off value of 230 determined whether a patient was a poor (PRU ≥230; resistant) or normal (PRU <230; sensitive) responder to clopidogrel.

Genotyping

TaqMan SNP genotyping assays were run to ascertain genotypes at two SNPs included in the CDS algorithm: CYP2C19*2 (rs4244285) and PON1 p.R192Q (rs662). StepOne Real-Time PCR System and software V.2.3 were used to determine genotype calling following manufacturer instructions.

Clinical endpoints

The primary endpoint (MACCEs) was a composite of cardiovascular death, non-fatal myocardial infarction, ischaemic stroke, definite/probable stent thrombosis and hospitalisation for revascularisations over 6 months after index PCI, based on standard definitions.18 Reports of major and minor bleeding episodes were used as secondary endpoints. Major bleeding events included a need for blood transfusion, as well as any medical attention, admission to hospital, medical intervention or disability caused by the bleeding. In addition, minor bleeding events included bleeding gums, blood in urine, dark stools, bruises, nosebleeds and haematemesis when they do not meet the above criteria for major bleeding (eg, blood transfusion, medical attention and hospital admission). Primary and secondary endpoints were assessed at hospital discharge, by telephone (ie, 1 and 3 months after PCI) and at the scheduled 6-month follow-up visit. If patients did not attend the scheduled follow-up visit or could not be reached by telephone after multiple attempts, a medical record review to assess follow-up was conducted. All endpoints were reviewed and adjudicated by cardiologists who were blinded to study groups, risk scores (online supplemental figure S2) and treatments. Only study-related events confirmed by the cardiologists to be clinical endpoints were included in the analyses. All time-to-event endpoints were defined with time of intervention as time zero. Adherence medication score was measured using the Medical Outcomes Study (MOS) General Measures of Patient Adherence Survey Instrument.19 Incentives were provided to enhance compliance and retention.

Interventions

For comprehensive details regarding interventions tailored to each study subcohort and the specific dual antiplatelet therapy (DAPT) regimens administered, as well as the procedural flow charts (online supplemental figure S3A,B) outlining the clinical protocol for stable CAD/SIHD patients and those with documented extracardiac vascular disease (ie, PAD) or ACS patients undergoing PCI, refer to online supplemental materials. These flow diagrams and the accompanying narrative provide a complete description of dosing schemes, referrals, procedures and the number of participants at each stage of the study, including the number of candidates who met the study criteria and were referred by a cardiologist, examined for eligibility, confirmed as eligible, included in the study, completing follow-up and analysed. The interventions took place within 3–5 days following either elective or urgent PCI.

Briefly, patients were non-randomly separated into two groups (ie, SoC and genotype-guided, PGx-CDS) on a 1:1 ratio (150 each), and stratified by risk scores.18 Patients in the SoC group were all prescribed clopidogrel according to drug label instructions and current medical guidelines but regardless of their risk score. Actionable recommendations guided by the CDS algorithm were implemented in the genotype-guided group instead, with high-risk patients (scores ≥2) escalated to ticagrelor and those with low risk or inconclusive results kept or de-escalated to clopidogrel, as shown in online supplemental figures S1 and S3A,B. Recommendations for optimising individual antiplatelet therapy were available through our custom mobile phone application (app) within 2 weeks of hospital discharge. To ensure full engagement, cardiologists were educated on proper use of the mobile app. This is a clinician-oriented, PGx-based point-of-care app that was developed in-house by our bioinformatics and technology (IT) team to allow the integration of patient’s genotypes into informed clinical decision-making (ie, CDS tool) so that clinicians may easily apply algorithmically guided DAPT plans in real-time, on-demand, in clinical settings. The following information was collected at study checkpoints (ie, hospital discharge, by phone and follow-up visit) and through record review: occurrence of MACCEs and bleedings, sex, age, body mass index (BMI), comedications (ie, aspirin, proton pump inhibitors (PPIs), statins), comorbidities (ie, diagnosis of type 2 diabetes mellitus (T2DM), obesity, hypertension, dyslipidaemia), smoking status, length and number of adjoined stents ≥30 mm, left ventricular ejection fraction %, start of clopidogrel treatment and time elapsed since starting therapy. Genotyping and platelet function test results were also used for risk score calculations. Details on risk scores calculation and the PGx-CDS app used can be found in online supplemental figure S2.

Statistical analysis

A study sample size of 140 patients per group was calculated based on an expected effect size (HR=0.70) to have over 80% of power to detect statistical differences with alpha of 0.05. To account for a potential drop-out rate of 5%, enrolment of 300 patients (n=150 per group) was planned. To characterise the study cohort, a descriptive analysis of all demographics and clinical parameters was performed. Categorical data were summarised as frequencies (percentages). Continuous variables were reported as mean±SD, and as median and IQR. Statistically significant differences between groups were assessed, using either χ2 or Fisher’s exact probability tests for categorical variables, and two-tailed unpaired Student’s t-test or Mann-Whitney U tests, as appropriate (ie, when normality assumption was violated), for continuous variables (independent samples). Matched groups and adjustment for covariates were used to minimise potential bias due to non-randomisation.

Event rates were calculated and cumulative survival (event-free time) curves for the occurrence of the clinical endpoints during the follow-up period were constructed using the Kaplan-Meier method, and differences were assessed using the log-rank test. The length of survival (event-free time) was defined as the time of entry into the study (intervention) until the occurrence of an event (primary or secondary endpoints) or termination of antiplatelet therapy. Termination of treatments was considered censored observations at the time of receiving their last pills. The Kaplan-Meier analyses were truncated at 6 months of follow-up. Patients completing follow-up through the scheduled 6-month follow-up visit or telephone contact were censored at 182 days after index PCI. Patients who withdrew or who were lost to follow-up were treated as censored at the date of last contact. The HRs with 95% CI for time to first occurrence of the primary or secondary endpoints were calculated using univariate analysis. In addition, adjusted HRs were calculated using multivariate Cox proportional hazards regression analysis that included age, sex, BMI, T2DM, stent length, statins, aspirin and PPIs users as covariates in the model. Selected covariates were chosen from previous studies.16 18 20 21 A p≤0.05 was used to determine statistical significance. All analyses were performed with STATA statistical software for data science (V.18, StataCorp).

Patient and public involvement

Through our Community Engagement Core Unit of the Center for Collaborative Research in Health Disparities (Research Centers in Minority Institutions) at the UPR MSC, patients and the public were partially involved in the design and dissemination plans of our research by discussing this protocol with community leaders, members of local health organisations (eg, AHA), patients and patient advocates who provided feedback on how to conduct the study as well as the best way to disseminate findings from this experimental protocol.

Deviations from the published protocol

We decided to use the MOS General Measures of Patient Adherence Survey Instrument instead of the Morisky, Green and Levine (MAQ) Medication Adherence Scale in the public domain.

Results

Table 1 summarises baseline characteristics of participants in the study. The average age (median) of all participants in this study was 67 years old, with 52% self-identified as males. There was a high prevalence of conventional risk factors (ie, high BMI of 27.8 kg/m2; 59% with T2DM diagnosis). Overall, 14% of participants were smokers, 88% were using statins (mainly simvastatin) to treat hyperlipidaemia, 31% were on PPIs (mainly pantoprazole) and 71% were maintained on aspirin as part of DAPT. Among participants, 72% had ACS and 18% had SIHD. Moreover, 9.7% reported the occurrence of MACCEs and 11% reported a major bleeding episode. Carriers of at least one CYP2C19*2 allele represented 23% of all participants in both groups (MAF=12.8%; 95% CI 10.7% to 15.1%), whereas carriers of the PON1 rs662 (p.Q192R) variant represented about 60% of participants (MAF 47%; 95% CI 42.7% to 51.5%). Of 150 patients assigned to genotype-guided therapy, 36 received a recommendation for switching to ticagrelor (24%, risk score ≥2). All participants assigned to the SoC received clopidogrel.

Table 1. Baseline characteristics of study participants (n=300, Caribbean Hispanics).

Variables All patients (n=300) SoC control (n=150) PGx CDS guided (n=150) P value*
Mean±SD Median (IQR) Mean±SD Median (IQR) Mean±SD Median (IQR)
PRU 172.8±66.97 182.5 (91.3) 151.0±65.3 145.5 (90.0) 201.9±57.8 202.5 (62.3) 0.005
BMI (kg/m2) 27.8±6.81 27.5 (6.7) 28.13±6.03 27.4 (5.4) 27.36±7.74 28.0 (7.7) 0.517
Age (years) 66.65±11.58 67.0 (17.0) 67.43±11.58 68.0 (17.0) 65.59±11.55 65.0 (17.5) 0.203
Risk Score 0.69±1.10 0.00 (1.00) 0.26±0.68 0.00 (0.5) 1.26±1.29 1.00 (1.6) 0.001
N % N % N %
Type 2 diabetes mellitus 178 59.3 83 55.33 95 63.39 0.158
Hypertension 255 85.0 124 82.7 131 87.3 0.257
Dyslipidaemia 215 71.7 107 71.3 108 72.1 0.894
PAD 74 24.6 35 23.4 39 26.0 0.592
Gender (males) 156 52.0 79 52.67 77 51.34 0.816
Smoking 42 14.0 22 14.67 20 13.39 0.738
Length adjoined stents ≥30 mm 113 37.7 60 40.1 53 35.3 0.404
LVEF≥30% 23 7.67 12 8.00 11 7.40 0.828
MACCEs 29 9.67 16 10.70 13 8.67 0.557
Bleedings events 83 27.70 52 34.67 31 20.67 0.0067
Major bleedings 34 11.34 18 12.00 16 10.67 0.714
Aspirin user§ 213 71.00 108 72.00 105 70.00 0.702
Statins user 265 88.34 133 88.67 132 88.59 0.858
PPI user 94 31.34 46 30.67 48 32.21 0.803
CYP2C92 carriers 71 23.67 33 22.40 38 25.30 0.497
PON1 rs662 (p.Q192R) carriers 179 59.70 94 62.70 85 56.60 0.289

Summary descriptive statistics and differences between treatment groups (SoC and Ggenotype-guided groups) were statistically tested by either two-tailed, unpaired t-tests for independent samples (continuous variables), or the chi-squareχ2/Fisher’s exact probability tests (categorical variables, %).

Due to rounding errors, percentages may not equal 100%.

These variables were retrieved from patients’ medical records in accordance with the experimental design specified in the initially approved study protocol.18 Other variables not included in the approved protocol cannot be collected from medical records.

*

Alternatively, two-tailed Mann-Whitney U tests (unpaired Wilcoxon rank-sum) for continuous variables wasere considered when normality assumption was violated.

Some patients diagnosed with CAD/ACS, along with documented extracardiac vascular disease, also exhibit concurrent PAD.

Bleeding event is a combination of major and minor events.

§

Aspirin users over the entire follow-up period.

ACSacute coronary syndromesBMIbody mass indexCADcoronary artery diseaseCDSclinical decision supportLVEFleft ventricular ejection fractionMACCEsmajor adverse cardiovascular and cerebrovascular eventsPADperipheral artery diseasePGxpharmacogeneticPPIproton pump inhibitorsPRUplatelet reactivity unitPRUP2Y12 platelet reactivity unitsSoCstandard of care

Patient adherence scores above 82% were reported in both groups over the entire follow-up period. The major reasons for non-adherence to treatment in this study were consistent with previously reported studies.22 23 In general, we observed high adherence scores for patients in both groups and for the whole period of assessment (ie, 90%, 82% and 84% with high levels of adherence after the first, third and sixth month of follow-up, respectively). The most common reason for non-adherence to treatment was ‘forgetting to take the medication’ (23%), followed by ‘stopping taking the medication when they feel worse’ (9%) within the first 6 months of therapy. Other causes contributing to non-adherence were medication costs, transportation and lack of communication with the cardiologist. Furthermore, no significant association was found between the levels of adherence and time of treatment (p=0.0570). Strategies to ensure medication adherence can help reduce the worst outcomes.

Kaplan-Meier curves (survival analysis) and log-rank tests were used to assess the probability of participants being event-free after a follow-up period of 6 months when comparing the occurrence of endpoints over time between SoC and genotype-guided groups. The median follow-up period was 4.5 months. The occurrence of MACCEs at 6 months was reported in 13 and 16 patients in the genotype-guided and the SoC groups, respectively. Genotype-guided group had a clinically lower but not significantly different risk of MACCEs (8.67% vs 10.7%, p=0.56; HR=0.56, 95% CI 0.19 to 1.56) and major bleeding (10.7% vs 12.0%, p=0.71; HR=0.52, 95% CI 0.12 to 2.23) compared with those in the SoC group. Among all Caribbean Hispanic patients who underwent PCI, genotype-driven CDS algorithm-guided selection of an oral P2Y12 inhibitor (ie, clopidogrel vs ticagrelor), compared with SoC therapy, did not significantly reduce MACCEs occurrence over time based on the effect size that our study was powered to detect at 6 months. Our findings also showed that the overall frequency of MACCEs and major/minor bleeding episodes combined in the SoC was higher than that in the genotype-guided group (ie, 45.3% and 29.3%, respectively). However, the incidence rate of these major adverse events (ie, MACCEs and bleedings) was not significantly different between these two groups (ie, 2.13 cases per 1000 person-days vs 1.5 cases per 1000 person-days, respectively; HR 0.67, 95% CI 0.41 to 1.08, p=0.097) (figure 1). Importantly, the unadjusted OR for the association between these combinatorial adverse endpoints and treatment groups at 6 months was 2.205 (95% CI 1.26 to 3.77, p=0.0021), indicating that the odds of experiencing either an MACCE or a bleeding was two times higher among patients whose antiplatelet therapy was SoC versus those that were genotype guided (figure 2).

Figure 1. Kaplan-Meier plot of time free of both primary and secondary endpoints combined at 6 months of follow-up, grouped by treatment interventions (ie, SoC vs genotype-guided groups). The adjusted HR and the corresponding log-rank test p value are presented. Age, sex, BMI, T2DM, stent length, statins, aspirin and PPIs users were considered as covariates in the analysis. Data were censored for 6 months. BMI, body mass index; SoC, standard-of-care; T2DM, type 2 diabetes mellitus.

Figure 1

Figure 2. The PGx-guided CDS algorithm reduced the incidence rate of MACCEs and major/minor bleedings in Caribbean Hispanic patients. The incidence of MACCEs and major/minor bleeding events in the SoC group was 40% while in the genotype-guided group was 23%. The OR (95%CI) and p value of the association were computed using a one-sided χ2 test. Data are presented as frequency of events in patients. CDS, clinical decision support; MACCEs, major adverse cardiovascular and cerebrovascular events; PGx, pharmacogenetic; SoC, standard of care. ** indicates a very significant result (p < 0.01).

Figure 2

Furthermore, when stratifying by patient risk score (ie, high risk ≥2 vs low <2), a significant difference in MACCEs occurrence at 6 months of follow-up was detected between the SoC and genotype-guided treatment groups among patients with high risk but not those with low-risk scores (figure 3). Among high-risk patients, the incidence rate of MACCEs was 1.4 cases per 1000 person-days in the genotype-guided group (8 out of 36 cases with events reported within a total person-time at risk of 5772 days), whereas it was 11.39 cases per 1000 person-days in the SoC group (11 of 14 cases with events reported in a total person-time at risk of 965 days). Escalation to ticagrelor in the genotype-guided group resulted in a significant reduction in the risk of MACCEs compared with the SoC group who continued receiving DAPT with clopidogrel (unadjusted HR 0.15; 95% CI 0.06 to 0.36). The log-rank test for equality showed a statistically significant difference in event-free time functions between the two groups (genotype guided vs SoC, p<0.0001). Significant differences were also detected using multivariable Cox regression model, as shown in figure 4 (adjusted HR 0.104; 95% CI 0.037 to 0.293; p< 0.0001). Notably, the genotype-guided risk score discriminated between patients with and without high rates of MACCEs, indicating that this polygenic risk score is a useful predictor of poor clopidogrel outcomes among Caribbean Hispanics.

Figure 3. Kaplan-Meier plots of time free of primary endpoint (MACCEs) grouped by risk scores (6 months of follow-up). (A) The comparison between SoC and genotype-guided treatment groups for low-risk patients (risk score<2); (B) the comparison between SoC and genotype-guided treatment groups for high-risk patients (risk score ≥2). HRs and corresponding log-rank test p values are shown, and data are censored for 6 months. Age, sex, BMI, T2DM, stent length, statins, aspirin and PPIs users were considered as covariates in the analysis. BMI, body mass index; MACCEs, major adverse cardiovascular and cerebrovascular events; PPIs, proton pump inhibitors; SoC, standard of care; T2DM, type 2 diabetes mellitus.

Figure 3

Figure 4. Forest plot displaying HRs for MACCEs (with 95% CIs), absolute differences in percentages, and corresponding p values obtained from the multivariate Cox proportional hazards regression model adjusted for relevant covariates (sex, age, BMI, T2DM, stent length, statins, aspirin and PPIs users). The results illustrate the comparison between high-risk patients in both treatment groups (ie, PGx CDS vs SoC). #Median length of adjoined stents ≥30 mm. BMI, body mass index; CDS, clinical decision support; MACCEs, major adverse cardiovascular and cerebrovascular events; PGx, pharmacogenetic; PPIs, proton pump inhibitors; SoC, standard of care; T2DM, type 2 diabetes mellitus.

Figure 4

Discussion

In this study, the genotype-guided group exhibited a lower risk of MACCEs compared with the SoC group, although this difference did not reach statistical significance (8.7% vs 10.7%, HR=0.56, p=0.27). However, within the subset of patients with high-risk scores, genotype-guided antiplatelet therapy statistically demonstrated clear superiority over SoC in reducing the incidence rate of MACCEs 6 months post-PCI (adjusted HR=0.104; p<0.0001). Among high-risk patients, MACCEs occurred 1.8 times less frequently in the genotype-guided group, with an incidence rate of 1.4 cases per 1000 person-days vs 11.39 cases per 1000 person-days in the SoC group, indicating a substantial risk reduction when guiding therapy by genetically driven risk status. These results suggest that the use of our risk score-based algorithm to individually guide antiplatelet treatment in Caribbean Hispanics offers superior outcomes in terms of reducing MACCEs in high-risk patients compared with the SoC approach. However, this benefit was not observed for patients at low risk or for the secondary endpoint of bleeding episodes.

Of note, the thrombotic/bleeding trade-off of switching to ticagrelor/prasugrel versus continuing clopidogrel after PCI for high-risk patients remains unknown. In this study, significant differences in the occurrence of major/minor bleeding events were observed between the genotype-guided and SoC groups (p=0.0067), but not between high (escalated to ticagrelor) and low (maintain/de-escalated to clopidogrel) risk patients (p=0.37). However, there was a trend in the expected direction of fewer reported bleeding episodes among patients who maintained or de-escalated to clopidogrel compared with patients who switched to ticagrelor. Published studies in other populations suggest that ticagrelor is associated with a higher risk of bleeding.24 25

Kaplan-Meier survival analyses and Cox proportional-hazards regression models were used to assess the time-to-event endpoints and identify potential risk factors associated with MACCE occurrence in this specific population. Kaplan-Meier method served as a crucial tool in this study to estimate the cumulative probability of MACCE over time among the Caribbean Hispanic cohort. The findings yielded promising outcomes, as evidenced by survival curves displaying a clear separation between high-risk patients in the genotype-guided and SoC groups. This observation strongly supports the favourable effect of guiding antiplatelet therapy by a PGx-CDS algorithm to mitigate adverse cardiovascular events in this subpopulation. However, this study also highlighted that subgroups within the population might experience varying benefits. As shown in table 1, baseline PRU was on average significantly lower in the SoC than in the genotype-guided group (ie, 148 vs 202, respectively, p=0.005), which could have reduced the net effect of PGx-CDS guidance in the entire cohort.

Multivariate Cox proportional-hazards regression models were instrumental in identifying potential predictors of MACCEs among Caribbean Hispanics on DAPT (figure 4). Several demographic, clinical and genetic factors were assessed, offering valuable insights into the multifaceted nature of MACCE occurrence risk in this specific population. The identification of these risk factors could aid in risk stratification and individualised treatment plans for Caribbean Hispanics, ultimately leading to improved patient outcomes. Further investigations are warranted to fully comprehend the underlying mechanisms and interactions between these risk factors to enhance the precision of preventive measures and therapeutic interventions.

In the comparative analysis of high-risk patients (with scores ≥2) between both subcohorts, a statistical power exceeding 80% was achieved. This calculation was based on a total sample size of n=50 and an estimated HR for the primary endpoint of 0.10. However, our study was not powered to detect significant differences with an alpha of 0.05 among patients with and without the non-functional CYP2C19*2 allele or the PON1 rs662 (p.Q192R) variant. Nonetheless, among those carriers of at least one risk allele, a significant association with MACCEs was earlier found by our team (OR: 8.17, p=0.041).26 No significant effects were observed for the odds of having a single event in the study cohort. A similar CYP2C19*2 MAF of 13.7% was reported in another cohort of Caribbean Hispanics.27 Our results also showed a PON1 rs662 MAF of 47%, which is consistent with previous literature reports.21 28 29 The inclusion of the PON1 rs662 (p.Q192R) variant in our PGx-guided CDS algorithm is based on our prior findings in Caribbean Hispanics and those by others.18 30 31 Therefore, this study supports in part the observed association of this variant with higher PON1 enzymatic activity and a lower risk of cardiovascular conditions.

Based on our findings, we conclude that the implementation of a PGx-guided CDS algorithm has a potential to reduce MACCEs and improve health outcomes in high-risk Caribbean Hispanic CV patients undergoing PCI. This clinical study demonstrated that guiding individual antiplatelet therapy by a novel PGx-CDS algorithm is associated with a reduced incidence rate of MACCEs among high-risk Caribbean Hispanics (risk scores ≥2). The multivariate Cox proportional-hazards regression models provided important information on risk factors (eg, T2DM, BMI, PPIs users), enabling a better understanding of cardiovascular risk in this population and guiding personalised treatment approaches. These findings contribute to the growing body of knowledge on the management of cardiovascular diseases in Caribbean Hispanics and underscore the importance of tailored interventions to optimise patient care and outcomes in this under-represented population. Since age-adjusted cardiovascular mortality rate reduction across the USA is lagging among populations experiencing significant health disparities as well as economic and social vulnerability, tailored interventions targeting Caribbean Hispanics are needed to close the gap.32

Consequently, the adoption of a precision medicine paradigm that considers this genetic risk score-based strategy is expected to become a fundamental part of SoC when tailoring DAPT in this minority population with a high risk of thrombotic events. This study also contributes to address under-representation of minority populations in the field of clinical pharmacogenomics by being the first to validate a CDS tool for tailoring antiplatelet therapy in a Caribbean Hispanic population.

Over the last decade, genotype-guided escalation and de-escalation of antiplatelet therapy in post-PCI patients has become a standard practice in real-world clinical settings.20 33 In this study, antiplatelet therapy switches were similar to those previously reported (ie, 24% vs 19%),33 with escalation to ticagrelor occurring mainly among patients with the nonfunctional CYP2C19*2 allele. PGx-CDS has become a very useful tool to integrate individual genotyping information into existing clinical workflows and facilitate clinical translation of PGx. The use of our PGx-CDS platform as a mobile phone app to inform and guide cardiologists on DAPT prescribing decisions in real time also represents a novelty in the incorporation of digital health among Caribbean Hispanics.

Clopidogrel resistant patients are at an increased risk of MACCEs after PCI.12 14 Continuation of clopidogrel in these high-risk patients is, therefore, associated with adverse outcomes. The POPular Risk Score study reported a reduction of thrombotic events when applying a risk score-guided treatment in patients who underwent PCI.16 PGx-guided implementation studies in Europeans have found evidence of lower incidence of these ischaemic events and improved outcomes, but data from under-represented populations are lacking.15 16 This clinical study aimed to investigate the efficacy of guiding anti-platelet therapy using a PGx-CDS algorithm in reducing the incidence rate of MACCEs among Caribbean Hispanics. Due to logistical constraints and a small patient population, a non-randomised trial was more feasible. The findings from this research shed light on the impact of guiding anti-platelet therapy using a PGx-CDS algorithm in the context of Caribbean Hispanics’ cardiovascular health and provide valuable insights for future treatment strategies.

Finally, it is important to recognise the limitations of this study. First, the sample size, though adequate for many analyses, was not large enough to detect subtle differences related to CYP2C19*2 status and individual events. There is also the possibility that some participants may not report minor bleeding, which could introduce bias into the data collection process. Furthermore, the use of a matched non-concurrent cohort as SoC controls34 and potential selection bias need careful consideration in future research efforts. Since our study examines endpoints of varying severity, future work will focus on the interpretation and analysis of competing risk data. Therefore, it is crucial to interpret the results with caution and acknowledge the need for further confirmatory studies to validate and expand on these findings.

supplementary material

online supplemental file 1
bmjopen-14-9-s001.pdf (759KB, pdf)
DOI: 10.1136/bmjopen-2024-084119

Acknowledgements

The authors thank the patients for voluntarily participating in this study protocol. A special acknowledgement to the Research Design and Biostatistics Core service of the Hispanic Alliance for Clinical and Translational Research (Alliance), for helping with study design, sample size calculations and statistical analyses. Additionally, we also thank the Genomics Core of the CCRHD-RCMI Programme at the UPR-MSC, for assistance with genotyping. Special thanks to Drs Ednalise Santiago, Hilton Franqui-Rivera, Damian E. Grovas-Abad, Laura Ileana Fernandez-Morales, Luis Antonio Velez-Figueroa, Orlando Arce, Gretchen Gutiérrez, Mariela Loyola, Paola Pereira, as well as Frances Marín-Maldonado and Andrés López-Reyes for helping with sample and data collection.

Footnotes

Funding: This work was supported in part by CCRHD-RCMI grant #2U54 MD007600 from the National Institute on Minority Health and Health Disparities (NIMHD) of the National Institutes of Health (NIH), the Postdoctoral Master in Clinical and Translational Research Program of the Hispanic Clinical and Translational Research Education and Career Development (HCTRECD) award (grant #R25 MD007607, NIMHD, NIH) and by the National Institute of General Medical Sciences (NIGMS)-Support for Research Excellence (SuRE) Programme award 1R16 GM149372-01 and Research Training Initiative for Student Enhancement (RISE) Program grant R25 GM061838. The Hispanic Alliance for Clinical and Translational Research (Alliance) is supported by the National Institute of General Medical Sciences (NIGMS), NIH, under award #U54 GM133807.

Prepublication history and additional supplemental material for this paper are available online. To view these files, please visit the journal online (https://doi.org/10.1136/bmjopen-2024-084119).

Provenance and peer review: Not commissioned; externally peer reviewed.

Patient consent for publication: Consent obtained directly from patient(s).

Ethics approval: This study involves human participants and was approved by an Ethics Committee(s) or Institutional Board(s). Institutional Review Board of the University of Puerto Rico Medical Sciences Campus (Federal-wide Assurance #00005561). IRB-protocol number A4070417.

Data availability free text: The data that support the findings of this study are available from the corresponding author, JD, on reasonable request.

Patient and public involvement: Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.

Contributor Information

Hector J Nuñez-Medina, Email: hector.nunez@upr.edu.

Mariangeli Monero, Email: mariangeli.monero@upr.edu.

Lorna M Torres, Email: lorna.torres2@upr.edu.

Enrique Leal, Email: enrique.leal@upr.edu.

Lorena Gonzalez-Sepulveda, Email: lorena.gonzalez2@upr.edu.

Ángel M Mayor, Email: angel.mayor@uccaribe.edu.

Jessicca Y Renta, Email: jessicca.renta@upr.edu.

Edgardo R González-García, Email: edgardo.gonzalez13@upr.edu.

Ariel González, Email: arielgonzalezcordero@gmail.com.

Kyle Melin, Email: kyle.melin@upr.edu.

Stuart A Scott, Email: sascott@stanford.edu.

Gualberto Ruaño, Email: Gualberto.Ruano@hhchealth.org.

Dagmar F Hernandez-Suarez, Email: dagmar.hernandez@upr.edu.

Jorge Duconge, Email: jorge.duconge@upr.edu.

Data availability statement

Data are available on reasonable request.

References

  • 1.Centers for Disease Control and Prevention Heart disease facts. 2022. https://www.cdc.gov/heartdisease/facts.htm#:~:text=due%20to%20death.-,Coronary%20Artery%20Disease,killing%20382%2C820%20people%20in%202020.&text=About%2020.1%20million%20adults%20age,have%20CAD%20(about%207.2%25).&text=In%202020%2C%20about%202%20in,less Available.
  • 2.Virani SS, Alonso A, Aparicio HJ, et al. American heart association council on epidemiology and prevention statistics committee and stroke statistics subcommittee. Heart disease and stroke statistics-2021 update: A report from the American Heart Association. Circulation. 2021;143:e254–743. doi: 10.1161/CIR.0000000000000950. [DOI] [PubMed] [Google Scholar]
  • 3.Daviglus ML, Talavera GA, Avilés-Santa ML, et al. Prevalence of major cardiovascular risk factors and cardiovascular diseases among Hispanic/Latino individuals of diverse backgrounds in the United States. JAMA. 2012;308:1775–84. doi: 10.1001/jama.2012.14517. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Brook Stony Cardiovascular disease in hispanics/latinos in the United States and on long Island. Renaissance school of medicine at Stony Brook University. 2019. https://renaissance.stonybrookmedicine.edu/surgery/blog/cardiovascular-disease-in-hispanics-latinos Available.
  • 5.Lawton JS, Tamis-Holland JE, Bangalore S, et al. 2021 ACC/AHA/SCAI guideline for coronary artery revascularization: Executive summary: A report of the American College of Cardiology/American Heart Association joint committee on clinical practice guidelines. Circulation. 2022;145:e4–17. doi: 10.1161/CIR.0000000000001039. [DOI] [PubMed] [Google Scholar]
  • 6.Levine GN, Bates ER, Blankenship JC, et al. 2015 ACC/AHA/SCAI focused update on primary percutaneous coronary intervention for patients with ST-elevation myocardial infarction: An update of the 2011 ACCF/AHA/SCAI guideline for percutaneous coronary intervention and the 2013 ACCF/AHA guideline for the management of ST-elevation myocardial infarction. J Am Coll Cardiol. 2016;67:1235–50. doi: 10.1016/j.jacc.2015.10.005. [DOI] [PubMed] [Google Scholar]
  • 7.Bittl JA, Baber U, Bradley SM, et al. Duration of dual antiplatelet therapy: A systematic review for the 2016 ACC/AHA guideline focused update on duration of dual antiplatelet therapy in patients with coronary artery disease: A report of the American College of Cardiology/American Heart Association task force on clinical practice guidelines. Circulation. 2016;134:e156–78. doi: 10.1161/CIR.0000000000000405. [DOI] [PubMed] [Google Scholar]
  • 8.Herbert JM, Savi P, Maffrand JP. Biochemical and pharmacological properties of clopidogrel: A new ADP receptor antagonist. Eur Heart J. 1999;1:A31–40. [Google Scholar]
  • 9.Kazui M, Nishiya Y, Ishizuka T, et al. Identification of the human cytochrome P450 enzymes involved in the two oxidative steps in the bioactivation of clopidogrel to its pharmacologically active metabolite. Drug Metab Dispos. 2010;38:92–9. doi: 10.1124/dmd.109.029132. [DOI] [PubMed] [Google Scholar]
  • 10.Gent M. A randomised, blinded, trial of clopidogrel versus aspirin in patients at risk of ischaemic events (CAPRIE) The Lancet. 1996;348:1329–39. doi: 10.1016/S0140-6736(96)09457-3. [DOI] [PubMed] [Google Scholar]
  • 11.Quinn MJ, Fitzgerald DJ. Ticlopidine and clopidogrel. Circulation. 1999;100:1667–72. doi: 10.1161/01.cir.100.15.1667. [DOI] [PubMed] [Google Scholar]
  • 12.Simon T, Verstuyft C, Mary-Krause M, et al. Genetic determinants of response to clopidogrel and cardiovascular events. N Engl J Med. 2009;360:363–75. doi: 10.1056/NEJMoa0808227. [DOI] [PubMed] [Google Scholar]
  • 13.Wojcik GL, Graff M, Nishimura KK, et al. Genetic analyses of diverse populations improves discovery for complex traits. Nature New Biol. 2019;570:514–8. doi: 10.1038/s41586-019-1310-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Garabedian T, Alam S. High residual platelet reactivity on clopidogrel: its significance and therapeutic challenges overcoming clopidogrel resistance. Cardiovasc Diagn Ther. 2013;3:23–37. doi: 10.3978/j.issn.2223-3652.2013.02.06. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Sibbing D, Aradi D, Alexopoulos D, et al. Updated expert consensus statement on platelet function and genetic testing for guiding P2Y12 receptor inhibitor treatment in percutaneous coronary intervention. JACC Cardiovasc Interv. 2019;12:1521–37. doi: 10.1016/j.jcin.2019.03.034. [DOI] [PubMed] [Google Scholar]
  • 16.Janssen PWA, Bergmeijer TO, Vos G-JA, et al. Tailored P2Y12 inhibitor treatment in patients undergoing non-urgent PCI—the POPular Risk Score study. Eur J Clin Pharmacol . 2019;75:1201–10. doi: 10.1007/s00228-019-02696-z. [DOI] [PubMed] [Google Scholar]
  • 17.Duconge J, Ruaño G. Preventing the exacerbation of health disparities by iatrogenic pharmacogenomic applications: lessons from warfarin. Pharmacogenomics. 2018;19:875–81. doi: 10.2217/pgs-2018-0055. [DOI] [PubMed] [Google Scholar]
  • 18.Hernandez-Suarez DF, Melin K, Marin-Maldonado F, et al. Implementing a pharmacogenetic-driven algorithm to guide dual antiplatelet therapy (DAPT) in Caribbean Hispanics: protocol for a non-randomised clinical trial. BMJ Open. 2020;10:e038936. doi: 10.1136/bmjopen-2020-038936. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Hays RD. The Medical Outcomes Study (MOS) measures of patient adherence. https://www.rand.org/health-care/surveys_tools/mos/patient-adherence.html Available.
  • 20.Cavallari LH, Lee CR, Beitelshees AL, et al. Multisite investigation of outcomes with implementation of CYP2C19 genotype-guided antiplatelet therapy after percutaneous coronary intervention. JACC Cardiovasc Interv. 2018;11:181–91. doi: 10.1016/j.jcin.2017.07.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Hernandez-Suarez DF, Botton MR, Scott SA, et al. Pharmacogenetic association study on clopidogrel response in Puerto Rican Hispanics with cardiovascular disease: a novel characterization of a Caribbean population. Pharmgenomics Pers Med. 2018;11:95–106. doi: 10.2147/PGPM.S165805. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Ferdinand KC, Senatore FF, Clayton-Jeter H, et al. Improving medication adherence in cardiometabolic disease: practical and regulatory implications. J Am Coll Cardiol. 2017;69:437–51. doi: 10.1016/j.jacc.2016.11.034. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Muntner P, Mann DM, Woodward M, et al. Predictors of low clopidogrel adherence following percutaneous coronary intervention. Am J Cardiol. 2011;108:822–7. doi: 10.1016/j.amjcard.2011.04.034. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Harris J, Pouwels KB, Johnson T, et al. Bleeding risk in patients prescribed dual antiplatelet therapy and triple therapy after coronary interventions: the ADAPTT retrospective population-based cohort studies. Health Technol Assess. 2023;27:1–257. doi: 10.3310/MNJY9014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Jacobsen MR, Jabbari R, Engstrøm T, et al. Bleeding risk and P2Y12 inhibitors in all-comer patients with ST-segment elevation myocardial infarction treated with percutaneous coronary intervention: a single-centre cohort study. Eur Heart J Cardiovasc Pharmacother. 2023;9:617–26. doi: 10.1093/ehjcvp/pvad048. [DOI] [PubMed] [Google Scholar]
  • 26.Duconge J, Santiago E, Hernandez-Suarez DF, et al. Pharmacogenomic polygenic risk score for clopidogrel responsiveness among Caribbean Hispanics: A candidate gene approach. Clin Transl Sci. 2021;14:2254–66. doi: 10.1111/cts.13124. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Melin K, Moon JY, Qi Q, et al. Prevalence of pharmacogenomic variants affecting the efficacy of clopidogrel therapy in the Hispanic Community Health Study/Study of Latinos cohort. Pharmacogenomics. 2019;20:75–83. doi: 10.2217/pgs-2018-0148. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Luo JQ, Ren H, Liu MZ, et al. European versus Asian differences for the associations between paraoxonase-1 genetic polymorphisms and susceptibility to type 2 diabetes mellitus. J Cell Mol Med. 2018;22:1720–32. doi: 10.1111/jcmm.13453. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Macharia M, Kengne AP, Blackhurst DM, et al. Paraoxonase1 genetic polymorphisms in a mixed ancestry African population. Mediators Inflamm. 2014;2014:217019. doi: 10.1155/2014/217019. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Mackness B, Mackness MI, Arrol S, et al. Effect of the molecular polymorphisms of human paraoxonase (PON1) on the rate of hydrolysis of paraoxon. Br J Pharmacol. 1997;122:265–8. doi: 10.1038/sj.bjp.0701390. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Mackness M, Mackness B. Human paraoxonase-1 (PON1): Gene structure and expression, promiscuous activities and multiple physiological roles. Gene. 2015;567:12–21. doi: 10.1016/j.gene.2015.04.088. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Terry K, Makhlouf M, Altarabsheh SE, et al. Trends in cardiovascular disease mortality by county-level social vulnerability index in the United States. J Am Heart Assoc. 2023;12:e030290. doi: 10.1161/JAHA.123.030290. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Martin J, Williams AK, Klein MD, et al. Frequency and clinical outcomes of CYP2C19 genotype-guided escalation and de-escalation of antiplatelet therapy in a real-world clinical setting. Genet Med. 2020;22:160–9. doi: 10.1038/s41436-019-0611-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Yang G, González P, Moneró M, et al. Discovery of ancestry-specific variants associated with clopidogrel response among caribbean hispanics. Pharmacology and Therapeutics . 2023 doi: 10.1101/2023.09.29.23296372. Preprint. [DOI]

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    online supplemental file 1
    bmjopen-14-9-s001.pdf (759KB, pdf)
    DOI: 10.1136/bmjopen-2024-084119

    Data Availability Statement

    Data are available on reasonable request.


    Articles from BMJ Open are provided here courtesy of BMJ Publishing Group

    RESOURCES