Artificial-Intelligence Cloud-Based Platform to Support Shared Decision-Making in the Locoregional Treatment of Breast Cancer: Protocol for a Multidimensional Evaluation Embedded in the CINDERELLA Clinical Trial

Abstract

Background

Shared decision-making (SDM) plays a crucial role in breast cancer care by empowering patients and reducing decision regret. Patient decision aids (PtDAs) are valuable tools for facilitating SDM, now available in digital and artificial intelligence (AI)-powered formats to offer increasingly personalized contents. The ongoing CINDERELLA clinical trial (ClinicalTrials.gov: NCT05196269) evaluates an innovative AI cloud-based approach using a web platform and a mobile application (CINDERELLA APProach) versus the conventional approach to support SDM in breast cancer patients undergoing locoregional treatment. This protocol outlines a trial-based multidimensional evaluation, encompassing economic, financial, implementability, and environmental considerations associated with the CINDERELLA APProach.

Methods

A within-trial cost-consequence and cost-utility analysis from a societal perspective will be performed using patient-level data on outcomes and resource use. The latter will be valued in monetary terms using country-specific unit costs or patient valuations. A budget impact analysis will be performed over 1 and 5 years from the budget holder perspectives. The CINDERELLA APProach implementability will be assessed through an evaluation of its usability, acceptability, organizational impact, and overall feasibility. The environmental impact will be quantitatively assessed across several dimensions, such as quantity, appropriateness, and emissions, supplemented by qualitative insights. Overall, data for the evaluation will be gathered from patient questionnaires, interviews with patients and managers, focus groups with healthcare professionals, and app electronic data.

Discussion

A thorough understanding of the broad consequences of the CINDERELLA APProach may foster its successful translation into real-world settings, hopefully benefiting breast cancer patients and clinical practice.

Supplementary Information

The online version contains supplementary material available at 10.1007/s41669-024-00519-1.

Background

Breast cancer is the most commonly diagnosed malignancy in women worldwide, and its incidence is expected to rise steadily in the next decades [1]. Treatment of early breast cancer is complex and consists in a combination of locoregional treatments, such as surgery and radiotherapy, and systemic neoadjuvant and adjuvant therapies [2]. Breast cancer diagnosis and treatment often cause emotional distress, anxiety, depression, and have a negative impact on health-related quality of life (HRQoL) [3]. To properly address the psychological consequences that follow a diagnosis of breast cancer, healthcare professionals should provide tailored and comprehensive information to patients for them to cope successfully with the disease [4]. Shared decision-making (SDM) in breast cancer is key to fostering patients’ empowerment, improving their knowledge about the disease and the different therapeutic options available, and ultimately decreasing decision regret [5, 6]. SDM may be particularly important and appropriate in clinical situations where decisions are highly preference sensitive and/or there is equipoise, i.e., there are multiple treatment options available and no one is clearly superior to the others [7]. The decision-making process in the surgical treatment of early breast cancer exemplifies this context [8], justifying and calling for the application of SDM. Despite the recognized relevance of SDM in breast cancer’s locoregional treatment, its implementation in current clinical practice is far from widely diffused. According to a recent systematic review and meta-analysis, only 38% of breast cancer patients benefited from a SDM process in regards to their surgical treatment [9]. The successful implementation of SDM in breast cancer depends on several factors, including also the availability of decision-support tools such as patient decision aids (PtDAs) [10]. PtDAs are evidence-based tools designed to help patients make choices among healthcare options that align with their preferences and values, fostering their participation in the decisional process [11]. Over the years, several PtDAs have been developed for different diseases, with many formats that range from paper-based aids to computer-based, interactive decision tools [12, 13]. Digital PtDAs have the potential to be individually tailored, present a more complete content, and a greater degree of interaction and scalability [14, 15]. The advancement of artificial intelligence (AI) and its growing integration into the healthcare domain [16], including breast cancer [17], represent a further opportunity for augmenting PtDAs potentialities [18, 19].

The CINDERELLA clinical trial, built within the European Union (EU) funded CINDERELLA project (HORIZON-HLTH-2021-DISEASE-04. Project number 101057389), is a prospective multicenter randomized controlled trial (ClinicalTrials.gov: NCT05196269) aimed at comparing an innovative AI cloud-based approach (CINDERELLA APProach) versus the conventional approach to support SDM in breast cancer patients planned for locoregional treatment. The clinical protocol for the trial is described in detail elsewhere [20]. In the CINDERELLA arm, patients are provided with a PtDA, namely a mobile health application that simulates visually through AI the possible aesthetic outcomes of a certain type of surgical treatment, accompanied by an objective evaluation (e.g., good/fair). The primary objective of CINDERELLA APProach is to improve the match of expectations about the aesthetic outcome before and after treatment, with an expected positive impact on patients’ psychosocial distress and HRQoL [21]. Moreover, secondary objectives of CINDERELLA APProach include the increase in patient’s body image satisfaction after surgery, patients’ general HRQoL, and the reduction in health resource consumption.

Hence, the multifaceted nature and broad spectrum of objectives of this intervention call for a comprehensive assessment that carefully considers both costs and benefits across several domains. The construction of an extensive framework for the evaluation represents also an opportunity to enrich the literature on PtDAs, where little attention has been posed so far on an assessment that goes beyond clinical effectiveness.

Currently, there is no consensus on an economic framework for evaluating PtDAs, resulting in substantial heterogeneity of methods and uncertainty regarding their cost-effectiveness [25]. A paucity of economic evaluation studies is also registered for AI applications in healthcare, as reported in a recent systematic review [22]. This in turn generates uncertainty on the cost-effectiveness profile of such technologies and raises questions about the context in which they could generate savings.

Beyond economic considerations, the implementation of digital health interventions, which are complex and dynamic interventions [23], can pose significant challenges that need to be monitored. Implementation strategies are in fact multifaceted and must adapt to local contexts, which have multiple interacting actors (for example, patients and healthcare providers), leading to considerable variability across different settings [24]. Despite extensive research into digital health interventions, the question of how to effectively integrate these technologies into clinical practice remains largely unanswered [25].

Furthermore, although PtDAs have the potential to optimize resource utilization by better understanding patient preferences and values [26], their environmental impact remains unexplored [27].

Hence, to enrich the literature on methods for the assessment of health technologies and to contribute to the CINDERELLA clinical trial with a comprehensive understanding of the consequences following the deployment of an AI cloud-based platform to support SDM in the treatment of breast cancer, this protocol illustrates a trial-based multidimensional evaluation, which encompasses the economic and financial impact, the implementability, and the environmental impact of the CINDERELLA APProach.

Methods

Figure 1 illustrates a summary of the data collection process, sources, different methodologies, and timeline throughout the CINDERELLA clinical trial.

Fig. 1.

Timeline, methodologies, and sources of data collection throughout the CINDERELLA trial. eCRFs electronic case report forms, EQ-5D-5L EuroQol 5 dimensions 5 levels, HCP healthcare professionals, ICHOM International Consortium for Health Outcomes Measurement, MAUQ mHealth Apps Usability Questionnaire, TDABC time-driven activity-based costing, uMARS user version of the Mobile App Rating Scale

Economic Evaluation

The objective of the economic evaluation is to evaluate the costs and consequences of the CINDERELLA APProach versus the standard of care over the trial period through trial-based cost-consequence and cost-utility analyses, conducted in accordance with the Professional Society for Health Economics and Outcomes Research (ISPOR) Good Research Practices [28].

The patients will be followed from randomization to 12 months after the end of locoregional treatment (either surgery or radiotherapy, depending on the last treatment received). The total duration of the trial will be approximately 36 months, with recruitment spanning approximately 18 months. The expected sample size is 1030 patients (515 in each arm), calculated based on the two primary objectives (i.e., (i) differences in pretreatment expectations and the final aesthetic outcomes between the trial arms; (ii) agreement of the pretreatment AI evaluation and patients’ post therapy self-evaluation in treatment arm) and adjusted for potential dropouts [20].

The economic evaluation analyses will be carried out from a societal perspective and will cover the jurisdictions involved in the clinical trial (i.e., Portugal, Germany, Italy, Israel, and Poland). The Consolidated Health Economic Evaluation Reporting Standards for Interventions That Use Artificial Intelligence (CHEERS-AI) will be followed when reporting the health economic evaluation [29, 30].

Identification, Measurement, and Valuation of Costs

According to the societal perspective, all cost categories will be included in the analysis (direct healthcare costs, direct nonhealthcare costs, and indirect costs). Table 1 provides, for each resource use input, a summary of the sources and timing of measurement, as well as valuation sources for the different jurisdictions. Resource use data will be collected until 12 months follow-up through different sources: (i) trial electronic case report forms (eCRFs); (ii) online questionnaires filled in by the healthcare professionals (hereafter, HCP questionnaires) in presence of the patients; (iii) online questionnaires filled in directly by the patients (hereafter, patient questionnaires), which will be translated in the main language in use at the trial centers; (iv) interviews conducted with the staff and healthcare personnel in participating clinical centers, adopting the Time-Driven Activity-Based Costing (TDABC) methodology for estimating personnel cost; and (v) interviews with CINDERELLA platform developers (i.e., CANKADO).

Table 1.

Measurement and valuation of resource use

Cost category	Measurement		Valuation
	Source	Timing	Portugal	Germany	Italy	Poland	Israel
Direct healthcare costs
Hospitalizations	HCP questionnaires	Presurgery; wound healing; 6 months follow-up; 12 months follow-up	Diagnosis-related group (DRG) tariffs (Portuguese National Health Service–Central Administration of the Health System)	Diagnosis-related group (DRG) tariffs (Institute for the hospital remuneration system–InEK)	Diagnosis-related group (DRG) tariffs (Italian Ministry of Health)	Diagnosis-related group (DRG) tariffs (National Health Fund)	Procedure-related group (PRG) tariffs (Israeli Ministry of Health) Per-diem rate (Israeli Ministry of Health)
Access to emergency department	HCP questionnaires	Presurgery; wound healing; 6 months follow-up; 12 months follow-up	Benchmark terms for hospital contractualization (Portuguese National Health Service–Central Administration of the Health System)	Diagnosis-related group (DRG) tariffs (Institute for the hospital remuneration system–InEK)	Literature	Lump sum tariff (National Health Fund)	National price list (Israeli Ministry of Health)
Radiotherapy	eCRFs	Randomization (baseline); wound healing (eCRFs)	Benchmark terms for hospital contractualization (Portuguese National Health Service–Central Administration of the Health System) Annual payer contracts (Contratos Programa–CP) Hospital records	Diagnosis-related group (DRG) tariffs (Institute for the hospital remuneration system–InEK) Outpatients fees (Einheitlicher Bewertungsmaßstab–EBM; Gebührenordnung für Ärzte–GOÄ) Hospital records	National tariffs (Italian Ministry of Health) Hospital records	Diagnosis-related group (DRG) tariffs (National Health Fund) Hospital records	National price list (Israeli Ministry of Health) Hospital records
Neoadjuvant therapy	eCRFs	Presurgery; wound healing (eCRFS)	Diagnosis-related group (DRG) tariffs (Portuguese National Health Service–Central Administration of the Health System) Centralized purchasing database (Portal BASE) Hospital records	Diagnosis-related group (DRG) tariffs (Institute for the hospital remuneration system–InEK) Hospital records	Diagnosis-related group (DRG) tariffs (Italian Ministry of Health) National price list (AIFA) Hospital records	Diagnosis-related group (DRG) tariffs (National Health Fund) Hospital records	National price list (Israeli Ministry of Health) Hospital records
Breast surgery	eCRFs; HCP questionnaires	Randomization (baseline); wound healing (eCRFs) Surgery (HCP questionnaires)	Hospital records	Hospital records	Hospital records	Hospital records	Hospital records
Imaging tests	HCP questionnaires	Presurgery; wound healing; 6 months follow-up; 12 months follow-up	National tariffs (Portuguese National Health Service-Central Administration of the Health System)	National tariffs (Einheitlicher Bewertungsmaßstab-EBM; Gebührenordnung für Ärzte-GOÄ)	National tariffs (Italian Ministry of Health)	National tariffs (National Health Fund)	National price list (Israeli Ministry of Health)
Laboratory tests	HCP questionnaires	Pre-surgery; wound healing; 6 months follow-up; 12 months follow-up	National tariffs (Portuguese National Health Service-Central Administration of the Health System)	National tariffs (Einheitlicher Bewertungsmaßstab–EBM; Gebührenordnung für Ärzte–GOÄ)	National tariffs (Italian Ministry of Health)	National tariffs (National Health Fund)	National price list (Israeli Ministry of Health)
Visits with specialists	HCP questionnaires	Pre-surgery; wound healing; 6 months follow-up; 12 months follow-up	Benchmark Terms for Hospital Contractualisation (Portuguese National Health Service–Central Administration of the Health System)	Consultation fees (Einheitlicher Bewertungsmaßstab–EBM; Gebührenordnung für Ärzte–GOÄ)	National tariffs (Italian Ministry of Health)	National tariffs (National Health Fund)	National price list (Israeli Ministry of Health)
Visits with general practitioner	HCP questionnaires	Presurgery; wound healing; 6 months follow-up; 12 months follow-up	National Auditing Body	Consultation fees (Einheitlicher Bewertungsmaßstab–EBM; Gebührenordnung für Ärzte–GOÄ)	Publicly available report	National tariffs (National Health Fund)	Capitation fees
Medicinal products	HCP questionnaires	Presurgery; wound healing; 6 months follow-up; 12 months follow-up	National drug price list (INFARMED)	Reference drug price list (BfArM)	National drug price list (AIFA)	National drug price list (National Health Fund)	National drug price list (Israeli Ministry of Health)
Personnel	Interviews (TDABC); Patient questionnaires	Alongside clinical trial (interviews) Presurgery; wound healing; 6 months follow-up; 12 months follow-up (patient questionnaires)	Salary tables (Portuguese Ministry of Health) Labour trade union contracts	Publicly available reports	Salary tables (Italian Ministry of Economics and Finance–National Accounts) Labor trade union contracts	Lump sum fees (National Health Fund)	Average yearly salaries (OECD database)
CINDERELLA APProach cost (e.g., training, equipment)	Interviews	Alongside clinical trial	Financial/administrative records	Financial/administrative records	Financial/administrative records	Financial/administrative records	Financial/administrative records
Direct nonhealthcare costs
Travel expenses	Patient questionnaires	Presurgery; wound healing; 6 months follow-up; 12 months follow-up	Patient questionnaires	Patient questionnaires	Patient questionnaires	Patient questionnaires	Patient questionnaires
Accommodation expenses	Patient questionnaires	Presurgery; wound healing; 6 months follow-up; 12 months follow-up	Patient questionnaires	Patient questionnaires	Patient questionnaires	Patient questionnaires	Patient questionnaires
Indirect costs
Patients’ productivity losses	Patient questionnaires	Presurgery; wound healing; 6 months follow-up; 12 months follow-up	Average country-specific yearly salaries (OECD database)	Average country-specific yearly salaries (OECD database)	Average country-specific yearly salaries (OECD database)	Average country-specific yearly salaries (OECD database)	Average country-specific yearly salaries (OECD database)
Patients’ leisure time lost	Patient questionnaires	Presurgery; wound healing; 6 months follow-up; 12 months follow-up	Average country-specific yearly salaries (OECD database)	Average country-specific yearly salaries (OECD database)	Average country-specific yearly salaries (OECD database)	Average country-specific yearly salaries (OECD database)	Average country-specific yearly salaries (OECD database)
Informal care	Patient questionnaires	Presurgery; wound healing; 6 months follow-up; 12 months follow-up	Average country-specific yearly salaries (OECD database)	Average country-specific yearly salaries (OECD database)	Average country-specific yearly salaries (OECD database)	Average country-specific yearly salaries (OECD database)	Average country-specific yearly salaries (OECD database)

eCRFs electronic case report forms, HCP healthcare professionals, OECD Organisation for Economic Co-operation and Development, TDABC time-driven activity-based costing

All resource use will be valued in monetary terms using appropriate country-specific unit costs or participant valuations (e.g., for travel and accommodation expenses). Unit costs will be derived from national costing manuals (e.g., formularies, official national list pricing), the hospitals’ financial and administrative records (e.g., sourced from the management and control office and/or from the clinical pharmacy), publicly available datasets [e.g., Organisation for Economic Co-operation and Development (OECD) data] or sourced from the literature when not available from the previously listed sources. In case multiple sources are available for the same cost item, scenario analyses will be conducted using different cost sources. All costs will be expressed in Euros and referred to 2023, the year of trial start.

Identification, Measurement, and Valuation of Benefits

The benefits of the intervention will be estimated by considering patient-reported outcomes (PROs), that will be measured in the trial using questionnaires administered to patients in both arms at different time points (Table 2). For conducting the cost-consequence analysis, a variety of outcomes will be considered, namely expectations about the aesthetic outcome, Quality-Adjusted Life Years (QALYs) and patient satisfaction. For cost-utility analysis, only QALYs will be considered.

Table 2.

Outcomes for economic evaluation

Type of outcome	Questionnaire	Outcome measure	Timing	Used in
Expectation about aesthetic outcome	Expectations Questionnaire	Expectation score	Presurgery; wound healing; 6 months follow-up; 12 months follow-up	Cost-consequence analysis
Health-related quality of life	EQ-5D-5L	QALYs	Presurgery; wound healing; 6 months follow-up; 12 months follow-up	Cost-consequence analysis Cost-utility analysis
Satisfaction	BREAST-Q ICHOM	Domain scores	Presurgery; wound healing; 6 months follow-up; 12 months follow-up	Cost-consequence analysis

EQ-5D-5L EuroQol 5 dimensions 5 levels, ICHOM International Consortium for Health Outcomes Measurement, QALYs Quality-Adjusted Life Years

The patients’ match of expectations about the aesthetic outcome of locoregional treatment, before and after treatment, will be measured through the Expectations Questionnaire, developed ad hoc for the CINDERELLA trial. Patients’ expectations will be recorded using a five-point scale: much worse, worse, same, better, much better [20]. QALYs will be calculated from responses to the EQ-5D-5L questionnaire [31]. The EQ-5D-5L questionnaire is a generic quality of life instrument with questions covering five domains: mobility, self-care, usual activities, pain and discomfort, anxiety and depression. Each dimension is divided into five levels of severity: no problems, slight problems, moderate problems, severe problems, and extreme problems. First, the responses on the five domains of the EQ-5D-5L will be converted into a synthetic index using utility scores. This index is anchored between 0, corresponding to dead, and 1, corresponding to perfect health. Country-specific utility values will be used for Portugal, Germany, Italy, and Poland. If not available at the time of data analysis, a value set from a comparable country will be used for Israel. QALYs will be computed by multiplying the duration of time spent in a determined health state by the utility of that health state. Patient satisfaction will be recorded through the BREAST-Q International Consortium for Health Outcomes Measurement (ICHOM) [32, 33], a disease-specific PRO measure. In its complete version, it aims at measuring the satisfaction and HRQoL in patients undergoing breast surgery. In this trial, only the subdomain concerning patients’ satisfaction with breasts and breast cancer treatment will be considered. Patient satisfaction will be measured on a four-point scale: very dissatisfied, somewhat dissatisfied, somewhat satisfied, very satisfied. Responses will be converted into a synthetic score, ranging from 0 to 100 (the higher the score, the better the outcome), using questionnaire-specific conversion tables.

Statistical Analysis

The main analyses will be carried out considering the “intention to treat” (ITT) population to avoid affecting the randomized allocation. In the ITT analysis, all randomized patients will be included, regardless of actual intervention received or adherence to the intervention protocol [34]. This includes patients who did not use the CINDERELLA APProach in the intervention arm and those who dropped out from either arm. This is a realistic approach that reflects the clinical scenario in the real-world, where noncompliance, dropouts and protocol deviations may frequently happen. A per-protocol analysis will also be conducted [35]. This analysis will include only a subset of the ITT population who completed the study without any major protocol violations (i.e., patients who did not adhere to treatment, switched groups, missed measurements, or dropped out will be excluded).

Descriptive statistics will be presented along with statistical tests of difference, either parametric (for normally distributed data) or nonparametric (for nonnormally distributed data) as appropriate.

The use of resources will be estimated for both arms. Differences in the consumption of resources will be qualitatively described but not compared statistically. Mean and median total costs, also disaggregated by cost category, will be computed and appropriate measures of dispersions (e.g., 95% confidence intervals) will be reported. Multivariable regression analyses will be performed to assess the impact on costs and outcomes of different baseline variables that, according to the literature, are expected to have an impact on SDM, patient outcomes, and/or costs. The set of variables includes age, education level, marital status, comorbidity profile (Charlson or Elixhauser index), employment status, digital health literacy, and trial country. As an example, marital status is used as a proxy of family support, which has been documented as significantly associated with patients’ HRQoL in breast cancer patients [36]. Employment status is considered given that recent works [37] showed that, during oncology consultations, employed patients with a recent diagnosis of breast cancer asked significantly more questions than nonemployed and retired patients, which may suggest a more pronounced attitude toward SDM. The threshold for statistical significance will be set at 0.05.

Trial data will be examined for any missing data. The appropriate method for dealing with missing data will depend on the proportion of missing data and likely mechanism of missingness [i.e., missing completely at random (MCAR); missing at random (MAR); missing not at random (MNAR)] [38]. Multiple imputation will be undertaken to account for MAR and MCAR.

Analyses will be conducted using StataSE, R and Excel (latest versions available).

Cost-Consequence and Cost-Utility Analyses

The mean differences in costs and outcomes for the two arms will be provided. The results of the cost-consequence analysis will be presented in a disaggregated format, with costs and outcomes (i.e., expectations about the aesthetic outcome, QALYs and patients’ satisfaction) reported separately. Estimates (mean and measure of dispersion) will be provided for each arm, along with the incremental differences between arms. Trade-offs between costs and effects will not be explicitly provided. For the cost-utility analysis, the incremental cost–utility ratio (ICUR) will be estimated, dividing the difference in total mean costs between the CINDERELLA APProach and the standard of care arm by the differences in QALYs. The ICUR will be interpreted using appropriate cost-effectiveness thresholds. As none of the jurisdictions involved in the clinical trial provides an explicit, discrete value for cost-effectiveness threshold, the cost-utility analysis will use two generic cost-effectiveness thresholds of 50,000 €/QALY and 100,000 €/QALY. Discounting will not be applied to either costs or benefits as the economic evaluation will be performed over a time horizon of 12 months.

Sensitivity analyses will be conducted to explore uncertainty surrounding key parameters and determine the reliability of the results from the base-case analysis [28]. Univariate sensitivity analysis will be performed by varying one key parameter at a time across its plausible range, while the remaining values are held at their baseline values. Results will be presented in table format and/or graphically through a tornado plot. Probabilistic sensitivity analysis will be performed by assigning to each key input parameter a specified distribution and by drawing randomly from this distribution. The nonparametric bootstrapping approach will be used to determine the level of sampling uncertainty surrounding the mean costs, outcomes, and cost-effectiveness summary measure (ICUR) by generating 10,000 estimates of (incremental) costs and benefits. To represent decision uncertainty in the cost-utility analysis, cost-effectiveness acceptability curves (CEACs) will be used. CEACs display the proportion of the estimates produced by bootstrapping that would be considered acceptable over a range of willingness-to-pay (WTP) thresholds. Results will be presented in table format and/or graphically through a cost-effectiveness plane when appropriate (e.g., for cost-utility analysis).

Subgroup analyses will be conducted to investigate how costs and outcomes vary between different patient subgroups. Following the same rationale explained in the previous section, several baseline variables will be used for conducting these analyses, including age, education level, marital status, comorbidity profile, employment status, digital health literacy, trial country.

Budget Impact Analysis

The economic evaluation will be complemented by a budget impact analysis to assess the financial impact of the CINDERELLA APProach implementation. This analysis will be conducted from the healthcare provider and the healthcare system perspectives (i.e., the budget holders’ perspectives) over two different time horizons: 1 year (short term) and 5 years (medium term), in accordance with the ISPOR Good Research Practices [39].

The cost analysis will be drawn from that conducted for economic evaluation, although only hospital-related costs will be included when considering the healthcare provider perspective, and direct healthcare costs will be included when considering the healthcare system perspective. Discounting will not be applied as the budget holder is interested in the expected financial streams at each point in time rather than their net present values.

To conduct the budget impact analysis, which compares the status quo (without the CINDERELLA APProach) with a future scenario entailing the gradual uptake of the CINDERELLA APProach, additional parameters besides resource use and costs will be estimated, namely the eligible population and the uptake of the new intervention. The eligible population consists in all patients eligible for the new intervention, i.e., the CINDERELLA APProach, during the time horizon of interest, given any access restrictions. The definition of the eligible population will start with an analysis of country-specific epidemiological data, namely disease prevalence and incidence, which will be sourced from either European-based databases (e.g., European Cancer Information System [40]) or national cancer registries (e.g., AIRTUM–Italian Network of Cancer Registries [41]). This will be complemented with an estimate of the number of patients covered by the locally approved indications for the new technology (i.e., breast cancer patients eligible for locoregional treatment). The uptake of the CINDERELLA APProach is, by definition, not known at the time of analysis, therefore some hypotheses on its diffusion need to be made and included in the future scenario. More specifically, two different changes will be included in the analysis: (i) the CINDERELLA APProach is added to the standard of care (combination) and (ii) the CINDERELLA APProach replaces the standard of care (substitution). Expert opinions will be collected to hypothesize credible levels of uptake for the new intervention.

The total costs of the status quo and the future scenario will be estimated and compared to estimate the financial impact of the uptake of the CINDERELLA APProach.

Implementability

The effectiveness of an intervention is critically influenced by its implementation in a given context [42]. Implementation science has emerged as the scientific study of methods to promote the systematic dissemination of research results in clinical routines, with the aim of improving the quality and effectiveness of health services [43]. By embedding an implementation research component into the CINDERELLA trial, we aim to foster the successful diffusion of the CINDERELLA APProach via a thorough understanding of the factors that affect its translation into real-world settings and an evaluation of its usability, acceptability, organizational impact, and overall feasibility.

Usability

Usability can be defined as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” [44]. User evaluation is one of the most commonly used method for assessing the usability of mobile applications [45], with several rating scales developed to evaluate the usability of mHealth apps [46, 47]. The usability of the CINDERELLA mobile application will be evaluated through an online questionnaire administered to patients in the intervention arm at 6 and 12 months follow-up. The questionnaire was developed starting from two previously validated questionnaires, namely the standalone version of the mHealth Apps Usability Questionnaire (MAUQ) for patients [48] and the user version of the Mobile App Rating Scale (uMARS) [49]. The MAUQ standalone version for patients contains 18 items, referred to three dimensions of app usability (i.e., ease of use, interface and satisfaction, usefulness). For each item, patients will be asked to express their degree of accordance on a seven-point scale (strongly agree, agree, somewhat agree, neither agree nor disagree, somewhat disagree, disagree, strongly disagree). From the uMARS questionnaire, we extracted only the four questions concerning the information quality subscale, whose answers are on a five-point scale. The questionnaires were translated into the main language in use at the trial centers.

Data on CINDERELLA usability derived from the questionnaire will be analyzed quantitatively. First, a principal components analysis (PCA) will be performed [50] to evaluate the psychometric properties of the questionnaire obtained from the integration of MAUQ and uMARS questionnaires, separately for each trial country to accommodate differences in questionnaire language, and the resulting subscales will be tested for internal consistency (Cronbach’s alpha). Then, a descriptive analysis will be conducted on the collected data. The means and standard deviations (SD) for individual statements, subscales and the entire questionnaire will be computed. Based on data distribution, parametric (e.g., t-test) or nonparametric tests (e.g., Kruskal–Wallis test) will be used to investigate differences among patients according to different baselines characteristics (e.g., age, education level, digital health literacy).

Acceptability

Acceptability is a multifaceted concept and, to date, there is still no consensus around its definition [51]. Sekhon and colleagues (2017) defined acceptability of healthcare interventions as “the extent to which people delivering or receiving a healthcare intervention consider it to be appropriate, based on anticipated or experienced cognitive and emotional responses to the intervention” [52], and Weiner and colleagues (2017) as “the perception among implementation stakeholders that a given treatment, service, practice, or innovation is agreeable, palatable, or satisfactory” [53]. Acceptability can be related to several concepts, including users’ attitude (e.g., how they feel about the intervention), usage intentions (e.g., willingness to engage), actual usage (e.g., number of interactions), and satisfaction after having engaged with the intervention [54]. The users’ perception of acceptability can be measured at different points in time, namely before (prospective acceptability), whilst (concurrent acceptability), and after (retrospective acceptability) engaging with the intervention [52]. For the purpose of this study, the acceptability of the CINDERELLA APProach will be measured according to two different user perspectives, i.e., the patients’ and the healthcare professionals’ perspective, as the successful implementation of a healthcare intervention depends on its acceptability to both intervention deliverers and recipients [55]. Moreover, data collection will be performed at different time points to capture both concurrent and retrospective acceptability.

Acceptability for patients will be referred strictly to the CINDERELLA mobile application and will be measured alongside different dimensions and using different data collection methods. First, the app’s actual usage will be evaluated using data collected electronically by the CANKADO system (e.g., number of log ins, app retention rates) throughout the clinical trial (concurrent acceptability). Second, patient satisfaction will be derived from the related section in the MAUQ questionnaire [48] at 6 and 12 months follow-up (concurrent and retrospective acceptability).

Quantitative data on CINDERELLA acceptability for patients (i.e., app’s usage and patients’ satisfaction) will be summarized using medians, interquartile ranges, means, standard deviations, counts, and percentages as appropriate. Based on data distribution, parametric (e.g., t-test) or nonparametric tests (e.g., Kruskal–Wallis test) will be used to investigate differences among patients according to different baselines characteristics (e.g., age, education level, digital health literacy).

Third, semi-structured interviews will be conducted with a sample of patients to measure retrospective acceptability at the end of the trial period (12 months follow-up). The final sample size for this study phase is not determined a priori but will be based on the data saturation principle [56]. We will set an initial sample equal to ten patients; after the first ten interviews, three further interviews will be conducted to verify whether there are any emerging themes [57]. The interview process will continue until no new insights emerge from the interviews. The enrolment will be done by the healthcare professionals at each clinical center, and the interviews will be conducted remotely with commonly used teleconferencing tools (e.g., Microsoft Teams) to accommodate the geographical dispersion of the interviewees. Every interview will last approximately 40 minutes. The interviews will be conducted making efforts to ensure that the interviewers speak the language of the interviewees. All interviews will be audio-recorded and transcribed. When needed, the interview transcripts will be translated into English. Back translation will be done to ensure the quality and accuracy of the translated text.

For healthcare professionals, acceptability of the overall CINDERELLA APProach will be evaluated based on qualitative data from focus groups. As the care pathway of breast cancer patients usually requires the coordination among different professional figures, focus groups are deemed an appropriate method to capture interaction, stimulate discussions and explore different experiences and points of view [58]. Two types of stakeholders will be primarily involved in the focus groups at each trial site: (i) breast surgeons/oncologists and (ii) nurses. The focus groups will be conducted either face-to-face or remotely (with commonly used teleconferencing tools, e.g., Microsoft Teams), according to the healthcare professionals’ availability. Focus groups will be conducted at two different time points, namely at 12 months and 24 months after trial start to perform a longitudinal analysis of experienced acceptability. Focus groups will be audio-recorded after obtaining verbal consent from each participant. The recordings will be verbatim transcribed. The outlines of the interview (Supplementary Material 1) and focus group (Supplementary Material 2) will be pilot tested and, if needed, modified iteratively.

The transcripts of the interviews and focus groups will be analyzed using thematic analysis [59], and the themes will be driven by a theoretical framework of acceptability [52]. Findings for the qualitative data from the interviews and focus groups will be reported using the Consolidated Criteria for Reporting Qualitative Research (COREQ) checklist [60].

Organizational Impact

The assessment of the organizational impact is critical for complex interventions, whose adoption in clinical practice frequently requires substantial organizational investments, multidisciplinary teams, need for supervision, change in the relationships between different organizations, and training for patients and professionals [61]. The evaluation of the organizational impact of the CINDERELLA APProach will be performed by conducting focus groups at each clinical center at different time points. One focus group will be held at the beginning of the trial with the aim of describing the current organizational model and in particular the delivery and service model (“as is” scenario). Two additional focus groups will be conducted at 12 months and 24 months after the trial start to understand how the organizational model has changed with the introduction of CINDERELLA APProach (“to be” scenario). Specifically, we aim to investigate whether and how work processes, human resources availability and skills, relationships between healthcare professionals, resource need, and involvement of patients have changed following the implementation of the intervention. Focus groups will be audio-recorded, and the recordings will be verbatim transcribed. The outline of the focus groups (Supplementary Material 3) will be pilot tested and, if needed, modified iteratively. In analyzing the data using thematic analysis [59], we will rely on the HTA Core Model proposed by the European Network for Health Technology Assessment (EUnetHTA) [61]. Findings will be reported using the COREQ checklist [60].

Feasibility

Feasibility of an intervention can be related to intervention’s content and delivery, acceptability, adherence, likelihood of cost-effectiveness, or capacity of providers to deliver the intervention [62]. Evaluating the feasibility of an intervention allows to determine whether the intervention is appropriate, and to identify what (if anything) in the research methods or protocols needs modification and how changes might occur [63]. The feasibility of the CINDERELLA APProach will be evaluated by integrating the evidence collected in previous phases, namely economic evaluation, usability, acceptability, and organizational impact. Although a comprehensive and unique framework for assessing feasibility does not exist, our analyses will be driven by a recent conceptual framework developed by Klaic and colleagues (2022) [64].

Environmental Impact

The environmental impact assessment of the CINDERELLA APProach will employ the framework proposed by MacNeill and colleagues (2021) [65]. The framework has several strengths, including a clear delineation of the dimensions addressed, a comprehensive perspective on the opportunities for the healthcare system to mitigate its diverse and heterogeneous environmental impacts, and its linkage with dimensions that that can be monitored with well-known indicators in ongoing studies. Grounded in three fundamental principles, the framework focuses on (i) reducing demand for health services; (ii) matching the supply of health services with demand, ensuring appropriate care delivery; and (iii) reducing emissions associated with the provision of health services. Consequently, the environmental impact of the CINDERELLA APProach will undergo quantitative evaluation across three key dimensions: quantity, appropriateness, and emissions.

The quantity dimension requires an estimation of the amount of care received by patients in the CINDERELLA versus the standard of care arm. To provide a comprehensive estimate of the different healthcare services received, their costs will also be considered. Hence, data on the healthcare services consumed together with their costs will be drawn from same sources employed for conducting the economic evaluation and analyzed accordingly for comparisons.

To assess appropriateness, a set of key performance indicators (KPIs) (Table 3) will be calculated for the two arms starting from the data collected through the eCRFs and compared using statistical tests of difference with 95% confidence intervals. The indicators have been recently developed [66, 67] and focus specifically on avoiding inappropriate use of healthcare services after breast cancer interventions.

Table 3.

Description of KPIs for the “appropriateness” dimension of the environmental impact analysis

KPI	Description of measurement
KPI 1: post surgery	The percentage of patients with stage I–II BC in whom, within 60 days after the index breast surgery, at least one of the following examinations was performed: hepatic ultrasound, CT, MRI, bone scan, or PET. The percentage of patients with BC who underwent PET must also have been evaluated for stage III BC. Target: the percentage for this KPI should not exceed 5%
KPI 2: follow-up	The percentage of patients with stage I–II BC who, starting from 60 days after the index breast surgery and up to 365 days after this surgery, underwent at least one of the following examinations: CT, MRI, bone scan, or PET. Target: the percentage for KPI should not exceed 5%
KPI 3: subsequent breast reconstruction/axillary dissection	The percentage of patients who, within 90 days after the index mastectomy, underwent subsequent surgery for breast reconstruction and/or axillary dissection. Target: the percentage for this KPI should be close to 0%
KPI 4: treatment timing	The percentage of patients who, as candidates for chemotherapy and/or hormone therapy and without evidence of disease, initiated adjuvant treatment (chemotherapy and/or hormone therapy as the only pharmacological therapy in the adjuvant setting) within 60 days after the index breast surgery. The percentage for this KPI should be close to 100%
KI P5: radiotherapy timing	The percentage of patients who underwent partial resection surgery and who, as candidates for radiotherapy, initiated radiotherapy within 180 days after the last surgery, if no adjuvant treatment was administered, or within 270 days, if adjuvant treatment was administered. Target: the percentage for this KPI should be close to 100%

BC breast cancer, CT computed tomography, KPI key performance indicator, MRI magnetic resonance imaging, PET positron emission tomography

Emissions resulting from patients’ travel to access healthcare services will be estimated using data collected from HCPs and patient questionnaires at the designated time points for economic evaluation. Data will be derived from close-ended questions investigating average travel distances to reach healthcare facilities (e.g., for hospital visits) and modes of transport (e.g., car, public transportation). The total distance traveled will be calculated by summing the declared kilometers for each healthcare service, considering the mean of the range reported by the patient. The emissions associated to different travel modes will be based on the most recent literature [68]. A multivariable linear regression model, with kilometers traveled as the dependent variable, will be fitted to identify variables with the greatest impact on costs, using baseline covariates.

Given the absence of a unified framework for environmental impact analysis, the quantitative assessments will be supplemented by qualitative insights from healthcare professionals and managers. A focus group with healthcare professionals at participating clinical centers at 24 months after trial start will investigate their perceived commitment in terms of practices adopted and information shared with patients. Additionally, interviews will be conducted with healthcare managers at participating clinical centers to address issues related to the healthcare sector's awareness of its responsibility in reducing environmental impact and increasing awareness among healthcare workforce and patients. The outline of the interviews and focus groups are provided in Supplementary Material 4 and 5, respectively. The qualitative data pertaining to environmental impact will be analyzed using the same methodology previously described for assessing acceptability and organizational impact.

In light of the ongoing debate on how to incorporate environmental impact within health technology assessment, we opted for a parallel evaluation [69], a flexible approach that entails analyzing and presenting environmental data alongside established health economic analyses.

Discussion

The CINDERELLA APProach holds great promise for a variety of different users and stakeholders in the breast cancer care pathway and, more broadly, in the healthcare system. For this reason, on top of the expected benefits for the breast cancer patients undergoing locoregional treatment, who may potentially experience higher level of awareness, SDM, greater matching between expectations and results, and improved HRQoL, the CINDERELLA trial aims to evaluate the broader consequences of the innovative approach for the healthcare professionals, healthcare facilities, and the society at large, from an organizational, economic, and environmental standpoint.

The generalizability of the CINDERELLA trial results depends on several factors, including differences in the healthcare system characteristics, clinical practices, and the availability and accessibility of healthcare services. The collection of data from countries with different healthcare systems will enable conclusions that are applicable to heterogeneous settings. Moreover, a comprehensive assessment of the CINDERELLA APProach’s implementability will shed light on the contextual and individual factors that might hinder or facilitate its translation into real-world settings, thus providing a nuanced understanding of how the intervention can be adapted to different contexts and jurisdictions. To ensure further transferability of results from the economic evaluation, future research should adopt appropriate methods for adjusting the information to reflect local conditions [70], such as differences in unit costs and healthcare utilization patterns.

The evidence emerging from the various dimensions investigated within the CINDERELLA project will be presented separately rather than being aggregated using structured methods such as multicriteria decision analysis (MCDA). Through this approach, we aim to ensure that the decision-making process remains deliberative and flexible, rather than shifting toward a fully quantitative model. However, future research may consider developing and using a MCDA framework for this complex intervention, ensuring that the weights chosen adequately reflect the different priorities and perspectives of all stakeholders involved.

The multidimensional evaluation described in this protocol will gather data alongside the CINDERELLA clinical trial for the next 3 years, through questionnaires, focus groups, and interviews with patients, healthcare professionals, and healthcare managers. The width of this protocol mirrors the ambition of the CINDERELLA trial to deliver substantial value first and foremost to the breast cancer patients involved, and to facilitate the widespread adoption of the CINDERELLA APProach into clinical practice internationally, thereby benefiting a larger patient population. More specifically, through the analysis of the economic, organizational, and environmental impact of the CINDERELLA APProach, our study will support its replicability and sustainability in various and potentially heterogeneous contexts. We ultimately believe that the results of these ancillary evaluations will provide further evidence to nourish the international debate on the sustainability and ethical use of AI in healthcare.

Supplementary Information

Below is the link to the electronic supplementary material.

Declarations

Funding

The CINDERELLA project was funded by the European Union (EU grant HORIZON-HLTH-2021-DISEASE-04. Project number 101057389). The funder did not have a role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Ethics approval

Ethics approval for the clinical trial was obtained from the Ethics Committees of all clinical centers involved (Sheba Tel Hashomer, Fundação Champalimaud, Gdański Uniwersytet Medyczny, Ospedale San Raffaele, Universitatsklinikum Heidelberg). Ethics approval for the focus groups and interviews with healthcare professionals and managers was obtained from the Ethics Committee of Bocconi University. The project itself was approved by European Commission's Ethics Committee.

Consent to participate

Not applicable.

Consent for publication (from patients/participants)

Not applicable.

Data availability statement

Data sharing is not applicable in this article as no datasets were generated or analyzed.

Code availability

Not applicable.

Author contributions

LB: conceptualization, methodology, writing (original draft), writing (review and editing), visualization. EL: conceptualization, methodology, writing (review and editing). OC: conceptualization, methodology, writing (review and editing), supervision, funding acquisition.