Preliminary Qualitative and Quantitative Evaluation of DESIREE, a Decision Support Platform for the Management of Primary Breast Cancer Patients

Abstract

The DESIREE project has developed a platform offering several complementary therapeutic decision support systems (DSSs) to improve care quality for breast cancer patients. A first assessment of the system was carried out in close-to-real tumor boards (TBs). Fourteen TB sessions were organized corresponding to a total of 125 exploitable decisions previously made without the system and re-played with the system after a washout period in three pilot sites. Results show an overestimation of declared compliance with guidelines when not using the system as compared to measured compliance with the recommendations issued from the guideline-based DSS of DESIREE. After using the system, measured compliance rate of decisions with guidelines was significantly improved from 74.4% to 89.6%. Most of the changes in decisions when using the guideline-based DSS were associated with non-compliant decisions that became compliant. Qualitative analysis and interviews showed that despite maturity issues, clinicians found DESIREE DSSs innovative and promising.

1. Introduction

Breast cancer is one of the most common types of cancer in European countries and the second leading cause of cancer death. In the US, an estimated 268,600 new cases of invasive breast cancer will be diagnosed among women in 2019, with 41,760 women expected to die from this disease.¹ If mortality rates are globally currently declining due to both improvements in treatment and earlier detection, not all women benefit equally from these advances and disparity are observed likely to reflect a combination of factors that are difficult to parse, including unfavorable tumor characteristics (especially triple negative tumors), obesity, and poor access to high-quality prevention, early detection, and treatment. A breast cancer diagnosis impacts upon the health, lifestyle, work, and family life of affected individuals.² It carries with it not only a risk of severe morbidity and mortality, but side-effects of treatments that may leave physical and psycho-social complications beyond treatment completion.

Managing breast cancer patients is a complex task that requires collaboration among professionals with complementary skills who work together to share the latest evidence and pool their expertise in order to customize patient treatment plans. Consequently, over the last decades, numerous countries have recommended the implementation of multidisciplinary tumor boards (TBs) to gather clinical expertise involved in the management of cancer patients, thus improving the quality of therapeutic decisions.³ TBs are a fundamental part of a complex care pathway, during which clinical cases are discussed to collectively achieve a definite cancer staging and elaborate a shared patient-centered treatment and follow-up plan in compliance with best available evidence.⁴

Given the medical complexity of breast cancer and the limited clinician time, there is a need for accurate tools to support personalized decisions and improve patient prognosis, survival, and quality of life whilst also reducing associated healthcare costs.⁵ A major advance is to use Clinical Decision Support Systems (CDSSs) to assist and support TB physicians in clinical decision-making, thus improving the quality of decisions and overall patient care. CDSSs have proven to have potential benefits in clinical practice, specifically in oncology⁶ and for the management of breast cancer patients.⁷ CDSSs can potentially improve patient safety and care quality by promoting the application of patient-specific clinical pathways and guidelines,^8,9 facilitating the use of up-to-date clinical evidence, essential for minimizing errors and maximizing patient satisfaction.¹⁰

DESIREE is a European project funded under the H2020 program. The objective is to develop a web-based software ecosystem dedicated to the personalized, collaborative, and multidisciplinary management of primary breast cancer, from diagnosis, to therapy, and follow-up. The DESIREE platform offers several functionalities among which three decision support (DS) modules developed to enrich decision support proposed to TB clinicians.¹¹ The first decision support module is based on the proposition of guideline-based recommendations from patient data (GL-DSS). Since clinical practice guidelines (CPGs) have flaws,¹² e.g., they may be incomplete, ambiguous, and they do not take into account patient preferences or specific contra-indications, it happens that some non-guideline-compliant TB decisions are legitimate. Such decisions have a clinical value that should be capitalized as another source of knowledge, beyond CPGs. Thus, DESIREE offers a second decision support module based on the clinical experience gained from non-compliant TB decisions (EX-DSS).¹³ Finally, a third decision support module has been developed based on the implementation of a case-based (CB) reasoning process where the goal is to reuse past TB decisions made for patients similar to a new patient case (CB-DSS).¹⁴ After the development and the integration of the three DS modules in the whole DESIREE platform, and with the objective of collecting feedback from end users at a milestone of the project, a first assessment of the platform was carried out in close-to-real TBs in July 2019, taking into account the maturity level of each DS module. This paper presents the preliminary assessment of (i) TB decision compliance with CPGs when using and not using the GL-DSS, (ii) the perceived usefulness and benefits of the GL-DSS, EX-DSS and CB-DSS, and (iii) the user experience with the whole DESIREE platform.

2. Description of the DS modules

If the three DS modules differ in the knowledge resources they use and the reasoning process they implement, all are articulated around a common breast cancer knowledge model, formalized as an ontology including both the clinical information model used to describe patient data and guideline knowledge and the termino-ontological aspects of the domain to describe data semantics and allow for reasoning at different levels of abstraction.¹⁵

2.1. Guideline-based decision support system: the GL-DSS

The GL-DSS integrates three different CPGs that TB clinicians may choose as the resource for guideline-based decision support: the US NCCN (National Comprehensive Cancer Network) Breast Cancer guidelines (NCCN),16 the French Paris Public Hospital breast cancer guidelines (APHP),17 and local breast cancer guidelines implemented by Onkologikoa (ONK), a Spanish oncology hospital, partner of the DESIREE consortium. Guidelines were formalized as decision rules. Using patient data recorded in the EHR, the GL-DSS yields patient-specific recommendations issued by the CPGs selected through the user interface (NCCN, APHP, or ONK). Recommendations are made of care procedures, e.g., surgeries, radiotherapies, chemo- and endocrine therapies, that may be grouped in ordered care plans. Once recommendations are returned and displayed, TB clinicians may either decide to comply with guidelines by selecting one of the GL-DSS propositions, or to not comply with guidelines by entering a different decision. In this later case, the reason why recommendations were not followed has to be given. Figure 1 provides an illustration of recommendations displayed in the user interface when the GL-DSS is launched on a clinical case with NCCN CPGs. In this example, several different concurrent therapeutic options are provided, i.e., two types of surgeries and multiple options of systemic neo-adjuvant therapies, with the recommendation evidence level (e.g. “2A”) and the recommendation expected conformance level (e.g. “SHOULD”, “MAY”).

Figure 1.

Display of guideline-based recommended care plans in the DESIREE user interface.

2.2. Experience-based decision support system: the EX-DSS

The EX-DSS overcomes the limitations of pure guideline-based DSSs adding new experience-based and clinical outcome-adjusted rules that model the clinical know-how expressed in TB non-compliant decisions. The principle is to analyze patient data profiles for which TB decisions do not comply with guidelines along with the criteria that justified the non-compliance. In each case, taking into account these criteria allows for the generation of a new experience-based rule. This rule integrates the new knowledge coming from the patient case for whom the TB decision was not compliant, and the final decision made. Experience-based rules, generated from a given center local practices and patients, are center-specific. The analysis does not compile patient data from different centers. The confidence value of the new rule is computed according to the clinical outcomes observed for the patients managed by this rule (e.g., with real-world patient-generated data during follow-up). Experience-based rules are stored aside guideline-based rules and are executed by the GL-DSS engine. EX-based recommendations are displayed with the same look as GL-DSS recommendations except they are tagged "EX" instead of “NCCN” or “ONK” or “APHP” to indicate their origin (Figure 2).

Figure 2.

Display of experience-based recommended care plans in the DESIREE user interface.

2.3. Case-based decision support system: the CB-DSS

The CB-DSS uses previously solved cases stored in the patient cases database to provide a solution to a new query case discussed by TB clinicians. Using various similarity functions, the similarity analyzer compares the query case with previous patient cases, As for EX-DSS, case-based reasoning only operates on patients from the same center. Finally, the retrieval algorithm retrieves the most similar patient cases. The retrieved patient cases are then graphically visualized in the user interface using the rainbow boxes visualization technique¹⁸ as displayed in Figure 3. The left part of the visualization shows a Multi-Dimensional Scaling scatter plot. The central white circle represents the query case and the other small colored shapes represent the similar cases. Colors indicate the type of treatment prescribed to the similar cases: red for surgery, yellow for endocrine therapy, green for chemotherapy, etc. The distance between two points corresponds to the similarity between the two clinical cases. The right part of the visualization qualitatively compares the two main treatment options provided for the 10 more similar cases (k-nearest neighbors algorithm with k=10), i.e., surgery and endocrine therapy. Each case is represented by a column and columns are grouped by treatment type. The two more probable treatments are displayed with the query case in the middle. Column widths are proportional to the level of similarity with the query. Colored boxes below column headers represent the characteristics shared by patient cases. Each box covers the columns corresponding to the patients sharing a given characteristic: e.g. patients #1456, #1576, and #1254 have all a “Biggest metastatic focus size between 32 and 38 mm”. Boxes are colored to follow the mean value of the colors associated with the columns they cover (weighted by column width), except when characteristics do not cover the query patient, boxes are in grey. In the example displayed in Figure 3, most characteristics of the query patient (i.e. the colored boxes) are associated with red or reddish colors. Therefore, red is predominant in the query case column, and surgery seems to be the most appropriate type of treatment.

Figure 3.

Rainbow boxes visualizing the cases similar to the query case as issued by the CB-DSS.

3. Material and methods

The evaluation of the DESIREE platform was based on the evaluation of the three DS modules by three clinical partners of the project: Onkologikoa (ONK, a cancer center located in San Sebastian, Spain), ERESA (ERE, delivering breast cancer imaging, and radiotherapy in Valencia, Spain), and Assistance Publique – Hôpitaux de Paris (APHP, the first cancer care institution in Paris, France, with the Georges Pompidou European Hospital, GPEH as the pilot site). Most centers had a weekly TB (two per week for ONK) with about 15 patient cases discussed. During TBs, ERE and ONK were already using an EHR but without decision support, whereas GPEH used a paper-based form printed from the EHR.

The evaluation protocol proceeded in three different steps: (i) selection of past TB clinical cases, anonymization and recording of the clinical cases in the DESIREE platform, collection of the decisions made without DESIREE and denoted D_without, (ii) organization of close-to-real (simulated) new TBs to re-discuss the same clinical cases but with the support of DESIREE, the decision previously made being hidden, and recording of the new decisions made with DESIREE and denoted D_with, and (iii) a final debriefing including User Experience Questionnaire (UEQ)¹⁹ scoring and clinician interviews to gather their opinion on the perceived usefulness of each component.

The three DS modules have been evaluated according to their maturity level at the time of the evaluation. The GL-DSS was fully operational. Thus, the evaluation of the GL-DSS was centered on the assessment of TB decision compliance rate with CPGs when not using and when using the GL-DSS, and the quality of these decisions. Since, the EX-DSS was still in development for the integration of EX-based rules, the clinical validation of the generated rules could not be performed, and the evaluation was focused on the a priori acceptability of the EX-DSS. In the same way, the CB-DSS module was in progress for the similarity analysis, and the evaluation aimed at gathering a first feedback on the display of the scatter plot and rainbow boxes to assess whether this visualization was understood and accepted by clinicians and what was the perceived usefulness of the CB-DSS.

3.1. Selection and recording of retrospective clinical cases

All pilot sites selected a sample of retrospective clinical cases previously discussed in real TBs, i.e., without any decision support, and for which TB decisions (D_without) were made. Selected clinical cases were representative of the cases usually discussed in TBs in each pilot site to cover the different stages of breast cancer and the different clinical pathways and management scenarios of breast cancer patients (e.g., patients after diagnosis but treatment not yet initiated, patients after neo-adjuvant therapy, patients after surgery, etc.). Clinical cases were then entered in the EHR embedded within the DESIREE platform as if they were “new” cases, i.e., the actual decision already made was not entered. Cases with bilateral tumors and mixed multifocal tumors (with different pathological types) were excluded because they were not properly handled by the platform at the time of the evaluation. Additionally, for each selected case, clinicians were asked whether the decision they made was compliant with their practices / CPGs.

3.2. Organization of close-to-real TBs

For evaluation purposes, close-to-real TBs were organized by the three pilot sites with the support of the DESIREE platform. Like regular TB meetings, they involved at least the three mandatory clinical specialties for breast cancer decisions (a radiotherapist, a medical oncologist, and a surgeon), but unlike regular TBs, they did not run under time pressure. Each pilot site worked on its own selection of clinical cases.

Prior to the beginning of the evaluation, a short user training video was presented to TB clinicians to describe (i) the main functionalities of the whole DESIREE platform, with a focus on (ii) how to register a patient to be discussed in TB, (iii) how data describing clinical cases is organized in user interfaces, and (iv) how both the GL-DSS and the EX-DSS should be used. A short description of the display of the information proposed by the CB-DSS (i.e. rainbow boxes and scatter plots) was also provided to the clinicians.

Then, in each close-to-real TB, the following steps were followed for all discussed clinical cases:

1. Read the patient clinical case on the DESIREE platform EHR.

2. Select the guidelines to be used by the GL-DSS (a priori APHP guidelines for GPEH, ONK guidelines for ONK, and NCCN guidelines for ERE but they could consult other CPGs when wished), run the GL-DSS, and take note of the recommendations provided.

3. For some of the clinical cases, once the decision was made by TB clinicians but before entering it into the system, clinicians were asked to look at the CB-DSS display. They had to comment about scatter plots and rainbow boxes used as the proposed visualization of the CB-DSS output.

4. Enter final decision (D_with), either selected among the guideline-based recommendations, or as a TB- specific choice, i.e., a decision considered non-compliant with the selected guidelines.

5. Assessment of the perceived usefulness and potential barriers and facilitators for using the recommendations issued from the EX-DSS on the basis of fake screenshots issued for six clinical cases from ONK retrospective cases (Figure 2).

3.3. Interviews of TB clinicians after using the DESIREE platform

At the end of all close-to-real TB sessions, TB clinicians were asked to fill out the User Experience Questionnaire (UEQ).¹⁹ The UEQ is a 26-item questionnaire containing six scales to cover a comprehensive impression of user experience: (i) Attractiveness: Overall impression of the system, do TBs clinicians like or dislike the system?, (ii) Perspicuity: Is it easy to get familiar with the system? Is it easy to learn how to use the system?, (iii) Efficiency: Can clinicians perform their tasks without unnecessary effort?, (iv) Dependability: Do clinicians feel in control with interaction modalities? Is it secure and predictable?, (v) Stimulation: Is it “exciting” and motivating to use the system?, and (vi) Novelty: Is the system innovative and creative? Does the system catch the interest of clinicians?

In addition, clinicians were also asked to answer a DS-specific questionnaire for each DS module:

• – What do you think about the overall clinical added value of the module?

• – What did you especially appreciate? Why?

• – What did you especially dislike? Why?

• – What could be the barriers for using the module in the future during TBs?

• – What could be the facilitators for using the module in the future during TBs?

3.4. Data collection and analysis

Quantitative analyses aimed at assessing the compliance of TB decisions with guidelines, often considered as a care quality indicator to be improved. Management of breast cancer patients may be expressed either as atomic care procedures or as care plans made of an ordered sequence of atomic care procedures. Examples of atomic procedures are lumpectomy, axillary lymph node dissection, paclitaxel x 12, epirubicin x 6, trastuzumab therapy, chest wall irradiation, accelerated partial breast irradiation, tamoxifen, etc. In some situations, care procedures may be expressed at a higher level of abstraction, e.g., taxanes, radiation therapy (RT), chemo+RT, or endocrine therapy. This was specially the case for decisions made without DESIREE. When comparing two care plans, we have distinguished three categories:

• - Care plans are compliant when both care plans are made of exactly the same therapeutic procedures in the same order, e.g. “lumpectomy + whole breast radiation”.

• - Care plans are potentially compliant when at least one element of a care plan is an acceptable abstraction of one or several elements of the other care plan, e.g., “Endocrine therapy” and “Tamoxifen for 5 years”, “chemotherapy” and “doxorubicin x 6, cyclophosphamide x 6, docetaxel x 6”.

• - Care plans are non-compliant when both care plans are different, i.e., they are neither compliant nor potentially compliant, e.g., “mastectomy + SLNB” vs “epirubicine x 4, cyclophosphamide x 4, paclitaxel x 12”, or “lumpectomy + SLNB” vs “lumpectomy + ALND”.

These categories were used to compare TB decisions and the GL-DSS recommendations to assess the compliance of TB decisions with CPGs, and to compare two TB decisions, i.e., between D_without and D_with decisions, to detect a decision change. We recorded: (i) the self-declared compliance with the site guidelines of TB decisions made without the system (D_without), (ii) the measured compliance with the GL-DSS propositions of decisions made without the system (D_without), (iii) the measured compliance with the GL-DSS propositions of decisions made with the GL-DSS (D_with). We compared D_without and D_with decisions and studied the changes between paired decisions to assess how these changes were related to compliance modifications.

Besides, interviews were performed and questionnaires were distributed. All interactions of TB clinicians with the DS modules during each close-to-real TB were recorded using FlashBack® software, e.g., comments on the system, time spent on each clinical case, and potential usability problems when using the system. In the special case of both EX-DSS and CB-DSS, comments on the proposed displays were collected.

4. Results

4.1. Characteristics of retrospective clinical cases and evaluation settings

A total of 136 retrospective clinical cases from past actual TBs involving 110 different patients were selected by the three pilot sites. Within each pilot site, clinical cases were selected to be representative of usual clinical cases discussed in TBs. As regards to histological types, there were 79% of invasive ductal breast cancers, 14% of invasive lobular breast cancers, and 7% of in situ tumors. The clinical stages ranged from stage 0 to stage IIIC corresponding to early breast cancer patients (non-advanced breast cancers). Clinical cases were different from one clinical site to the other, but distributions were comparable between the different clinical sites. Most of the different clinical pathways of breast cancer management were represented in GPEH and ONK clinical cases.

A total of 14 close-to-real TBs were organized, six by ONK, five by GPEH, and three by ERE. Each close-to-real TB included three clinicians to represent the required specialties. Eleven cases were not considered in the analysis (technical problems with the platform, usability issues, etc.) leading a total of 125 fully exploitable clinical cases.

4.2. Declared compliance with guidelines without the GL-DSS

While collecting the cases retrospectively, clinicians were asked to declare the guideline-compliance of the decisions they made during TBs. On the 125 decisions, three were declared as non-compliant, leading to a declared compliance rate of 97.6%. Table 1 includes the distribution by site of clinician-declared compliance of D_without decisions for the 125 cases.

Table 1.

Declared and measured compliance rates of D_without decisions, along with the measured compliance rate of D_with decisions by pilot site (C = compliant, PC = potentially compliant, NC = non compliant).

		Without GL-DSS							With GL-DSS
		Declared compliance		Measured compliance				Measured compliance
Site	# cases	C	NC	Rate	C	PC	NC	Rate	C	NC	Rate
ERE	15	15		100.0%	14		1	93.3%	15		100.0%
GPEH	31	31		100.0%	11	14	6	80.6%	24	7	77.4%
ONK	79	76	3	96.2%	52	2	25	68.4%	73	6	92.4%
Total	125	122	3	97.6%	77	16	32	74.4%	112	13	89.6%

4.3. Measured compliance with guidelines without the GL-DSS

Measured compliance without using the GL-DSS was assessed by comparing D_without decisions with the recommendations issued by the GL-DSS when TB clinicians selected the CPGs used in their site in close-to-real TBs. We observed a global compliance rate at 74.4%, gathering compliant and potentially compliant decisions (Table 1), a compliance rate significantly lower than the declared compliance of 97.6% (p<10⁵, using McNemar’s test for paired data). Agreement between declared and measured compliance reached 73.6% (92 compliant cases and one non-compliant case). In 31 cases (24.8%), TB clinicians declared their decision was compliant but it was not measured as such. For two cases, they declared their decisions were not compliant while they were measured compliant.

4.4. Measured compliance with guidelines with the GL-DSS

Measured compliance rate with the GL-DSS was assessed by comparing D_with decisions with the recommendations issued by the GL-DSS. The global compliance rate reached 89.6% (Table 1). We compared this number with the measured compliance rate of 74.4% obtained for D_without decisions. These measures were significantly different (p<10³, McNemar’s test for paired data). Among the 77 compliant D_without decisions, only one D_with was changed to a non-compliant decision. Among the 16 potentially compliant D_without decisions, 12 D_with decisions became compliant while four became non compliant. As for the 32 non-compliant D_without decisions, three quarters (24) had a compliant D_with decision while eight remained non-compliant.

4.5. Comparison of paired decisions without and with the use of the GL-DSS

For each of the 125 cases, D_without, and D_with decisions were compared. Seventy-six cases (60.8%) had similar decisions, meaning there was no change. Fifteen cases (12.0%) had potentially similar decisions. In these cases, D_without decisions were not entered in a structured way but specified as free text, describing treatments at different levels of abstraction, like the type of therapy (e.g., radiotherapy, endocrine therapy…), the class of the drug (e.g., taxanes, antiHer2…), or the drug without specifying the regimens (e.g., paclitaxel). On the contrary, D_with decisions were entered through DESIREE, thus following a structured and controlled entry, and leading to more detailed descriptions of care plans (for instance, Breast RT standard fractionation: 50 (Gy) / 50 fr, Boost radiation of the tumor bed: 16 (Gy) / 8 fr, Anastrozol). The remaining 34 cases (27.2%) were different, indicating a change of the TB decision. We analyzed how the changes from D_without decisions to D_with decisions distributed over guideline compliance. Table 2 reports the distribution of compliant and non-compliant decisions, without and with DESIREE according to the changes between D_without, and D_with decisions:

Table 2.

Distribution of cases according to the measured compliance for D_without and D_with decisions depending on their similarity category.

Similar decisions				Potentially similar decisions			Different decisions
	Dwith			Dwith			Dwith
Dwithout	C	NC	Total	C	NC	Total	C	NC	Total
C + PC	71		71	10	2	12	7	3	10
NC		5	5	1	2	3	23	1	24
Total	71	5	76	11	4	15	30	4	34

• – When both decisions were similar, there was no change in the compliance status of D_without, and D_with decisions.

• – When both decisions were potentially similar, the change between D_without and D_with decisions did not change the compliance status in 12 out of the 15 cases. In one case, a non-compliant D_without decision became a compliant D_with, and in two cases, potentially compliant D_without decisions became non-compliant D_with.

• – When both decisions were different, only eight cases on 34 had no change in their compliance status, seven remained compliant and one non-compliant. In 26 cases on 34, the change in decisions was associated with a change in compliance: 23 decisions initially non-compliant became compliant with DESIREE, and three compliant D_without decisions became non-compliant.

4.6. A priori acceptability of the EX-DSS

All clinicians acknowledged the recommendations issued from previous non-compliant decisions were interesting and they found the EX-DSS promising because EX-based recommendations go beyond the limitations of existing guidelines. In all pilot sites, for the same four patient cases over the six proposed from ONK retrospective clinical cases, all clinicians found the EX-DSS recommendation was suitable to the patient characteristics. However, only two clinicians would have chosen the EX-DSS recommendation whereas all the other clinicians would have chosen a guideline-based recommendation: “we have to respect CPGs as much as possible and avoid deviations”, "[EX-DSS recommendation] the source is unknown". Clinicians could have chosen to follow the EX-based recommendation if they had information about its level of confidence.

4.7. Assessment of rainbow boxes and scatter plots for the CB-DSS

Out of the 125 decisions, twenty-four were reviewed with the CB-DSS after a first decision was made (nine ONK cases reviewed by ONK clinicians, nine GPEH cases reviewed by GPEH clinicians, and six ERE cases reviewed by ERE clinicians). Rainbow boxes and scatter plots were correctly interpreted by all the clinicians and for all cases. The predominance of color in the choice of the treatment, the size of boxes, and the link between the scatter plot and the rainbow boxes were considered as understandable.

One of the characteristics of the display, however, has led to confusion: the grey boxes. They represent characteristics shared by similar patients, but not shared with the query case. Most doctors have been misled despite the grey color. They all thought that all the characteristics displayed were shared with the query case, whatever the color of the boxes. Once this had been re-explained, all clinicians emphasized the importance of the similarity metrics, because if patients do not share characteristics that are essential, then they should not be considered as similar. The non-shared characteristics may make the displayed patients different from the query case. For instance, “a stage IIB cancer cannot be compared to a stage IIA”, even if patient share a lot of other characteristics.

4.8. Qualitative assessment of the DESIREE platform: user experience and debriefing

Results show a very positive evaluation from the clinicians regarding the attractiveness, ease of learning (perspicuity), and control feeling (dependability), and a positive evaluation regarding the motivation to use the system (stimulation) and its novelty (values > 0.8, see Figure 4). The efficiency of the system was perceived as improvable, mainly due to some response times that were perceived to be too long.

Figure 4.

User Experience Questionnaire scales (mean values and variances).

All along the close-to-real TB sessions, no significant usability issues were observed. Clinicians found the data they need; they understood the provided information. The mean time per patient was 3’12’’ (SD=1’45’’) with no significant difference between the first and the last close-to-real TBs evidencing the system was easy to learn. Some additional specific display features were especially liked, such as a timeline display and a dashboard with all patients’ clinical data of a given center. “The system is quite easy to use with a user-friendly interface”, “I appreciated the user-friendly interfaces, their intuitiveness, and the provided synthetic views of patient data”.

The GL-DSS system caught the interest of clinicians who declared it would be a useful support for non-expert centers (3 clinicians), “very interesting tool for small hospitals without TBs”. The EX-DSS was perceived as a promising tool for expert centers for which it could be a real added value going beyond the limitations of existing guidelines (3 clinicians). The a priori acceptability was very good if the generated EX-based rules are correctly validated and the information about this validation is available to clinicians. The CB-DSS display was perceived as interesting, and easy to understand once it has been explained. A point of caution was raised regarding the CB-DSS display relevance in case of non-shared characteristics between patients (four clinicians).

As regards the potential barriers and facilitators for a future system, clinicians declared they were ready to use the GL-DSS in a daily practice (six clinicians): “when the system will be able to handle all the possible cases discussed during our TBs, it will be perfect for centers like ours that handle many cases, including complex cases”. The possibility to have access to different guidelines was especially appreciated (three clinicians). Clinicians insisted on the importance of (i) the DS interoperability with local EHR data, otherwise it would not be used (three clinicians) and (ii) the need to foresee the procedure and time to manage the continuous update of guidelines (three clinicians). Finally, four clinicians evoked an “excessively long time to get the recommendations” as a barrier.

Discussion and Conclusion

Among the provided functionalities, the DESIREE platform offers three decision support modalities to enrich decision support proposed to TB clinicians for the management of breast cancer patients. Taking into account the various maturity levels of each modality, the evaluation reported in this paper of the DS modules is preliminary. Results highlighted a positive effect of the GL-DSS on prescribing behavior and a great potential for enriched approaches for decision support (EX- and CB-DSSs).

The quantitative analysis focused on guideline adherence of TB decisions. First, it highlighted a significant over-estimation of self-declared compliance with guidelines of TB decisions made without any decision support (D_without) compared to their effective measured compliance obtained by comparing the same D_without decisions to recommendations issued by the GL-DSS (97.6% vs. 74.4%). This has been often reported in the literature. Considering decisions made with the support of the GL-DSS (D_with) for the same set of clinical cases, it is observed that the measured guideline compliance was significantly increased and reached a rate of 89.6% (vs. 74.4% without DSS). When decisions made with (D_with) and without (D_without) the GL-DSS were different, three quarters of non-compliant D_without decisions (24/32) became compliant with the use of the GL-DSS. Such an improvement suggests it could be an effect of the DSS, but this cannot be asserted because of the study protocol (limited sample size though three pilot sites, use by personnel involved in the DS development, no control group, no real TB conditions).

The qualitative analysis showed that the system as a whole was perceived as user-friendly, easy to learn, and providing a real added value for TB clinicians. One the one hand, in its current state of development, the GL-DSS was found to be useful but for non-expert centers, with easily accessible information and adapted recommendations. On the other hand, considering the more complex cases that can be discussed in TBs (e.g., bilateral cancers, or multifocal tumors), the tool might also be easily used in real TBs and bring a real added value also for expert centers when complex clinical cases would be managed by the system. Moreover, as regards the display of similar cases (CB-DSS) and the provision of recommendations based on past experience (EX-DSS), clinicians considered it was innovative, promising, and a real potential gain for expert teams. If the next integration steps are properly validated with clinicians, these two modules are considered to be a real plus as compared to existing tools.

This preliminary evaluation was conducted while the integration of the various components was still in progress. The study de facto presents certain limitations due to the poor maturity of some components. But the objective was to test the a priori acceptability of DS and its potential relevance during real TBs.

Despite some usability issues to be corrected, the DESIREE platform offering three complementary DS modules was appreciated and considered as having a great potential. The evaluation allowed to validate the GL-DSS which relies on acknowledged CPG contents, and to point out the critical issues for further steps for the CB-DSS and the EX-DSS. Both approaches, relying on prior data, faced the issues reported with machine learning and AI. Their explainability must be sought and the contents they deliver must be monitored constantly, and revised when appropriate. More specifically, there is a need for the CB-DSS to work on the similarity measure to analyze the specific characteristics that make patients different when non-shared. There is also a need for the EX-DSS to think about how to display/explain to clinicians the information about the validation of the rules and provide for each rule information such as a confidence level as it is done with GL-DSS rules. The next steps will be to validate with clinicians the similarity algorithm for the CB-DSS and the new generated rules of the EX-DSS before moving forward to a larger study in real-life TBs to confirm the promising preliminary results of this study.

Acknowledgements

This project has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 690238. The authors would like to thank all participants of the project who permitted the development of the DESIREE platform.