Breast Decisions: Recommender System for Appearance Counseling about Breast Reconstruction

Background:

Appearance counseling is an important component of the consent process for breast reconstruction. The purpose of appearance counseling is to help the patient form realistic expectations of what she might look like after breast reconstruction. In this article, we introduce a recommender system, “BreastDecisions,” for appearance counseling that suggests photographs of previous patients that are tailored to a specific patient to help her form realistic expectations of her own reconstruction.

Methods:

We present user specifications and algorithm parameters needed to incorporate the recommender system into the appearance counseling workflow. We demonstrate the system for a common counseling scenario using a knowledgebase of previous breast reconstruction patients. The medical appropriateness of the recommended photographs for use in appearance counseling was evaluated by experts using a four-point rating system.

Results:

The recommender system presents photographs that are medically appropriate for counseling a specific patient, depicts typical outcomes, and adapts to a variety of clinical workflows. For each of 33 patients taken as examples of breast reconstruction patients, we used the system to identify photographs for appearance counseling. The baseline average medical appropriateness of the recommended photographs was between mostly appropriate (some explanation needed) and medically appropriate (minimal explanation needed). We demonstrate filtering and ranking steps to reduce the number of recommended photographs and increase the average medical appropriateness.

Conclusions:

Our recommender system automatically suggests photographs of previous breast reconstruction patients for use in counseling a patient about appearance outcomes. The system is patient-specific and customizable to a particular surgeon’s practice.

Takeaways

Question: How might we improve appearance counseling about the expected outcomes of breast reconstruction?

Findings: The “BreastDecisions” system recommends photographs of breast reconstruction outcomes of previous patients to show to a specific patient when counseling them about what they may look like after breast reconstruction. The system can be customized to meet the needs of different surgeons and their clinics’ workflows.

Meaning: BreastDecisions has the potential to improve appearance counseling, the informed consent process, and patient psychosocial outcomes such as body image for breast reconstruction patients.

INTRODUCTION

Women who are involved in decisions about their care and have realistic expectations about their appearance after breast reconstruction are more likely to be satisfied with their reconstruction outcomes.^1–8 Appearance counseling is an important component of the consent process for breast reconstruction surgery. The purpose of appearance counseling is to help the patient form realistic expectations of what she might look like during and after breast reconstruction. A variety of visualizations can be used in appearance counseling, such as drawings on the patient’s body, photographs of previous patients,⁹ computer-simulated pictures of reconstructed breasts,¹⁰ or three-dimensional (3D) biomechanical simulations.¹¹

Often, a gallery of example photographs—conventional or 3D photographs, such as surface scans¹²—of reconstruction outcomes of previous patients is provided to patients to help them visualize potential outcomes (eg, https://www.plasticsurgery.org/photo-gallery/procedure/breast-reconstruction). However, these galleries often include examples of procedures that are not medically appropriate for every patient and outcomes that are not relevant to every body type. For example, if a patient is only a candidate for an implant-based reconstruction, viewing photographs of former patients who underwent autologous reconstruction could be counterproductive. Similarly, patients who view images of women with much larger or much smaller body and breast sizes may not be able to develop accurate mental images of the plausible outcomes of their own reconstruction.

Rather than offering every patient the same gallery of photographs of reconstruction outcomes, one could select specific photographs for each patient that would meet the appearance counseling needs of that individual. However, the time required to manually choose photographs to show each patient is prohibitive. We, therefore, sought to overcome this barrier by designing a recommender system, “BreastDecisions”, to identify and select a tailored set of medically appropriate and relevant photographs for each patient.

A recommender system is a machine learning algorithm that seeks to recommend items from a collection that will be useful to the user, based on a predicted preference or rating.^13–15 Many consumer services, such as Amazon and Netflix, use recommender systems to suggest products to the user. In this article, we introduce a recommender system for appearance counseling of breast reconstruction patients, which recommends photographs of previous patients on the basis of clinical and physiological variables, such as age and breast volume. Our goals for the BreastDecisions recommender system were for it to only present photographs that are medically appropriate (eg, do not show outcomes of procedures for which the patient is not a candidate), relevant to each patient’s body size and shape, and of typical outcomes (eg, do not show equal numbers of “good” and “bad” outcomes when “bad” outcomes are unlikely), and to be adaptable to a variety of clinical workflows. We tested our recommender system in hypothetical surgical and appearance counseling scenarios using real photographs of previous breast reconstructions.

METHODS

Figure 1 is a vignette illustrating appearance counseling for a fictional patient, Rita, in which her reconstructive surgeon, Dr. Lerner, uses photographs of previous patients selected by the BreastDecisions recommender system. In the following sections, we expand on each of the concepts introduced in the vignette. Figure 2 aggregates definitions used in this article into a glossary for quick reference.

Fig. 1.

A vignette illustrating appearance counseling for a hypothetical surgeon, Dr. Lerner, and his patient, Rita. The storyboards show what information is needed from Dr. Lerner and Rita for the BreastDecisions recommender system to recommend photographs of previous patients who are medically appropriate for Rita and depict typical reconstruction outcomes. The BreastDecisions recommender system is designed to be adaptable to a variety of clinical workflows. (Stock images: Irina Strelnikova/adobe.stock.com, dlyastokiv/adobe.stock.com.)

Fig. 2.

Glossary of terms for appearance counseling using photographs of previous patients recommended by a recommender system.

User Specifications for BreastDecisions Recommender System for Appearance Counseling

Effective appearance counseling must be customizable to the appearance counseling scenario and the surgical scenario (Fig. 2). Figure 3 summarizes provider considerations when determining the specifics for each scenario. For example, in the vignette (Fig. 1), Dr. Lerner and his colleagues decide that appearance counseling of all patients in their clinic will be based on 3D photographs and that the surgeon will discuss the photographs of previous patients with the current patient as part of the surgical consultation. The BreastDecisions recommender system is adaptable to different appearance counseling scenarios, including different photograph types (eg, knowledgebase of previous patients contains conventional photographs, but 3D photographs are taken of current patients) and settings (eg, view photographs at home versus in clinic with a nurse).

Fig. 3.

The user specifications required from the provider to implement the appearance counseling recommender system, BreastDecisions. The appearance counseling scenario summarizes the circumstances under which the recommended photographs will be used to counsel the patient. The appearance counseling scenario can be defined at an institutional level for all patients, or it can vary by patient. The surgical scenario is patient-specific as it summarizes the factors determining which photographs are medically appropriate to use in counseling the patient. (Stock images: Irina Strelnikova/adobe.stock.com.)

The surgical scenario summarizes the factors determining which photographs are medically appropriate to use in counseling the patient. For example, a patient being counseled about what her body might look like during tissue expansion would not be shown photographs of previous patients who already had their expanders exchanged for implants. The surgical scenario is necessarily patient-specific. For example, in the vignette (Fig. 1), Dr. Lerner defines Rita’s surgical scenario by reviewing her medical history and considering Rita’s preference to remain about the same size as she is now. The surgical scenario may include more than one procedure type if the patient being counseled is a candidate for more than one procedure. The quality of the recommendations of a recommender system depends on the degree of concordance between the query and the knowledgebase (Fig. 2). A recommender system cannot recommend appropriate photographs of previous patients if there are no previous patients in the knowledgebase whose surgical scenarios were similar to that of the current patient being counseled.

Algorithm Parameters for BreastDecisions Recommender System for Appearance Counseling

Figure 4 summarizes the process of specifying the algorithm parameters for the BreastDecisions recommender system for appearance counseling. As shown in blue rectangles, the system performs three main functions to recommend medically appropriate photographs for appearance counseling: filtering, ranking, and sorting. Filtering eliminates photographs of previous patients that are not medically appropriate. Ranking orders a set of filtered photographs from most to least relevant for counseling the current patient, based on the Euclidean distance between variables for previous patients and the current patient, as illustrated in Figure 5. Sorting is a process that groups filtered photographs when the information available is insufficient for ranking to be employed. Filtering, ranking, and sorting can use quantitative metrics from photographs, such as breast volume, breast symmetry, or breast shape, in addition to conventional demographic and medical variables, such as age, body mass index (BMI), or bra size, as long as the variables are routinely collected and recorded for the patients in the knowledgebase.

Fig. 4.

The BreastDecisions recommender system suggests photographs of previous patients from a knowledgebase to use in appearance counseling. The blue rectangles represent functions specified based on the surgical scenario, whereas the orange diamonds are directional indicators guided by the appearance counseling scenario. The first algorithm step, “filter by surgical scenario,” ensures the medical appropriateness of the selected photographs. The recommender system is adaptable to a variety of clinical workflows, and the user specifications define the criteria by which photographs are deemed to be medically appropriate and to depict typical outcomes.

Fig. 5.

Photographs of previous patients are ranked by calculating the Euclidean distance between the current patient and previous patients in the space of variables. The example figure uses three variables: age, BMI, and average breast volume. Note that patients A and B are visually more similar to each other than they are to patient C and that A and B are closer to each other than they are to C in the space defined by age, BMI, and average breast volume.

Workflow Using the BreastDecisions Recommender System for Appearance Counseling

The workflow for using the BreastDecisions recommender system for appearance counseling is summarized in Figure 6. BreastDecisions is designed to achieve three goals: (1) recommend photographs of previous patients that are medically appropriate for use in counseling the current patient, (2) recommend photographs that represent typical outcomes for the current patient, and (3) be adaptable to different appearance counseling scenarios. The first goal is achieved by suggesting photographs from the knowledgebase that are based on the surgical scenario specified by the provider (Fig. 2). The second goal of typicality stands in contrast to an alternative objective of presenting a range of poor to excellent outcomes, regardless of whether a poor outcome is very unlikely or vice versa. The algorithm parameters of the recommender system operationalize the concept of typicality, for example, the choice of variables used to rank photographs from the most relevant to least relevant for counseling the current patient. However, as with any machine learning method, the effectiveness of a recommender system ultimately depends upon the data quality and quantity. A recommender system for appearance counseling cannot recommend photographs of typical outcomes of a given surgery or for a given demographic group if the knowledgebase is sparse with respect to that procedure or demographic. The third goal of the recommender system is that it be adaptable to different appearance counseling scenarios (Fig. 2). For example, a practice may invest in a 3D camera with the intent of switching to using 3D photographs of previous patients for appearance counseling after the knowledgebase is populated. The algorithm parameters of the recommender system can be readily adapted to such changes in the appearance counseling scenario.

Fig. 6.

Workflow for using the BreastDecisions recommender system for appearance counseling. First, the provider specifies the appearance counseling scenario (Fig. 2). Second, information, such as the patient’s medical history, preferences, and photographs, is acquired from the patient to be counseled. Third, the provider specifies the surgical scenario (Fig. 2) for the patient to be counseled based on the information acquired in step 2. Fourth, the recommender system retrieves photographs of previous patients from the knowledgebase based on the appearance counseling scenario and surgical scenario. The photographs of previous patients can be conventional photographs or 3D photographs, such as surface scans (Fig. 2).¹² Quantitative metrics can be obtained from the photographs using automated or semiautomated analysis software.^17–20 Fifth, the recommended photographs are used for appearance counseling as specified in the appearance counseling scenario. Finally, photographs and other data from the current patient are incorporated into the knowledgebase to enhance its value for counseling future patients. (Stock images: Irina Strelnikova/adobe.stock.com, dlyastokiv/adobe.stock.com.)

Example of Implementing and Assessing the BreastDecisions Recommender System for Appearance Counseling

To assess the medical appropriateness of the photographs chosen BreastDecisions, we asked it to recommend photographs of previous patients from a knowledgebase of real breast reconstruction patients for use in counseling a hypothetical current patient (Fig. 1).

The knowledgebase consisted of 505 women who underwent breast reconstruction at the University of Texas MD Anderson Cancer Center from 2011 to 2014. The participants provided medical record data, completed psychosocial questionnaires, and had 3D photographs of their torso taken at a baseline study visit and 3, 6, 9, 12, 18, and 18+ months after breast reconstruction under IRB-approved protocol 2010-0321. Three-dimensional photographs were taken with a custom 3dMDTorso system (3dMD, LLC, Atlanta, Ga.). All participants provided informed consent.

The hypothetical appearance counseling scenario was, “the current patient’s care team would like to select 3D photographs for her to view with her reconstructive surgeon during a consultation appointment. The care team estimates that five photographs are a reasonable number to discuss in this setting.” The surgical scenario was, “the current patient is considering undergoing a bilateral, two-stage, implant-based reconstruction. She has her native breasts now. She would like to know what she may look like after she has her final implants.”

We evaluated BreastDecisions using leave-one-out cross-validation (Fig. 2). After the photographs were filtered by the surgical scenario, we chose one patient from the photographs to play the role of the current patient, while the remaining patients’ photographs were used as the knowledgebase for the subsequent functions of the recommender system. The process was repeated such that each patient record played the role of the current patient one time.

The recommended photographs of previous patients were evaluated for medical appropriateness for use in counseling the hypothetical current patient. Two reconstructive surgeons rated the medical appropriateness of each photograph of a previous patient for counseling the current patient on a scale of 1 (“not medically appropriate”) to 4 (“medically appropriate”). The surgeons were also asked to indicate their specific concerns about each previous patient photograph in relation to the current patient. Figure 7 shows an example of the photographs and rating scale used by the experts.

Fig. 7.

For each patient record designated as the current patient, two reconstructive surgeons independently rated (four-point scale) the medical appropriateness of a photograph of a previous patient for use in counseling the current patient. The surgeons also were asked to record any concerns they would have about showing the photograph of the previous patient to the current patient. The medical appropriateness scores were used to evaluate the effectiveness of each step of the BreastDecisions recommender system.

The average medical appropriateness of the recommended photographs of previous patients was calculated for each step in the BreastDecisions recommender system, that is, after filtering and again after ranking. In the filtering step, the photographs were filtered by ±3 BMI units (nonphotograph variable) and the current patient’s desired postreconstruction breast size (photograph variable). To filter by the current patient’s desired postreconstruction breast size, breast volume was measured from the current patient’s prereconstruction 3D photograph and postreconstruction 3D photographs of each previous patient. We determined whether the difference in breast size between the current patient (prereconstruction) and each previous patient (postreconstruction) matched the current patient’s desired change in breast size (larger, smaller, or about the same as her prereconstruction size). The photographs of previous patients were ranked by the similarity of each of the previous patient’s prereconstruction breast shape compared with the prereconstruction breast shape of the current patient, using spherical harmonics to quantify breast shape, as described by Sampathkumar et al.^21,22

RESULTS

Filter by Surgical Scenario

Filtering the knowledgebase of 505 patients by the surgical scenario specified in the vignette yielded 33 relevant cases (Fig. 1). (See figure, Supplemental Digital Content 1, which displays the diagram of knowledgebase filtering by the surgical scenario depicted in the vignette in Figure 1, http://links.lww.com/PRSGO/C206.) Two experienced plastic surgeons independently rated the medical appropriateness of the postreconstruction photographs of each of the 32 patients in the knowledgebase for use in counseling each current patient, for a grand total of 33 × 32 = 1056 ratings. For a given current patient, the two surgeons’ appropriateness scores were averaged for each previous patient in the knowledgebase. Then, the average appropriateness scores of the postreconstruction photographs for all 32 patients in the knowledgebase were averaged for the current patient. The mean appropriateness score across all patients in the current patient role was 3.16 (range, 2.56–3.48).

Filter by Photograph Variables and Nonphotograph Variables

Since the appearance counseling scenario specified in the vignette stated that the desired number of photographs for use in counseling was five, BreastDecisions determined that further filtering was needed to reduce the number of recommended photographs from 32 to five. We performed the two filtering processes described in the Methods: (1) filtering by prereconstruction BMI and (2) filtering by the current patient’s desired change in breast size.

Filter by Nonphotograph Variable (BMI)

The first filtering step was applied to each of the 33 sets of 32 recommended photographs. After filtering by BMI, the number of recommended photographs for each of the current patients ranged from 0 to 19 (mean, 12). For the 32 current patients with at least one recommended photograph, the mean appropriateness score of the recommended photographs was 3.35 (range, 2.67–3.75). After filtering by BMI, 26 current patients still had more than five recommended photographs.

Filter by Photograph Variable (Desired Change in Breast Size)

The 26 current patients with more than five recommended photographs after the first filtering step were further filtered by the current patient’s desired change in breast size. After this second filtering step, the number of recommended photographs ranged from zero to 16 (mean, 5), with 10 current patients having zero recommended photographs that met the filtering criteria. For the 16 current patients with at least one recommended photograph, the mean appropriateness score of the recommended photographs was 3.33 (range, 2.5–4). Nine current patients still had more than five recommended photographs, so BreastDecisions determined that a ranking step was needed to identify the five most relevant photographs.

Rank by Photograph Variable (Breast Shape)

For the nine current patients who still had more than five recommended photographs of previous patients after the filtering steps, the recommended photographs were ranked by breast shape relevance. More specifically, the similarity of the previous patients’ prereconstruction breast shapes to that of the current patient’s prereconstruction breast shape was used to rank the corresponding postreconstruction photographs of the previous patients (Fig. 5). Breast shape was quantified from the prereconstruction photographs using a method previously described.^21,22 The medical appropriateness score of the top five ranked postreconstruction photographs was averaged for each current patient. The mean appropriateness score across all current patients after this ranking process was 3.38 (range, 3–3.7). We also compared how many of the top five ranked postreconstruction photographs of previous patients overlapped with the top five postreconstruction photographs of previous patients as sorted by medical appropriateness. The mean number of photographs that overlapped was 2.55. Figure 8 displays the top five ranked postreconstruction photographs of previous patients recommended for use in counseling a given current patient. The prereconstruction photograph of the current patient is also shown in Figure 8.

Fig. 8.

Demonstrative results of the ranking procedure for a current patient. The figure shows the current patient’s preoperative photograph and the five most recommended postreconstruction photographs of previous patients. The current patient’s desired postreconstruction breast size was “bigger than I am now.” The variable used to rank the previous patients was preoperative SPHARM RMSD for the left and right breasts, a measure of breast shape.^19,20 The medical appropriateness of the top five recommended postreconstruction photographs is maintained as mostly medically appropriate. SPHARM RMSD indicates spherical harmonics root mean square distance.

Surgeon Evaluation Agreement

The agreement between the two surgeons on individual medical appropriateness scores was minimal (squared weighted Cohen’s kappa = 0.35). However, the surgeons mostly (47% of ratings) had mild disagreement (a difference of 1 between their ratings) and rarely (2% of ratings) had maximum disagreement (difference of 3 between ratings). Figure 9 displays the rates of agreement between the two surgeons and example patient photographs for which the surgeons had varying levels of agreement, along with their comments.

Fig. 9.

Examples of the medical appropriateness ratings of postreconstruction photographs of previous patients for use in counseling the current patient. The two surgeons agreed on the medical appropriateness for counseling of 39% of the photographs, had mild disagreement (difference of 1 between scores) for 47% of the photographs, had medium disagreement (difference of 2 between scores) for 12% of photographs, and had maximum disagreement (difference of 3 between scores) for 2% of the photographs. The three most common concerns the surgeons had about the medical appropriateness of the postreconstruction photographs of previous patients for use in counseling the current patient were body habitus mismatch, breast size mismatch, and unlikely scar patterns.

DISCUSSION

To help patients form more realistic expectations of breast reconstruction outcomes and to support informed consent, we proposed BreastDecisions, a recommender system that provides a set of medically appropriate and patient-centered photographs for discussion during appearance counseling. Using hypothetical surgical and appearance counseling scenarios and a real knowledgebase of photographs of previous breast reconstruction patients, the recommender system successfully filtered and ranked the photographs into a customized set that was small enough for clinical consultation and deemed medically appropriate by experienced surgeons. Moreover, the system’s filtering and ranking functions increased the average appropriateness scores by reducing the number of suggested photographs with lower appropriateness scores.

A key benefit of the BreastDecisions recommender system for appearance counseling is that it is fully customizable to a particular surgeon’s practice, and the surgeon is always in control of what photographs are presented to patients. The system can be set up to use only information from the medical record or to calculate information from patient photographs to inform the filtering and ranking processes. It can be set up using conventional clinical photographs or 3D photographs. Of course, the provider can also review the recommended photographs and eliminate some of them from the set based on their professional judgment before counseling the patient. While our hypothetical scenario focused on long-term postoperative photographs, the system can also be used to visualize intermediate stages of reconstruction, for example, during tissue expansion, to help patients mentally prepare for the reconstruction process. Notably, the appearance counseling scenario or surgical scenario can be adjusted on the basis of new information from the patient.

As with any recommender system, the primary constraint is the number and variety of the items in the knowledgebase. In other words, the BreastDecisions system can only recommend photographs about surgical scenarios for which the knowledgebase contains photographs of previous patients. Ideally, a patient would only be shown photographs depicting reconstruction outcomes of previous procedures performed by their own surgeon since outcomes vary by surgeon. However, to maximize knowledgebase size and scope, it may be preferable to combine sets of patient photographs from multiple surgeons who consider their practices to be similar. While pooling photographs across surgeons is not ideal, it may be better for a patient to view photographs from other practices that their surgeon deems as similar to their own rather than relying only on photographs from sources outside of their surgeon’s control, for example, photographs that a patient finds on the internet.²³ A larger knowledgebase also allows for more specific filters, which may be desirable for certain appearance counseling scenarios, such as viewing photographs at home.

Feasibility of implementing the BreastDecisions system will depend upon the appearance counseling scenario and the systems already in place at a particular institution. An information technology expert will be needed, and additional support such as a data analyst and software engineer may be required to create a more advanced implementation. A limitation of this study is our use of a new, expert-based evaluation metric; the results of any expert-based metric depend upon the experts who are consulted.

We recommend providers consider whether the BreastDecisions system may be incorporated into their practice to improve appearance counseling. Prospective testing in the clinic of the BreastDecisions recommender system is needed to assess its impact on patients and providers. Key metrics to investigate for patients are perceived autonomy support²⁴ and decisional conflict.²⁵ Important considerations for providers are usability²⁶ and workflow.²⁷ The system can be extended to other surgical procedures, such as ear reconstruction and aesthetic breast procedures.

ACKNOWLEDGMENT

The manuscript was edited by Dawn Chalaire of the Research Medical Library at The University of Texas MD Anderson Cancer Center.