Sensitivity Analyses of a Scoring System for a Contraception Decision Aid

Abstract

We conducted formal analyses of a scoring system for a contraception decision aid to support transgender and gender-nonconforming (TGNC) assigned female at birth (AFAB) individuals. For this purpose, we developed a methodology framework to assess the weights for each decision factor, to conduct univariate and multivariate sensitivity analyses, and to provide data visualization, which led to successful identification of critical values in weight assignment that can impact the recommendations generated by the system. These analyses made critical contributions to development and validation of the system’s knowledge base, providing explainable recommendations, and conducting additional research related to system users and functions. Future research is required to explore high-dimensional sensitivity analyses, to address technical issues identified, and to examine the generalizability of the methodology to other applications.

Introduction

Contraception decision-making is a multifaceted process influenced by personal, medical, and social factors. According to the World Health Organization (WHO), effective contraception decision-making requires a comprehensive understanding of individual preferences, medical conditions, and social contexts¹. Contraception is an essential aspect of reproductive healthcare, particularly for transgender and gender-nonconforming (TGNC) people assigned female at birth (AFAB). This population faces unique challenges with special medical considerations in selection of contraceptives². For example, many TGNC AFAB individuals need to consider the interactions between specific contraceptive methods and frequently reported issues in this population such as gender dysphoria, hormone replacement therapy (HRT), gender-affirming surgery, depression, and anxiety. To address these challenges, we have initiated a research project to develop and evaluate an online contraception decision aid for the TGNC AFAB individuals. As a preliminary study, we have built a pilot tool, “MyChoiceForAll”, which can provide contraception education, customized recommendation of contraceptives based on personal preferences and medical conditions, and links to additional resources to support the TGNC AFAB community³.

Scoring systems are integral to the functionality of many patient decision aids and clinical decision support systems, as they help quantify the different decision options based on specific criteria and provide recommendations customized to an individual patient. For instance, EyeChoose⁴, a patient decision aid for refractive eye surgery, employs a scoring algorithm to generate personalized recommendations based on medical history and personal preferences, achieving 99% accuracy in its recommendations. My Contraceptive Choice (MCC)⁵, a patient decision aid for college-aged women, uses a scoring system to address their preferences and needs, achieving 72% accuracy in recommendations. Some widely used clinical algorithms, such as the Patient Health Questionnaire (PHQ)⁶ and the Acute Physiology and Chronic Health Evaluation (APACHE)⁷, are also based on scoring systems. These examples highlight the importance of scoring systems in improving decision-making in healthcare.

Although some of the existing contraceptive decision aids have provided pragmatic evaluations on system performance, there is a lack of transparency in how these systems assign weights to different decision factors in their scoring algorithms, raising concerns on accuracy and reliability of the system-generated recommendations in guiding users toward optimal contraceptive choices⁸. Formal analysis of scoring systems, including the assignment of weights to different decision factors and their impacts to the final recommendations, are essential for developing high-quality knowledge bases and improving the accuracy and usability of these tools⁹. Our review of the literature did not find prior studies that provide systematic approaches for sensitivity analysis on scoring systems in decision aids, particularly in the context of serving the TGNC AFAB individuals.

To address these gaps in the literature, we have been conducting formal analyses of the scoring algorithms used by the MyChoiceForAll tool. For this purpose, we have been performing simulations and sensitivity analyses to assess how changes of the weight assigned to specific decision factors can impact the overall ranking of the different contraceptives. In this paper, we present the methodology of an ongoing study for sensitivity analyses. This research will lead to the development and validation of a knowledge base for a full-scale study to build a contraception decision aid for the TGNC AFAB community. Our approach can also provide a framework for comprehensive analysis of similar scoring algorithms for a variety of other applications.

Methods

One of the main features of the MyChoiceForAll tool is the personalized recommendations of contraceptives. For this purpose, the system collects a user’s background information and preferences, which are used as decision factors, in the following categories: (1) previous use and experience of contraceptives, including birth control methods that someone has used before (yes or no), experience with a specific method (negative, neutral, or positive), and whether they would consider to use the method again (no, yes, or I don’t know); (2) personal preferences, including a person’s prioritization in selecting contraceptives, with most variables (cost-effectiveness, pregnancy prevention, management of period and side effects, low chance of weight gain/loss, and partner’s willingness to participate in contraception) rated on a seven-point Likert-scale (for example, very important, important, slightly important, neutral, slightly unimportant, unimportant, or very unimportant) and the last variable (maintenance schedule) rated on four preference levels (daily, weekly, monthly, or yearly); (3) medical preferences, including a person’s medical related concerns, with most variables (STI prevention, willingness to insert foreign object into vagina, willingness to seek consultations by medical professionals, and willingness to have medical procedures) rated on a seven-point Likert-scale and the last variable (preferred level of hormone use) rated on three preference levels (none, progestin only, or estrogen and progestin); and (4) medical conditions, including a person’s current medical conditions (acne, anxiety, depression, top gender dysphoria, bottom gender dysphoria for periods, bottom gender dysphoria for vaginal insertion, and gender dysphoria for pregnancy), severity (mild, moderate, or severe) of the conditions, and history of medical problems (migraine, smoking, hypertension, blood clots, congestive heart failure or coronary artery disease, polycystic ovary syndrome, endometriosis, breast cancer, testosterone HRT, and top surgery), which may influence or interfere with the selection of specific contraceptives.

As these questions are answered, each contraceptive method associated with a specific decision factor will gain or lose points. The number of points to gain or lose after a specific question is answered depends on two parameters: (1) the correlations between the decision factor in the question and each contraceptive method, with 1 indicating a strong positive correlation, 0.5 indicating a weak positive correlation, 0 indicating no correlation, and -1 indicating a negative correlation; and (2) a multiplier that is defined by an individual’s choice or preference for the decision factor. Here the correlations are based on the properties of specific contraceptives and the related clinical evidence, while the multipliers reflect an individual’s personal situation. The multiplication of these two parameters derives the number of points to gain or lose by a specific contraceptive method after a question about a decision factor is answered. For example, the correlation between the decision factor pregnancy prevention and the contraceptive method condom is 0.5; if a person indicates that pregnancy prevention is very important, corresponding to a multiplier of 3, a total point of 1.5 (0.5 * 3) is gained by the contraceptive condom. By the end of the process when all the questions are answered, the three contraceptive methods that recorded the highest total points are selected as the recommendations. An overview of selected decision factors, their correlations to specific contraceptive methods, and the multipliers currently used by MyChoiceForAll is shown in Table 1.

Table 1.

An overview of selected decision factors, their correlations with contraceptives, and the multipliers.

^*If there is no concern on a specific medical condition or no history of a specific medical problem, no point will be gained/lost. No contraceptive method is positively correlated with top gender dysphoria. No contraceptive method is negatively correlated with endometriosis.

In a pilot study conducted previously, we showed good system performance of MyChoiceForAll, with appropriate recommendations for 94% of the 105 test cases that were developed using stratified sampling to ensure balanced representation across diverse user backgrounds, personal preferences, and medical conditions³. Yet the assignment of weights to specific decision factors in the scoring system was based on heuristics. For example, the seven-point Likert scale was mapped to a multiplier from -3 to 3, while a negatively correlated medical condition at different severity levels was assigned a weight of -15, -25, and -50. Although the overall conservative approach to heavily penalize a contraceptive method when it interferes with a medical condition can be justified, the specific values assigned to the multipliers require thorough examination and solid support.

To address this concern, we have been conducting simulation research and sensitivity analyses. For this purpose, we treat the multiplier of each decision factor as a variable that can be assigned to a specific value in a pre-defined range. We perform simulations on these variables to assess how their changes can impact the total scores of the contraceptive methods, which may further lead to the recommendations of different contraceptives.

According to the scoring algorithm, the total points for a specific contraceptive method can be given by the following formula:

$$TotalPoint={\displaystyle \sum _{i=1}^{n}multiplier\left(i\right)\ast correlation\left(i\right)}$$

Here i = 1, 2, 3,… n, indicating all the decision factors; multiplier(i) is a number (weight) assigned to the decision factor i, which further depends on a user’s answer to the question related to that decision factor; and correlation(i) is the correlation between decision factor i and the contraceptive method under discussion.

We started the sensitivity analyses with a single decision factor at a time by only changing the weight of multiplier for that decision factor, while keeping the multipliers of the other decision factors in constant values as they were used in the scoring system. Since the specific value of a multiplier in the scoring system depended on a specific choice or preference related to a decision factor (see Table 1), we leveraged the following strategies to reduce potential biases when assigning individual choices to the other decision factors not in simulation: (1) the answers to all Likert-scale questions were set to neutral; (2) for questions related to medical conditions and medical history, the answers were set to yes and no half and half; and (3) for questions related to medical conditions with a answer of yes, the severity level were set to moderate. With the assignment of multipliers to the other decision factors such that this part of the formula became a fixed value, we were able to focus on the univariate analysis for the decision factor selected for simulation. After the sensitivity analysis (details provided below) for this decision factor was completed, we repeated the process for the next decision factor until we completed the univariate analyses for all decision factors.

For simulation and sensitivity analysis with two decision factors (bivariate analysis), we selected two specific decision factors at a time and changed the weights of multipliers for both at the same time, while keeping the multipliers of the remaining decision factors in constant values as they were used in the score system. We used strategies similar to those in univariate analysis to reduce potential biases when assigning individual choices to the remaining decision factors not in simulation. After the sensitivity analysis (details provided below) for these two decision factors was completed, we repeated the process for another pair of decision factors.

For univariate analysis, the general formula to calculate the total points for a specific contraceptive method shown above can be simplified and rewritten as:

$$TotalPoint=correlationX\ast x+c$$

Here x is the value of multiplier for the decision factor under simulation, correlationX is the correlation between the contraceptive method under discussion and the decision factor in simulation, and c is a fixed value after calculating the remaining part of the general formula. This univariate linear function can be easily visualized as a line in a two-dimensional space.

To define the range of x for simulation, we reviewed the values of multipliers used by the scoring system, which spanned from -50 to 3. A reasonable selection would be to expand the positive side to at least 50 such as to assess its interplay with the negative points for a single decision factor. We also noted that the multiplier value of 3 was assigned to the most positive response to a seven-point Likert-scale. Since it would be typical to evenly distribute the points for a Likert-scale, we decided to extend the range on the positive side to 60, such that the other responses to the Likert-scale could be set to 40, 20, 0, -20, -40, and -60. With the positive side extended to 60, it would be natural to set the negative side in a symmetric range. We therefore defined the range of x as [-60, 60] for the purpose of simulation. This broad range could test all user choices, including each of the seven-point Likert-scale and every response to the other questions (see Table 1).

For bivariate analysis, the general formula to calculate the total points for a specific contraceptive method shown above can be simplified and rewritten as:

$$TotalPoint=correlationX\ast x+correlationY\ast y+c$$

Here x is the value of multiplier for the first decision factor under simulation, correlationX is the correlation between the contraceptive method under discussion and the first decision factor in simulation, y is the value of multiplier for the second decision factor under simulation, correlationY is the correlation between the contraceptive method under discussion and the second decision factor in simulation, and c is a fixed value after calculating the remaining part of the general formula. This linear function with two variables can be visualized as a plane in a three-dimensional space. We used a similar strategy to determine the ranges of x and y for simulation, with both set to [-60, 60].

Results

Univariate Analysis

We conducted univariate analyses on each decision factor. For a specific decision variable under analysis, we performed simulations when changing the weight (multiplier) of this decision variable within the predefined range and recorded the corresponding total point for each contraceptive method.

Figure 1 shows the simulation for pregnancy prevention, a decision factor under the category of personal preferences, with the plot of total point (vertical axis) for each contraceptive method. Here each contraceptive method is shown as a specific line in the graph. The slope of a line depends on the correlation between the decision factor under study (in this case, pregnancy prevention) and the contraceptive method represented by that line. For a correlation value of 1 (for example, hormonal IUD), the total point for that contraceptive method increases as the weight changes from -60 to 60. For a correlation value of 0.5 (for example, condom), the total point for that contraceptive method also grows but at a slower pace as the weight increases. For a correlation value of 0 (for example, cervical cap), the line is flat, which means that the total point for the contraceptive method remains the same as the weight changes.

Figure 1.

Univariate analysis for the decision factor of pregnancy prevention.

Figure 2 shows the univariate analysis for the decision factor low chance of weight gain/loss, with several contraceptives (for example, copper IUD) having a negative correlation of -1. In the graph, the lines associated with these contraceptive methods have a negative slope. Therefore, the total point for such a contraceptive method decreases as the weight of the decision factor increases.

Figure 2.

Univariate analysis for the decision factor of low chance of weight gain/loss

Figure 3 shows the simulation for willingness to have medical procedure, a decision factor under the category of medical preferences.

Figure 3.

Univariate analysis for the decision factor of willingness to have medical procedure.

For a decision factor under the category of medical condition, the simulation of the multiplier is based on different schemes, depending on whether it is positively or negatively correlated to contraceptives. When it has positively correlated contraceptives, the weight range of the multiplier for simulation is [3 – 60]. When it has negatively correlated contraceptives, the original assignments of weight based on the three severity levels of medical condition were -50 (serious), -25 (moderate), and -15 (mild). To make it symmetric to the positive side, we extended the weight range of the multiplier to [-60 – 0]. Figure 4 shows the simulation for the decision factor of gender dysphoria - pregnancy.

Figure 4.

Univariate analysis for the decision factor of gender dysphoria – pregnancy.

Figure 5 shows the simulation for endometriosis, a decision factor under the category of medical history. Since it only has positively correlated contraceptives, we set the range as [3 - 60] for the simulation.

Figure 5.

Univariate analysis for the decision factor of endometriosis.

With the plots of the lines representing specific contraceptive methods, we are now in the process of conducting sensitivity analyses by identifying the points where different lines of contraceptive cross each other. When the assigned weight for the decision factor under analysis is very close to the crossing point, it may impact the recommendations of top-scored contraceptives. For example, in Figure 4 when the weight for gender dysphoria – pregnancy is assigned to a value around 22, the three top-scored contraceptives may change between [sterilization, condom, vaginal gel] and [sterilization, condom, arm implant]. Figure 6 shows the zoomed-in view of this part from Figure 4. We are now working to identify all such crossing points to further assess how weight assignment may impact the recommendations generated by the system.

Figure 6.

Zoomed-in view of Figure 4 showing the change of top-scored contraceptives.

Bivariate Analysis

In addition to univariate analysis, we have been conducting bivariate analyses by selecting two decision factors at a time to examine their interactions and how such interactions impact the total score for each contraceptive method. For the two specific decision variables under analysis, we perform simulations when changing the weights (multipliers) for both decision variables within the predefined ranges and record the corresponding total score for each contraceptive method. During this process, we assign individual responses to the other decision factors so to keep the remaining part of the score calculation a fixed value.

The predefined range of simulation for a specific decision factor depends on the category it belongs to, which we have discussed earlier. For example, if both decision factors are under the category of medical conditions, the weight range of the multiplier for each is [-60 – 60]. Figure 7 shows the simulations with the two decision factors of acne and top gender dysphoria, visualized as a 3-D graph with the total score plotted on the vertical- or z-axis, the weights (multipliers) of the two decision factors plotted on the x-axis and y-axis, and each contraceptive method shown as a plane. When projecting the normal vector of this plane to the x-axis and y-axis, the slope is equivalent to the correlation between the decision factor represented by a specific axis and the contraceptive method represented by this plane. If both correlations happen to be 0 (for example, when the contraceptive method is sterilization), the plane is shown as horizontal (perpendicular to the vertical- or z-axis).

Figure 7.

Bivariate analysis for the two decision factors of acne and top gender dysphoria

With the plots of the planes representing specific contraceptive methods, we are in the process of conducting sensitivity analyses by identifying the intersection lines when two different planes cross each other. When the assigned weights to the two decision factors under analysis are very close to an intersection line, it may impact the recommendations of top-scored contraceptives.

Current Status

We have completed all the univariate analyses. We are now in the process: (1) to complete the sensitivity analyses for the univariate analysis to identify all the crossing points; (2) to complete the bivariate analyses by selecting important pairs of decision factors that may correlate or compete; and (3) to explore the feasibility of multivariate analyses with three or more variables. Due to space limitations, we won’t be able to present all the results in this paper. Visualizations for additional decision factors can be found at: https://github.com/asusanyamshah/simulation_visualizations. We will report additional results in future publications.

Discussion

Scoring systems to generate patient-specific recommendations need to be based on clinical evidence, to address personal preferences, and to serve medical needs. Given the wide range of differences in personal values, medical conditions, and individual situations among the TGNC AFAB people, assigning appropriate weights to specific decision factors is critical to develop a high-quality scoring system for the proposed contraception decision aid to support the TGNC AFAB community. Our ongoing work of simulation study and sensitivity analysis presented in this paper has demonstrated that weight changes for specific decision factors can impact the overall ranking of contraceptives and generate different recommendations to patients. The methodology contributions of this work include: (1) development of a scoring mechanism that separates the multipliers, which represent an individual’s preferences for specific decision factors, from the correlations between specific decision factors and particular contraceptive methods, which represent the underlying medical knowledge and the related properties of certain contraceptives; and (2) formulation of a methodology framework to conduct simulation and sensitivity analysis through assessment of the multipliers associated with each decision factor, univariate analysis, multi-variate analysis, data visualization, and identification of the critical values in weight assignment that may impact the recommendations generated by the system.

The application of the methodology to development of a contraception decision aid for the TGNC AFAB population is unique, representing one of the first efforts to build a digital health tool for the gender minority community. In particular, the transparency of the weight assignments to specific decision factors directly addresses the needs of the TGNC AFAB community. This approach can serve as a model to build decision assistance tools for other marginalized populations.

The analyses conducted in this ongoing study have made a series of contributions to the development of the contraception decision aid, including: (1) review of the clinical evidence associated with specific contraceptives and decision factors, and validating their correlations; (2) development of a high-quality knowledge base with the capacity to explain the rationales behind the recommendations generated by the system; (3) raising additional research questions to examine TGNC AFAB individuals’ decision making when encountering correlated or competing decision factors; and (4) providing additional system functions to refine the recommendations, for example, by asking a user to provide horizontal comparisons of specific decision factors to resolve tied or closely ranked contraceptives.

There are a few limitations in our study. First, the correlation factors presented here include only a few possible options (1, 0.5, 0, -1), which may not be able to handle complex correlations between certain contraceptives and decision factors. In addition, the computational complexity of the multivariate analysis increases exponentially as we include more variables. It is also challenging to visualize and to provide intuitive understanding of such analyses involving high-dimensional data. Lastly, it is possible that many critical values of weight assignments (for example, those close to the crossing points in univariate analysis and the intersection lines in bivariate analysis) can be identified and yet we may not be able to find a solution to resolve all the potential conflicts when determining the optimal values for weight assignments.

Our current and future work includes: (1) completing the univariate and bivariate sensitivity analyses; (2) exploring high-dimensional sensitivity analysis and data visualization, including applying dimensionality reduction techniques such as principal component analysis and t-distributed stochastic neighbor embedding^10-11; (3) developing methodology to resolve conflicts in weight assignments to avoid the critical points identified through sensitivity analysis; (4) conducting formal validation study for the scoring system; and (5) examining the generalizability of the methodology framework by applying it in development of scoring systems for other clinical problems.

Conclusion

We have presented a methodology framework to conduct formal sensitivity analysis on scoring systems used in patient decision aids. We have applied this framework to a contraception patient decision aid for TGNC AFAB individuals. Univariate and bivariate analyses have shown that we can leverage this approach to successfully identify critical values in weight assignments. These analyses have made important contributions in development and validation of knowledge base, providing explainable recommendations, and conducting additional research related to system users and functions. Future research is required to explore sensitivity analyses involving three or more variables, to address certain technical issues identified, and to examine the generalizability of the methodology to other applications.