Decision-Analytic Modeling to Assist Decision Making in Organizational Innovation: The Case of Shared Care in Hearing Aid Provision

Abstract

Objective

To illustrate the use of decision-analytic modeling to assist decision making in organizational innovations.

Study Setting/Data Sources

Regarding an organizational innovation (shared care in hearing aid provision) available evidence from different sources was synthesized.

Study Design

A probabilistic Markov model was constructed.

Data Collection/Extraction

We modeled the long-term cost-effectiveness of different organizational formats of shared care as opposed to the current organization. We assessed the expected value of perfect information (EVPI) for several groups of parameters in the model.

Principal Findings

The current organization had the highest probability of being cost-effective. Additional research is worthwhile, especially on access to care and safety (sensitivity to detect pathology).

Conclusions

Decision-analytic modeling in an early stage of organizational innovation is a valuable tool to facilitate evidence-based decision making.

Health technology assessment focuses increasingly more on organizational innovations than on specific technologies. With regard to decision making in organizational innovation there are two major issues: (1) there is not enough data to answer questions with enough precision for use in decision making and (2) collecting data is expensive and there is not enough budgeted to provide helpful answers (Hadley 2000). This paper illustrates how decision-analytic modeling can assist decision making in organizational innovations that are system changes rather than specific technologies. Especially in an early stage, organizational innovations can be designed in various formats, which makes it difficult to perform trials to assess safety and efficiency. However, this lack of evidence may not mean that decisions are postponed. It is better to inform a decision with the available evidence than without any evidence at all (Weinstein et al. 2003). Decision-analytic modeling is a useful tool to synthesize available evidence, and is increasingly used to estimate the safety and efficiency of health care interventions. It is not intended to reveal scientific truth (Weinstein 2006), but to guide decision making. Data from different sources can be combined to inform decision making under conditions of uncertainty (Briggs, Claxton, and Sculpher 2006). Additionally, one can assess the expected value of perfect information (EVPI), and specifically what type of additional evidence would be most valuable. It can therefore assist in making a two-fold decision: (1) whether or not an intervention should be implemented, given the current evidence, and (2) whether it is worthwhile to perform additional research to reduce the uncertainty surrounding the first decision (Claxton, Sculpher, and Drummond 2002).

The current paper aims to illustrate how decision-analytic modeling can assist decision making in organizational innovations. As an illustration, we examined shared care in hearing aid provision.

METHODS

Decision-Analytic Modeling for Organizational Innovation

A key step in decision-analytic modeling is defining the model structure. When a Markov model is used, mutually exclusive health states need to be defined. In organizational innovation these health states should not only be based on health outcomes and costs, but also on organizational aspects. Especially in an early stage of an innovation, many alternative formats may be possible. Therefore, it is also important that the model structure allows all relevant organizational formats to be compared.

The main advantage of decision-analytic modeling is that data from different sources can be synthesized. Especially in an early stage of organizational innovation little evidence is available, and the limited evidence is often found in different sources. This uncertainty can be incorporated in the model by assigning distributions to the model parameters. Incorporating uncertainty also makes results easier to interpret by decision makers, as one can calculate which format has the highest probability of being cost-effective. One can also calculate the worth of acquiring additional information through further research, by calculating the EVPI (Claxton and Posnett 1996). Additionally one can examine for which parameters further research is the most valuable, by calculating the EVPI for (groups of) parameters (Felli and Hazen 1998).

An Illustration

Shared Care in Hearing Aid Provision

As only a small proportion of hearing-impaired adults require management other than hearing aids, it is advocated that not all persons with hearing complaints need to be examined by an ear nose throat (ENT) specialist or audiological center (AC) (Swan and Browning 1994). Increasing attention is paid to shared care initiatives in hearing aid provision, including direct hearing aid provision by private dispensers. However, it is uncertain whether private dispensers are capable of identifying the subgroup of persons who require medical care (safety). This subgroup will be referred to as “patients” throughout this paper, while persons not in need of medical care are referred to as “clients.” Also, it is unclear whether the transfer of tasks will indeed reduce costs while maintaining the quality of care (efficiency). However, this lack of evidence did not mean that decisions were postponed. In fact, some health insurance companies in the Netherlands already allow private dispensers to take over tasks from ENT specialists and ACs. This illustration examines the long-term cost-effectiveness of different shared care formats as opposed to the current organization of hearing aid provision. Additionally, we assessed the EVPI, and for which topics further research is most valuable.

Model Structure

A Markov model was constructed with mutually exclusive health states. The model simulated the lifetime course of events in a hypothetical cohort of persons with hearing complaints, aged 50 years or older. The cycle length of the model was set to 1 year.

Health states were based on hearing aid use and pathology. Regarding hearing aid use, health states were “no hearing aid,”“hearing aid user,” and “ex-user.” Hearing aid users were subdivided into two health states: good and poor quality hearing aid fitting. Pathology states were divided into acoustic neuroma, chronic otitis media, otosclerosis, and psychosocial problems, either detected or undetected. Pathology states were further divided into hearing aid users and nonusers. The final absorbing state was “death.”Figure 1 presents a schematic diagram of the model.

Figure 1.

Simplified Schematic Diagram of the Markov Model

1. Process of hearing aid fitting. 2. Incidence and detection of pathology. 3. Successful treatment of pathology. 4. Becoming ex-user. 5. Detecting undetected pathology. ^*Hearing aid users are subdivided into users with a poor and a good quality hearing aid fitting. ^**The pathology state is subdivided into “acoustic neuroma,”“chronic otitis media,”“otosclerosis,” and “psychosocial problems.” These pathologies can be either detected or undetected. For detected acoustic neuroma, before and after treatment states are distinguished. Pathology states are further subdivided into hearing aid users and nonusers, as is illustrated for the no pathology states

In the “current format” persons seek help with the general practitioner (GP) and ENT specialist or AC, and all persons fitted with a hearing aid have an evaluation of their hearing aid fitting at the ENT specialist or AC. We defined three shared care formats. In the “total direct format” persons seek help with a private dispenser, and only patients have a follow-up at the ENT specialist to evaluate the hearing aid fitting. In the “triage format” persons seek help with the private dispenser, but all persons fitted with hearing aids have an evaluation at the ENT specialist or AC. In the “follow-up format” persons seek help with their GP, but for clients there is no need for an evaluation of the hearing aid fitting by the ENT specialist or AC. The organizational formats are presented in Table S1.

Data Sources

Input parameters for the model were derived from a multicenter prospective study, a patient cohort study, published literature, and, if no other source was available, expert opinion. Unless stated otherwise, expert opinion was provided by the authors through an informal process. In the prospective study shared care versus the current organization was examined. Details of this study (N=269) are presented elsewhere (Grutters et al. 2007). The cohort study consisted of 1,000 randomly selected patients consulting for hearing problems in a community hospital in the Netherlands in 2002 (J. A. Duijvestijn et al. unpublished data). All input parameters are listed in Table S2.

Analysis

The analysis was performed from the societal perspective, and all costs were reported in Euros (€1 is US $1.25, average 2006 conversion rate). Future costs and effects were discounted to their present value by a rate of 3 percent (Siegel et al. 1997).

We compared the cost-effectiveness of the four organizational formats. Incremental cost-effectiveness ratios (ICERs) were calculated, dividing the incremental costs by the incremental quality-adjusted life years (QALYs). ICERs were calculated by comparing each format with the next most effective format. Whether a format is deemed efficient depends on how much society is willing to pay for a gain in effect, which is referred to as the ceiling ratio. The National Institute for Health and Clinical Excellence in the United Kingdom uses a ceiling ratio between £20,000 and £30,000 per QALY (Buxton 2006), which is roughly €40,000.

We assigned distributions to the model parameters, to reflect the second-order uncertainty in the estimation of that parameter (Weinstein 2006). See Table S2 for the assigned distributions. Parameter values were drawn at random from the assigned distributions, using Monte Carlo simulation with 10,000 iterations. To illustrate the results of the simulation, cost-effectiveness acceptability curves (CEACs) were calculated (van Hout et al. 1994; Fenwick, Claxton, and Sculpher 2001). For different ceiling ratios, the net monetary benefit was calculated for each format by subtracting the costs from the effects, multiplied by the ceiling ratio. CEACs show the probability that a format has the highest net monetary benefit, and thus is deemed cost-effective, given different ceiling ratios.

In the base case analysis age was set at 70 years, as this was the median age of participants in the prospective study and was estimated to be the median age for persons seeking help for their hearing complaints. In a subgroup analysis, we varied age to 50 and 85 years.

As uncertainty exists, there is always a chance that the “wrong” decision will be made (Briggs, Claxton, and Sculpher 2006). The EVPI is the expected value of obtaining perfect knowledge of the “true” values of all parameters. We calculated the total EVPI by subtracting the net monetary benefit of the organizational format we would choose under conditions of uncertainty, from the net monetary benefit of the optimal decision we would make if we knew the “true” parameter values. Finally, we calculated the partial EVPI for (groups of) parameters (Felli and Hazen 1998), for a ceiling ratio of €40,000 per QALY, to examine for which parameters further research is the most valuable. We calculated this partial EVPI by subtracting the net monetary benefit of the format we would choose under conditions of uncertainty for a given parameter (group), from the net monetary benefit of the optimal decision we would make if we knew the “true” value of that parameter (group).

RESULTS

Cost-Effectiveness

For a 70-year-old population, the current format yielded the most QALYs (6.765; Table 1), followed by the follow-up format. The follow-up format was associated with a QALY loss of 0.004 and cost savings of €43, resulting in a mean ICER of €10,972 saved per QALY lost. Because the incremental results are negative (less effective and less expensive), in this case higher ICERs are more acceptable. That is, with higher ICERs more money is saved per QALY lost. As both the triage format and the total direct format were somewhat more expensive than the follow-up format and associated with a loss in QALYs, these formats were judged as inferior.

Table 1.

Base Case Results and Subgroup Analyses

	Expected Costs in € (95% CI)	Expected QALYs (95% CI)	Comparison	ICER (€ Saved per QALY Lost)
70 years (base case analysis)
Current format	1,604 (1,449 to 1,967)	6.765 (6.48 to 7.03)
Follow-up format	1,560 (1,415 to 1,928)	6.761 (6.48 to 7.03)	Current format	10,972
Triage format	1,608 (1,379 to 1,969)	6.754 (6.48 to 7.02)	Follow-up format	Inferior
Total direct format	1,562 (1,344 to 1,925)	6.750 (6.48 to 7.02)	Follow-up format	Inferior
50 years
Current format	2,656 (2,348 to 3,639)	12.161 (11.65 to 12.64)
Follow-up format	2,571 (2,287 to 3,567)	12.152 (11.64 to 12.63)	Current format	9604
Triage format	2,592 (2,260 to 3,593)	12.099 (11.65 to 12.63)	Follow-up format	Inferior
Total direct format	2,505 (2,197 to 3,509)	12.090 (11.64 to 12.62)	Follow-up format	1080
85 years
Current format	693 (636 to 780)	2.905 (2.78 to 3.02)
Follow-up format	679 (624 to 765)	2.904 (2.78 to 3.02)	Current format	15,512
Triage format	700 (597 to 816)	2.902 (2.78 to 3.02)	Follow-up format	Inferior
Total direct format	684 (583 to 799)	2.901 (2.78 to 3.02)	Follow-up format	Inferior

95% confidence intervals (CI) are based on probabilistic analysis.

In Figure 2 the CEACs show that considerable uncertainty surrounds the decision which format is deemed cost-effective. Given a ceiling ratio of €40,000, the current format had the highest probability of being cost-effective (51 percent), followed by the triage (40 percent), the follow-up (6 percent), and the total direct format (3 percent).

Figure 2.

Cost-Effectiveness Acceptability Curves for the Four Formats

The subgroup analysis for age showed that younger age (50 years) was associated with a higher probability (58 percent) that the current format is cost-effective than older age (85 years; 42 percent). Otherwise, age did not affect the results.

EVPI

For the base case analysis, the uncertainty surrounding the decision whether or not to implement shared care resulted in an EVPI of €87 per person, given a ceiling ratio of €40,000. Implementing shared care affects all persons with hearing complaints of 50 years and older without a hearing aid, being a total of 1,155,719 persons in the Netherlands in the next 10 years (Central Bureau for Statistics 2007). This makes the population EVPI €100 million, meaning that perfect information on this topic is worth €100 million. The population EVPI for different ceiling ratios is presented in Table S3.

Additionally, the EVPI for parameters was calculated. The parameter most valuable for further research was about access to care: whether persons with hearing complaints would seek help sooner when they can do so directly with the private dispenser. Other valuable parameters for further research were about the safety of care: the probability that the private dispenser refers patients to the ENT specialist (sensitivity), followed by the probability that undetected pathology will nevertheless be detected before any harm is done. Additional research with regard to the utility scores was also worthwhile. For the remaining (groups of) parameters the EVPI was relatively low (Table S4).

DISCUSSION

Some years ago Hadley presented two major issues in health services research: (1) there is not enough data to answer questions with enough precision for people to use in decision making, and (2) collecting data is expensive, and health services research does not have large enough budgets to provide helpful answers (Hadley 2000). The current paper shows that decision-analytic modeling can assist in both these issues. First, all available evidence can be synthesized in a model to guide decision making. Next, EVPI analyses can help to allocate research budgets in the most efficient way. This will obviously not answer all the questions that exist in health services research, but will at least help to guide the research agenda.

There are a number of limitations of the present study that need to be addressed. First, due to the lack of evidence in this early stage of the innovation, many uncertain parameters were included in the model, based on expert opinion or small study populations. However, a model should not be faulted because existing data fall short of ideal standards of scientific rigor (Weinstein et al. 2003). That is, decisions will be made; hence, it is better to inform the decision with the available evidence than without any evidence at all. Using decision-analytic modeling we were able to incorporate the uncertainty in the model, and results were shown with surrounding uncertainty. We did assume “perfect implementation,” although. Less than perfect implementation reduces efficiency, and it is unclear whether patients and professionals will change their behavior if shared care would be implemented. If shared care in hearing aid provision would be deemed efficient, it would therefore be interesting to examine the value of perfect implementation. This would allow for a tradeoff between the costs and benefits of implementation strategies (Fenwick, Claxton, and Sculpher 2008; Hoomans et al. 2007). Also beyond the scope of this paper, but worth considering if in a later stage shared care is found to be cost-effective and may be implemented, is the possible disadvantage that the prescriber of the hearing aid financially benefits from selling hearing aids.

Another limitation is that decision-analytic modeling is rather computer-intensive. Especially for the partial EVPI, many iterations need to be drawn. In the present study, we ran 200 Monte Carlo simulations by 200 draws for the parameter (group) of interest. When repeatedly running our analysis the ordering of which parameters were most valuable for further research remained stable. Thus, we could accurately estimate which parameters are more and less valuable for further research. One should, however, be aware that, to answer the question how valuable obtaining perfect information for each parameter exactly is, many more iterations would be required.

Finally, modeling is always a simplification of a complex reality. In our case, we incorporated four types of pathology, while other types also may affect hearing and may be missed by dispensers. However, we feel that we included the most important types of pathology in this study.

CONCLUSION

Overall, this study shows that decision-analytic modeling in an early stage of organizational innovation is a valuable tool to facilitate evidence-based decision making. Despite the limited evidence, the illustrative decision-analytic model provided valuable information for decision makers on whether or not to implement shared care in hearing aid provision. Moreover, it informed decision makers and researchers on whether or not to perform additional research and for which topics additional research is most valuable.

Supplementary material

The following material is available for this article online:

Appendix SA1 Author Matrix.

Table S1. Alternative Organizational Formats of Hearing Aid Provision.

Table S2. List of Input Parameters.

Table S3. Population Expected Value of Perfect Information Curve.

Table S4. Expected Value of Perfect Information for (Groups of) Parameters.

This material is a vailable as part of the online article form: This material is available as part of the online article from: http://www.blackwellsynergy.com/doi/abs/10.1111/j.1475-6773.2008.00872.x (this link will take you to the article abstract).

Please note: Blackwell Publishing is not responsible for the content or functionality of any supplementary materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.