Translating Medical Effectiveness Research into Policy: Lessons from the California Health Benefits Review Program

Abstract

Context: Legislatures and executive branch agencies in the United States and other nations are increasingly using reviews of the medical literature to inform health policy decisions. To clarify these efforts to give policymakers evidence of medical effectiveness, this article discusses the California Health Benefits Review Program (CHBRP). This program, based at the University of California, analyzes the medical effectiveness of health insurance benefit mandate bills for the California legislature, as well as their impact on cost and public health.

Methods: This article is based on the authors’ experience reviewing benefit mandate bills for CHBRP and findings from evaluations of the program. General observations are illustrated with examples from CHBRP's reports. Information about efforts to incorporate evidence into health policymaking in other states and nations was obtained through a review of published literature.

Findings: CHBRP produces reports that California legislators, legislative staff, and other major stakeholders value and use routinely in deliberations about benefit mandate bills. Where available, the program relies on previously published meta-analyses and systematic reviews to streamline the review of the medical literature. Faculty and staff responsible for the medical effectiveness sections of CHBRP's reports have learned four major lessons over the course of the program's six-year history: the need to (1) recognize the limitations of the medical literature, (2) anticipate the need to inform legislators about the complexity of evidence, (3) have realistic expectations regarding the impact of medical effectiveness reviews, and (4) understand the consequences of the reactive nature of mandated benefit reviews.

Conclusions: CHBRP has demonstrated that it is possible to produce useful reviews of the medical literature within the tight time constraints of the legislative process. The program's reports have provided state legislators with independent analyses that allow them to move beyond sifting through conflicting information from proponents and opponents to consider difficult policy choices and their implications.

The principles and tools of evidence-based medicine (EBM) are being increasingly applied to policy questions, and some countries with national or provincial health insurance systems are using systematic reviews of the medical literature to determine which specific health care services their systems will cover (Morgan et al. 2006; Wailoo et al. 2004). In the United States, both private and public health plans rely on literature reviews to develop coverage policies (Foote and Town 2007; Fox 2005; Garber 2001; Hartung, Ketchum, and Haxby 2006).

Reviews of the medical literature also are conducted to help state legislators evaluate legislation mandating that health plans and health insurers cover specific tests, procedures, medications, providers, diseases, or conditions. Such bills are often referred to as benefit mandates. This article reviews the efforts of the California Health Benefits Review Program (CHBRP), a program based at the University of California, to provide legislators with evidence regarding medical effectiveness and the cost and public health impacts of proposed benefit mandates.

CHBRP's reports offer relevant and timely analyses that state legislators and other major stakeholders use in deliberations regarding benefit mandate bills. These bills have addressed a wide range of topics, including preventive services, chronic disease management, behavioral health services, and complementary and alternative providers, and their legislative outcomes sometimes are based on the conclusions of CHBRP's medical effectiveness reviews. But medical effectiveness evidence does not always determine legislative outcomes; concerns about costs and other political considerations are important as well.

Here we focus on the lessons we have learned over the past six years from conducting medical effectiveness reviews for CHBRP. Our article builds on a series of papers published in 2006 that discuss the establishment of the program, the development of its analytic methods, and the first and second years of its operation (Halpin et al. 2006; Kominski et al. 2006; Luft et al. 2006; McMenamin, Halpin, and Ganiats 2006; Oliver and Singer 2006; Philip 2006). Over the ensuing years, the number of reports issued by CHBRP has grown from twenty-two to fifty-nine. Whereas the earlier articles presented examples of individual bill analyses, our article synthesizes the findings from CHBRP's medical effectiveness reviews across all reports issued through May 2009.

Mandated Benefit Review Laws

As of 2008, states had enacted 1,505 health insurance benefit mandates, the number ranging from a low of six in Idaho to a high of fifty-two in Maryland, with a mean of thirty per state (Laudicina, Gardner, and Crawford 2008). Thirty states, listed in table 1, enacted mandate review laws to help state legislators (and governors) make better-informed decisions about benefit mandates, with nearly two-thirds of these laws passed since 2000. Most mandate review laws specify that reviews be conducted before the state legislature considers mandate bills. A study of states with mandate review laws as of September 2004 found that twenty-five of twenty-six states stipulated that reviewers use specific criteria to evaluate the bills. Furthermore, twelve of these states (48 percent) required reviewers to examine the medical effectiveness of the technologies or interventions that the proposed mandates would require health plans and health insurers to cover (Bellows, Halpin, and McMenamin 2006).

TABLE 1.

States That Require Review of Health Insurance Mandate Bills

Arizona	Nevada
Arkansas	New Hampshire
California	New Jersey
Colorado	New York
Florida	North Carolina
Georgia	North Dakota
Hawaii	Ohio
Indiana	Oregon
Kansas	Pennsylvania
Kentucky	South Carolina
Louisiana	Tennessee
Maine	Texas
Maryland	Virginia
Massachusetts	Washington
Minnesota	Wisconsin

Source:Laudicina, Gardner, and Crawford 2008.

A Description of CHBRP and Its Medical Effectiveness Methods

CHBRP was established by the California legislature in 2002 to give state legislators rigorous and objective analyses of the medical, financial, and public health impacts of proposed health insurance benefit mandates and repeals (Philip 2006). The legislation requires the chairs of the Committees on Health of the California State Senate and California State Assembly to ask CHBRP to analyze all benefit mandate bills. The analyses must be completed within sixty days (effectively thirty days to complete a draft for review) to furnish information to the legislature before it considers the mandate bills. The analyses are funded through an assessment charged to licensed health plans and insurers.

Through May 2009, CHBRP had issued fifty-nine reports on proposed benefit mandates and repeals of existing mandates and eight formal follow-up letters supplying further details regarding completed analyses. Table 2 lists the bills that CHBRP has analyzed, indicates whether they were enacted into law, and summarizes the findings from CHBRP's medical effectiveness reviews. All reports are transmitted electronically to California legislators and their staff and are posted on the program's website (http://www.chbrp.org), from which they may be downloaded free of charge. Members of the public, including persons affiliated with benefit mandate review programs in other states, may register for a listserv through which CHBRP sends email alerts when it receives new requests from the legislature or issues new reports.

TABLE 2.

Health Insurance Benefit Mandate Bills Analyzed by the California Health Benefits Review Program (CHBRP)

Year	Bill No.	Topic	Status	Conclusion of CHBRP's Medical Effectiveness Review
2004	ABa 438	Osteoporosis screening	Died in the legislatureb	Evidence shows that screening women aged 65 or older is effective, but there is insufficient evidence to determine the effectiveness of screening younger women.
	AB 547	Ovarian cancer screening	Gutted/amendedc	Evidence is insufficient to determine whether screening improves outcomes.
	AB 1084	Vision services providers	Reintroduced as AB 1927	N/A: Relative effectiveness of optometrists and ophthalmologists was not assessed.
	AB 1549	Asthma self-management for children	Reintroduced as AB 2185	Evidence shows that pediatric asthma self-management education improves health outcomes and reduces use of acute care services.
	AB 1927	Vision services providers	Gutted/amended	Same as AB 1084.
	AB 2185	Asthma self-management for children and asthma devices	Enacted	Evidence shows that pediatric asthma self-management education improves health outcomes and reduces use of acute care services. Evidence shows that using a metered-dose inhaler (MDI) with a spacer is as effective as using a nebulizer. Evidence is insufficient to ascertain whether peak flow meters are effective and whether using an MDI with a spacer is more effective than using an MDI alone.
	SB 101	Substance use disorders	Reintroduced as SB 1192	Evidence shows that treatment of substance use disorders is effective but that effectiveness may vary by type of addiction.
	SB 174	Hearing aids for children	Reintroduced as SB 1158	Evidence suggests that treatment of hearing loss is effective, but evidence is insufficient to determine the relative effectiveness of different types of hearing aid circuitry.
	SB 897	Maternity services	Reintroduced as SB 1555	Evidence suggests that many individual maternity services are effective, but evidence is insufficient to determine which combination(s) of maternity services improve outcomes for pregnant women and infants.
	SB 1157	Elimination of intoxication exclusion	Vetoed by governor	Medical effectiveness could not be analyzed because this bill does not mandate coverage of a particular health care service but instead prohibits coverage exclusions and because there are no published data on the medical effects of removing coverage exclusions.
	SB 1158	Hearing aids for children	Vetoed by governor	Same as SB 174.
	SB 1192	Substance use disorders	Died in the legislature	Same as SB 101.
	SB 1555	Maternity services	Vetoed by governor	Same as SB 897.
2005	AB 8	Mastectomies and lymph node dissections: length of stay	Gutted/amended	Evidence is insufficient to determine whether longer length of inpatient stay is associated with better outcomes for women who have a mastectomy or lymph node dissection.
	AB 213	Lymphedema	Died in the legislature	Evidence shows that manual lymphatic drainage and compression therapy are associated with reduction in limb size, but evidence is insufficient to determine whether smaller limb size is associated with improvement in health outcomes (e.g., limb function).
	AB 228	Transplant services for persons with human immunodeficiency virus	Enacted	Evidence from case series and case reports suggests that patients with human immunodeficiency virus (HIV) undergoing kidney transplantation have survival rates similar to those of patients without HIV. Among persons who do not have hepatitis C, survival rates for liver transplantation also are similar for patients both with and without HIV. Few studies have assessed the impact of HIV on the success of transplantation of other organs.
	AB 1185	Chiropractic services	Died in the legislature	Evidence from studies with major methodological limitations suggests that chiropractic care reduces pain and improves functional status. Evidence is insufficient to determine whether chiropractic care affects quality of life.
	SB 415	Alzheimer's disease drugs	Gutted/amended	Evidence indicates that Alzheimer's disease drugs provide modest, short-term improvements in cognitive, functional, behavioral, and neuropsychological outcomes. Evidence of effects on nursing home placement is inconsistent.
	SB 572	Mental health benefits	Died in the legislature; reintroduced as AB 423	Evidence suggests that mental health parity laws are associated with reductions in out-of-pocket costs for consumers but do not substantially increase the numbers of persons receiving mental health services.
	SB 573	Elimination of intoxication exclusion	Vetoed by governor	Same as SB 1157.
	SB 576	Tobacco cessation services	Vetoed by governor	Much evidence indicates that tobacco cessation counseling and medication are effective.
	SB 749	Autism diagnosis	Died in the legislature	No studies have assessed the specific diagnostic process mandated in the bill. Expert opinion suggests the bill would increase the accuracy of autism diagnoses, decrease the time between first referral and diagnosis, and lower the average age at diagnosis.
	SB 913	Rheumatic disease drugs	Died in the legislature	Evidence indicates that biological response modifier (BRM) medications are effective treatments for rheumatoid arthritis, psoriatic arthritis, and ankylosing spondylitis.
2006	AB 264	Pediatric asthma education	Vetoed by governor	Evidence shows that pediatric asthma self-management education improves health outcomes and reduces use of acute care services.
	AB 2012	Orthotic and prosthetic devices	Enacted	The effectiveness of prosthetic devices compared with no treatment was not evaluated because use of conventional prosthetic devices has been established as the standard of care for improving physical and psychological functioning of persons with amputations and congenital limb deformities. Evidence from small observational studies suggests that new prosthetic technologies may benefit nonelderly adults with lower limb amputations or deformities who are healthy and active.
	AB 2281	High-deductible health plans: coverage for preventive services	Died in the legislature	Evidence shows that many preventive services improve health and well-being. No studies have assessed the effect of high-deductible health plans on use of preventive services. However, most studies of persons with other types of health insurance have found that lower cost sharing is associated with greater use of beneficial preventive services.
	SB 1223	Hearing aids for children	Vetoed by governor	Same as SB 174, with the addition of evidence from small studies suggesting that certain newer hearing aid technologies (e.g., dual microphones) are more effective than older technologies.
	SB 1245	Cervical cancer screening	Enacted	Evidence shows that use of human papillomavirus (HPV) testing as an adjunct to the conventional Pap test for cervical cancer increases the accuracy of screening and improves the efficiency of screening programs.
	SB 1508	Propofol for colonoscopies	Died in the legislature	Evidence shows that propofol is associated with shorter recovery time and fewer side effects than traditional methods for sedating persons undergoing colonoscopies. Evidence of effects on cognitive and physiological outcomes is inconsistent. The safety of propofol is similar to that of other sedative and analgesic agents.
2007	AB 30	Inborn errors of metabolism: coverage of medical formulas and foods	Vetoed by governor	No controlled studies of treatments for inborn errors of metabolism (IEM) were identified. However, IEM disorders are single-cause conditions for which the scientific basis and rationale for treatment with medical formulas and foods are strong.
	AB 54	Acupuncture	Vetoed by governor	Evidence shows that acupuncture is an effective treatment for lateral elbow pain, neck disorders, osteoarthritis of the knee, and postoperative nausea and vomiting and is more effective than other nonsurgical treatments for chronic headache, osteoarthritis of the knee, and pelvic pain associated with pregnancy.
	AB 368	Hearing aids for children	Vetoed by governor	Same as SB 1223.
	AB 423	Mental health and substance abuse services	Vetoed by governor; reintroduced as AB 1887	Same as SB 572, except findings from new studies were added that showed that mental health parity laws were associated with modest improvement in treatment of depression but did not affect suicide rates.
	AB 1214	Waiver of mandated benefits	Died in the legislature	Evidence shows that many of California's health insurance benefit mandates require health plans to cover medically effective services.
	AB 1429	Human papillomavirus vaccine	Vetoed by governor; reintroduced as AB 16	Evidence from the only clinical trial published to date suggests that for women and girls who received all three doses of the vaccine and were not previously exposed to the human papillomavirus (HPV), the vaccine is highly effective for prevention of precancerous cervical lesions and adenocarcinoma associated with the four types of HPV to which it is targeted (i.e., types 6, 11, 16, and 18).
	AB 1461	Elimination of intoxication exclusion	Enacted	Same as SB 1157.
	SB 24	Tobacco cessation services	Never introducedd	Same as SB 576.
	SB 365	Out-of-state carriers	Died in the legislature	No medical effectiveness analysis was undertaken.
2008	AB 16	Human papillomavirus vaccine	Vetoed by governor; reintroduced as SB 158	Same as AB 1429.
	AB 1774	Gynecological cancer screening	Died in the legislature	Findings differ across gynecological cancers. Evidence shows that cervical cancer screening reduces the incidence of cervical cancer in asymptomatic women who are sexually active and have not had a hysterectomy. The evidence suggests that screening asymptomatic women at average risk for ovarian cancer can detect ovarian cancer at an earlier stage, but evidence is insufficient to determine whether screening asymptomatic women at average risk reduces morbidity and mortality. No studies have assessed the effectiveness of screening asymptomatic women for endometrial cancer.
	AB 1887	Mental health and substance abuse services	Vetoed by governor; reintroduced as AB 244	Same as AB 423.
	AB 1894	Human immunodeficiency virus testing	Enacted	Substantial indirect evidence shows that screening for human immunodeficiency virus (HIV) is effective. Findings from multiple studies suggest that tests for HIV are highly accurate (i.e., have high sensitivity and specificity) and that highly active antiretroviral therapy (HAART), prophylaxis for opportunistic infection, and vaccination against hepatitis B and influenza reduce the risk of clinical progression of HIV, opportunistic infection, and death. Delivering infants born to HIV-positive mothers by elective cesarean section and feeding them formula instead of breast milk further reduce the risk of mother-to-child transmission above and beyond the reduction in risk achieved through use of HAART.
	AB 1962	Maternity services	Vetoed by governor; reintroduced as AB 98	The medical effectiveness analysis for this bill focused on prenatal care services. Randomized controlled trials (RCTs) have consistently found no association between the numbers of pregnant women's prenatal visits and birth outcomes for either infants or mothers. However, there is evidence from multiple RCTs that some individual prenatal care services are effective (e.g., smoking cessation counseling, screening for hepatitis B, treatment of women at increased risk for preterm delivery with corticosteroids and progestational agents).
	AB 2174	Amino acid–based elemental formulas for eosinophilic disorders and short bowel syndrome	Died in the legislature; revised and reintroduced as AB 163	The only evidence regarding the effectiveness of amino acid–based elemental formula for the treatment of eosinophilic disorders (ED) and short bowel syndrome (SBS) comes from studies with weak research designs and small sample sizes. Findings suggest that both elemental formula and elimination diet are effective treatments for ED. Among persons with SBS, elemental formula is associated with shorter duration of use of parenteral nutrition and fewer hospitalizations.
	AB 2234	Breast cancer screening	Died in the legislature; revised and reintroduced as AB 56	Evidence shows that screening mammography reduces breast cancer morbidity and mortality. Evidence also shows that digital mammography improves the rate of cancer detection among women with radiologically dense breasts, pre- and perimenopausal women, and women younger than age 50. Evidence suggests as well that breast magnetic resonance imaging (BMRI) may be useful for detecting cancers in high-risk women. Evidence shows that notifying women for whom screening mammography is recommended increases the rate at which women are screened.
	SB 1198	Durable medical equipment	Vetoed by governor; revised and reintroduced as AB 214	Few studies have assessed the effectiveness of durable medical equipment (DME). However, expert opinion and observation suggest that use of DME can improve health, functioning, and quality of life. Findings from studies on the effect of insurance coverage on use of DME are inconsistent.
	SB 1522	Standardization of coverage sold in the individual health insurance market	Died in the legislature; reintroduced as AB 786	No medical effectiveness analysis was undertaken.
	SB 1634	Orthodontic procedures for cleft palate	Vetoed by governor	Orthodontic services, coordinated with surgeries, are a central part of the standard of care for treatment of oral clefts. Expert consensus for treatment of oral clefts is that all care should be provided by multidisciplinary teams of experts. No studies were identified that directly addressed whether having additional coverage of orthodontic services beyond the previously mandated coverage of surgical care would affect health outcomes.
2009	AB 56	Mammography	Vetoed by governor	Same as AB 2234, except that this version of the bill addressed only coverage for mammography and notification of women for whom screening mammography is recommended.
	AB 98	Maternity services	Vetoed by governor	Same as AB 1962.
	AB 163	Amino acid–based elemental formula	Pending in the legislature	Same as AB 2174, except that this version of the bill only addresses eosinophilic disorders (i.e., does not address short bowel syndrome).
	AB 214	Durable medical equipment	Pending in the legislature	Same as SB 1198.
	AB 244	Mental health services	Vetoed by governor	Same as AB 423 and AB 1887, except for the addition of new studies suggesting that mental health parity laws are associated with small increases in use of mental health services by persons employed by moderately small firms (50 to 100 employees) who have poor mental health or low incomes.
	AB 259	Certified nurse midwives: direct access	Pending in the legislature	Evidence suggests that outcomes for pregnant women and infants cared for by certified nurse midwives are as good or better than outcomes for those treated by physicians.
	AB 513	Breastfeeding: lactation consultation and breast pumps	Vetoed by governor	Evidence as to whether providing extra lactation consultation beyond usual care increases duration of breastfeeding is inconsistent; evidence that coverage for breast pumps is associated with longer duration of breastfeeding among low-income women is limited.
	AB 786	Standardization of coverage sold in the individual health insurance market	Pending in the legislature	Same as SB 1522.
	SB 92	Health insurance reform	Pending in the legislature	Same as AB 1214.
	SB 158	Human papillomavirus vaccine	Vetoed by governor	Same as AB 1429, except for the addition of evidence from new clinical trials suggesting that the vaccine is less effective among females who have not completed all three doses of the vaccine and/or were exposed to the human papillomavirus (HPV) prior to vaccination than among females who received all three doses and were not previously exposed to HPV. Results from the new trials also suggest that the vaccine is not very effective in preventing precancerous cervical lesions associated with HPV types other than the four HPV types the vaccine targets. The evidence suggests that females who are vaccinated should continue to be screened for cervical cancer.
	SB 161	Orally administered anticancer medications	Vetoed by governor	Evidence-based guidelines recommend the use of 38 oral anticancer medications to treat more than 50 types of cancer. Many oral anticancer medications have no intravenous or injectable substitutes, although there are some important exceptions.

Notes:

^a“AB” indicates a bill that originated in the California State Assembly. “SB” indicates a bill that originated in the California State Senate.

^b“Died in the legislature” means that a bill was not approved by both houses of the California legislature before the end of the legislative session.

^c“Gutted/amended” indicates that the California legislature amended a health insurance benefit mandate bill to strip the bill of its contents and replace it with legislative language on a different topic.

^dCHBRP analyzed a proposed amendment to SB 24 that would have inserted provisions into the bill requiring health plans to cover tobacco cessation counseling and pharmacotherapy. After CHBRP issued its report, the bill's author elected not to amend the bill to add these provisions.

Sources:Legislative Counsel of the State of California 2009; Philip 2006.

The review program primarily analyzes legislation that would apply to commercial health plans and health insurance policies sold in the group and/or individual markets. In California, the Department of Managed Health Care regulates health maintenance organizations (HMOs) and some preferred provider organizations (PPOs), and the Department of Insurance regulates the remaining PPOs, indemnity plans, and other insurance products. Some mandate bills also apply to commercial health plans and health insurers that provide coverage for persons enrolled in Healthy Families (California's State Children's Health Insurance Program), Medi-Cal (California's Medicaid program) managed care plans, Access for Infants and Mothers (a program for pregnant women not eligible for Medi-Cal), and the Managed Risk Medical Insurance Program (a program for persons with preexisting conditions who cannot otherwise obtain health insurance). Mandate bills do not apply to Medicare or self-insured health plans sponsored by employers because the states do not have the authority to regulate those types of health plans.

CHBRP is administered by the University of California (UC), but it is institutionally independent and led by a small analytic staff at the UC's Office of the President, which coordinates with the state legislature, executive branch agencies, and the governor's office. A statewide faculty task force composed of clinician and nonclinician health services researchers from UC's five medical schools and two public health schools, as well as Loma Linda University, Stanford University, and the University of Southern California, carry out the analyses. CHBRP contracts with a certified actuarial firm to help the task force with its cost-impact analyses, and a national advisory council composed of experts from outside California who represent major stakeholders with an interest in health insurance benefit mandates provides guidance (Philip 2006).

Those faculty and staff who are responsible for the medical effectiveness sections of the reports review the medical literature to determine the clinical effectiveness of the technologies and interventions addressed in the proposed mandates. For each bill, faculty and staff work with one or more content experts to examine the technology or intervention addressed, specify the population to which the technology or intervention would likely apply, identify the relevant outcomes, and list pertinent literature sources. Faculty and staff draw up specifications for the literature search that are used by a medical librarian to search databases likely to contain the relevant peer-reviewed and gray literature (i.e., literature not published by commercial publishers). Once a search has been completed, the faculty and staff review abstracts of the articles and, if necessary, the full text to find studies that meet the inclusion criteria (CHBRP 2008d). Information regarding the characteristics (e.g., sample size, population studied) and findings of included studies is then abstracted and analyzed.

Studies are selected for inclusion in reviews based on a hierarchy of evidence, shown in table 3, that ranks the studies according to the strength of their research designs (CHBRP 2008d). Ideally, for each bill, CHBRP relies on the studies with the strongest research designs available and uses studies with weaker designs solely when they contain the only evidence available about important outcomes or populations. In addition to the research design, the following four criteria are used to assess the strength of the evidence regarding the effectiveness of the technologies or services addressed by a proposed benefit mandate: (1) statistical significance, (2) direction of effect, (3) size of effect (i.e., clinical significance), and (4) generalizability of results (CHBRP 2008d). CHBRP analyzes the possible effects of the proposed mandates on outcomes stipulated by the legislation or, if none are specified, the outcomes identified by qualified content experts as most important to evaluating the mandate's effectiveness. These findings are summarized in both text and tables (Luft et al. 2006).

TABLE 3.

California Health Benefits Review Program (CHBRP) Hierarchy of Evidence of Medical Effectiveness

1. High-quality meta-analysesa

2. Systematic reviews

3. Well-implemented randomized controlled trials (RCTs) and cluster RCTsb

4. RCTs and cluster RCTs with major weaknesses

5. Nonrandomized studies with comparison groups and time series analyses

6. Case series and case reports

7. Narrative reviews and clinical guidelines based on consensus or opinion

Notes:

^a“High-quality” meta-analyses are meta-analyses that have clear objectives and hypotheses, apply appropriate inclusion/exclusion criteria, assess meaningful outcomes, and use sound methods to find, select, and evaluate studies and to generate pooled estimates of an intervention's effects.

^b“Well-implemented” RCTs and cluster RCTs are defined as studies that have (1) sample sizes that are sufficiently large to detect statistically significant differences between the intervention and control groups, (2) low attrition rates (less than 20 percent) or use intent-to-treat methods, and (3) intervention and control groups that are statistically equivalent before the intervention with respect to baseline measures of the outcome and important factors associated with the outcome. To be considered well implemented, a cluster RCT must also use appropriate statistical methods to take into account the clustering of observations at the level at which randomization occurs.

Source:CHBRP 2008d.

The key differences between CHBRP's reviews and those of organizations that perform systematic reviews of the medical literature, such as the Agency for Healthcare Research and Quality's (AHRQ) Evidence-Based Practice Centers (EPCs), the Cochrane Collaboration, and the Drug Effectiveness Review Project, are that the CHBRP reviews (1) are intended to be tailored to the bill's specific populations and services; (2) rely, whenever possible, on previous reviews in order to meet the tight time requirements; and (3) are subject to a less-intensive review process. Comprehensive meta-analyses and systematic reviews take much time and effort, and a new review addressing the specific issues raised in a bill would be impossible to complete within sixty days. This was clear when the legislation establishing CHBRP was passed, and so it was understood that timely, albeit less than perfect, information was preferable to no information at all.

CHBRP, therefore, relies heavily on previous meta-analyses and systematic reviews to streamline its reviews of the literature. If a high-quality meta-analysis or systematic review is available, such as a review conducted by the Cochrane Collaboration or one of the AHRQ EPCs, CHBRP will rely mainly on its findings. In such cases, CHBRP reviews only those individual studies published after the meta-analysis or systematic review. For example, CHBRP's analysis of a bill mandating coverage for screening for the human immunodeficiency virus (HIV) was based largely on three systematic reviews conducted by the Oregon EPC (Chou and Huffman 2007; Chou et al. 2005a, 2005b). Similarly, CHBRP's analyses of bills mandating coverage of pediatric asthma self-management education drew extensively on a Cochrane review (Wolf et al. 2002). In fact, meta-analyses and/or systematic reviews were available on topics addressed by thirty-one of the fifty-one mandate bills for which CHBRP has completed medical effectiveness reviews.

Although its reliance on previous meta-analyses and systematic reviews allows CHBRP to produce reports within sixty days, it also has some important limitations. For example, the previous reviews may not be complete, and their methods may not be transparent (Whitlock et al. 2008). Furthermore, the research questions that previous reviews answer are not always well matched to the subjects of mandate bills. The study populations assessed may be narrower or broader than the population to which the bill would apply. That is, although a bill may apply to all persons with a particular disease or condition, some meta-analyses and systematic reviews synthesize only those studies of persons who were hospitalized for the disease or condition. The previous reviews also may not have examined the effects on the outcomes of greatest interest to California's policymakers. If available meta-analyses or systematic reviews or individual studies have only limited relevance to a bill, CHBRP explicitly notes these limitations in its reports and focuses on other available evidence.

Owing to time constraints, fewer content experts participate in the development and review of CHBRP's reports than in the meta-analyses and systematic reviews completed by the AHRQ EPCs, the Cochrane Collaboration, and the Drug Effectiveness Review Project Review (AHRQ 2009; Drug Effectiveness Review Project 2009; Higgins and Green 2009). Only one or two content experts contribute to CHBRP's protocol for the literature review. Draft reports are reviewed by only one or two content experts and three to five members of CHBRP's National Advisory Council, who include health policy researchers, clinicians, journalists, health care executives, and leaders of professional societies and trade associations. In addition, unlike the Drug Effectiveness Review Project, CHBRP does not offer opportunities for the public to comment on draft reports, although the public is given the opportunity to supply information relevant to the analysis before the report is completed. Streamlining the review process in these ways helps CHBRP finish its reports within the required sixty days, although CHBRP's reports may miss subtle but important nuances in evidence that might have been identified in a more extensive review by a larger group of experts.

The reviews of the medical literature inform CHBRP's analyses of the cost and the public health impacts of the benefit mandate bills. The faculty and staff responsible for analyzing the proposed mandates’ effects on health insurance premiums incorporate the medical effectiveness reviews’ findings in their estimates. For example, if the medical effectiveness review finds that the use of the intervention or technology addressed in a mandate bill (e.g., training in the self-management of a chronic illness) is associated with less use of other, more expensive services (e.g., hospital and emergency department use), the cost team will factor this information into its analysis (Kominski et al. 2006). If the cost team determines that a proposed mandate is likely to lead to the increased use of an intervention or technology, the faculty and staff working on the public health analysis will use the findings from the medical effectiveness review to estimate the proposed mandate's effect on Californians’ health (McMenamin, Halpin, and Ganiats 2006).

Some proposed mandates have such broad scopes that synthesizing the relevant literature in sixty days is not possible. Examples are two bills regarding coverage for durable medical equipment (DME) that would have required health plans and health insurers to cover a large number of devices used by persons with a wide variety of diseases and conditions (CHBRP 2009c). Other examples are a bill that would require parity in the coverage of orally administered and intravenously administered/injectable anticancer medications. The bill would have applied to thirty-eight drugs used to treat more than fifty different types of cancer. For this bill, CHBRP relied on the National Comprehensive Cancer Network's guidelines to identify the recommended uses for orally administered anticancer drugs and discussed the drugs that were most widely prescribed in California and accounted for the largest share of total spending for such drugs (CHBRP 2009f).

In addition, some bills address the design of health insurance benefits, a topic that does not lend itself to a traditional medical effectiveness analysis. For example, CHBRP was asked to analyze a bill that would have required high-deductible health plans to provide first-dollar coverage (i.e., coverage not subject to a deductible) for preventive services (CHBRP 2006b). In this case, the key policy question was not whether preventive services were effective but whether having first-dollar coverage for preventive services increased the use of effective preventive services. Similarly, the critical policy question for bills that would have mandated parity in coverage for physical health conditions and for substance use disorders and less severe mental health conditions was whether parity in coverage was associated with the greater use of effective mental health and substance use disorder services and not whether such services were effective(CHBRP 2009d). (These bills have focused on parity in coverage for substance use disorders and less severe mental health conditions because California already mandates parity in coverage of severe mental health conditions as specified in a statute.)

CHBRP does not make recommendations to the California legislature regarding the merits of mandate bills. Rather, the program's faculty and staff present their findings and leave the application of the information to the legislators. In some cases, the information that CHBRP reports leads to clear conclusions about the effectiveness of a technology or intervention for which coverage would be mandated. But in other cases, CHBRP's findings reflect inconsistencies in evidence or gaps in evidence that make it difficult to determine whether an intervention would be effective. Refraining from making recommendations thus has helped CHBRP's reports gain credibility with stakeholders across the ideological spectrum and demonstrates that the purpose of the program is not to advance a particular policy agenda.

Lessons Learned from Reviewing the Medical Effectiveness of Proposed Benefit Mandates

Over the years, the faculty and staff affiliated with CHBRP have learned four main lessons from conducting medical effectiveness reviews for California policymakers that may help individuals and organizations interested in establishing similar programs.

Lesson 1: Recognize the Limitations of the Medical Literature

Although the literature on the effectiveness of medical care has grown substantially over the past several decades, CHBRP faculty and staff have encountered several major limitations of the literature on the interventions and technologies that the program has been asked to assess.

The Amount of Evidence Varies across Diseases and Conditions Addressed by a Mandate

The strength and quantity of evidence presented in the medical literature often vary across the diseases and conditions relevant to a particular mandate. A bill that would have mandated coverage of acupuncture to treat any condition is a case in point. Many studies have evaluated the effectiveness of acupuncture for headaches and musculoskeletal conditions, but few studies have looked at its effects on other conditions for which it also is used, such as asthma and epilepsy (CHBRP 2007b). As a consequence, CHBRP could not give legislators a simple yes or no answer to the question of whether acupuncture is effective. CHBRP also could not give an uncomplicated answer to the medical effectiveness question posed by a bill that would have mandated coverage for gynecological cancer screening. The literature review identified large numbers of studies of the effectiveness of screening asymptomatic women for cervical and ovarian cancers but found no studies of screening for endometrial cancer (CHBRP 2008a).

Treatments Assessed Vary across Studies Pertinent to a Mandate

CHBRP's medical effectiveness team also has had difficulty synthesizing findings from individual studies of particular topics because the studies have not always examined the same treatment. Studies of the effectiveness of ovarian cancer screening, for instance, did not always look at the same screening protocol. Some studies assessed the effectiveness of transvaginal ultrasound (TVU), whereas others examined the CA-125 blood test for tumor markers. Still others evaluated screening protocols that incorporated both the TVU and the CA-125 tests (CHBRP 2008a). The differences in the screening protocols assessed thus made it difficult to synthesize these studies’ findings.

Another bill for which literature synthesis was challenging concerned coverage for pediatric asthma self-management education (CHBRP 2006a). Some studies of asthma education interventions looked at group education, whereas others examined individual, in-person education or interactive computer programs. Similarly, some asthma education interventions addressed a wide range of self-management topics, whereas others focused exclusively on the proper use of inhalers or the reduction of exposure to allergens.

Important Clinical Outcomes Are Not Always Addressed

For some proposed mandates, few studies have been published on important clinical outcomes. In such cases, CHBRP must examine a chain of indirect evidence to decide whether a service is likely to be effective. CHBRP's analysis of a bill regarding coverage for HIV screening for asymptomatic persons is a good illustration. No studies directly link screening asymptomatic persons for HIV to reductions in morbidity and mortality, but there is evidence that tests for HIV are highly accurate. In addition, there is substantial evidence that giving HIV-positive persons both antiretroviral medications and prophylaxis and vaccination for opportunistic infections can reduce morbidity and mortality, especially if those persons are diagnosed early. These two bodies of literature offer indirect evidence that screening asymptomatic persons for HIV could reduce morbidity and mortality by increasing the numbers of HIV-positive persons who are diagnosed and treated early (CHBRP 2008b).

Studies with the Strongest Research Designs May Not Assess Effects on the Most Relevant Populations

In some cases, CHBRP confronts trade-offs between internal validity and external applicability. Those studies with the strongest research designs are not always the most generalizable to the population to which a bill would apply. CHBRP's analysis of a bill that would allow women to obtain services directly from a certified nurse midwife (CNM) without a physician's referral illustrates this dilemma. Most randomized controlled trials (RCTs) in developed countries that compare outcomes for women and infants cared for by midwives with those of women and infants cared for by physicians were conducted in Australia, Canada, New Zealand, and the United Kingdom. Midwives in these countries work within health systems that are quite different from that of the United States. The level and type of education required for midwives in these countries also differ from those required of CNMs in the United States. CHBRP therefore decided that its literature review should go beyond RCTs to include observational studies with comparison groups that were conducted in the United States (CHBRP 2009e). Although the observational studies were weaker methodologically (in particular, they could be subject to selection bias), their findings were more generalizable to the providers to which the bill would apply (i.e., CNMs) than were the non-U.S. studies.

There May Be Little Evidence of Any Kind

In some cases, the literature on the effectiveness of a treatment is sparse. A notable example is a bill that would have mandated coverage for amino acid–based elemental formula to treat persons with eosinophilic gastrointestinal disorders. Evidence of the effectiveness of elemental formula for these rare conditions is limited. Most studies are case reports and case series; studies with large sample sizes and strong research designs are nonexistent. Although eosinophilic gastrointestinal disorders affect both adults and children, no studies were found that examined the effectiveness of elemental formula for treating adults. Eosinophilic esophagitis and eosinophilic gastroenteritis are the only eosinophilic disorders for which studies of the effectiveness of elemental formula could be found. The only study of the effectiveness of this treatment for eosinophilic gastroenteritis is a case study of a single child (CHBRP 2009b). Accordingly, the medical effectiveness section of the report on this bill summarized these studies’ findings but emphasized their limitations. CHBRP faced similar challenges when analyzing a bill that would have required health plans to cover formula and medical foods for persons with inborn errors of metabolism, a group of rare genetic disorders that affect a person's ability to metabolize nutrients (CHBRP 2007a).

Lesson 2: Anticipate the Need to Inform Legislators about the Complexity of Evidence

Most state legislators and staff members have little formal training in health policy or health services research. In California, the ability of legislators and some staff to develop expertise in health care on the job is constrained by term limits of six years in the assembly and eight years in the senate, which result in the frequent turnover of committee chairs and staff (Jewell and Bero 2008; Oliver and Singer 2006). Consequently, one of CHBRP's most important roles is teaching legislators and their staff the complexities of the literature on some of the topics addressed in the benefit mandate bills.

CHBRP's informational role has been especially important for bills mandating coverage of vaccination or screening. Advocates for vaccination and screening mandates seem to concentrate exclusively on the potential benefits to persons for whom diseases or conditions may be prevented or detected early. Their arguments for vaccination and screening often downplay the reality that the majority of persons who receive a vaccination or a screening test will not develop the disease or condition, regardless of whether they are vaccinated or screened. Nor are the risks of vaccination and screening well understood. Even if a screening test is not invasive, false positive results (such as with a PSA test for prostate cancer) may lead to interventions that can have serious side effects.

The importance of assessing both the benefits and the harms of screening is illustrated by CHBRP's literature reviews for two bills that would have mandated coverage for ovarian cancer screening. Although there is evidence that screening women with the TVU and/or the CA-125 tests can detect ovarian cancers at an earlier stage, the only randomized controlled trial (RCT) on the long-term impact of ovarian cancer screening published to date concluded that screening did not improve survival (Jacobs et al. 1999). That is, screening appears simply to increase the time the woman knows she has the cancer. Assessment of the effectiveness of ovarian cancer screening is further complicated by the fact the screening tests are imprecise. Even with a positive test, ovarian cancer usually cannot be diagnosed definitively without abdominal surgery, an invasive procedure with the risk of mortality and morbidity. CHBRP thus concluded that for women at average risk for ovarian cancer, the evidence suggests that the benefits of screening to a small number of women with ovarian cancer would not outweigh the harms to a larger number of women who would undergo tests and surgeries that would ultimately be found to be unnecessary and could have harmful effects (CHBRP 2004, 2008a). These reports helped legislators understand that screening tests may not always yield net benefits.

In some cases, CHBRP helps legislators understand that some bills may appear to be straightforward but actually are not. For example, legislators sometimes introduce bills requiring that health plans adhere to national guidelines, not realizing that there are often multiple and sometimes inconsistent guidelines for treatment of a disease or condition. (Draft legislation is frequently generated by advocacy groups who may reasonably be expected to focus on their own guidelines.) For example, CHBRP was asked to analyze a bill that would have mandated coverage for screening and diagnostic tests for breast cancer “in accordance with national guidelines.” In its medical effectiveness review, CHBRP identified five national organizations that had issued guidelines for screening asymptomatic women without a history of breast cancer. These guidelines differed markedly with regard to the use of breast magnetic resonance imaging (BMRI). One guideline recommended the use of BMRI, in conjunction with mammography, to screen women who have a genetic mutation associated with an elevated risk of breast cancer or who have a family history of breast cancer. Another guideline recommended that BMRI be used to screen all women with a lifetime risk of breast cancer of 20 percent or greater as determined by a risk assessment instrument. Three guidelines did not address the use of BMRI at all. By summarizing these guidelines, CHBRP alerted legislators to the existence of multiple guidelines and their lack of consistency regarding the use of BMRI. CHBRP also reviewed studies of BMRI and concluded that there was insufficient evidence to determine whether BMRI screening reduced breast cancer morbidity and mortality (CHBRP 2008c).

Lesson 3. Have Realistic Expectations Regarding the Impact of Medical Effectiveness Reviews on Legislative Decisions

In 2004 CHBRP commissioned a faculty member at an academic institution outside California to conduct an independent evaluation of the program. The evaluator interviewed CHBRP's key personnel as well as legislators, legislative staff, executive branch staff, and representatives of other major stakeholders. The findings from these interviews indicated that the primary users of CHBRP reports were legislative committee staff, personal staff of legislators proposing mandate bills, staff of the executive branch agencies regulating health plans and health insurers, the governor's chief staff person on health issues, and lobbyists for health plans, health insurers, professional societies, disease-focused organizations, and consumers. All the interviewees, regardless of their party affiliation or general position on benefit mandates, stated that CHBRP made a positive contribution to state policymaking. The evaluation also found that CHBRP's reports were cited extensively in legislative analyses prepared for committee and floor votes and by proponents and opponents of mandate bills. Legislative staff particularly appreciated the reports as an independent source of critical technical information. Before CHBRP was created, they were dependent on information supplied by proponents and opponents. The medical effectiveness and public health analyses were especially useful to legislative staff because they provided context for the cost estimates on which legislators usually focused (Oliver and Singer 2006).

Every year since the independent evaluation was conducted, CHBRP's analytic staff in the UC Office of the President has conducted interviews with legislative staff, executive branch staff, health plan personnel, and representatives of organizations that have sponsored mandate bills to learn their perspectives on CHBRP's work and obtain their suggestions for improvement. Findings from these interviews have been generally consistent with those of the independent evaluation (CHBRP 2005c; Philip 2009).

CHBRP reviews appear to have also contributed to the passage of some bills for which there is strong evidence of effectiveness. In 2008, for example, legislators approved and the governor signed the bill requiring health plans to cover HIV screening for asymptomatic persons. Bill analyses prepared by the California Assembly and Senate Health Committees cited CHBRP's findings that the tests for HIV are accurate and that the early detection and treatment of HIV improve health outcomes (CHBRP 2008b).

In other cases, legislators have decided not to act on bills for which CHBRP found little evidence of effectiveness. One example is a bill regarding coverage for drugs for Alzheimer's disease. CHBRP's report on the bill, which found that the drugs produced only modest benefits that, furthermore, were not sustained over time, may have contributed to the legislators’ decision to not take action on the bill (CHBRP 2005a). CHBRP's report may also have contributed to the lack of action on the bill regarding screening for gynecological cancers. The report found that cervical cancer is the only gynecological cancer for which there is evidence that screening is effective and that the bill would not lead to substantial improvement in women's health because coverage for cervical cancer screening was already required under existing law (CHBRP 2008a).

CHBRP's reports may also influence legislators to revise legislation. In 2008, CHBRP analyzed a bill that would have mandated coverage for all types of screening and diagnostic tests for breast cancer. CHBRP found that the evidence of effectiveness varied across breast cancer screening tests. There is evidence that using mammography to screen asymptomatic women reduces breast cancer mortality among women aged forty or older. In contrast, there is insufficient evidence to determine whether screening asymptomatic women with BMRI or ultrasound decreases breast cancer mortality or morbidity (CHBRP 2008c). The bill was not approved by the state legislature before the end of the 2007/2008 legislative session, and the bill's author subsequently reintroduced it in 2009. Whereas the initial version of the bill addressed all types of screening and diagnostic tests, the revised bill focused on mammography (CHBRP 2009a). CHBRP's findings may have played a role in the bill author's decision to limit the proposed mandate to the test for which there is evidence of effectiveness.

Not surprisingly, California policymakers’ decisions about benefit mandate bills are not always based on clinical effectiveness, since legislators and the governor often have other priorities when making decisions about these bills. For example, in 2007 the governor and the legislators were concentrating on comprehensive health care reform proposals. Accordingly, the governor vetoed all benefit mandates passed by the state legislature that year because he believed that mandates for coverage for individual services should not be approved until broader health care reform legislation was enacted, and for similar reasons, the state legislature did not act on several additional benefit mandate bills. Interest in the mandates grew, however, in 2008 and 2009, after the governor and legislature failed to reach agreement on health care reform.

Concerns about cost also influence decisions regarding proposed benefit mandates. A bill that would have required coverage for smoking cessation services is a noteworthy example. CHBRP found extensive evidence from meta-analyses and systematic reviews of numerous well-designed studies that persons who receive brief advice, counseling, or pharmacotherapy are more likely to stop smoking than are persons who try to stop without assistance (CHBRP 2005b). In addition, CHBRP estimated that the proposed mandate would have resulted in a short-term increase in health care expenditures. The estimated increase in premiums was high relative to most benefit mandates because in 2005, 14 percent of Californians smoked (UCLA Center for Health Policy Research 2005); many health plans and health insurers do not cover smoking cessation counseling or pharmacotherapy; and providing coverage is likely to increase the use of these services. (CHBRP's estimates of the marginal cost associated with most benefit mandates have been small; in most cases, a mandate would result in an increase in total health insurance expenditures of less than 0.1 percent.) Although CHBRP also estimated savings in expenditures for health insurance premiums owing to reductions in the numbers of heart attacks and low birth weight deliveries, in the short run these benefits did not outweigh the costs associated with the bill.

The governor cited CHBRP's cost estimate as his primary rationale for vetoing this bill (Oliver and Singer 2006). A study conducted by the UC San Francisco Center for Tobacco Control Research and Education found that the bill's advocates also cited CHBRP's cost estimate as one of the main reasons why the bill was not enacted, although opposition to the bill by employers, health plans, and tobacco companies also was responsible (Hong, Barnes, and Glantz 2007). Similarly, the legislative staff interviewed for the independent evaluation of CHBRP reported that CHBRP's cost estimate was one of several factors contributing to the demise of a bill that would have mandated coverage for a wide range of substance use disorders (Oliver and Singer 2006).

Such developments are not surprising. As indicated earlier, CHBRP is required to analyze the effects of bills on health insurance premiums as well as evidence regarding medical effectiveness and consequences for public health. These independent estimates of the effects of proposed mandates on costs as well as benefits are sometimes consistent with health plans’ and health insurers’ conclusions that mandate bills would substantially increase costs. At a time at which health care costs are rising rapidly and economic conditions are poor, such considerations understandably resonate with elected officials in California.

Lesson 4. Understand the Consequences of the Reactive Nature of Mandated Benefit Reviews

Legislators set the agenda for CHBRP and other programs that review legislation. The strength of this approach is that CHBRP produces reports that are directly relevant to the bills under consideration, which enhances the likelihood that legislators, legislative staff, and other stakeholders will use them. But this inherently reactive approach has important limitations. Legislators introduce bills that would mandate coverage for interventions or technologies that are of interest to them and/or their constituents, even though these interventions or technologies may not always represent the most pressing questions regarding medical effectiveness. For example, unlike the Institute of Medicine's Committee on Comparative Effectiveness Research (IOM 2009), CHBRP does not have the authority to look across the spectrum of diseases and interventions to identify those for which there is the greatest need for new studies or syntheses of existing studies. Nor does CHBRP have the discretion to decide to review the literature on new technologies (e.g., CT angiography, virtual colonoscopy) for which legislators have not introduced mandate bills.

Implications for Policymakers

CHBRP's experience suggests that useful reviews of the medical literature can be produced within the tight time constraints of the legislative process, particularly when recent meta-analyses and systematic reviews are available. The program has provided state legislators with evidence regarding the medical effectiveness of the interventions for which coverage would be mandated and impacts of proposed mandates on public health and costs that they can take into account when considering mandate bills. Its approach could easily be adapted by other states or the federal government or for other types of health and social policies.

We believe that CHBRP's design has contributed to its largely positive reception. Earlier studies have found that policymakers value information that is relevant, timely, credible, and accessible (Colby et al. 2008; Jewell and Bero 2008; Oliver and Singer 2006). All the CHBRP reports are relevant to California legislators because all address bills under consideration. The analyses are completed before the committee hearings so that legislators and their staff can review them before debating the proposed mandates. CHBRP's reports are regarded as highly credible because the program uses standardized procedures to conduct its analyses. Finally, CHBRP strives to make its reports easy for policymakers to use. The main findings are summarized in bullet points in the executive summary and in text boxes in the report's main body. Moreover, the authors strive to write concisely and avoid jargon.

Several additional features of CHBRP are worth noting by those considering replication. Legislators and their staff value the full-time staff in the Office of the President who are readily accessible to them and provide continuity in leadership and contact. Each of the campuses leading the analytic effort (Berkeley, Los Angeles, and San Francisco) has full-time analytic staff, part-time leadership from senior faculty, and the ability to bring in on short notice a wide range of content experts as consultants. The sixty-day deadline for completing the reports would be difficult to accommodate in an academic setting without such an infrastructure. Our experience to date is that it is possible to maintain high quality and timely reporting in such an environment if the requests are staggered and advance notice of requests is provided. These aspects of the program are facilitated by the ongoing connections between legislative staff and the full-time staff in the Office the President.

Based on our experience, we believe that CHBRP could be improved in several areas. First, extending the time by which CHBRP is required to complete reports would enable it to obtain feedback on its reports from more experts and to rely less heavily on previous systematic reviews and meta-analyses. As discussed earlier, the research questions addressed by existing systematic reviews and meta-analyses are not always well matched to the foci of mandate bills. In addition, conducting a new review of all individual studies reduces dependence on the judgments of other reviewers. Conversely, longer review times would make the CHBRP reports less timely. Occasionally extending the review time for a bill that would benefit substantially by a more thorough review would be a possible solution. Second, the program could better integrate its review of the medical effectiveness literature with its review of the cost-effectiveness literature pertinent to a bill. These reviews are currently conducted separately by two different teams of faculty and staff. Better coordination of the two teams could improve these aspects of CHBRP's reports. Third, the program would provide additional benefits to California policymakers if it compared the effectiveness of technologies and interventions more broadly. CHBRP has tended to consider comparative effectiveness only when head-to-head comparisons of similar treatments have been published (e.g., comparisons of drugs used to treat Alzheimer's disease) and typically has not compared different types of treatment for the same disease or condition (e.g., surgical versus nonsurgical interventions). Taking a more expansive view of comparative effectiveness could generate additional information that would be useful to policymakers. This suggestion might be addressed by issuing “off-cycle” reports, perhaps in anticipation of a bill's resubmission.

CHBRP's faculty and staff also have learned some important lessons about translating medical literature for policymakers. Some of these lessons reflect the difficulties inherent in the communication between elected officials and academics in any field. Others are unique to CHBRP's task. One of the program's greatest contributions has been to tell elected officials and their staff that there are no definitive answers to many of the medical effectiveness questions that interest them. Despite the growth in studies over the past several decades, evidence is still lacking for many health care services, especially treatments for rare diseases and long-standing treatments for common ones that have never been subjected to rigorous study. When evidence is available, it can be difficult to synthesize because the interventions assessed may vary widely. The studies may not evaluate the outcomes of greatest concern to policymakers. Some critical topics, such as benefit design, go beyond the clinical questions that medical researchers typically address. Our experience underscores the need for researchers to continue conducting and synthesizing research on the effectiveness of health care services and for public and private funders to support these efforts. The $1.1 billion included in the federal economic stimulus bill for comparative effectiveness research is an important source of additional funding for this work.

Finally, we have come to appreciate the implications of estimating the cost of mandate bills as well as their effects on health outcomes. CHBRP's reviews sometimes uncover strong evidence for both greater effectiveness and greater cost. Such “good news/bad news” reviews are obviously viewed differently by opponents and proponents of the bills. With the CHBRP reports in hand, however, the deliberations can move beyond debates between “dueling” experts to discussions of the real policy choices elected officials must make.