SUMMARY

Medical Effectiveness Analysis

Introduction
As discussed in the About CHBRP section of this web site, the California Health Benefits Review Program (CHBRP) was originally established to implement AB 1996 and was reauthorized by SB 1704. These laws amended Section 127660 et seq. of the Health and Safety Code.

This summary describes:

Medical effectiveness provisions of SB 1704
General approaches to medical effectiveness analysis
CHBRP's approach to analyzing medical effectiveness
Differences between CHBRP's medical effectiveness reviews and other medical effectiveness reviews
Content of the medical effectiveness sections of CHBRP reports

For a more detailed description of the procedures CHBRP follows when conducting medical effectiveness reviews, see Medical Effectiveness Analysis Research Approach.

Medical Effectiveness Provisions
The following provisions of SB 1704 describe CHBRP's responsibilities with regard to the preparation of medical effectiveness analyses.

(a)(2)(A) "The extent to which the benefit or service is generally recognized by the medical community as being effective in the screening, diagnosis, or treatment of a condition or disease, as demonstrated by a review of scientific and peer reviewed medical literature."

CHBRP's approach to addressing this provision is discussed later in this summary under the heading CHBRP's Approach to Medical Effectiveness Analysis and in more detail in the Medical Effectiveness Analysis Research Approach document.

(a)(2)(B) "The extent to which the benefit or service is generally available and utilized by treating physicians."

CHBRP addresses this provision in its medical effectiveness analyses by discussing physician practice patterns, standards of care, and technologies approved by the Food and Drug Administration that are pertinent to the screening, diagnostic, or treatment intervention in question.

General Approaches to Medical Effectiveness Analysis
CHBRP's approach to medical effectiveness analysis is grounded in the principles of evidence-based medicine (EBM), which has been defined as "a set of principles and methods intended to ensure that to the greatest extent possible, medical decisions, guidelines, and other types of policies are based on and consistent with good evidence of effectiveness and benefit" (DM Eddy, Health Affairs, 2005;24(1):16). The practice of EBM requires systematic review of the best available evidence from medical research. CHBRP applies the principles of EBM to health insurance mandates by performing systematic reviews to assess the medical effectiveness of proposed mandates.

Reviews of new medical services or procedures initially address issues of efficacy, or how well an intervention works under ideal conditions. EBM studies usually try to go beyond an examination of efficacy in ideal conditions to an examination of effectiveness, or how well an intervention works under usual conditions of clinical practice. Organizations that conduct EBM studies include the US Preventive Services Task Force, the Centers for Medicare & Medicaid Services, and the Cochrane Collaboration, among others.

CHBRP's Approach to Medical Effectiveness Analysis
CHBRP's approach to medical effectiveness analysis is similar to that of other organizations that synthesize medical literature. Once CHBRP receives a request from the State Legislature, the medical effectiveness team defines the parameters for a search of the medical literature in consultation with a medical librarian and an expert on the disease or condition to which the proposed mandate would apply. The parameters for the literature review encompass the entire causal pathway of a potential intervention. For example, the pathway may include administration of a mandated screening test, additional tests that may be ordered as a result of a mandated screening test, treatments that may be provided if tests indicate the presence of a disease or condition, and health outcomes that might result from receipt of the test and treatment.

Once the literature search is completed, the medical effectiveness team selects studies for inclusion in the review based on a hierarchy of evidence that ranks studies by the strength of the evidence they present. For further information about the hierarchy of evidence, see Medical Effectiveness Analysis Research Approach.

Team members systematically evaluate evidence across five domains:

• Research design
• Statistical significance
• Direction of effect
• Size of effect
• Generalizabilty of results

Studies with strong research designs are more likely to yield accurate information about an intervention's effects. Statistical significance indicates whether the association between an intervention and an outcome is stronger than that which might occur by chance. The direction of effect reveals whether the intervention is associated with better or poorer outcomes or has no effect on outcomes. The size of effect suggests whether an intervention's effect is sufficiently large to be clinically meaningful to patients and/or their caregivers. Generalizability concerns the applicability of a study's findings to the population to which a proposed mandate would apply. The degree of generalizability is greatest when the study population is similar to the population covered by a mandate and when the intervention is provided in a community-based setting (as opposed to an academic health center or other tertiary care setting).

Conclusions regarding an intervention's effects on outcomes are based on the strength of the evidence across all five domains. For further information about CHBRP's approach to grading evidence of effectiveness, see Medical Effectiveness Analysis Research Approach.

Medical effectiveness findings may relate to any one of a number of types of outcomes including the following:

• Physiological (e.g., blood pressure)
• Behavioral (e.g., smoking cessation)
• Cognitive (e.g., improved short-term memory)
• Functional status (e.g., activities of daily living)
• Quality of life (e.g., overall sense of well-being)
• Morbidity (e.g., specific complications, progression of disease, restricted activity days)
• Mortality (e.g., years of life lost)
• Health care utilization (e.g., emergency department visits)

Differences between CHBRP Reviews and Other Medical Effectiveness Reviews
If a mandate bill specifies particular outcomes that are expected to result from the proposed mandate, CHBRP must assess the intervention's effect on these outcomes. This requirement distinguishes CHBRP reviews from typical medical effectiveness reviews in which the reviewers select outcomes based on availability of studies and the importance of outcomes to patients' health and well-being.

CHBRP medical effectiveness reviews differ from other medical effectiveness reviews in several other important respects. Most notably, the California Legislature is primarily concerned with determining the effect of an intervention on California's diverse population as part of usual community-based care rather than under ideal circumstances, such as in a randomized controlled trial (RCT) in an academic setting. CHBRP usually attempts to focus attention on those patients for whom the disease or condition in question is a major health problem. Such patients, however, may have other medical conditions (i.e., co-morbidities) or other characteristics that limit their response to an intervention. For example, researchers often do not enroll pregnant women in RCTs of medications due to concern about potential adverse effects on their fetuses. However, if a mandate would apply to pregnant women, CHBRP must consider this population.

In addition, CHBRP's medical effectiveness analyses are usually not as simple as assessing whether drug A is better than drug B or screening test A is better than screening test B. A proposed mandate may include a collection of services, some of which may entail behavioral modification and education programs. In many instances, the medical literature may evaluate a particular device or test, but a mandate may refer to a class of devices, tests, and procedures with varying degrees of effectiveness. The available evidence on a topic rarely assesses all possible combinations of services encompassed by a mandate that proposes coverage for a collection of services, or all items addressed by a mandate for coverage of a class of devices, tests, or procedures.

In some cases, very few studies address the outcomes most pertinent to assessing an intervention's effectiveness. For example, RCTs often focus on intermediate physiological endpoints, such as a cholesterol level or lung function, rather than on disability days or mortality. If no RCTs have been published on an important outcome, CHBRP reviews observational studies (i.e., studies in which subjects are not randomly assigned to intervention and control groups). Although observational studies are less rigorous than are RCTs, CHBRP reviews their findings, if they are the only source of information about important outcomes.

Content of the Medical Effectiveness Sections of CHBRP Reports
Key findings from the review of the medical evidence are presented in the Executive Summary of each CHBRP report. The Executive Summary also includes caveats or limitations to the medical effectiveness analysis. These may include discussions about gaps in information, the methodological quality of studies, and implications of evidence for current practice guidelines.

More detailed findings are presented in the medical effectiveness section of the text of the report. The medical effectiveness section includes information regarding the:

• Services covered under the proposed mandate
• Outcomes of interest
• Methods used to gather evidence
• Evidence for each outcome measure assessed
• The medical effectiveness team's conclusion regarding the effectiveness of the intervention.

All CHBRP reports contain a qualitative synthesis of the medical literature on the outcomes of interest. In some cases, the effectiveness team also produces quantitative estimates of effectiveness for select outcomes. The criteria and guidelines for quantitative estimates are discussed in Medical Effectiveness Analysis Research Approach.

The reports also include a table that summarizes the effectiveness team's findings for each outcome with regard to research design, statistical significance, direction of effect, size of effect, and generalizability, as well as the team's conclusion regarding the intervention's effectiveness.

Further information about the effectiveness analyses is presented in two appendices. The first Appendix describes the methods used to conduct the literature review. The second Appendix consists of a table that lists the studies included in the medical effectiveness analysis and their major characteristics, such as the specific screening test, diagnostic test, or treatment assessed, the research design, the sample size, the population studied, and the location at which the study was conducted.