Improving Quality and Efficiency in Health Care through Comparative Effectiveness Analyses: An International Perspective by Gerry Fairbrother, Ph.D.; Ellen O’Brien, Ph.D.; Rosina Pradhananga, M.P.H.; Kalipso Chalkidou, M.D., Ph.D. November 2014 www.academyhealth.org 1 Table of Contents Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 HTA in Europe, Canada, and Australia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 What is Health Technology Assessment (HTA)? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 HTA and cost effectiveness analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 How are priorities set? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 What is the role of patients and other stakeholders? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 What are the methods for synthesizing evidence? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 How does HTA affect policy and practice? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 CER and PCOR in the United States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 What is Comparative Effectiveness Research (CER)? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Why do we need more CER? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 What is Patient-Centered Outcomes Research (PCOR)? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 How are PCOR priorities set? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 PCORI’s funded projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 What are the methods for generating evidence? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 PCORnet: The promise of “big data” networks for conducting patient-centered research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Evidence synthesis and appraisal: What is PCORI’s role? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 What strategies are needed to promote uptake of research findings? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Discussion and Concluding Thoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2 Executive Summary Figure 1: Comparative Effectiveness Decision Framework: From CER to Coverage and Practice The combination of rising health care costs, efforts to achieve universal or near-universal coverage globally, and growing drive for better outcomes brings an urgent demand to spend health care funds efficiently and in accordance with each country’s priorities. A response to such demand requires, first, Comparative HTA Coverage an understanding of what technologies and Effectiveness Valve Assessment Decisions Medical Practice Research interventions (drugs, devices, procedures, diagnostics, and health care services) increase How well does it work? Is it worth it? Do we pay for it? How is it implemented? the quality and value of health care and, Source: Based on Docteur and Berenson. How Will Comparative Effectiveness Research Affect the Quality of Health Care? 2010.2 second, knowledge of the policy levers that encourage health care systems to adopt appropriate technologies. Comparative CER and HTA activities. CER can provide by private health plans, state Medicaid Effectiveness Research (CER) and Health evidence to inform patients’ medical programs, and occasional national Technology Assessment (HTA) are important decisions and clinicians’ recommendations coverage decisions in the Medicare tools used in different ways by countries to to their patients, but CER also may feed program in a decentralized insurance achieve these goals. into HTA and then into health plan system; each develops its own appraisal of coverage decisions, which shape care a technology’s value and, as a consequence, CER is primary research that compares the delivery and clinical decisions. decisions may vary widely across plans and effectiveness of alternative interventions, payers. with the intent of determining whether In high-income countries other than one technology or service works better the United States, most comparative HTA in Europe, Canada, and than another for a given population or effectiveness analysis activities focus Australia group of patients. In the United States, on HTA, the second step in Figure The topics and scope of HTA activities CER studies have been publicly funded 1 and are conducted by government vary significantly across countries. for a number of years;1 however, public agencies. In essence, HTA programs in For example, some HTA entities focus investment significantly expanded Europe, Canada, and Australia, among exclusively on the relative value of drugs beginning in 2009. In contrast, the publicly others, assess the comparative clinical (Australia), while others concentrate on financed comparative effectiveness entities effectiveness of technologies while asking drugs as well as medical devices and other in most other high income countries are if a particular technology is worth its health care interventions for diagnosis focused on HTA, which typically involves cost to that country’s national health and treatment. Some also develop or coupling the synthesis of existing evidence system. Clinical practice guidelines deal with clinical practice guidelines (the on clinical comparative effectiveness with for incorporating the results of these United Kingdom and Germany). The an assessment of the cost-effectiveness. analyses are often developed concurrently United Kingdom’s HTA body, the National with HTA. Policy decisions about what Institute for Health and Care Excellence This paper provides an overview of HTA technology or service is to be covered by (NICE), also conducts assessments of activities in Europe, Canada, and Australia insurance then follow. public health interventions and programs and examines the new public investments and, more recently, has begun developing in CER in the United States. It also seeks Medical practice in most countries is guidance on social care. In France, the to place the new United States federal affected in a direct way by policy decisions HTA organization has a wide-ranging investments in evidence generation in the on coverage by (often national) insurance remit: from reviewing the value of context of the rather different investments and other payment policies, which flow pharmaceutical products to hospital that are predominantly focused on HTA in from HTA. The result might be expanded accreditation and clinician certification. other industrialized countries. access to new technologies at no charge or restricted access to new technologies Analyses of cost-effectiveness are Overview of CER and HTA Activities either through refusal of coverage or an important feature of HTA. Cost- in the United States and Other placement of limits on use to a subset effectiveness analysis using the Quality Countries of the population.3 In contrast to other Adjusted Life Year (QALY) as the main Figure 1 illustrates a simplified pathway nations examined in this report, payment outcome measure is the norm in Canada, from CER to medical practice to and coverage decisions in the United States Australia, the United Kingdom, and Sweden, demonstrate the relationship between are made by a complex mix of decisions with France recently also considering this i metric. Many countries then set a threshold The new investments are designed to reorient Discussion and Concluding amount that they are willing to pay for a research to make it more relevant, especially Thoughts QALY; the threshold can vary considerably for patients. In fact, the organizing vision for Differences in health care systems in the across countries. For example, in the United PCORI is “research done differently.”10 United States and other countries lead Kingdom the cost-effectiveness threshold to variations in scope, organization, and range is about $34,000 to $51,000 per QALY, The new CER effort is designed to address financing of CER or HTA activities and while in Sweden the range is about $77,000 to patients’ questions, to assess the outcomes of mechanisms for moving research findings $85,000 per QALY (both in 2014).4,5 Thus, in interventions in real patients and real-world into practice and policy. Some of the HTA these countries, if the cost of new technology settings, and to learn about what forms of entities in Europe, Australia, and Canada have is under the threshold, it would be health care are most effective in improving the authority to determine or recommend incorporated in the insurance coverage. Since health. Furthermore, it focuses not only on coverage decisions as well as to develop overall health budgets do not increase with generating new evidence, but also on uptake of clinical guidelines for how the technology is to the adoption of a technology with a cost per that evidence into decision-making. The term be implemented. These direct mechanisms for QALY value above the threshold, inclusion patient-centered outcomes research (PCOR) bringing into practice results of effectiveness of such technology would need to be at the was meant to emphasize that the CER they analyses for technologies are far more complex expense of other health care expenditures. fund focuses on topics and outcomes that are in the United States. The PCORI strategy Although QALY is broadly accepted as a highly relevant to patients, draws on patient encourages adoption through widespread and measure of health benefit, debate continues data, and engages patients in the research strategic dissemination of findings as well as about its exclusive use, particularly because and decision-making processes by answering strategic involvement of the stakeholders who it does not capture the social value of questions such as: would adopt the technologies at every step of interventions nor quality of life.6,7 the research lifecycle.12 • “Given my personal characteristics, conditions, and preferences, what should CER and PCOR in the United States An issue for the United States, going I expect will happen to me?” The expansion of publicly financed HTA forward, is whether the indirect approach activities in most other industrialized • “What are my options and what are the will sufficiently induce broad system- countries stands in contrast to the United potential benefits and harms of those level adoption of more effective drugs, States experience. In Europe, Canada, and options?” devices, and interventions. Given the many Australia, an authoritative body—publicly differences between the countries included financed, transparent, and at arm’s length • “What can I do to improve the outcomes in this review, much remains to be learned from payers—synthesizes and assesses that are most important to me?” of the relative strengths, weaknesses, and existing research, offers an appraisal of • “How can clinicians and the care delivery impact of alternative approaches to CER alternative health care interventions, and systems they work in help me make the and HTA. The enhanced CER and HTA makes recommendations for coverage. In the best decisions about my health and health activities in the United States, European United States, HTA has been fragmented, care?”11 nations, and other countries underscore decentralized, and uneven. Although the need for overall assessments of the many federal health care agencies produce PCORI’s emphasis is on generating new effects of CER and HTA on health care or use CER, there had historically been evidence, which requires funding prospective systems and their cost, quality, and access. no government agency (or other publicly trials or observational studies of patient Countries increasingly want to know if financed entity) to coordinate research. outcomes from data available in insurance CER and HTA efforts are improving clinical claims systems, electronic medical records, or practice, patient care, and health outcomes. However, significant new public investments clinical registries. Methodological rigor is a Do we see appropriate use of health care in CER were made in 2009 through The central part of PCORI’s work. technologies and delivery of evidence- American Recovery and Reinvestment Act based, high-value care? Have these (ARRA) ($1.1 billion)8 and in 2010 through Given how recently PCORI was established comparative effectiveness analyses helped The Affordable Care Act (ACA) with the and PCOR defined, the field is evolving and improve quality and efficiency in health establishment of the Patient-Centered maturing. Examining HTA processes in other care? There have been calls for overall Outcomes Research Institute (PCORI) as an countries reveals several similar dimensions evaluations in Europe. The Government independent nonprofit corporation (with to their processes which can yield some Accountability Office (GAO) is conducting an estimated $3.5 billion available over 10 interesting observations for the United States such an evaluation for PCORI efforts in the years).9 These new federal investments focus as we continue to evolve our approach to United States which will be released in early on improving the quality and relevance of evidence generation and use. Specifically, this 2015. Evaluation of the effects of CER and primary research available mainly to patients paper examines approaches to priority setting, HTA on the health care system is important and their care providers but also to health patient and other stakeholder engagement, and could inform the future direction of plan decision-makers and policy makers. and dissemination of results to users. CER and HTA efforts internationally. ii Introduction Box 1. Definitions In all health care systems, there is increasing Comparative Effectiveness Research (CER) The definition of comparative effectiveness research published by the Institute of Medicine commit- pressure to improve health system tee (IOM) charged (in 2009 federal legislation) with identifying priorities for such research follows: performance—to do more to ensure that • Comparative effectiveness research is the generation and synthesis of evidence that compares health care plans and programs pay for the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical interventions that work—with the goals condition or to improve the delivery of care. The purpose of CER is to assist consumers, pur- chasers, and policy makers to make informed decisions that will improve health care at both the of improving health and moderating individual and population levels.13 cost growth. Budget austerity, aging populations, and increased emphasis on Patient-Centered Outcomes Research (PCOR) In the United States, the Patient-Centered Outcomes Research Institute (PCORI) refers to its value in health care have served to both CER activities as “patient-centered outcomes research” in order to emphasize its strong patient emphasize the importance and increase the focus.14,15 PCOR involves CER focused on topics that are highly relevant to patients or that engage use of comparative effectiveness analyses— patients in various research-related decision-making processes. both evidence generation and evidence Health Technology Assessment (HTA) synthesis—to shed light on what works in The definition used by the European Network for Health Technology Assessment (EUnetHTA), the health care and for whom. umbrella group of HTA entities in Europe, follows: • Health technology is the application of scientific knowledge in health care and prevention. 16 Comparative effectiveness research (CER) • Health technology assessment is a multidisciplinary process that summarizes information about the medical, social, economic, and ethical issues related to the use of a health technology in a is the generation and synthesis of evidence systematic, transparent, unbiased, robust manner. Its aim is to inform the formulation of safe, ef- that compares the benefits and harms of fective, health policies that are patient-focused and seek to achieve best value. Despite its policy goals, HTA must always be rooted in research and the scientific method.16 alternative methods—to prevent, diagnose, treat, and monitor a clinical condition or improve the delivery of care.13 [See Box 1 Overview of CER and HTA Activities The purpose, scope, and function of CER for complete Institute of Medicine’s (IOM) in the United States and Other and HTA activities in the United States definition.] In essence, CER compares Countries and other countries vary widely, with the effectiveness of alternative treatment In recent years, HTA has advanced rapidly the differences rooted in the structure of options, with the intent of determining in other high income countries—in the respective health care systems. However, the whether a new technology, for example, United Kingdom, Germany, Sweden, France, ultimate goal of all of these national HTA works better than the one currently in Canada, and Australia, for example. In the and CER activities is the same: to improve use. In the United States, major public United States, there has been substantial the health of individual patients and of investments are centered on CER. In expansion in public investment in CER with populations and to increase the efficiency contrast, the publicly financed entities in the establishment of the Patient-Centered and effectiveness of health care. most industrialized countries are focused on Outcomes Research Institute (PCORI). health technology assessment (HTA), which The driving forces behind the creation of The difference between the United States and typically involves the synthesis of existing this new institute were: beliefs that clinical other countries in their approach to CER research evidence to arrive at an assessment studies had, in fact, not been sufficiently or HTA activities reflects, at least in part, of clinical comparative effectiveness as well comparative, nor had they mainly evaluated different approaches to health care financing. as an assessment of the cost effectiveness, effectiveness in general populations (as In the United States, health insurance is a or value, of drugs, devices, or other opposed to selected groups) or addressed complex mix of public and private payers, interventions. heterogeneity well. Further, the research with payment and coverage decisions made objectives and approaches themselves by the individual payers—including Medicare, Given the recent expansion of CER had not been selected with consideration Medicaid, as well as private insurance investments in the United States, examining of patient needs and preferences (in the companies. Health insurance plans in both HTA processes in other countries can yield United States or elsewhere), let alone their the public and private sectors conduct HTA, some interesting observations for the United engagement in the research process itself. but there is little transparency in how those States of similar processes as we continue to The overriding rationale for the creation of decisions are made and how evidence is evolve our approach to evidence generation PCORI was that more targeted, comparative, incorporated and judged. There has been less and use. Specifically, this paper examines and patient-centered outcomes research publicly financed HTA in the United States. approaches to priority setting, patient (PCOR) was necessary for better decision- In Europe, Canada, and Australia, on the and other stakeholder engagement, and making at all levels. other hand, broadly speaking, health care is dissemination of results to users. organized and financed centrally, and there is a fixed budget within which allocations 1 must be made. A single body or set of bodies Figure 1: Comparative Effectiveness Decision Framework: From CER to Coverage makes decisions about the adoption of new and Practice drugs, devices, procedures, or interventions. A HTA body—with a federal mandate and federal funding—evaluates evidence and makes decisions (or recommendations) about coverage and payment decisions. Thus, the United States’ approach emphasizes Comparative HTA Coverage improving the evidence base in part to Effectiveness Medical Practice Valve Assessment Decisions inform the HTA processes of a diverse set Research of payers and decision-makers. Indeed, the How well does it work? Is it worth it? Do we pay for it? How is it implemented? need for better evidence for these coverage Source: Based on Docteur and Berenson. How Will Comparative Effectiveness Research Affect the Quality of Health Care? 2010.2 decisions is implicit in the funding strategy for PCORI: the majority of the budget comes from a fee levied on health plans based decisions regarding payment for specified income countries. This paper provides on the number of covered lives. Our peer health outcomes. an overview of HTA activities in Europe, nations with robust HTA capabilities tend to Canada, and Australia and examines the conduct effectiveness analyses from mostly Medical practice in most countries is new public investments in CER in the non-comparative efficacy studies, using affected in a direct way by policy decisions United States, which are now primarily led modeling and other analytic techniques on what is covered by (often national) by PCORI. We conclude with some brief on secondary data to estimate comparative insurance and other payment policies comments about common concerns and effectiveness and value. which flow from HTA. The result might areas for collaboration across countries. be expanded access to new technologies Figure 1 illustrates a simplified pathway at no charge or restricted access to new HTA in Europe, Canada, and from CER to medical practice to illustrate technologies either through refusal of Australia the relationship between CER and HTA coverage or placement of limits on use to Adoption of HTA has advanced steadily activities. CER can directly inform a subset of the population.3 In contrast in the last several decades in Europe, patients’ medical decisions and clinicians’ to other nations examined in this report, Canada, and Australia as countries recommendations to their patients, but payment and coverage decisions in the have established agencies and programs CER also may feed into HTA and then United States are made by a complex dedicated to HTA. The number of HTA into health plan coverage decisions, which mix of decisions by private health plans, bodies and the breadth of their activities shape care delivery and clinical decisions. state Medicaid programs, and occasional have expanded, with one or more entities national coverage decisions in the Medicare assuming a key regulatory or advisory role In high-income countries other than program in a decentralized insurance in policymaking, typically with regard the United States, most comparative system. Some plans have sophisticated HTA to coverage and reimbursement and, effectiveness analysis activities focus on methodologies, while others base coverage sometimes, pricing decisions. There are HTA, the second step in Figure 1 and are decisions on cost alone. In any case, each historical, structural, and cultural factors conducted by government agencies. That develops its own appraisal of a technology’s affecting the way in which HTA bodies is, HTA programs in Europe, Canada, value and, as a consequence, decisions may have evolved and the way and extent to and Australia, among others, assess the vary widely across plans and payers. which their recommendations are linked comparative clinical effectiveness of to decisions about standards, coverage, technologies while asking if a particular Given the level of recent activity both in the and payment within each system. While technology is worth its cost to that United States and abroad, it is reasonable there are many differences in the form and country’s national health system. Clinical to take stock of evolving practice function of the organizations, there is one practice guidelines for incorporating internationally and to examine what can common theme that binds them all: their the results of these analyses are often be learned in a mutually beneficial manner role is to turn evidence into policy with developed concurrently with HTA. Policy from global experiences, especially from the ultimate objective of sustaining their decisions about what technology is to be new directions in CER and HTA activities. countries’ universal access to care. As covered by insurance then follow and is It is useful, as well, to put the new United such, they all consider, in different ways, sometimes carried out by the same body States federal investments in evidence economics—value for money is central that conducts HTA and sometimes by generation in the context of the rather when it comes to defending the principle of another council or body. Payment decisions different investments in HTA in other high universal access. may also take place at this stage, such as 2 Box 2. HTA Entities in Europe, Canada, and Australia Country HTA Entity Scope of HTA United Kingdom National Institute for Health and Care Excel- Drugs, devices, diagnostics, interventional procedures, clinical and public health lence (NICE) interventions, service delivery, social care, pay-for-performance schemes, human resources and staffing norms Australia Pharmaceutical Benefits Advisory Commis- Drugs and vaccines sion (PBAC) Germany German Institute for Quality and Efficiency in Drugs, devices, surgical procedures, quality control interventions, diagnostic tests, Healthcare (IQWiG) clinical practice guidelines of disease management programs, lay information for patients Canada Canadian Agency for Drugs and Technolo- Health care technologies gies in Health (CADTH) France French High Health Drugs, devices, surgical procedures, and diagnostic tests; clinical guidelines for Authority (HAS) disease management; public health guidance on disease prevention and health care system organization, accreditation, and QA of providers Sweden Swedish Council on Technology Assessment Health care technologies in Health Care (SBU) Sources: Chalkidou and Anderson. Comparative Effectiveness Research: International Experiences and Implications for the U.S., 2009; 23 Sorenson and Chalkidou. Reflections on the Evolution of Health Tech- nology Assessment in Europe, 2012; 7 What is Health Technology The topics and scope of HTA activities Service (NHS) executive for health Assessment (HTA)? vary across countries. [Box 2] For example, services guidance and has recently Health technology assessment (HTA) is a some HTA entities focus exclusively on the developed guidance on safe staffing method of evidence synthesis that considers relative value of drugs (Australia), while levels for secondary and tertiary care evidence regarding clinical effectiveness, others focus on drugs as well as medical providers. This difference may be due to safety, cost-effectiveness, and, when broadly devices and other health care interventions the history of the different entities. From applied, includes social, ethical, and legal that focus on areas such as diagnosis the beginning, NICE brought together aspects of the use of health technologies. and treatment. Some also develop the conventional HTA functions on drugs The precise balance of these inputs depends guidelines (Germany and the United with the clinician-led guidelines program. on the purpose of each HTA. A major use Kingdom). The French HAS has a wide Over the years, the two converged of HTA is in informing reimbursement ranging remit, from reviewing the value methodologically, with economics now a and coverage decisions, in which case HTA of pharmaceutical products to hospital core component of clinical guidelines and should include benefit-harm assessment accreditation and clinician certification. HTA applied from whole pathways of care and economic evaluation.17 Set up in 1987, The United Kingdom’s NICE also conducts to deriving quality standards and audit Sweden’s Council on Technology Assessment assessments of public health interventions measures.20 Conversely, in Australia, the in Health Care (SBU) is one of the oldest and programs and, more recently, has PBAC has a specific drug focus with other HTA agencies. Australia’s Pharmaceutical developed guidance on social care. Since committees concentrating on devices and Benefits Advisory Commission (PBAC) NICE is well-established, and is one of the guidelines. is another pioneer in applying HTA to HTA entities better-known globally, and pharmaceutical policy, with cost-effectiveness because one of the authors of this paper is HTA and cost effectiveness analysis analysis becoming a mandatory requirement with NICE, we draw many of our examples Most countries undertaking HTA for listing of pharmaceutical products in from NICE and use its experiences to in a public-sector agency include an 1993.18,19 Canada’s Agency for Drugs and make overall points. assessment of cost-effectiveness when Technologies in Health (CADTH) was set up comparing alternative treatment in 1990 and the United Kingdom’s National While some have proposed that HTA approaches. Methods and approaches Institute for Health and Care Excellence entities should also assess alternative to evaluating cost-effectiveness vary. (NICE) followed in 1999.20 The French service delivery models, most countries’ Cost-effectiveness analysis using the High Health Authority (HAS) and German HTA programs have not expanded their Quality Adjusted Life Year (QALY) as the Institute for Quality and Efficiency in activities to service delivery or quality main outcome measure is the norm in Healthcare (IQWiG) were established shortly and safety, at least not yet.23 In the United Canada, Australia, the United Kingdom, thereafter.21,22 Kingdom, however, NICE is increasingly and Sweden, with France recently also receiving requests by the National Health considering this metric. Germany, on 3 other form of care. All of these issues reflect Box 3. What is a QALY? Measuring the Value of Interventions societies’ values and need to be balanced in HTA answers the question: does treatment result in improved health outcomes (outcomes that matter for patients) by extending life, reducing disability, increasing function, or reducing pain? the framework of the HTA.28 To determine the value of a wide range of treatments, standard measures of value are essential for assessing and comparing various treatments across various diseases. A convenient metric How are priorities set? is the Quality Adjusted Life Year (QALY). HTA is driven by the needs of decision- QALY represents a person’s health over time as a series of “preference-weighted” health states, makers in national health programs. where the quality weights reflect the desirability of living in a given health state—typically from perfect health (weighted 1.0) to death (weighted 0.0). Once the weights are obtained for each Different agencies have established different state, they are multiplied by the time spent in each state; the products are summed to obtain models for topic selection. They range from the QALY. A QALY resulting from an intervention may be derived from an improvement in a considering every new drug or indication, patient’s survival, an improvement in quality, or both. as in Australia’s case, where listing by PBAC QALYs may be used in cost-effectiveness analysis to guide decisions about resource allocation. is a prerequisite for use in the country’s The cost-to-QALY ratio may be compared across interventions to determine the most cost- effective way to deliver health benefits.27 public system, to a more selective approach where topic referrals are triggered by payers, as in Germany’s case. IQWIG the other hand, has developed a different and, as such, it carries what is known as an receives topics from G-BA, the Federal Joint system for assessing the cost/benefit opportunity cost. Committee that comprises representatives equation of new pharmaceutical products, of physicians, dentists, hospitals, and the efficiency frontier. Although QALY is broadly accepted health insurance funds and is responsible as a measure of health benefit, debate for the benefit catalogue of the statutory Broadly speaking, a QALY—the metric continues about its exclusive use, health insurance funds across Germany. In most commonly used—is a year in full particularly because it does not capture the between the two models, there are selective health.7 [Box 3] Many, but not all (e.g. social value of interventions nor quality processes such as NICE’s, which considers France and Germany), countries then set of life.6,7 Social value considerations are all cancer drugs and effectively every new a threshold amount that they are willing especially relevant for decisions about drug or technology that is likely to lead to to pay for a QALY [Box 4] which serves as the cost and value of technology for very a significant investment by the NHS. The a decision rule to guide judgments about expensive treatments, such as those at end- health service through its executive, NHS cost-effectiveness.4 The threshold could be of-life care or for seriously ill individuals. England, and the British government refer set based on political, ethical, and societal non-technology specific topics to NICE, considerations though it reflects actual HTA agencies are working on how to such as the development of cost-effective spending levels and the productivity of include value judgments in the decision- metrics to populate the NHS’s pay-for- the health care system and can be derived making process alongside economic performance schemes for primary care using empirical evidence as has recently evaluations. France, for example, is providers or the newly launched financial been done in the United Kingdom. The developing a social benefit measure incentives scheme for regional payers, the level at which the threshold is set can vary that not only evaluates the therapeutic Clinical Commissioning Groups. Some of considerably. For example, in the United benefits of interventions but also accounts NICE’s programs, targeting potentially cost Kingdom the cost-effectiveness threshold for economic endpoints and important saving innovations, rely on topic referrals range is £20,000 to £30,000 per QALY (or ethical, social, and legal considerations.7 by manufacturers while others, focusing about $34,000 to $51,000 in 2014), while Similarly, “Value Based Assessments” aim on safety of new surgical and diagnostic in Sweden the range is £45,000 to £50,000 to incorporate societies’ values on end-of- procedures, run on referrals from the per QALY (or about $77,000 to $85,000 in life care and other related topics, bringing country’s surgical or clinical community. 2014).4,25,26 together two characteristics not necessarily captured by the QALY: disease severity Topic selection mechanisms vary across In theory, adoption of technologies (Burden of Illness—proportional QALY countries and reflect the structure and role with incremental cost-effectiveness shortfall) and the impact of a disease on of the HTA agency in the broader health ratios (ICERs) in these ranges would productivity (Wider Societal Benefits— care landscape. For example, the United not displace other more cost-effective absolute QALY shortfall). Usually, such Kingdom’s NICE has more independence health care services. Since overall health additional weights would only be used if as it issues its recommendations directly to budgets do not increase with the adoption the group that would receive the technology the NHS, including payers and providers, of a technology with a cost per QALY was small. The application of such weights whereas the French HAS needs to work value above the threshold, inclusion of within an existing national health care through the insurers of the country’s such technology would need to be at the budget would mean that the new more social security system as does the German expense of other health care expenditures expensive technology is displacing some IQWIG. Overall, however, there are core 4 insurers, and patients) to ensure that the Box 4. Thresholds research question is the right one and data The threshold is a decision rule which, combined with an incremental cost-effectiveness ratio (ICER) coming out of a cost-effectiveness evaluation, will guide the decision as to whether a sources other than conventional ones are technology is good value for money and ought to be paid for. Thresholds are therefore linked to consulted, including expert opinion and budgets and to the demand on those budgets by the various technologies (new and existing) grey literature. Further, in order to be and the health care needs of the population. While there have been several studies on willing- ness to pay thresholds, in real life, and unless population wishes can be translated directly into relevant to the real-world, the synthesis health care budgets, thresholds reflect the productivity of the health care system at the margin, needs to incorporate both observational given the budget, and, therefore, can be empirically derived if adequate data on inputs and and experimental data, including patient outputs are available.29,30 testimonies, unit cost, patient preference data, and, increasingly, individual patient- level data as seen by the regulators. similarities across systems. They include house, outsourced to universities and a focus on reviewing technologies and consultancies, or submitted by the services where there currently is, or there manufacturers. Australia’s PBAC relies fully HTA entities across most countries is potential for, significant variation, where on industry submissions critiqued by the stress methodological rigor and the need there is evidence of effectiveness, and academic groups it subcontracts with, while to adhere to high standards. However, where there is potential for significant Sweden’s SBU does not call for industry across countries, there are differences in health gains and/or budgetary impact in submissions. interpretation of methodological rigor case of country-wide adoption. and in how much weight to place on evidence from randomized trials—the Patients and the public have been the “gold standard” for evidence. The United What is the role of patients and least involved in the HTA processes, but a Kingdom’s NICE, which has long been other stakeholders? number of countries have sought to provide “recognized for its methodological rigor, Across entities, opportunities for opportunities for more substantive patient and in particular, its assessment guidelines, stakeholder involvement vary, but most engagement in recent years. The United which set standards for consistency and have focused on the need to engage Kingdom’s NICE established a Citizens’ compatibility of the studies submitted,”7 stakeholders—industry representatives, Council in 2002 to gather information has increasingly been divesting itself health professionals, and patients and on social and ethical issues, including the of evidence hierarchies. Its German patient groups. Stakeholders generally application of QALY thresholds. The French counterpart tends to follow a more have opportunities to submit topics for HAS and German IQWiG have established traditional evidence-based medicine consideration and manufacturers have similar opportunities.7 NICE is also rather or Cochrane model with randomized opportunities to submit data and analysis unique in having a dedicated team of experts controlled trials at the top of their evidence for consideration. to help identify, train, and support patients rankings—though this more traditional in articulating their views as well as offering approach of the latter is also gradually For example, the United Kingdom’s NICE training to senior clinicians who head up being relaxed.33 has encouraged extensive stakeholder many of its committees, to support them involvement in all areas of its work. In the in empowering patients in the context of committee meetings. NICE also has lay The role in setting standards has recently case of technology appraisals, stakeholders people (patients and care-givers) chair its been enhanced for NICE. At the request of are involved in defining the key questions specialist committees. the Gates Foundation, NICE has recently and the clinical alternatives to be exam- embarked on an effort to set standards for ined. Manufacturers have the opportunity carrying out cost-effectiveness analysis to to submit data and analyses for the assess- What are the methods for inform investments in global health. This ment group to consider. All stakeholders, synthesizing evidence? is a unique opportunity to standardize the including manufacturers, professional Study designs and methodological methods of economic evaluations, which in groups, and patient organizations, and the techniques flow from the overarching turn drive billions of dollars of spending in NHS have the opportunity to comment on purposes of the HTA entities. Information aid globally.34 the assessment and appraisal and the final on clinical effectiveness and cost are the appraisal determination. Stakeholders can building blocks of HTA, which comes also appeal the final appraisal.31 from a synthesis of available evidence from How does HTA affect policy and CER studies. The United Kingdom’s NICE, practice? for example, commissions joint teams Ultimately, the goal of HTA is to affect Stakeholders have different roles in the of systematic reviewers and modelers or medical practice and to provide high value analysis stage. For example, in the case of health economists to carry out its HTA.32 for the health care system. This happens the United Kingdom’s NICE, depending The systematic reviewers need to work through linking the results of HTA to on the program, evidence syntheses and closely with the modelers or economists decisions on coverage, reimbursement, economic models can be produced in and with the decision-makers (clinicians, and sometimes pricing. There are some 5 important differences in the role these real-world use of a technology can address case studies of best practice in the organizations play, in particular, their roles these limitations and can be linked to implementation of NICE guidance.23 in decision-making related to coverage. payment subject to data collection. Increasingly, provider payment (pay-for- In the United Kingdom, the HTA entity performance in primary care; Healthcare is akin to a regulatory body vested with HTA entities in the United Kingdom, Resource Groups in secondary care), decision-making authority and is involved Sweden, Germany, and France are contractual arrangements, and quality in both assessment and appraisal processes, experimenting with new approaches to regulation are linked to compliance with especially on pharmaceutical products generate better pre- and post-marketing NICE guidance.38,39 A common theme going through the technology appraisal data. For example, many of these countries across countries is an increased emphasis process. As a result, the United Kingdom’s have developed some form of coverage on measuring compliance and then trying NICE is closely linked to and therefore that involves evidence development, risk- to hold payers and providers accountable influential in the policy process. For sharing arrangements, or patient access through payment schemes, contractual example, NICE’s positive recommendations schemes, whereby coverage of a technology arrangements, peer pressure, media and for pharmaceutical products are mandatory is conditional upon arrangements for patient information campaigns, and other and are issued directly to budget holders additional post-marketing evidence relevant levers. and providers across the NHS. NICE’s collection or the achievement of certain health services delivery and clinical practice recommendations are advisory health or financial targets.7 However, perhaps because of the burden of evidence CER and PCOR in the but are increasingly included in payment collection and analysis, the emphasis—at United States systems. In the NHS, insurance programs least in the United Kingdom—increasingly The expansion of publicly financed HTA have to cover drugs approved by NICE. addresses price deals in the beginning as activities in most other countries stands in Similarly, in Australia approval by PBAC opposed to coverage conditional on HTA contrast to the United States experience. is required (though it is not enough as the arrangements post-approval. Such trends In Europe, Canada, and Australia, an Minister can still say “no”) to list a given in Europe may be driven to some extent by authoritative body—publicly financed, drug on the national formulary. the role of the European Medicines Agency transparent, and at arm’s length from (EMA), the European regulator, which payers—synthesizes and assesses existing In contrast, the German and French is increasingly getting into effectiveness research, offers an appraisal of alternative HTA bodies assume advisory roles that evidence and is keen to help accelerate health care interventions, and makes align with the structure of their social market access post approval. European recommendations for coverage. In the insurance-based health care systems. In Commission initiatives such as the United States, HTA has been decentralized these countries, HTA agencies conduct the European network for Health Technology leading to substantial variation in the assessments and make recommendations Assessment (EUnetHTA) has also played degree to which evidence is used to inform on coverage, which also inform a positive role in encouraging some assessments and coverage decisions. In reimbursement. In Germany, the HTA standardization across HTA agencies and particular, the assessment and appraisal body transmits its recommendations to a more experimentation with risk-sharing, of new health care interventions are often group of insurers and providers to make again working closely with the EMA in the made without adequate evidence about coverage decisions (G-BA). In France, the context of joint early EMA-HTA scientific comparative effectiveness and sometimes HTA entity also advises other individual advice to product manufacturers.37 even without adequate information about insurance plans about the appropriateness safety and efficacy. Although many federal of coverage of health services. Coverage decisions may not always have health care agencies produce or use CER,1 the intended effect of changing clinical there has been no single government Recognizing that data may not be sufficient practice. In the United Kingdom, for agency (or other publicly financed entity) at the point that the HTA is conducted, example, even though positive NICE to coordinate research on comparative most countries provide an option for guidance on technologies is mandatory, effectiveness. However, significant new coverage subject to a requirement for continuing evidence of variation in public investments in CER were made additional data acquisition (or “coverage medical practice has drawn attention in 2009 and 2010. These new federal with evidence development”35,36). A HTA’s to the need to both monitor changes in investments focus on improving the quality or CER’s initial time frame may not have clinical practice and develop additional and relevance of the primary research allowed for the manifestation of all benefits tools that bring practice into line with available mainly to patients and their care and harms or the statistical power may not evidence-based guidelines. In addition, providers, but also to health plan decision- have been sufficient to detect rare adverse NICE’s “implementation directorate” is makers and policy makers. effects. Post-marketing studies that follow responsible for collecting and sharing 6 The implementation of these public invest- Why do we need more CER? outcomes. As such, it is being implemented ments is new and evolving as processes are In the United States, there historically to be “research done differently.”10 implemented, evaluated, and improved. Here, has been relatively little CER. Part of the we highlight key features of this new CER reason for the lack of true comparative The new CER effort focuses both on program, which is led by PCORI. This sec- effectiveness studies has to do with generation of new evidence and uptake tion describes the goals and expectations for methodological and data related challenges of that evidence into practical decision- CER and, more specifically, patient-centered and the cost of conducting large clinical making. Stakeholder engagement, which outcomes research (PCOR), a brief summary trials, which have been the primary brings stakeholders—patients, as well as of PCORI’s funding announcements to date, research method for CER studies. However, others—into the research process from and the principles and strategies that PCORI lack of public funding has been the more the beginning, is also important for has developed or augmented to guide the significant challenge. Publicly financed maximizing the likelihood that findings effort. We also summarize those processes efforts designed to evaluate the relative from the research will be incorporated which we described in other countries’ HTA value of health care technologies have met into practice. There have been CER studies efforts, namely priority setting, engagement of resistance from some sectors and ultimately in the past that have not made a major patients and other stakeholders, methodologi- have been de-funded or eliminated.40,41 impact on clinical and policy decisions for cal approaches, and dissemination. a number of reasons. It is difficult to change The significant new federal investment in clinical practice, the findings do not always What is Comparative Effectiveness CER is designed to fill this gap and avoid help patients and clinicians make decisions, Research (CER)? some of the challenges encountered in past and often the findings are not disseminated Clinical research is often focused on efforts. The new investments are being effectively or there is no infrastructure for determining the efficacy and safety of medical implemented by PCORI to reorient the building them into clinical practice and the care treatments. Randomized, placebo- research to what patients care about and policy decision-making process. Involving controlled trials are considered the gold to make it more relevant to patients. It is stakeholders in the research process is standard for regulators and the basis for designed to address patients’ questions, part of PCORI’s strategy to promote determining which treatments are efficacious to assess the outcomes of interventions in implementation. and in which patients. In contrast, CER real patients and real-world settings, and to asks about the relative risks and benefits of learn about what forms of health care are Building a sustained CER effort is a very alternative approaches to diagnosis, treatment, most effective in order to improve health. large undertaking. PCORI is establishing and prevention. CER generally requires “The goal is to learn what forms of health research priorities and at the same time the comparison of two or more treatments. care work best, so that we can abandon building a research program that is wide- CER focuses on questions such as “does the those that are ineffective and adopt those ranging, ongoing, and responsive to treatment work better than alternatives?” diagnostic tests, treatments, and approaches changes in health care delivery. The goal and “does it have some additional benefit in to prevention that do the most to improve is to move the United States’ health care all patients or in a subset of patients?” CER health.”42 Hence, patient engagement is system toward a “learning system” that addresses these questions directly, through viewed as critical to the overall success of has an infrastructure for learning about primary research, and CER also includes the CER investments, in helping to shape what works and mechanisms for applying evidence synthesis. the research questions and define the that knowledge effectively in health care decision-making. Box 5. Recent Federal Investments in CER in the United States The American Recovery and Reinvestment Act (ARRA) of 2009 invested $1.1 billion for compara- What is Patient-Centered Outcomes tive effectiveness research (CER), with funding for the National Institutes of Health (NIH), the Office of the Secretary in the Department of Health and Human Services (HHS), and the Agency for Research (PCOR)? Healthcare Research and Quality (AHRQ).8 A Federal Coordinating Committee (FCC) convened by The ACA calls for PCORI to “assist patients, HHS and the Institute of Medicine (IOM) both identified priorities for CER in statutorily mandated clinicians, purchasers and policymakers reports to the Congress that were issued on June 30, 2009. The IOM developed a working defini- tion for CER, created a list of 100 priority topics for research to address, and provided 10 recom- in making informed health decisions by mendations to implement the infrastructure for a sustained CER effort.13 advancing the quality and relevance of In 2010, a substantially larger federal investment in CER was made, with a 10-year funding stream evidence concerning the manner in which and authorization for a new publicly financed entity to coordinate and manage these investments. diseases, disorders and other health conditions The Affordable Care Act (ACA) established the Patient-Centered Outcomes Research Institute can effectively and appropriately be prevented, (PCORI) as an independent nonprofit corporation and makes an estimated $3.5 billion available (over 10 years) for CER through the Patient-Centered Outcomes Research Trust Fund (PCORTF). diagnosed, treated, monitored, and managed PCORI is governed by a board of directors representing diverse stakeholder groups—patients, through research and evidence synthesis that payers, providers, and drug and device manufacturers—as well as the heads of the NIH and considers variations in patient subpopulations, AHRQ. This new program seeks to reshape the way health care research is done to meet the practical needs of patients and their health care providers. and the dissemination of research findings with respect to the relative health outcomes, 7 clinical effectiveness, and appropriateness of most from alternative interventions.”42 It Box 6. Statutory Criteria for medical treatments, items and services….”43 must be tailored, much more than it has PCORI-Funded Research The term “research” is defined as research been in the past, to the clinical situation and 1.Effect on individuals and populations evaluating and comparing health outcomes to patient priorities. and the clinical effectiveness, risks and benefits 2.Probability of improvability through research for two or more medical treatments, services, How are PCOR priorities set? 3.Inclusiveness of different populations and items.43 There have been several efforts to identify 4.Current gaps in knowledge/variation in care research priorities in the United States— 5.Effect on health care system performance PCORI further defined PCOR through a topics that will address the most pressing process led by their (statutorily required) concerns of patients and clinicians. The IOM, 6.Potential to influence decision making Methodology Committee and informed by with diverse stakeholder input, narrowed a 7.Patient-centeredness diverse stakeholder input. The definition list of suggested topics from more than 2,000 8.Rigorous research methods begins with the ultimate purpose of to just 100 initial priorities for research.13 9.Efficient use of research resources the research: “PCOR helps people and These topics include a wide range of their caregivers communicate and make interventions, populations, and approaches informed health care decisions, allowing and include research on the dissemination of research priority areas to capture its goals their voices to be heard in assessing the CER and on CER methods. for changing the way research is done and value of health care options.” It then goes disseminated. The five priority areas are: on to offer examples of the types of patient Rather than identify priorities or require questions answered by this type of research: PCORI to develop a research agenda based 1.Assessment of Prevention, Diagnosis, on the IOM priorities, the ACA identified and Treatment Options • “Given my personal characteristics, criteria to help define PCOR. [Box 6] conditions, and preferences, what should 2.Improving Healthcare Systems The ACA further called upon PCORI to I expect will happen to me?” prioritize studies based on their likely 3.Addressing Disparities • “What are my options and what are the effects on health care decision-making potential benefits and harms of those and health, and to select studies with an 4.Communication and Dissemination options?” eye toward their ability to improve health Research system performance, their likelihood of 5.Accelerating PCOR and Methodological • “What can I do to improve the outcomes influencing decision-making, and their that are most important to me?” Research 47 ability to be patient-centered.45 • “How can clinicians and the care delivery The scope of PCORI’s research agenda is systems they work in help me make the These criteria guide PCORI’s assessment thus quite broad. PCOR includes traditional best decisions about my health and health of research proposals and their funding CER topics (the assessment of prevention, care?”44 decisions. PCORI focused on identifying diagnosis, and treatment options), but PCOR research questions with the greatest prob- also comprises studies of health care system The emphasis is on research that answers interventions and research designed to assess ability of changing clinical decisions. The real-world questions for patients and interventions to reduce health disparities. questions include: (1) What is the potential clinicians and acknowledges the need to for new information to improve care and modify existing approaches to health care PCORI’s funded projects patient-centered outcomes? (2) What are research. To provide this information, Four years into implementation, PCORI the facilitators and barriers that would affect CER must assess a comprehensive array of has made more than 350 awards, investing the implementation of new findings in prac- health-related outcomes for diverse patient $519.2 million in CER, data infrastructure tice? (3) How likely is it that new CER on populations. Interventions compared may and methods, and dissemination and the topic would provide better information include medications, procedures, medical implementation research [Table 1]. To to guide clinical decision-making? (4) How and assistive devices and technologies, date, 65% of PCORI’s research funding has likely is it that the results of new research behavioral change strategies, and delivery focused on CER, 26% has been focused on would be implemented “right away?” 46 system interventions.44 The research data infrastructure building and methods must consider “variations in patient research, and 9% on communication and Rather than developing a list of research subpopulations,” which means it needs to dissemination research. questions or adopting the IOM priority uncover clinical differences among patients list, PCORI developed five overarching that help predict which patients will benefit 8 Table 1. PCORI Research Award Funding, by Priority Area Priority Area Funding Percent of Total Funding CER $ 338,087,798 65% Assessment of Prevention, Diagnosis, and Treatment Options $129,716,941 25% Improving Healthcare Systems $120,748,218 23% Addressing Disparities $87,622,639 17% Communication and Dissemination Research $48,761,292 9% Accelerating PCOR and Methodological Research $132,395,902 26% Total Awards (to date) $519,244,992 100% Source: Authors’ calculation based on PCORI, “Complete List of Funding Awards,” pfaawards.pcori.org, downloaded on September 12, 2014. Awards for Pilot Projects and Engagement Awards are not included. PCORI has funded research that reflects which standards have been or are being The Methodology Committee also set a wide range of priority areas, diseases, developed are: 1) prioritizing research forth standards for using large databases treatments, delivery approaches, and questions; 2) using appropriate study in observational studies. There is a need populations. The funded projects include designs and analyses; 3) incorporating for large-scale observational databases studies likely to produce results quickly, as well patient perspectives throughout the from claims, electronic health records, as longer-term initiatives. PCORI solicited research continuum; and 4) fostering or registries. These are needed to provide proposals through relatively open funding efficient dissemination and implementation a sufficiently large population to detect announcements and selected for funding of results.49 A Congressionally-mandated rare events and to provide a resource for proposals by whether they answered high Methods Report describes the standards.50 rigorous observational studies that yield priority questions, met PCOR criteria, and had It is important to note that the methods results more rapidly than randomized trials. appropriate research designs. These projects developed in the context of PCOR are However, while existing claims and clinical are investigator-initiated, but more recently important and pertinent broadly and not databases and registries offer a convenient PCORI has released more targeted funding just in the PCORI context. resource, observational studies often do not announcements for specific high-priority control for important differences in groups, questions—related to asthma treatment in The work of the Methodology Committee such as individuals who adhere to a drug African American and Hispanic populations was intended to overcome some of the regimen compared to those individuals who and approaches to preventing falls in the weaknesses of methods frequently used and do not. elderly, for example. [Box 7] to ensure that CER produces information that is meaningful, unbiased, timely, and Some of the most important methodological PCORI has funded a number of studies actionable.49 With respect to randomized advances set forth by the Methodological designed to develop and evaluate tools, controlled trials (RCTs), serious problems Committee that address these issues include structures, and incentives that can help make concern the time and cost it takes to pragmatic trials, adaptive designs, and results available for “real-time” decision- complete them. Because participants in assessment of causal inference. Table 2 making. For example, this implementation- a clinical trial are enrolled prospectively, describes the advantages and limitations of the focused research is designed to assess the RCTs can require years to enroll patients research methods used for primary CER.50-52 potential role of communication tools and and to observe the outcomes of alternative shared decision-making models in motivating treatments. A second problem is that, in PCORnet: The promise of “big data” patients and providers to make evidence- an attempt to make comparisons as “clean” networks for conducting patient- based choices when considering alternative as possible, studies often impose exclusion centered research approaches to diagnosis, treatment, and care criteria that result in a homogeneous study PCORI has also funded projects designed management.48 population that is not representative of the to improve the data and methods for “real world.” For example, for any number primary CER studies. Projects are focused What are the methods for generating of reasons, a study population might on the measurement of patient-centered evidence? exclude individuals with comorbidities or outcomes, on methods for pragmatic trials PCORI’s emphasis is on generating new exclude the elderly. Further, studies may not and observational research, and on building evidence. Its Methodology Committee is take place in settings where care is routinely a data infrastructure and research network charged with developing methodological delivered. for PCOR. standards for PCOR. The four general areas identified by the committee in 9 The National Patient-Centered Clinical Box 7. Examples of PCORI Funding Awards, by Priority Area Research Network (PCORnet) is one of Assessment of prevention, diagnosis and treatment options the major and signature programs in the •Comparative Effectiveness of Broad vs. Narrow Spectrum Antibiotics for Acute Respiratory Tract PCORI portfolio. It aims to build a national Infections in Children research network, linked by a common data •Benchmarking the Comparative Effectiveness of Diabetes Treatments Using Patient-Reported platform and embedded in clinical care Outcomes and Socio-Demographic Factors delivery systems. This network will enable •Comparing Patient-Centered Outcomes after Treatment for Uterine Fibroids studies, and in particular randomized trials, •Generating Critical Patient-Centered Information for Decision-Making in Localized Prostate that have been impractical to conduct Cancer to date. PCORnet currently includes 11 •Shared Decision-Making in the Emergency Department: The Chest Pain Choice Trial Clinical Data Research Networks (CDRNs), •Comparative Effectiveness of Rehabilitation Services for Survivors of an Acute Ischemic Stroke 18 Patient-Powered Research Networks (PPRNs), and a coordinating center. The 11 •Comparative Effectiveness of Adolescent Lipid Screening and Treatment Strategies CDRNs are large data networks from health •Promoting Informed Decisions about Lung Cancer Screening plans, academic medical centers, outpatient •Smoking Cessation Versus Long-Term Nicotine Replacement among High-Risk Smokers and inpatient hospitals, and others, such as •Physical Therapy vs. Internet-Based Exercise Training for Patients with Knee Osteoarthritis Kaiser Permanente, Oregon Community •Comparative Effectiveness of Behavioral Interventions to Prevent or Delay Dementia Health Information Network, and The Children’s Hospital of Philadelphia.53 Each CDRN is committed to building a Improving healthcare systems large patient cohort with comprehensive, •Randomized Trial of a Multifactorial Fall Injury Prevention Strategy: A joint initiative of PCORI and longitudinal electronic clinical data the National Institute on Aging of the National Institutes of Health* and building the capacity to participate •An Integrative Multilevel Study for Improving Patient-Centered Care Delivery among Patients with Chronic Obstructive Pulmonary Disease successfully in multi-network randomized trials and observational studies.53 •Changing the Healthcare Delivery Model: A Community Health Worker/Mobile Chronic Care Team Strategy •Increasing Healthcare Choices and Improving Health Outcomes Among Persons with Serious The PPRNs consist of 18 disease-specific Mental Illness data repositories, with 9 representing •Improving the Quality of Care for Pain and Depression in Persons with Multiple Sclerosis common conditions and 9 representing rare diseases.54 Each PPRN consists of patients, • A Comparative Effectiveness Trial of Optimal Patient-Centered Care for US Trauma Care Systems caregivers, or families who are linked •Redesigning Ambulatory Care Delivery to Enhance Asthma Control in Children by the experience of a shared condition •Improving Healthcare Systems for Access to Care and Efficiency by Underserved Patients (e.g., inflammatory bowel disease, major •Advance Planning for Home Services for Seniors depressive disorder/bipolar disorders, sleep apnea, primary immunodeficiencies).53 Addressing disparities Each CDRN and PPRN will be self- •Asthma Treatment Options for African Americans and Hispanics/Latinos (7 projects) governed, and each network will securely •A Helping Hand to Activate Patient-Centered Depression Care among Low-Income Patients maintain its own data. The hope is that (AHH) these PPRNs will help empower patients •Impact of Patient Navigators on Health Education and Quality of Life in Formerly Incarcerated and their families and caregivers to Patients generate large amounts of data about their •Peer Health Navigation: Reducing Disparities in Health Outcomes for the Seriously Mentally Ill conditions. Further, by shifting research • Improving Health Outcomes among Native Americans with Diabetes and Cardiovascular Disease control from traditional researchers and funders to patients who own their data and •Integrative Medicine Group Visits: A Patient-Centered Approach to Reducing Chronic Pain and Depression in a Disparate Urban Population can choose to share them, PPRNs have the potential to more systematically address •Telehealth Self-Management Program in Older Adults Living with Heart Failure in Health Disparity Communities the question of genuine importance to patients.55 •Nueva Vida Intervention: Improving QOL in Latina Breast Cancer Survivors and Their Caregivers •Rural Options At Discharge Model of Active Planning (ROADMAP) Together the CDRN/PPRN repositories are • A Patient-Centered Intervention to Increase Screening of Hepatitis B and C Among Asian-Americans geographically dispersed, with patients in 50 states.53 They are organized as distributed * The IOM identified research on falls in older adults in the top quartile of its final list of 100 priority topics: “Compare the effectiveness of primary prevention methods, such as exercise and balance training, versus clinical treatments in preventing falls in older adults at data networks, each with its own governing varying degrees of risk” 10 PCORnet is tackling many Box 8. Examples of PCORI Funding Awards: Dissemination and Methods fundamental challenges to the conduct Research of multisite research. These include the Communication and Dissemination Research technical challenges of collecting and •Shared Decision-making and Renal Supportive Care harmonizing longitudinal data from •Randomized Trial to Increase Adherence to Cervical Cancer Screening Guidelines for Young multiple and fast-growing sources, Women as well as the ethical and regulatory •Comparing Traditional and Participatory Dissemination of a Shared Decision-making Intervention challenges of conducting research in this •Improving Communication for Chemotherapy: Addressing Concerns of Older Cancer Patients new environment, including obtaining and Caregivers informed consent, use of central •Patient-Identified Personal Strengths (PIPS) vs. Deficit-Focused Models of Care institutional review boards, and the •Presenting Patient-Reported Outcomes Data to Improve Patient and Clinician Understanding protection of patient privacy.53,56 Thus, and Use PCORnet holds the promise to transform •Shared Medical Decision-making in Pediatric Diabetes clinical research. •Relapsed Childhood Neuroblastoma as a Model for Parental End-of-Life Decision-Making •Creating a Patient-Centered Tool to Help Medicare Beneficiaries Choose Prescription Drug Evidence synthesis and appraisal: Plans What is PCORI’s role? PCORI is authorized to conduct primary Accelerating PCOR and Methodological Research research—to generate additional and •Facilitating Patient Reported Outcome Measurement for Key Conditions improved evidence about what works— •Statistical Methods for Missing Data in Large Observational Studies and to engage in research synthesis. [Box 9] The PCORI investments have been •Improving the Use of Patient Registries for Comparative Effectiveness focused on the primary evidence generation •Measuring Patient-Centered Communication for Colorectal Cancer Care and Research component of CER and not on the synthesis •Developing Patient-Centered Outcomes for Dementia: Goal Setting and Attainment and appraisal activities. However, these •Understanding Treatment Effect Estimates When Treatment Effects Are Heterogeneous for More evidence reviews are a central feature of Than One Outcome HTA, and are conducted by other entities •Evaluating Methods to Engage Minority Patients and Caregivers as Stakeholders (e.g., AHRQ, Cochrane Review). •Improving Patient Engagement and Understanding Its Impact on Research through Community Review Boards Historically, reviews of existing evidence •The National Patient-Centered Clinical Research Network (PCORnet) have been conducted by the AHRQ under the Effective Healthcare Program which was established in 2003 by the Medicare board but adhering to common data effective? Modernization Act.58 However, support for standards. They use a distributed query this effort is in question with the advent approach, which permits analyses to be • What are the best treatment strategies for low back pain? of PCORI and elimination of funding for conducted behind institutional firewalls.53 the Effective Healthcare Program separate For researchers, this network of electronic • Which interventions are most effective from PCOR funds transfers. PCORI may medical records will make observational for reducing disparities in hypertension yet extend its role to reviews of evidence. and interventional trials easier to launch, outcomes? more representative of diverse, real-world PCORnet has recently set its first research A central feature of HTA is review and populations, and capable of providing target–a $10 million pilot clinical trial on synthesis of evidence. HTA is conducted much-needed answers to comparative the use of aspirin to prevent heart disease. by (or on behalf of) a variety of private and effectiveness research questions with Participants will take daily doses of public payers including the Veteran’s Ad- greater accuracy. Some examples of the aspirin that fall within the range typically ministration, Medicare, and Medicaid, and types of questions that can be answered prescribed for heart disease and will be private health insurance issuers and large, through PCORnet are:56 monitored to determine whether one self-insured employer-based plans. [Box • What are the best strategies for dosage works better than the others.57 10] They may undertake their own as- managing localized prostate cancer? The aspirin trial, while important in its sessments, relying on inputs from entities own right, is also a “proof of concept” for that have expertise in evidence synthesis • Which of the available primary care and make appraisals for coverage, benefit conducting research through PCORnet. strategies for children with attention- design, and payment. deficit hyperactivity disorder are most 11 Table 2. Methods for Conducting Primary CER (Generating New Evidence) Study Design Features Advantages Limitations or Other Considerations Randomized Controlled Gold standard of comparative effectiveness With sufficiently large sample size, RCTs usually require considerable time; Trials (RCT) research; randomize individuals or groups confounders are balanced across imposition of exclusion criteria results in a to different study arms, usually treatment different study arms, enabling homogeneous study population that is not and control. assessment of unbiased treatment representative of real-world population; effect(s). study settings may not reflect places where routine care is delivered. Pragmatic Randomized Pragmatic or practical trials involve: (1) Research in real-world setting with Large number of participants and diversity Controlled Trials comparison of clinically relevant alterna- diverse populations. of study sites; can use routinely occurring tive interventions, (2) diverse population of variation in health care systems. study participants, (3) participants from het- erogeneous practice settings, and (4) data on broad range of health outcomes.51,52 Propensity Scores Strategy to approximate randomized con- Can be conducted more rapidly Individuals or groups that received the and Causal Inference trolled trials; use of large electronic clinical and with less cost compared to intervention or treatment may differ sys- Methods and administrative databases; use of pro- randomized trials; in large data- tematically from those that did not; cannot pensity scores to create a similar covariate bases, researchers can “observe” confer rigor of RCTs. distribution between groups. the differences between those using and not using a treatment or intervention. Adaptive and Bayes- Adaptive trials build on the approaches Reduces the rigidity of design Improper adaptations may give rise to bias. ian Trial used in most randomized trials but allow specifications in rapidly chang- for changes during the course of the study ing world and minimizes time to for parameters such as what proportion of completion. participants is randomized to which group, sample size, eligibility criteria, and end points 50 Sources: The PCORI Methodology Report, 2013 50. Tunis et al. Comparative effectiveness research: Policy context, methods development and research infrastructure, 2010 51. Chalkidou et al. The role of pragmatic randomized controlled trials (pRCTs) in comparative effectiveness research, 2012 52. PCORI will not undertake the kind of What strategies are needed to audiences, including patients, clinicians, appraisal activities that are common in promote uptake of research findings? and policy makers to encourage uptake. the HTA agencies in other high-income In addition to more and better evidence, countries. The ACA specifically excludes enhanced patient choices and outcomes, To address these weaknesses, certain kinds of appraisal activities from and faster uptake, we need more uniform implementation considerations are a factor PCORI’s purview. For example, the ACA uptake of evidence. The myriad reasons in each step of the PCORI process, from forbids the use of cost per QALY “as a for poor uptake and use by decision- the prioritization of research questions, to threshold to establish what type of health makers have been well-documented the selection of a study design and choice care is cost effective or recommended.”43 and include: the evidence may not be of outcome measures, to the consideration The ACA also states that the research compelling to patients or clinicians; of how the evidence is likely to be used findings of PCORI funded research can clinicians may have a hard time changing in decision-making. A core component “not be construed as mandates for practice practice patterns; contextual factors such of the PCORI strategy for maximizing guidelines, coverage recommendations, as current payment incentives hinder uptake is intensive, interactive patient and payment, or policy recommendations.”43 behavior change; or the findings may not stakeholder engagement from priority Nevertheless, there is an expectation that be widely disseminated.62,63 Patients, for setting, to inclusion in the research the new CER will make higher quality example, may have the greatest incentive to team itself, to deep involvement in merit evidence available to plan decision-makers, use CER, but “relatively few are equipped review of applications. The organizing enabling health plans to orient coverage to make use of the highly technical vision is that this deep involvement and payment policies toward health scientific evidence generated through will enhance uptake by producing care interventions and delivery system CER and to understand how it applies in “dissemination ready” findings. To further strategies that produce better outcomes at their particular situation.”2 Consequently, maximize the likelihood of uptake from lower cost. even when CER findings are relevant research findings, PCORI is also funding and informed by patient and stakeholder dissemination and implementation engagement, they must be packaged and research studies and the PCORTF provides disseminated in ways that ensure that funding directly to AHRQ to undertake they meet the needs of different target this type of research. While PCORI has no 12 and implementation. Understanding Box 9. Evidence Synthesis: Definitions the processes through which research Research synthesis, which is often included under the rubric of CER, is a process through which findings are implemented in general, and “researchers seek to summarize the information from multiple studies addressing similar research ques- in particular the influence of stakeholder tions.”13 involvement in uptake, will be crucially “A comparative effectiveness systematic review summarizes available scientific evidence in which important in the coming years. investigators collect, evaluate, and synthesize studies in accordance with an organized, structured, explicit and transparent methodology. They seek to provide decision-makers with accurate, inde- pendent, scientifically rigorous information for comparing the effectiveness and safety of alternative clinical options, and have become a foundation for decision-making in clinical practice and health Discussion and policy including informing coverage decisions for therapeutics in health care.” The methods for systematic review include: activities related to how the questions are framed, how the quality of Concluding Thoughts the available evidence is assessed, and the standards that are applied to existing research. This The combination of rising health care evidence assessment or synthesis activity often feeds into an appraisal process in which questions costs and the move toward universal or about cost are addressed.13 near-universal coverage globally brings an urgent demand to spend health care funds on effective technologies and interventions and to do so efficiently and in accordance Box 10. HTA in the United States with each country’s priorities. A response The Federal Agency for Healthcare Research and Quality (AHRQ) is the lead source of systematic reviews to such demand requires an understanding which are used for HTA.59 A mix of federal and nonfederal partners nominates topics, and research- of what technologies increase the quality ers who participate in a network of AHRQ-funded Evidence-based Practice Centers (EPCs) produce and value of health care along with the evidence synthesis reports. EPCs are housed in academic centers (at the Oregon Health Sciences University and at Johns Hopkins University, for example), but AHRQ also funds EPCs at the Blue Cross/ knowledge of the policy levers that Blue Shield Association Technology Evaluation Center, the Kaiser Permanente Research Center, and the encourage health care systems, patients, Minneapolis VA. The AHRQ evidence based practice center reports are intended to provide informa- and clinicians to adopt appropriate tion useful to patients and clinicians to improve care as well as to health plan decision-makers to inform coverage and payment decisions. technologies. These forces enhance the importance of CER and HTA activities and Many large private insurers and Pharmacy Benefits Management programs (PBMs) have sophisticated HTA programs staffed by clinical experts and financial analysts and supported by sophisticated data the mechanisms that promote the adoption systems.59 Smaller health plans have much more limited staffing and often depend on technology as- of best practices. sessments produced by outside private or public agencies. Little information is available regarding the scope, internal processes and conduct of HTAs by private insurers and PBMs. WellPoint makes its HTA guidelines publically available, but while the output is available, the HTA process itself is typically a black Although the organization and financing of box: there is often little transparency regarding the evidence and approaches used to arrive at coverage health care are different in the United States decisions.59 Coverage decisions based on the HTA activities of private insurers can vary widely. and Europe, Australia, and Canada, which The nation’s major public payers—Medicare, Medicaid, and the Veterans’ Health Administration—also lead to diverse mechanisms for both CER use CER to inform coverage decisions. The Medicare Coverage Division within CMS is responsible for undertaking or commissioning HTA reports to support considerations for a national coverage decision.60 and HTA, there are common issues and Most evidence-related coverage determination efforts rely on synthesizing existing clinical and often opportunities for improvements in methods health services research literature by means of formal or informal systematic reviews.40 Medicare can that can inform HTA and PCOR across request a formal evidence report from AHRQ.61 The ultimate coverage decisions, along with evidence dossiers and, other meeting materials, are placed on the CMS website for transparency, and issued countries. The opportunities for international through National Coverage Decision memoranda. Notably, the Coverage Division is explicitly prohibited collaboration are perhaps greatest in four by law from considering evidence related to the cost or cost-effectiveness of technologies when making areas: patient engagement, observational coverage decisions.61 Medicare Part D outpatient drug program operates separately from the Medicare Coverage Division, with all coverage decisions made by Part D contractors. research methods, meta-analysis, and implementation research. [Box 11] State Medicaid programs often purchase HTA’s from private organizations that specialize in this area. In general, the operating budgets for state-level HTA activities are insufficient for the workload.59 For HTA International exchange and collaboration can around drugs, 14 states use the comparative effectiveness reports on drugs produced by the Drug Effec- be key for improving and establishing HTA. tiveness Review Process at the Oregon Health and Sciences University.40 One key area for future collaboration direct oversight on how AHRQ uses these stakeholders in the health care system is approaches to implementation—that funds, there are ongoing efforts to, and will likely be needed. While PCORI has a is, undertaking deliberate strategies to expectation of, coordination of these linked mandate to disseminate PCOR findings encourage the implementation of evidence- responsibilities. (the ACA set out a 90-day timeframe based clinical decision-making and for the dissemination of results from evidence-based coverage and payment While PCORI can set the stage for completion of the research), more than policy. In Europe, Canada, and Australia, uptake and maximize conditions that dissemination is needed to ensure good there are direct mechanisms for bringing will encourage uptake, PCORI has uptake. PCORI has set in motion a set results of effectiveness analyses for limited policy levers at its disposal and of strategies and research to understand technologies into practice that are not partnerships with a broad range of factors that influence dissemination present in the United States. However, 13 Do we see appropriate use of health care Box 11. Areas within CER and HTA with Potential for Mutual Learning technologies and the delivery of evidence- Patient engagement: How do you include patient and family perspectives and benefit from them? based, high-value care? Have these How do you coach and train people to be good participants? comparative effectiveness analyses helped Observational research methods: How can observational data be used to make CER more improve quality and efficiency in health care? practical, relevant, and timely? How will they be generated and analyzed? There have been calls for overall evaluations in Europe.7 GAO is conducting such an Meta-analysis methods: Network meta-analysis and indirect comparisons can be looked into by all HTA bodies and PCORI evaluation for PCORI’s efforts in the United States. Evaluation of the effects of CER and Approaches to implementation: How do you translate evidence and knowledge in a more HTA on the health care system is important decentralized practice system? and could inform the future direction of these efforts internationally. Further engagement by researchers and stakeholders there is also a need for additional activities by March 2015. The primary objectives in the United States, including PCORI’s focused on local implementation. for the review are to examine: (1) whether leadership, in international developments An issue for the United States, going PCORI established research priorities will promote mutual learning and could forward, is whether the dissemination and funded research in accordance with enhance the impact of PCOR investments in and implementation activities to be its legislative requirements, and (2) the this country. undertaken by PCORI will be sufficient extent to which PCORI has established to encourage widespread change in health plans and undertaken efforts to evaluate care coverage and delivery. the effectiveness of its work.64 The time Acknowledgements frame for this evaluation is appropriate The authors are grateful to the following for examining issues around use of funds groups and individuals who provided Across countries, the question of the (e.g., whether funded studies addressed valuable insight and feedback on the impact of CER and HTA is never far from the most pressing issues in the most manuscript: view. This leads to questions like: how do we assess whether we have had an impact effective ways), but it is too short to assess • Panel with 17 domestic and in clinical practice? How does the impact the quality and relevance of the findings international experts who provided compare with desired outcomes? And from the research funded, what effect feedback on the manuscript equally important, if there is no impact, the CER has had on medical practice, and certainly too short to look at effect • Lisa Simpson, AcademyHealth an investigation about barriers to impact would be necessary. Was the time frame on health care outcomes. This question • Katherine Griffith, AcademyHealth too short? Were the incentives for adoption is particularly important for PCORI, not strong enough or, worse, were perverse since an impact assessment will be done • Raj Sabharwal, AcademyHealth incentives in place? Of course, financial by GAO in 2017, just before PCORI’s • Kristin Rosengren, AcademyHealth incentives—like payments for particular authorization and funding expires in outcomes or actions—need to be aligned 2019.43,65 • Emily Holubowich, CRD Associates with the HTA decisions for these to have The authors would also like to thank maximum impact on practice. Methods In sum, the enhanced CER and HTA Health Technology Assessment and strategies from the emerging field of activities in the United States, European International (HTAi) for allowing them to Research Impact Assessment (RIA) may be nations, and other countries underscore the hold a meeting to discuss the manuscript useful here. need for overall evaluations of the effects with the expert panel during their annual of CER and HTA on health care systems. conference. Another significant aspect of impact Increasingly, countries want to know if CER assessment is timing. When is the right and HTA efforts are improving or likely The views expressed herein are those of time to evaluate the impact of CER to improve clinical practice, patient care, the authors and not of AcademyHealth, or HTA? In the United States, a GAO and health outcomes while enhancing the the expert panel, or the sponsors of this evaluation of PCORI is due to Congress performance of their health care systems. analysis. Funding Recognition AcademyHealth gratefully acknowledges the Kaiser Permanente Institute for Health Policy and the National Institute for Health Care Management Foundation for their generous support of this work. See www.academyhealth.org for more information. FOUNDATION 14 References 17. Luce B, Drummon M, Jönsson B, et al. EBM, 33. IQWiG. IQWiG publishes new version of its 1. Holve E, Pittman P. A First Look at the Volume HTA, and CER: Clearing the Confusion. General Methods 2013; https://www.iqwig. and Cost of Comparative Effectiveness Research Milbank Q. 2010;88(2):256-276. de/en/press/press_releases/press_releases/ in the United States. AcademyHealth. 2009. iqwig_publishes_new_version_of_its_general_ 18. Birkett DJ, Mitchell AS, McManus P. A Cost- methods.5346.html. Accessed June 5, 2014. 2. Docteur E, Berenson R. How Will Comparative Effectiveness Approach To Drug Subsidy Effectiveness Research Affect the Quality of And Pricing In Australia. Health Affairs. 34. NICE. Methods for Economic Evaluation Project Health Care? Urban Institute 2010:1-15. 2001;20(3):104-114. and the Gates Reference Case 2014; http:// www.nice.org.uk/About/What-we-do/NICE- 3. Drummond M, Tarricone R, Torbica A. 19. Lopert R. Evidence-Based Decision-Making International/NICE-International-projects/ Assessing the added value of health technologies: Within Australia’s Pharmaceutical Benefits Methods-for-Economic-Evaluation-Project-and- reconciling different perspectives. Value Health. Scheme. The Commonwealth Fund. 2009. the-Gates-Reference-Case. Accessed July 15, 2013;16(1 Suppl):S7-13. 2014. 20. Chalkidou K. Comparative Effectiveness Review 4. Rawlins MD, Culyer AJ. National Institute for Within the U.K.’s National Institute for Health 35. Tunis SR, Pearson SD. Coverage options for Clinical Excellence and its value judgements. and Clinical Excellence. The Commonwealth promising technologies: Medicare’s coverage BMJ. 2004;329:224-227. Fund. 2009. with evidence development’. Health Aff (Millwood). 2006;25(5):1218-1230. 5. Devlin N, Parkin D. Does NICE have a cost- 21. Rochaix L, Xerri B. National Authority for effectiveness threshold and what other factors Health: France. The Commonwealth Fund. 2009. 36. Daniel GW, Rubens EK, McClellan M. Coverage influence its decisions? A binary choice analysis. with evidence development for Medicare 22. Nasser M, Sawicki P. Institute for Quality Health Econ. 2004 May;13(5):437-452. beneficiaries: challenges and next steps. JAMA and Efficiency in Health Care: Germany. The Intern Med. 2013 Jul 22;173(14):1281-1282. 6. Johnson FR. Editorial: Moving the QALY Commonwealth Fund. 2009. forward or just stuck in traffic? Value Health. 37. EUnetHTA. EMA. 2014; http://www.eunethta. 23. Chalkidou K, Anderson G. Comparative Mar 2009;12 Suppl 1:S38-39. eu/ema. Accessed June 30, 2014. Effectiveness Research: International 7. Sorenson C, Chalkidou K. Reflections on the Experiences and Implications for the United 38. Sutcliffe D, Lester H, Hutton J, Stokes T. NICE evolution of health technology assessment States. AcademyHealth. 2009:1-19. and the Quality and Outcomes Framework in Europe. Health Econ Policy Law. 2012 (QOF) 2009-2011. Quality in Primary Care. 24. Chalkidou K, Tunis S, Lopert R, et al. Jan;7(1):25-45. 2012;20:47-55. Comparative effectiveness research and 8. Lauer MS, Collins FS. Using Science to Improve evidence-based health policy: experience from 39. NICE: About the Quality and Outcomes the Nation’s Health System: NIH’s Commitment four countries. Milbank Q. 2009 Jun;87(2):339- Framework (QOF) 2014; http://www.nice.org. to Comparative Effectiveness Research. 367. uk/aboutnice/qof/qof.jsp. Accessed July 1, 2014. JAMA : the journal of the American Medical 25. Devlin N, Parkin D. Does NICE have a cost- 40. Luce B, Cohen RS. Health technology assessment Association. 2010;303(21):2182-2183. effectiveness threshold and what other factors in the United States. Int J Technol Assess Health 9. Hirsch JA, Barr RM, McGinty G, et al. Affordable influence its decisions? A binary choice analysis. Care. Jul 2009;25 Suppl 1:33-41. care 2014: a tale of two boards. J Neurointerv Health Econ. 2004;13(5):437-452. Surg. 2014 Jun 24. 41. Perry S, Thamer M. Medical Innovation and the 26. Earnshaw J, Lewis G. NICE Guide to Methods Critical Role of Health Technology Assessment. 10. PCORI: Research Done Differently. http://www. of Technology Appraisal Pharmacoeconomics. JAMA : the journal of the American Medical pcori.org/sites/default/files/PCORI-Research- 2008;26(9):725-727. Association. 1999;282(19):1869-1872. Done-Differently.pdf. Accessed July 8, 2014. 27. Newmann PJ, Greenberg D. Is the United 42. Implementing Comparative Effectiveness 11. PCORI. Patient-Centered Outcomes Research. States Ready for QALYs? Health Affairs. Research: Priorities, Methods, and Impact. 2014; http://www.pcori.org/research-we- 2009;28(5):1366-1371. Brookings. 2009:1-84. support/pcor/. Accessed July 8, 2014. 28. Office of Health Economics. Value-Based 43. The Patient Protection and Affordable Care Act. 12. Hamilton Lopez M, Holve E, Rein A, Winker J. Pricing. 2014; http://news.ohe.org/category/ Public Law 111-148, 124 Stat. 1192010. Involving Patients and Consumers in Research: pricing-and-reimbursement/value-based- New Opportunities for Meaningful Engagement pricing/. Accessed June 5, 2014. 44. Rich EC. Past as prologue: how comparative in Research and Quality Improvement. effectiveness research became patient- 29. Culyer A, MaCabe C, Briggs A, et al. Searching centered outcomes research. J Comp Eff Res AcademyHealth. 2012 June. for a Threshold, Not Setting One: the Role 2012;1(6):475-477. 13. Initial National Priorities for Comparative of the National Institute for Health and Effectiveness Research. National Academy of Clinical Excellence J Health Serv Res Policy. 45. Selby J, Beal AC, Frank L. The Patient-Centered Sciences 2009. 2007;12(1):56-58. Outcomes Research Institute (PCORI) National Priorities for Research and Initial Research 14. Selby JV, Fleurence R, Lauer M, Schneeweiss S. 30. Claxton K, Martin S, Soares M, et al. Agenda. JAMA : the journal of the American Reviewing hypothetical migraine studies using Methods for the Estimation of the NICE Cost Medical Association. 2012;307(15):1583-1584. funding criteria from the Patient-Centered Effectiveness Threshold. The University of York Outcomes Research Institute. Health Aff 2013:1-119. 46. PCORI. Research Prioritzation Topic Briefs (Millwood). Oct 2012;31(10):2193-2199. 2014; 1-89. Available at: http://www.pcori.org/ 31. Drummond M, Sorenson C. Can NICE Be assets/2014/04/PCORI-Improving-Healthcare- 15. Selby JV. The researcher-in-chief at the Patient- Nicer? In a World of Budget Constraints Systems-Topic-Brief-050814.pdf Accessed June Centered Outcomes Research Institute. There Are No Easy Solutions. Value Health. 30, 2014. Interview by Susan Dentzer. Health Aff 2009;12(5):634-636. (Millwood). Dec 2011;30(12):2252-2258. 47. Patient-Centered Outcomes Research Institute: 32. Sutton AJ, Abrams KR, Jones DR, Sheldon National Priorities for Research and Research 16. EUnetHTA. 2014; http://www.eunethta.eu/ TA, Song F. Systematic Review of Trials and Agenda. 2012:1-21. about-us/faq#t287n73. Accessed June 30, 2014. Other Studies. Health Technology Assessment. 1998;2(19):1-276. 15 48. PCORI Funding Awards. 2014; http://pfaawards. 54. NIH. PCORnet: Meeting Clinical Trial’s Need 60. Neumann PJ, Divi N, Beinfeld MT, et al. pcori.org. Accessed September 18, 2014. for Speed 2014; http://directorsblog.nih. Medicare’s national coverage decisions, gov/2013/12/17/pcornet-meeting-clinical-trials- 1999-2003: quality of evidence and review 49. Methdological Standards and Patient- need-for-speed/. Accessed September 4, 2014. times. Health Aff (Millwood). 2005 Jan- Centeredness in Comparative Effectiveness Feb;24(1):243-254. Research: The PCORI Perspective. JAMA : the 55. Fleurence RL, Beal AC, Sheridan SE, Johnson journal of the American Medical Association. LB, Selby JV. Patient-powered research networks 61. Neumann P, Rosen A, Weinstein M. Medicare 2012;307(15):1636-1640. aim to improve patient care and health research. and Cost-Effectiveness Analysis N Engl J Med. Health Aff (Millwood). 2014 Jul 1;33(7):1212- 2005;353:1516-1522. 50. Hickam D, Totten A, Berg A,Rader K, Goodman 1219. S, Newhouse, R. The PCORI Methodology 62. Timbie JW, Fox DS, Van Busum K, Schneider Report. 2013. 56. Collins F, Hudson K, Briggs J, Lauer M. EC. Five reasons that many comparative PCORnet: Turning a Dream Into Reality J Am effectiveness studies fail to change patient care 51. Tunis SR, Benner J, McClellan M. Comparative Med Inform Assoc. 2014:1-2. and clinical practice. Health Aff (Millwood). effectiveness research: Policy context, methods 2012 Oct;31(10):2168-2175. development and research infrastructure. Stat 57. Optimal Aspirin Dose for Patients with Med. 2010 Aug 30;29(19):1963-1976. Coronary Artery Disease Approved as Topic 63. Avorn J, Fischer M. ‘Bench to behavior’: for First PCORnet Research Trial. 2014; http:// translating comparative effectiveness research 52. Chalkidou K, Tunis S, Whicher D, Fowler www.pcori.org/2014/optimal-aspirin-dose- into improved clinical practice. Health Aff R, Zwarenstein M. The role for pragmatic for-patients-with-coronary-artery-disease- (Millwood). 2010 Oct;29(10):1891-1900. randomized controlled trials (pRCTs) in approved-as-topic-for-first-pcornet-research- comparative effectiveness research. Clin Trials. trial/. Accessed September 10, 2014. 64. PCORI. Board of Governors Meeting via 2012 Aug;9(4):436-446. Teleconference/Webinar. 2014; http://www. 58. Medicare Prescription Drug, Improvement, and pcori.org/assets/2014/05/PCORI-Board- 53. Fleurence RL, Curtis LH, Califf RM, Platt R, Modernization Act of 2003. Meeting-Slide-Presentation-050514.pdf. Selby JV, Brown JS. Launching PCORnet, a Accessed June 30, 2014. national patient-centered clinical research 59. Sullivan SD, Watkins J, Sweet B, Ramsey SD. network. J Am Med Inform Assoc. 2014 Jul- Health technology assessment in health-care 65. National Pharmaceutical Council. PCORI Aug;21(4):578-582. decisions in the United States. Value Health. Outlines Research Funding Efforts for 2013. 2009 Jun;12 Suppl 2:S39-44. http://www.npcnow.org/newsletter/content/ evidently-february-2013-pcori-outlines- research-funding-efforts-2013. Accessed June 30, 2014. 16