United States Government Accountability Office Report to Congressional Requesters VA HEALTH CARE May 2017 Improvements Needed in Data and Monitoring of Clinical Productivity and Efficiency GAO-17-480 May 2017 VA HEALTH CARE Improvements Needed in Data and Monitoring of Clinical Productivity and Efficiency Highlights of GAO-17-480, a report to congressional requesters Why GAO Did This Study What GAO Found VA has faced challenges managing its In 2013, the Department of Veterans Affairs (VA) implemented clinical budget and ensuring veterans’ access productivity metrics to measure physician providers’ time and effort to deliver to health care, generating procedures. VA also developed statistical models to track clinical efficiency at VA congressional interest in asking GAO medical centers (VAMC). Data collected under the metrics and models are used to examine VA’s use of its productivity to identify sub-optimal clinical productivity and inefficiency at VAMCs. GAO found and efficiency metrics. that contrary to federal internal control standards for information, VA’s metrics This report assesses (1) whether VA's and models may not provide quality information because the information is clinical productivity metrics and incomplete and may not accurately reflect clinical productivity and efficiency. efficiency models provide complete GAO identified limitations with VA’s metrics and models that limit VA’s ability to and accurate information on provider assess whether resources are being used effectively. Specifically, productivity and VAMC efficiency, and (2) VA’s efforts to monitor and improve • Productivity metrics are not complete because they do not account for all clinical productivity and efficiency. providers or clinical services. Due to data systems limitations, the metrics do not capture all types of providers who deliver care at VAMCs, including GAO reviewed VA documentation, contract physicians and advanced practice providers, such as nurse such as policies and guidance, and practitioners, serving as sole-providers. In addition, the metrics do not 2015 data on clinical productivity and capture providers’ workload evaluating and managing hospitalized patients. efficiency, the most recent data available. GAO also interviewed VA • Productivity metrics may not accurately reflect the intensity of clinical Central Office officials about VA’s workload. A 2016 VA audit shows that VA providers do not always accurately metrics and models and monitoring code the intensity of—that is, the amount of effort needed to perform— efforts. GAO visited six VAMCs and clinical procedures or services. As a result, VA’s productivity metrics may not their corresponding Veterans accurately reflect provider productivity, as differences between providers may Integrated Service Networks, selected represent coding inaccuracies rather than true productivity differences. based on geographic diversity, variation of facility complexity, and • Productivity metrics may not accurately reflect providers’ clinical staffing differences in productivity and levels. Officials at five of the six selected VAMCs GAO visited reported that efficiency levels. GAO examined VA’s providers do not always accurately record the amount of time they spend efforts to monitor and improve clinical performing clinical duties, as distinct from other duties. productivity and efficiency in the context of federal standards for internal • Efficiency models may also be adversely affected by inaccurate workload control related to information and and staffing data. To the extent that the intensity and amount of providers’ monitoring. clinical workload are inaccurately recorded, some of VA’s efficiency models examining VAMC utilization and expenditures may also be inaccurate. For What GAO Recommends example, the model that examines administrative efficiency requires accurate GAO is making four recommendations data on the amount of time VA providers spend on administrative tasks; if the to improve the completeness and time providers allocate to clinical, administrative, and other tasks is incorrect, accuracy of VA’s productivity metrics the model may overstate or understate administrative efficiency. and efficiency models and to strengthen VA’s oversight of VAMCs’ GAO found that VA Central Office has taken steps to help VAMCs monitor use of these metrics and models. VA provider productivity by developing a comprehensive analytical tool VAMCs can concurred or concurred in principle use to identify the drivers of low productivity. While VAMCs are required to with GAO's recommendations and monitor VA’s productivity metrics, GAO found that VA does not require VAMCs described its plans to implement them. to monitor VA’s efficiency models. Further, VA does not systematically oversee VAMCs’ efforts to monitor clinical productivity and efficiency. As a result, VA cannot systematically identify best practices to address low productivity and View GAO-17-480. For more information, inefficiency as well as determine the factors VAMCs commonly identify as contact Randall B. Williamson at contributing to low productivity and inefficiency. This approach is inconsistent (202) 512-7114 or williamsonr@gao.gov. with federal standards for internal control related to monitoring. United States Government Accountability Office Contents Letter 1 Background 4 VA’s Metrics and Models May Not Provide Complete and Accurate Information on Clinical Productivity and VAMC Efficiency 10 VA Central Office Has Taken Steps to Help VAMCs Monitor and Improve Clinical Productivity, but Does Not Systematically Oversee Productivity and Efficiency across VA 18 Conclusions 24 Recommendations for Executive Action 25 Agency Comments and Our Evaluation 25 Appendix I Comments from the Department of Veterans Affairs 28 Appendix II GAO Contact and Staff Acknowledgments 32 Related GAO Products 33 Tables Table 1: Overview of the Department of Veterans Affairs (VA) Fiscal Year 2017 Productivity Performance Standards for Selected Clinical Specialties 7 Table 2: Overview of the Department of Veterans Affairs (VA) 12 Efficiency Opportunity Grid Models 9 Figure Figure 1: Calculation of the Department of Veterans Affairs’ (VA) Physician-Provider Productivity Metrics for 32 Clinical Specialties 6 Page i GAO-17-480 VA Health Care Abbreviations ACSC Ambulatory Care Sensitive Condition CDI Clinical Documentation Improvement CPT Current Procedural Terminology EOG Efficiency Opportunity Grid FTEE full-time employee equivalent OIG Office of Inspector General OPES Office of Productivity, Efficiency, and Staffing SAIL Strategic Analytics for Improvement and Learning Value Model VA Department of Veterans Affairs VAMC Department of Veterans Affairs medical center VHA Veterans Health Administration VISN Veterans Integrated Service Network wRVU work-relative value unit This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page ii GAO-17-480 VA Health Care Letter 441 G St. N.W. Washington, DC 20548 May 24, 2017 Congressional Requesters The Department of Veterans Affairs’ (VA) Veterans Health Administration provided care to about 6.9 million veterans and obligated about $65 billion for their care in fiscal year 2016. To help VA ensure the effective use of its budgetary resources in light of a growing number of veterans seeking health care services from VA, VA’s Office of Inspector General (OIG) recommended that the department establish clinical productivity metrics for providers at VA’s 168 medical centers (VAMC). 1 Under the metrics, VA would collect clinical productivity data and establish productivity performance standards for at least five clinical specialties by the end of fiscal year 2013, with the goal of expanding data collection and associated standards to all specialties. The purpose of collecting such data and establishing associated standards is to increase VAMCs’ ability to identify sub-optimal productivity and to inform decisions about resource needs—such as determining the number of providers required to deliver timely access to VA health care services. 2 VA began implementing the OIG’s recommendations in fiscal year 2013. In addition, VA developed statistical models to track VAMCs’ clinical efficiency—that is, VAMCs’ ability to provide high-cost and high-volume clinical services in a manner that optimizes the use of resources. Recent budgetary challenges and revelations of untimely access to VA health care services have renewed congressional interest in assessing 1 VA OIG, Veterans Health Administration: Audit of Physician Staffing Levels for Specialty Care Services. 11-01827-36. (Washington, D.C.: Dec. 27, 2012). Clinical productivity refers to the workload performed by VA’s clinical providers over a given time period. 2 VA uses its Veterans Equitable Resource Allocation model to allocate resources among its 18 regional Veterans Integrated Service Networks (VISN), each of which manages and oversees several VAMCs. These networks then allocate resources to their respective VAMCs. VA’s productivity metrics can help VISNs determine VAMCs’ resource needs, for example, by providing data on the extent to which VAMCs are employing an appropriate number of providers in each clinical specialty. Page 1 GAO-17-480 VA Health Care clinical productivity and efficiency. 3 We were asked to examine VAs use of productivity metrics and efficiency models. In this report, we assess 1. whether VA’s clinical productivity metrics and efficiency models provide complete and accurate information on provider productivity and VAMC efficiency; and 2. VA’s efforts to monitor and improve clinical productivity and efficiency. To assess whether VA’s clinical productivity metrics and efficiency models provide complete and accurate information on provider productivity and VAMC efficiency, we examined the types of providers and clinical services captured by—and the processes for recording—the underlying clinical workload and staffing data used in these metrics and models. To do this, we reviewed VA documentation, including handbooks, directives, and other policy documents. 4 We also interviewed officials from the Office of Productivity, Efficiency, and Staffing (OPES), an office within VA Central Office that is responsible for developing VA’s clinical productivity metrics and efficiency models. We also conducted site visits at six VAMCs which we selected based on geographic diversity, differences in facility complexity, and variation in their providers’ performance on VA’s productivity metrics as well as variation in the VAMCs’ performance on VA’s efficiency models for fiscal year 2015 (we selected the VAMCs to include both high and low performers). 5 The data 3 For an overview of the budgetary and access challenges affecting VA, see, for example, GAO, VA’s Health Care Budget: In Response to a Projected Funding Gap in Fiscal Year 2015, VA Has Made Efforts to Better Manage Future Budgets. GAO-16-584 (Washington, D.C.: Jun. 3, 2016) and GAO, VA Health Care Management and Oversight of Consult Process Need Improvement to Help Ensure Veterans Receive Timely Outpatient Specialty Care. GAO-14-808 (Washington, D.C.: Oct. 30, 2014). 4 See VHA Handbook 1065.01, Productivity and Staffing Guidance for Specialty Provider Group Practice (May 4, 2015); VHA Directive 1082, Patient Care Data Capture (March 24, 2015); and VHA Directive 1231, Outpatient Clinic Practice Management (November 15, 2016). 5 VA assigns each VAMC to one of five complexity groups based on patient population served, clinical services offered, education and research complexity, and administrative complexity. The six VAMCs we selected had varying levels of complexity, as defined by VA. In addition, of the six VAMCs, we selected three VAMCs that VA data showed had low clinical productivity—that is, four or more clinical specialties with productivity levels more than one standard deviation below the applicable productivity performance standard—and high overall inefficiency in fiscal year 2015. In addition, we selected three VAMCs that VA data showed had high clinical productivity—that is, zero clinical specialties with productivity levels more than one standard deviation below the applicable productivity performance standard—and low overall inefficiency in fiscal year 2015. Page 2 GAO-17-480 VA Health Care showing the performance on the productivity metrics and efficiency models for fiscal year 2015 were the most recent data available at the time of our analysis. At each of the six VAMCs, we interviewed facility officials about steps taken to record the clinical workload and staffing data that serve as the basis of the productivity metrics and efficiency models, how the officials use these metrics and models to determine their facilities’ productivity and efficiency, and any limitations of the metrics and models. The six VAMCs we selected are located in Atlanta, Georgia; Baltimore, Maryland; Harlingen, Texas; Las Vegas, Nevada; Saginaw, Michigan; and Salem, Virginia. Finally, we assessed whether VA’s clinical productivity metrics and efficiency models provide complete and accurate information on provider productivity and VAMC efficiency in the context of federal standards for internal control related to information. 6 To examine VA’s efforts to monitor and improve clinical productivity and efficiency, we assessed the monitoring and improvement efforts of VA Central Office, the six selected VAMCs, and the Veterans Integrated Service Networks (VISN) that are responsible for overseeing the six VAMCs. 7 To do this, we first reviewed VA documentation, such as handbooks, directives, and guidance and identified VA’s requirements for monitoring and improving clinical productivity and efficiency across all VAMCs. We also interviewed VA Central Office officials from the Deputy Undersecretary for Health for Operations and Management —who are responsible for managing and overseeing VAMCs and VISNs—to assess VA Central Office’s role in monitoring and improving clinical productivity and efficiency across VA. At the six VAMCs and their VISNs, we interviewed officials and examined any efforts to monitor clinical productivity and efficiency, any steps taken to identify the causes of low clinical productivity and inefficiency, and any remediation plans developed for addressing low clinical productivity and inefficiency. We assessed VA’s efforts in the context of federal standards for internal control for information and monitoring. 8 6 GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1 (Washington, D.C.: November 1999). Internal control is a process effected by an entity’s oversight body, management, and other personnel that provides reasonable assurance that the objectives of an entity will be achieved. 7 VA Central Office is responsible for managing and overseeing the VA health care system and delegates certain responsibilities to its VISNs. 8 See GAO, Standards for Internal Control in the Federal Government, GAO-14-704G (Washington, D.C.: September 2014). Page 3 GAO-17-480 VA Health Care We also assessed the reliability of the data showing VA providers’ performance on the productivity metrics and VAMCs’ performance on the efficiency models for fiscal year 2015. Specifically, we conducted checks for missing values and outliers and interviewed VA officials knowledgeable about the data, including officials from the six selected VAMCs and OPES. Based on these steps, we identified several limitations with the clinical workload and staffing data that serve as the basis of the metrics and models, as we discuss in the report. However, we determined that the performance data on the metrics and models were sufficiently informative to use as one criterion for selecting VAMCs to visit. We conducted this performance audit from June 2016 to May 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. OPES, within VA Central Office, has developed metrics designed to help Background assess physician providers’ clinical productivity as well as models to assess VAMCs’ overall clinical efficiency. The productivity metrics and efficiency models are also intended to help VAMCs conduct comparisons with their peers and determine resource needs. VA’s productivity metrics and efficiency models are part of VA’s Strategic Analytics for Improvement and Learning Value Model (SAIL), a system for assessing facility performance in terms of quality, access, patient satisfaction, productivity, and efficiency. 9 Clinical Productivity Metrics VA uses its clinical productivity metrics to measure physician providers’ time and effort to deliver individual procedures. VA measures productivity for specified clinical specialties at each medical facility on a bi-weekly basis. These metrics are based on the work-relative value units (wRVU) generated by VA physician providers. A wRVU represents providers’ 9 SAIL uses a star rating system from 1 to 5 to assess each VAMC’s performance relative to other VAMCs. Furthermore, according to VA Central Office officials, VAMC and VISN leadership officials’ performance pay is based, in part, on performance on their facilities’ performance on SAIL metrics. According to these officials, VA’s productivity and efficiency metrics are not included in the calculation of the star rating and performance pay. Page 4 GAO-17-480 VA Health Care workload—defined as the time, mental effort and judgment, technical skill and effort, and stress involved in delivering a health care procedure. wRVU data are automatically calculated based on the procedures providers record in patients’ electronic medical records, and the data are captured in a national database. OPES calculates provider productivity for each clinical specialty in each medical facility on a bi-weekly basis by dividing the number of wRVUs generated by physician providers by the number of full-time employee equivalent (FTEE) physician providers. 10 (See figure 1 for a description of this calculation.) According to an OPES official, OPES began developing the clinical productivity metrics in fiscal year 2013 for 32 different clinical specialties, such as cardiology and surgery. 10 For the purposes of this calculation, the number of FTEE physicians only includes the time physicians allocate to their clinical duties, as distinct from their research, education, and administrative duties. For example, a full-time physician who performs clinical work 4 days per week and research duties 1 day per week would be counted as 0.8 FTEE. Page 5 GAO-17-480 VA Health Care Figure 1: Calculation of the Department of Veterans Affairs’ (VA) Physician-Provider Productivity Metrics for 32 Clinical Specialties a A work-relative value unit represents providers’ workload defined as the time, mental effort and judgment, technical skill, physical effort, and stress involved in delivering a health care procedure. b For the purposes of this calculation, the number of FTEE physician providers only includes the time physicians allocate to their clinical duties, as distinct from their research, education, and administrative duties. For example, a full-time physician who performs clinical work 4 days per week and research duties 1 day per week would be counted as 0.8 FTEE. To measure VAMC providers’ performance in terms of productivity, OPES has developed productivity performance standards for each of the 32 clinical specialties. These standards are based on average productivity levels across all VAMCs from a prior fiscal year. 11 According to VA 11 Productivity performance standards for fiscal years 2014 and 2015 were based on average productivity levels from fiscal year 2013 while standards for fiscal years 2016 and 2017 were based on average productivity levels from fiscal year 2015. Page 6 GAO-17-480 VA Health Care officials, VA uses internal benchmarks to establish productivity performance standards for its VAMCs, rather than basing these standards on external benchmarks, such as average productivity levels in the private sector, which are generally higher than productivity levels for VA providers. Internal benchmarking allows VA to account for characteristics unique to VA, including its goal of improving veterans’ access to specialty care. VAMCs with clinical specialties performing within one standard deviation above or below the standard are considered to be performing within the acceptable range of productivity. In contrast, VAMCs with any clinical specialties performing more than one standard deviation below the applicable productivity performance standard are considered to have low productivity. See table 1 for an overview of these standards for selected clinical specialties for fiscal year 2017. Table 1: Overview of the Department of Veterans Affairs (VA) Fiscal Year 2017 Productivity Performance Standards for Selected Clinical Specialties Productivity performance standards (in terms of work-relative value units) by VA a Clinical Specialty medical center (VAMC) complexity group 1a 1b 1c 2 3 Cardiology 7,627 5,938 5,815 4,508 3,931 Neurology 3,893 3,730 3,731 3,180 3,038 Surgery 4,774 3,832 4,246 3,664 3,388 Source: GAO analysis of VA information | GAO-17-480 a A work-relative value unit represents providers’ workload defined as the time, mental effort and judgment, technical skill, physical effort, and stress involved in delivering a health care procedure. VA assigns each VAMC to one of five complexity groups based on patient population served, clinical services offered, education and research complexity, and administrative complexity. VAMCs assigned to complexity group 1a are the most complex, while those assigned to group 3 are the least complex. VAMCs with clinical specialties performing within one standard deviation above or below the standard are considered to be performing within the acceptable range of productivity. In contrast, VAMCs with any clinical specialties performing more than one standard deviation below the applicable productivity performance standard are considered to have low productivity. VAMCs considered to have low productivity must develop and implement a remediation plan that describes the planned steps taken for addressing the low productivity. Since fiscal year 2015, VA Central Office has Page 7 GAO-17-480 VA Health Care required VAMCs to submit any remediation plans to their respective VISNs for approval. 12 Clinical Efficiency Models VA uses its clinical efficiency models to compare, on an annual basis, each VAMC’s actual utilization and expenditures for health care services with the facility’s expected utilization and expenditures. Specifically, VA Central Office measures clinical efficiency using its Efficiency Opportunity Grid (EOG)—a series of 12 models that focus on overall efficiency and on efficiency for different high volume or high expenditure components of health care services, such as emergency department and urgent care. Under the models, OPES in VA Central Office calculates annually VAMCs’ utilization and expenditures in each of the 12 areas and the extent to which these differ from expected levels, based on the characteristics of each VAMC and the patient population served. The EOG models provide each VAMC with a tailored annual report showing the facility’s overall level of inefficiency—defined as utilization and expenditures that exceed expected levels—which the facilities can use to help identify the specific factors contributing to inefficiency as well as opportunities for improvement. The model calculations are based on data from the previous fiscal year. VA Central Office encourages VAMCs to use these models to monitor clinical efficiency and identify the underlying causes of inefficiency at their facilities. See table 2 for an overview of VA’s efficiency models. 12 In addition, VA requires facilities with any clinical specialties performing below the 25th percentile productivity performance standard or above the 75th percentile to determine whether the workload and staffing data that inform the productivity metrics are correct. For a description of VA’s requirements pertaining to its clinical productivity metrics, see VHA Directive 1065, Productivity and Staffing Guidance for Specialty Provider Group Practice (May 4, 2015). Page 8 GAO-17-480 VA Health Care Table 2: Overview of the Department of Veterans Affairs (VA) 12 Efficiency Opportunity Grid Models Efficiency Model Description Stochastic Frontier Analysis Measures each VA medical center’s (VAMC) overall efficiency—including clinical and administrative efficiency. A score of 1 is equal to the ideal, most efficient score whereas higher scores indicate inefficiency. For example, a score of 1.1 indicates that a VAMC has 10 percent inefficiency compared to the ideal VAMC. Congestive Heart Failure Ambulatory Compares each VAMC’s admissions for congestive heart failure to expected admissions, a Care Sensitive Condition (ACSC) based on a risk-adjusted model. Hospitalizations due to ACSCs such as congestive heart Admission Model failure can be largely avoidable or preventable if primary care is provided in a timely and effective manner. Pneumonia ACSC Admission Model Compares each VAMC’s admissions for pneumonia to the expected number of a admissions, based on a risk-adjusted model. Specialty Care Visit Model Compares each VAMC’s utilization of specialty care services—in terms of the number of a visits—to expected utilization of these services, based on a risk-adjusted model. Medical and Surgical Specialty Encounter Compares each VAMC’s utilization of medical/surgical care services—in terms of the Model number of visits—to expected utilization of these services, based on a risk-adjusted a model. Emergency Department and Urgent Care Compares each VAMC’s utilization of emergency department and urgent care services— Utilization Model in terms of the number of visits per 1,000 patients—to expected utilization, based on a a risk-adjusted model. Acute Bed Day of Care Model Compares each VAMC’s bed days of care (i.e., days patients spend overnight in an inpatient status) for acute care admissions to expected bed days of care, based on a risk- a adjusted model. Administrative Full Time Employee Compares each VAMC’s number of administrative FTEE to expected number of a Equivalent (FTEE) Model administrative FTEE, based on a risk-adjusted model. Pharmacy Expenditure Model Compares each VAMC’s expenditures on drugs to expected expenditures, based on a a risk-adjusted model. Radiology Expenditure Model Compares each VAMC’s expenditures on radiology services—including screening, diagnostic, and treatments—to expected expenditures for these services, based on a risk- a adjusted model. Non-VA Care Expenditure Model Compares each VAMC’s expenditures on care veterans receive from community providers (referred to as non-VA care) to expected expenditures for this care, based on a a risk-adjusted model. End of Life Expenditure Model Compares each VAMC’s expenditures on a per-veteran basis during the veteran’s last six a months of life to expected expenditures, based on a risk-adjusted model. Source: GAO analysis of VA information | GAO-17-480 a Each efficiency model is risk-adjusted, meaning that it takes into account factors that may contribute to higher or lower utilization and expenditures that VAMCs cannot control, such as regional variations in labor costs. A VAMC is deemed to be inefficient if its actual expenditures or utilization exceed the expected expenditures or utilization. Page 9 GAO-17-480 VA Health Care Clinical Productivity and To help VAMCs monitor and take the necessary steps to improve clinical Efficiency Monitoring productivity and efficiency, in 2016, VA Central Office began requiring each VAMC to hire at least one group practice manager. 13 Group practice managers are responsible for analyzing data, including those related to productivity and efficiency, and taking steps to ensure VAMCs’ outpatient clinics are set up in a way that enhances access to care for veterans. To do this, group practice managers work with clinical service line leaders, such as the Chief of Surgery, and administrative staff. VA has developed a series of datasets and reports that allow VAMC staff, including group practice managers and clinical service line leaders, to examine potential factors contributing to sub-optimal clinical productivity and inefficiency identified through VA’s metrics and models. For example, VA has developed datasets that include data on productivity and other workload measures for each provider, as well as the number of wRVUs generated by each procedure. In addition, VA has developed reports on clinic utilization, which include information on the number of unfilled appointment slots and no-shows (e.g., when a patient does not attend a scheduled appointment), which can lead to lower productivity and inefficiency. We found that VA’s metrics and models may not provide complete and VA’s Metrics and accurate information on clinical productivity and efficiency at VAMCs. Our Models May Not findings concerning VA’s productivity metrics are similar to those of an independent assessment of VA’s health care delivery systems and Provide Complete management processes issued in September 2015, which also identified and Accurate concerns about the completeness and accuracy of these metrics. 14 Not having quality information runs counter to federal standards for internal Information on control related to information, which call for management to use timely Clinical Productivity and quality information to achieve the agency’s objectives. To the extent that VA’s productivity metrics and efficiency models do not provide and VAMC Efficiency complete and accurate information, they may misrepresent the true level of productivity and efficiency across VAMCs and limit VA’s ability to 13 See VHA Directive 1231, Outpatient Clinic Practice Management (November 15, 2016). 14 See RAND Corporation. Assessment B (Health Care Capabilities). (Santa Monica, CA: September 1, 2015). Page 10 GAO-17-480 VA Health Care determine the extent to which its resources are being used effectively to provide health care services to veterans. 15 Our review of VA’s national data and information we collected from our site visits identified several limitations in the data that inform VA’s productivity metrics and efficiency models. Specifically, we found the following limitations: Productivity metrics are not complete because they do not account for all providers or clinical services. VA’s productivity metrics do not cover all types of providers who deliver care at VAMCs, nor do the metrics capture all types of care. Specifically, the metrics do not capture data for • providers who are not employed by VA, such as contract physicians; • providers who work in multiple specialties (e.g., a provider who performs care in both the dermatology and pathology specialties); • advanced practice providers, such as nurse practitioners, acting as the sole provider; and • certain types of inpatient care delivered by all providers. 16 During our site visits, VAMC officials described the limitations created by the exclusion of certain providers from VA’s productivity metrics. For example, VAMC officials told us that excluding contract providers who are not employed by VA limits VA’s ability to accurately measure the clinical productivity of specialties that rely on these types of providers. VAMC officials explained that VAMCs often contract with outside providers to 15 For example, based on our review of fiscal year 2015 data, we found significant variation in clinical specialties’ productivity among VAMCs within the same complexity group, which may be partly attributed to incomplete and inaccurate information rather than true differences in productivity. Specifically, for the cardiology specialty at VA’s most complex facilities (i.e., group 1A facilities), the highest performing VAMC was more than six times as productive as the lowest performing VAMC. 16 VAMCs utilize contract physicians who are not employed by VA to help supplement the number of physicians employed by the VAMC. These contract physicians either provide services that the VAMC does not currently offer—such as highly specialized procedures— or they supplement the capacity of the VAMC by providing additional physicians to treat veterans. Advanced practice providers are clinical professionals who perform similar duties as physicians. In many states, nurse practitioners are licensed to diagnose and treat patients without physician involvement, whereas in other states they are licensed to practice under the supervision of a physician. Page 11 GAO-17-480 VA Health Care deliver highly-specialized procedures—such as complex surgeries—that generate a high number of wRVUs. For example, at one of the six VAMCs we visited, officials told us that that complex neurosurgery and general surgery procedures were performed by contract physicians, who had the requisite expertise to perform these procedures, while VA surgeons performed relatively less-complex procedures. With the workload performed by contract providers not employed by VA excluded, the clinical productivity of specialties that rely on these providers appears to be lower than it otherwise would be if these providers were included in the metrics, VA officials said. VA Central Office officials explained how VA data system limitations and other factors have made it difficult for VA’s productivity metrics to capture the workload for all types of providers. Specifically, • According to the officials, while VA captures wRVU data for providers who are not VA employees, it is presently unable to calculate the clinical productivity for these providers, because VA’s staffing system only captures clinical FTEE data for VA physician providers. (As described earlier, clinical FTEE data are necessary for calculating providers’ clinical productivity.) According to VA data, in fiscal year 2015, about 10 percent of all wRVUs across VA were generated by providers not employed by VA, although some clinical specialties at certain VAMCs relied entirely or almost entirely on non-VA employed providers. • In addition, VA Central Office and VAMC officials explained that VA’s data system does not allow VA providers to be assigned to—and have their workload recorded for—more than one clinical specialty. As a result, VA’s productivity metrics only include partial workload data for providers who work in multiple specialties. VA Central Office officials told us that this limitation only affects a small proportion of providers, as most providers practice only within one clinical specialty. • Furthermore, while VA captures wRVU data for advanced practice providers who act as the sole provider, VA’s productivity metrics exclude these providers because VA is not always able to assign advance practice providers to particular clinical specialties as it does for physicians, according to VA officials. 17 As a result, officials said 17 VA Central Office officials also stated that provider taxonomies in the National Plan & Provider Enumeration System, which was developed by the Department of Health and Human Services and assigns unique identifiers to providers, has limited specialty designations for advanced practice providers. Page 12 GAO-17-480 VA Health Care that VA has not developed productivity performance standards for these providers; however, according to these officials, VA intends to improve VAMCs ability to monitor the productivity of advanced practice providers in the future. According to VA data, in fiscal year 2015, about 10 percent of all wRVUs were generated by advanced practice providers VA-wide. 18 In addition to not capturing all providers, VA Central Office officials explained that VA’s data systems are not designed to fully capture providers’ workload delivering inpatient services that do not involve procedures—in particular, evaluating and managing patients who are hospitalized. As a result, VA does not require providers to record this workload in VA’s electronic medical records. 19 Productivity metrics may not accurately reflect the intensity of clinical workload. An internal VA audit conducted in 2016 found that across VA’s 168 VAMCs, providers did not always accurately code the intensity of their clinical workload—that is, the amount of effort needed to deliver the procedures or services they perform. 20 In June 2016, VA Central Office began requiring VAMCs to audit a sample of every provider’s clinical coding at least once annually and report the accuracy of providers’ coding to VA Central Office. 21 According to VA officials, the data show that between June and December 2016 providers across VA coded only 64 to 69 percent of their procedures accurately. More 18 In addition, VA’s productivity metrics do not capture workload performed by medical residents, who are incorrectly listed as the primary provider in veterans’ medical records. VA policy requires that a medical resident’s supervising provider be listed as the primary provider in the medical record; however, VA data show that in fiscal year 2015, about 1 percent of all wRVUs across VA were generated by medical residents incorrectly listed as the primary provider. 19 VA Central Office officials stated that, on average, approximately 13 percent of physician providers’ clinical time is dedicated to inpatient workload. Time spent performing inpatient workload is excluded from VA’s productivity metric calculations and therefore does not have an impact on VA’s productivity metrics. 20 Coding is the process of assigning a current procedural terminology (CPT) code for the clinical procedures or services providers perform. Each clinical procedure or service may have several different CPT codes providers can choose to denote the level intensity of the procedure or service. For example, there are five CPT codes corresponding to outpatient office visits for established patients. Each CPT code is associated with a defined number of wRVUs. Providers must include in veterans’ electronic medical records documentation that supports the CPT codes they choose. 21 For these audits, VAMCs reviewed clinical documentation to determine if providers selected the appropriate CPT code. Page 13 GAO-17-480 VA Health Care specifically, VA’s internal audit found that the coding accuracy is even lower—about 60 percent VA-wide—for evaluation and management procedures, which are the most commonly performed types of procedures, according to VA officials. Because of these inaccuracies, VA’s productivity metrics may not accurately reflect provider productivity, as differences in the number of wRVUs providers generate may represent coding inaccuracies rather than true productivity differences. For example, if a provider mistakenly chooses a code for an outpatient office visit that is one intensity level higher than the code chosen by another provider for the same service, the number of wRVUs generated by the two providers will vary by about 50 percent, even though they both performed the same service. We could not quantify the effect of these clinical coding inaccuracies on each of VA’s productivity metrics without conducting an audit of each provider’s clinical coding across VAMCs. During our site visits, VAMC officials identified two key reasons for coding inaccuracies. • Officials at four of the six VAMCs attributed coding errors to providers’ lack of training on coding clinical procedures. According to officials from one of these VAMCs, VA has not emphasized the importance of accurate clinical coding, as only a small portion of the procedures VAMCs perform are submitted to insurance companies for reimbursement. 22 As a result, according to these officials, some VA providers do not fully understand the importance of accurate clinical coding. • Furthermore, officials at three of the six VAMCs attributed coding errors to providers’ lack of time for carefully coding procedures, which the officials said makes the providers more susceptible to errors. To improve coding accuracy, VA Central Office has provided guidelines for VAMCs to implement Clinical Documentation Improvement (CDI) programs and hire CDI specialists to provide coding training to providers. VA officials reported that as of the end of 2016, 84 VAMCs had implemented a CDI program, and 66 of these VAMC’s had hired staff dedicated to such a program. However, officials from VA Central Office told us that VA does not require that VAMCs implement a CDI program because doing so typically requires VAMCs to hire additional staff, and these decisions are the purview of VAMCs. Officials at three of the five 22 Under certain conditions, VA receives third-party payments from veterans’ private health care insurance for the treatment of non-service connected conditions. Page 14 GAO-17-480 VA Health Care VAMCs we visited that had implemented a CDI program cited success in their efforts to improve coding accuracy. For example, officials at one of these VAMCs told us that they improved provider coding accuracy by issuing a monthly CDI newsletter containing guidance on good coding practices. Officials at another VAMC told us that they posted a color- coded procedural coding chart in every exam room to help providers more accurately code the intensity of clinical procedures and services. These officials reported an improvement in coding accuracy from 24 percent to 65 percent at their facility since they started implementing these efforts. Productivity metrics may not accurately reflect providers’ clinical staffing levels. We found that providers do not always accurately record in VA’s Managerial Cost Accounting System the amount of time they allocate to clinical duties, as distinct from research, education, and administrative duties. 23 As we have reported, VA’s productivity metrics are calculated for providers’ clinical duties only. However, officials at five of the six VAMCs we visited reported that clinical FTEE data are not always accurate due to errors in how providers record the amount of time they have allocated to various duties. Furthermore, several remediation plans for addressing low clinical productivity we reviewed identified inaccurate clinical staffing data.VA Central Office officials attributed the inaccuracies to the manual process used to record staffing data in VA’s Managerial Cost Accounting System, and officials from some VAMCs attributed the inaccuracies to unfamiliarity with the criteria used to categorize the aforementioned duties. According to VA policy, all activities related to patient care, not just face-to-face time with patients, should be categorized as clinical duties. While we were unable to assess the extent to which inaccuracies in provider staffing level data exist nationally—and by extension, quantify the effect of such errors on VA’s productivity metrics—fiscal year 2015 VA data show substantial variation across all VAMCs in the percentage of time allocated to administrative duties. This variation may indicate that VAMC providers are not categorizing their duties consistently. For example, among the most complex VAMCs, the percentage of time allocated to administrative duties ranged from 7 23 VA refers to the allocation of providers’ time among these duties as “labor mapping.” Labor mapping allocations are recorded in VA’s Managerial Cost Accounting System, which captures expenditure, workload, and utilization data VA-wide. Page 15 GAO-17-480 VA Health Care percent to about 17 percent in fiscal year 2015. The level of variation was typically even greater among less complex VAMCs. 24 To help address inaccuracies in recording provider staffing levels, VA Central Office issued a more detailed policy for all VAMCs to help them accurately capture providers’ clinical FTEE data. 25 This policy specified which types of activities are considered to be clinical, research, education, and administrative duties. The policy also specified the maximum amount of time that providers may allocate to various administrative assignments, such as serving as the Chief of Staff, a service chief, or a member of a committee. 26 VA Central Office has also developed a tool VAMCs can use to identify incorrect clinical staffing levels. Officials at four of the six VAMCs we visited stated that they were taking steps to help ensure data accuracy going forward. For example, officials at one VAMC stated that they review data on provider FTEE levels on a monthly basis and examine the extent to which unusually high or low performance on VA’s productivity metrics may have been caused by inaccurate categorization of FTEE data. Officials from another VAMC stated that when they identified inaccurate FTEE data, they made immediate corrections to the allocation of physicians’ time to clinical, research, education, and administrative workload. However, the officials stated that inaccurate FTEE data could not be corrected retroactively; therefore, any resulting inaccuracies in VA’s productivity metrics could also not be addressed retroactively. Productivity metrics may not accurately reflect providers’ clinical specialty assignments. Because VA’s clinical productivity metrics are calculated for 32 different clinical specialties, it is important that VA’s data 24 OPES officials told us that OPES generates a monthly report that identifies changes in the percentage of time providers allocate to clinical, research, education, and administrative duties at each VAMC. While these officials stated that the allocations have remained stable over time across the VA health care system as a whole, we found that the portion of providers’ time allocated to administrative duties has increased by about 10 percent VA-wide between fiscal years 2014 and 2016 and by over 20 percent at 48 VAMCs. 25 See VHA Handbook 1065.01, Productivity and Staffing Guidance for Specialty Provider Group Practice (May 4, 2015). 26 For example, VA’s guidance specifies that a VAMC chief of staff may allocate up to 100 percent of their time to administrative duties, while service chiefs, such as chiefs of surgery, may allocate up to 70 percent of their time to administrative duties. Page 16 GAO-17-480 VA Health Care systems assign providers to the correct clinical specialty. 27 However, officials from three of the six VAMCs we visited reported cases in which providers were assigned to the wrong clinical specialty, which resulted in inaccuracies in VA’s productivity metrics. For example, at one VAMC, officials told us that a provider who was performing compensation and pension examinations for veterans seeking VA disability benefits was erroneously assigned to the surgery specialty in VA’s data system. These officials attributed this error to the fact that the provider was a board certified surgeon and therefore was likely assigned to the surgery specialty by the VAMC’s human resources department. To identify similar errors, officials from this VAMC reviewed all provider assignments to clinical specialties in 2016 and made needed corrections. We could not determine the extent to which providers were assigned to the wrong clinical specialty VA-wide, and therefore we were unable to quantify the effect of such errors on VA’s productivity metrics. Efficiency models may also be adversely affected by inaccurate workload and staffing data. The inaccuracies in VA’s clinical productivity metrics we have identified—inaccurate coding as well as inaccurate clinical FTEE data—may also adversely affect the accuracy of some of VA’s clinical efficiency models. For example, the model that examines administrative efficiency requires accurate data on the amount of time VA providers spend on administrative tasks; if the time physicians allocate to clinical, research, education, and administrative tasks is incorrect, the metric will overstate or understate administrative efficiency. Similarly, the inaccurate coding of veterans’ medical conditions can impact VA’s efficiency models that examine admissions for ambulatory care sensitive conditions. 28 For example, if providers inaccurately document the severity of the condition of patients, the model may understate or overstate the expected and actual number of admissions for these conditions. VA Central Office officials told us that the statistical analyses used to develop its models are robust; as a result, the reliability of the models is not significantly impacted by inaccuracies in the underlying data, so long as the inaccuracies are evenly distributed among 27 VA refers to the assignment of providers to clinical specialties as “person class” assignments. 28 In addition to coding clinical procedures or services, providers assign patients a diagnosis code, which reflects the medical condition or conditions that a patient is experiencing. VA uses the tenth revision of the International Statistical Classification of Diseases and Related Health Problems, the standard code set used in the United States for documenting patient medical diagnoses and inpatient medical procedures. Page 17 GAO-17-480 VA Health Care VAMCs. However, officials also said that to the extent that there is significant variation in the quality of the underlying data across VAMCs, the models may produce erroneous results. VA Central Office Has Taken Steps to Help VAMCs Monitor and Improve Clinical Productivity, but Does Not Systematically Oversee Productivity and Efficiency across VA VA Central Office Is VA Central Office has recently taken steps to help VAMCs in their Developing a ongoing efforts to monitor VA’s clinical productivity metrics and identify and address any drivers of low productivity at their facilities. 29 In their Comprehensive Analytical monitoring efforts, each of the six VAMCs we visited use a range of Tool to Help VAMCs with available data, including data on provider workload and clinic utilization— Ongoing Monitoring and such as the length of appointment slots, the number of unfilled slots, and Efforts to Improve Clinical the number of missed appointments—in conjunction with data collected Productivity under VA’s productivity metrics. Based on their analysis of VA’s clinical productivity metrics and the factors contributing to suboptimal clinical productivity, the six VAMCs have taken steps to improve productivity. In addition to addressing inaccurate clinical workload and staffing data, in general, these steps have focused on aligning veterans’ demand for clinical services with the availability of services. For example, • Officials from two of the six VAMCs told us that they have created additional appointment slots by decreasing the length of time for appointments—e.g., from 45 minutes to 30 minutes—to accommodate an increased number of patients seeking care in particular clinical 29 VA policy also requires VAMCs to examine the drivers of unusually high clinical productivity identified by VA’s metrics and to identify potential inaccuracies in the underlying data used to calculate these metrics. Page 18 GAO-17-480 VA Health Care specialties. At one VAMC, for example, officials told us that they were able to decrease the length of appointments by allocating additional exam rooms for patients receiving care in the podiatry specialty, which was experiencing relatively high levels of demand. As a result, officials said that the VAMC was able to provide care to more veterans. • Officials from two VAMCs told us that, in order to meet the demand for certain types of services, they are using providers from specialties with relatively lower demand who are also qualified to deliver care in other specialties. For example, officials at one VAMC told us that to free up orthopedic surgeons’ time to perform a greater number of surgeries, they transferred joint injections—which has been performed by orthopedic surgeons—to the rheumatology specialty, which had excess capacity. • Officials from two VAMCs told us that they leveraged partnerships between their facilities and community providers or military health care facilities to improve veterans’ access to care and providers’ productivity. 30 For example, at one facility officials referred veterans requiring foot nail care—a low-cost service that generates a relatively low number of wRVUs—to community providers, which allowed the facility to free up providers’ capacity to perform more complex surgical procedures in its podiatry service. • Officials from two VAMCs told us that they began using telehealth services to deliver care when a clinical specialty has excess capacity to provide care. 31 For example, officials at one VAMC established agreements with other facilities within the VISN that enabled an infectious diseases physician to deliver care at those facilities using telehealth services. While the six VAMCs have taken steps to improve productivity, officials at these VAMCs told us it can be challenging to identify the underlying causes of low productivity, due to the large amount and complexity of data that could be used for this purpose. For example, officials at one VAMC said that a VA web-based portal was available that provided 30 VA may also authorize care through community providers to meet the needs of the veterans it serves, such as when a VA facility is unable to provide certain specialty care services. Services provided by community providers must generally be authorized by a VAMC provider prior to a veteran receiving care. 31 For this report, we define telehealth as clinical services that are provided remotely via telecommunications technologies. Page 19 GAO-17-480 VA Health Care access to multiple datasets related to productivity, but the officials did not know how to effectively analyze the data. As a result of these challenges, the data approaches used by VAMCs to identify and address the drivers of low productivity have varied. For example, officials at some VAMCs told us that they were in the process of developing customized analytical tools, referred to as data “dashboards,” which allow VAMCs to perform in- depth analyses of the data. Officials at other VAMCs told us that they conducted more limited analyses using spreadsheets. To address the data challenges VAMCs face in improving clinical productivity, VA Central Office began developing a comprehensive analytical tool in December 2016 to help VAMCs improve and monitor clinical productivity. According to VA Central Office officials, the comprehensive analytical tool VA is developing—in the form of a data dashboard—is intended to help address these challenges by centralizing relevant data elements, including data on clinic utilization, veterans’ access to care, and provider workload, and thereby allow VAMC officials, including group practice managers and clinical service line leaders, to more easily examine factors contributing to low productivity. Furthermore, officials stated that VAMCs will be able to customize this data dashboard tool to fit their specific needs. In developing this analytical tool, VA Central Office officials solicited input from VAMC group practice managers and reviewed 21 analytical tools developed by individual VAMCs. VA Central Office officials told us that they expect the new data dashboard to eliminate the need for VAMCs to develop their own data approaches for identifying the drivers of low clinical productivity, which in turn will help ensure consistency in identifying and correcting sub-optimal productivity VA-wide. These officials also told us that they expect the data dashboard to be developed piecemeal and rolled out to all VAMCs and VISNs over the course of 2017. VA Central Office Does While VA Central Office officials encourage VAMCs to monitor VA’s Not Require VAMCs to efficiency models and take steps to improve clinical inefficiency at their facilities, VA does not require VAMCs to monitor these models and Monitor Efficiency Models address any inefficiencies identified by them. For example, VA has not or Address Inefficiencies established performance standards based on these models and does not Identified by Them require remediation plans to address inefficiency. 32 VA Central Office 32 As described in this report, VA’s efficiency models are used to track VAMC utilization and expenditures for various health care services and compare these expenditures to expected levels. Page 20 GAO-17-480 VA Health Care officials we interviewed stated that they have not required VAMCs to monitor these models and address any inefficiencies because they view the models solely as a tool to guide VAMCs in managing their resources. Federal standards for internal control for information and monitoring state that management should assess the quality of their performance over time and evaluate monitoring efforts to help ensure they meet their strategic goals. Without a requirement to assess, monitor, and, if necessary, improve VAMCs’ clinical efficiency, VA increases its risk that the drivers of clinical inefficiency will not be identified and appropriately addressed. In the absence of a monitoring requirement, we found that some VAMCs we visited had not taken steps to address inefficiencies identified by VA’s efficiency models. While officials from each of the six VAMCs we visited told us that they were aware of their facility’s performance on VA’s models, officials at two VAMCs told us that they had not taken steps to identify and then address the drivers of inefficiencies identified by these models. Officials at both of these VAMCs stated that they did not find the models useful for monitoring purposes, as they viewed the use of the models as “overly theoretical” and as such not helpful for identifying specific ways to address inefficiencies. Furthermore, officials at one of these VAMCs—which officials say was experiencing high levels of staff turnover—stated that they lacked the time to identify and address the drivers of inefficiencies identified through these models. In contrast, officials from the four VAMCs that were using VA’s efficiency models to identify and address inefficiencies reported that they changed their clinical processes to address identified inefficiencies. For example, officials from one of the VAMCs told us that the VAMC had conducted an analysis to determine the factors contributing to high utilization of services for congestive heart failure and pneumonia and, based on this analysis, identified a need for better follow-up with patients discharged from the facility. Among other steps, the VAMC is using telehealth services to manage these patients and prevent inpatient readmissions. At another VAMC, officials told us that they are addressing an inefficiency identified by the Emergency Department and Urgent Care Utilization EOG model by increasing the number of primary care providers and support staff, in an effort to reduce the number of emergency department visits. VAMC and VISN officials said that the usefulness of these models for monitoring and improving clinical efficiency could be improved through more frequent reporting of the models’ data. For example, officials from three of the six VAMCs—including one of the VAMCs that had not taken Page 21 GAO-17-480 VA Health Care steps to address the drivers of clinical inefficiency—reported that VA’s efficiency models are only somewhat useful because the models are not timely, as they are only released annually. According to these officials, this serves as a limitation because 1) the data may show evidence of an inefficiency that has already been addressed or 2) a VAMC may address an inefficiency in one year and have to wait until the next year before there are available data showing whether the effort has been successful. An OPES official we spoke with acknowledged these concerns which had been previously shared with OPES by VAMC officials. In response, OPES developed additional training materials to help VAMCs better understand how these models can be used to identify and address inefficiency. In particular, OPES officials stated that—while the efficiency models themselves are only released annually—most of the underlying data that inform these models are regularly updated, which allows VAMCs to examine trends in utilization and expenditures that impact efficiency throughout the year. VA Central Office Does While VAMCs monitor and may take steps to improve clinical productivity Not Systematically and, in some cases, efficiency, VA Central office does not have an ongoing process to systematically oversee these efforts. As we have Oversee VAMCs’ Efforts to discussed in this report, VA policy requires VAMCs to develop Monitor and Improve remediation plans to address low productivity in their clinical specialties Productivity and Efficiency and submit these plans to their VISN. Based on our analysis, we found across VA that the VAMCs we reviewed had developed the required remediation plans and officials from these VAMCs stated that they submitted the plans to their respective VISNs for review. 33 However, VA’s policy does not stipulate that VAMCs or VISNs are to submit approved remediation plans to VA Central Office; nor does the policy stipulate that VISNs or VA Central Office must monitor the implementation of these remediation plans to ensure their success. As a result, for example, officials at one of the VISNs we interviewed told us the VISN does not monitor the implementation of VAMCs’ remediation plans to address low productivity. In the case of clinical efficiency, as we noted above, VAMCs are not required to monitor their efficiency and as a result, they do not develop remediation plans to address inefficiencies identified by VA’s efficiency models. Federal standards for internal control related to monitoring call for management to establish and operate monitoring activities to evaluate 33 Three of the six VAMCs we reviewed were required to develop remediation plans. Page 22 GAO-17-480 VA Health Care whether a specific function or process is operating effectively and take corrective actions as necessary. OPES officials told us that they monitor productivity metric and efficiency model data VA-wide. However, because VA Central Office does not monitor VAMCs’ clinical productivity remediation plans and does not require remediation plans for addressing clinical inefficiency, it cannot systematically identify best practices VAMCs have developed for addressing low productivity and inefficiency as well as determine the factors VAMCs commonly identify as contributing to low productivity and inefficiency and have not addressed. 34 For example, at the VAMCs we visited, officials told us that they face numerous staffing and infrastructure challenges that officials say have impeded productivity and efficiency. Specifically, these challenges include the following: • Staffing challenges. Officials at three of the six VAMCs we visited reported that a lack of support staff has impeded their efforts to improve clinical productivity. In particular, officials from one of these VAMCs stated that a lack of medical support assistants, who are responsible for scheduling appointments and other administrative tasks, led to lower clinical productivity, because veterans’ appointments were not being scheduled in a timely manner—leading to unfilled appointment slots. • Infrastructure challenges. Officials at four of the six VAMCs we visited reported that infrastructure challenges, including a lack of exam or procedure rooms and inadequately equipped facilities, have impeded facility efforts to address suboptimal productivity and inefficiency. For example, at one VAMC, the lack of an infusion center for administering chemotherapy treatments has limited the facility’s ability to improve the productivity within the Hematology Oncology specialty. Instead of performing chemotherapy procedures at the facility, it referred patients requiring these procedures to community providers. Another VAMC was limited in its ability to improve the clinical productivity of its surgery specialty because it lacked updated software used to plan and perform certain complex neurosurgery procedures. 34 In its 2012 report, the VA OIG noted that information on productivity can help VA identify best practices and those practices that should be changed or eliminated. See VA OIG, Veterans Health Administration: Audit of Physician Staffing Levels for Specialty Care Services. 11-01827-36 (Washington, D.C.: Dec. 27, 2012). Page 23 GAO-17-480 VA Health Care VA’s productivity metrics and efficiency models can provide VA with Conclusions important information on the clinical productivity of its providers and the efficiency of health care utilization and expenditures at VAMCs. However, our review shows that VA’s productivity metrics do not capture information on all providers who deliver care at VAMCs and certain types of inpatient care and that both VA’s productivity metrics and its efficiency models may be compromised by inaccuracies in their underlying data. While VA has taken some steps to improve the reliability of the data that inform its metrics and models—such as auditing these data for accuracy and issuing a more detailed policy for recording provider staffing levels— –concerns about their reliability persist. Using incomplete and potentially inaccurate data on clinical productivity and efficiency at VAMCs could prevent VAMCs from identifying the true causes of suboptimal productivity and efficiency and deter VAMCs from pursuing effective solutions to them. Ultimately, without better productivity and efficiency data, VA may be missing opportunities to both maximize the use of its resources and provide greater health care access and services to veterans. Likewise, providing strong oversight by VA Central Office is vital to better ensure that all VAMCs have the right analytical tools, the ability to identify and address suboptimal clinical productivity and efficiency, and are fully engaged in addressing identified problems and implementing effective solutions. While VA Central Office is in the process of developing a comprehensive analytical tool to help VAMCs in their ongoing efforts to monitor and, if necessary, address the drivers of low clinical productivity, the use of VA’s efficiency models appears to be more limited. VA does not require VAMCs to use VA’s efficiency models to monitor and address clinical inefficiencies. Even if all VAMCs used VA’s productivity metrics and efficiency models to monitor and address the drivers of low productivity and inefficiency, VA Central Office does not collect information on these efforts, ensure that they have been implemented and are operating successfully, and position itself to monitor trends across VA. Just as having reliable data is important to VAMCs to improve their clinical productivity and efficiency, so too is having a robust oversight process to help and encourage VAMCs to make productivity and efficiency gains. Achieving such gains will better ensure that VA is using its resources wisely and maximizing the health care services it provides for veterans. Page 24 GAO-17-480 VA Health Care To improve the completeness and accuracy of VA’s productivity metrics Recommendations for and efficiency models and strengthen the monitoring of clinical Executive Action productivity and efficiency VA-wide, we recommend that the Secretary of the Department of Veterans Affairs direct the Undersecretary for Health to take the following four actions: 1. expand existing productivity metrics to track the productivity of all providers of care to veterans by, for example, including contract physicians who are not VA employees as well as advance practice providers acting as sole providers; 2. help ensure the accuracy of underlying staffing and workload data by, for example, developing training to all providers on coding clinical procedures; 3. develop a policy requiring VAMCs to monitor and improve clinical efficiency through a standard process, such as establishing performance standards based on VA’s efficiency models and developing a remediation plan for addressing clinical inefficiency; and 4. establish an ongoing process to systematically review VAMCs’ remediation plans and ensure that VAMCs and VISNs are successfully implementing remediation plans for addressing low clinical productivity and inefficiency. VA provided written comments on a draft of this report, which we have Agency Comments reprinted in appendix I, and provided technical comments, which we have and Our Evaluation incorporated as appropriate. In its comments, VA concurred with our fourth recommendation to establish an ongoing process to systematically review VAMCs’ remediation plans and described the efforts it plans to take to address our recommendation. In addition, VA concurred in principle with our remaining three recommendations, citing efforts already undertaken or plans already in place to address them. However, for two of these three recommendations, VA’s existing efforts or plans may not fully address our recommendations. In particular, • To address our first recommendation to expand existing productivity metrics, VA stated that it plans to establish productivity performance standards for advanced practice providers, using available productivity data, by October 2017. In its response, however, VA did not provide information on whether it plans to expand its productivity metrics to include providers who are not employed by VA, such as contract physicians. As we Page 25 GAO-17-480 VA Health Care recommended, VA should capture information on these providers in its productivity metrics to help ensure that the metrics provide complete information on all providers’ clinical productivity. • In response to our second recommendation that VA help ensure the underlying accuracy of staffing and workload data, VA reiterated its existing efforts to improve clinical coding accuracy and stated that the department would reissue existing policy to VAMCs by June 2017 as well as continue to provide need-based, focused coding training to providers, as appropriate. However, VA did not provide information on how it plans to improve the accuracy of provider staffing data, which inform VA’s productivity and efficiency metrics and models. As we recommended, VA should take steps to ensure that its staffing data are accurate, which in turn will improve the accuracy VA’s efficiency metrics and models. VA also commented that our report does not provide context for variation in productivity as evidenced through external benchmarks. In response to this comment, we incorporated a brief discussion on internal versus external benchmarking in the background section of this report. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of the Department of Veterans Affairs, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or williamsonr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II. Randall B. Williamson Director, Health Care Page 26 GAO-17-480 VA Health Care List of Requesters The Honorable Johnny Isakson Chairman The Honorable Jon Tester Ranking Member Committee on Veterans Affairs United States Senate The Honorable Phil Roe Chairman The Honorable Tim Walz Ranking Member Committee on Veterans Affairs House of Representatives The Honorable Derek Kilmer House of Representatives The Honorable Mark Takano House of Representatives Page 27 GAO-17-480 VA Health Care Appendix I: Comments from the Department Appendix I: Comments from the Department of Veterans Affairs of Veterans Affairs Page 28 GAO-17-480 VA Health Care Appendix I: Comments from the Department of Veterans Affairs Page 29 GAO-17-480 VA Health Care Appendix I: Comments from the Department of Veterans Affairs Page 30 GAO-17-480 VA Health Care Appendix I: Comments from the Department of Veterans Affairs Page 31 GAO-17-480 VA Health Care Appendix II: GAO Contact and Staff Appendix II: GAO Contact and Staff Acknowledgments Acknowledgments Randall B. Williamson, (202) 512-7114 or williamsonr@gao.gov GAO Contact In addition to the contact named above, Rashmi Agarwal, Assistant Staff Director; Michael Zose, Analyst in Charge; Krister Friday; Hannah Grow; Acknowledgments Jacquelyn Hamilton; and Amanda Pusey made key contributions to this report. Page 32 GAO-17-480 VA Health Care Related GAO Products Related GAO Products Federal Health Care Center: VA and DOD Need to Develop Better Information to Monitor Operations and Improve Efficiency. GAO-17-197 (Washington, D.C.: Jan. 23, 2017). Veterans Health Care: Improvements Needed in Operationalizing Strategic Goals and Objectives.GAO-17-50 (Washington, D.C.: Oct. 21, 2016). Veterans Affairs Contracting: Improvements in Policies and Processes Could Yield Cost Savings and Efficiency. GAO-16-810 (Washington, D.C.: Sep. 16, 2016). VA’s Health Care Budget: In Response to a Projected Funding Gap in Fiscal Year 2015, VA Has Made Efforts to Better Manage Future Budgets. GAO-16-584 (Washington, D.C.: Jun. 3, 2016). VA Primary Care: Improved Oversight Needed to Better Ensure Timely Access and Efficient Delivery of Care. GAO-16-83 (Washington, D.C.: Oct 8, 2015). VA Health Care Management and Oversight of Consult Process Need Improvement to Help Ensure Veterans Receive Timely Outpatient Specialty Care. GAO-14-808 (Washington, D.C.: Sep. 30, 2014). VA Health Care: Methodology for Estimating and Process for Tracking Savings Need Improvement. GAO-12-305 (Washington, D.C.: Feb. 27, 2012). (100908) Page 33 GAO-17-480 VA Health Care The Government Accountability Office, the audit, evaluation, and investigative GAO’s Mission arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no cost is Obtaining Copies of through GAO’s website (http://www.gao.gov). Each weekday afternoon, GAO GAO Reports and posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to http://www.gao.gov Testimony and select “E-mail Updates.” Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, http://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, LinkedIn, Twitter, and YouTube. Connect with GAO Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. Visit GAO on the web at www.gao.gov and read The Watchblog. Contact: To Report Fraud, Website: http://www.gao.gov/fraudnet/fraudnet.htm Waste, and Abuse in E-mail: fraudnet@gao.gov Federal Programs Automated answering system: (800) 424-5454 or (202) 512-7470 Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400, Congressional U.S. Government Accountability Office, 441 G Street NW, Room 7125, Relations Washington, DC 20548 Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707 Strategic Planning and U.S. Government Accountability Office, 441 G Street NW, Room 7814, External Liaison Washington, DC 20548 Please Print on Recycled Paper.