United States Government Accountability Office Testimony Committee on Veterans’ Affairs, House of Representatives VA DISABILITY For Release on Delivery Expected at 10:00 a.m. ET Wednesday, December 12, BENEFITS 2018 Planning Gaps Could Impede Readiness for Successful Appeals Implementation Statement of Elizabeth H. Curda, Director, Education, Workforce, and Income Security GAO-19-272T December 12, 2018 VA DISABILITY BENEFITS Planning Gaps Could Impede Readiness for Successful Appeals Implementation Highlights of GAO-19-272T, testimony before the Committee on Veterans' Affairs, House of Representatives Why GAO Did This Study What GAO Found VA’s disability compensation program In a March 2018 report, GAO made four recommendations to address planning pays cash benefits to veterans with gaps in the Department of Veterans Affairs’ (VA) November 2017 plan for disabilities connected to their military changing its appeals process for disability compensation claims. Since then, VA service. In recent years, veterans who has updated its appeals reform plan and taken steps to address aspects of these appealed VA decisions on their claims recommendations, but further steps could enhance its readiness for have waited an average of 3 years. implementation: The subset of appeals resolved by the Board of Veterans Appeals—a • Address all legally required elements. VA’s November 2017 plan did not separate VA agency that provides a address one and only partially addressed four of 22 elements required by the higher level of appeals review—took Veterans Appeals Improvement and Modernization Act of 2017 (Act); GAO on average 7 years to resolve. recommended VA fully address all 22. As of November 2018, VA addressed The Veterans Appeals Improvement one element related to projecting productivity and took steps to partially and Modernization Act of 2017 makes address the other four. VA is still missing information the agency needs to changes to VA’s current (legacy) certify that it has the resources needed to successfully implement appeals process, giving veterans options to reform. have their claims reviewed by VA or to appeal directly to the Board. The Act • Articulate plans for performance monitoring and assessment. GAO requires VA to submit to Congress and recommended VA clearly articulate how it will monitor and assess the new GAO a plan for implementing a new appeals process relative to the legacy process, including, for example, appeals process (which VA submitted specifying timeliness goals for the five new appeals options, and measures in November 2017) and periodic for decision accuracy in processing appeals. As of November 2018, VA progress reports (which VA submitted officials stated their intention to use productivity, timeliness, accuracy, and in February, May, August, and veteran satisfaction metrics to assess the new versus the legacy appeals November 2018). The Act also processes. However, VA has yet to specify a complete set of goals or includes a provision for GAO to assess measures for monitoring and assessing the relative efficacy of the new VA’s original plan. process or articulate detailed steps and timeframes for establishing them. In March 2018, GAO found that VA could help ensure successful • Augment master schedule. GAO recommended VA augment its master implementation of appeals reform by schedule for appeals reform to reflect sound practices for guiding addressing gaps in planning and made implementation of reform. Although VA’s updated schedule reflected four recommendations, with which VA progress since VA’s original 2017 plan, it still did not fully meet sound agreed. This testimony focuses on the practices for project management. For example, the schedule does not steps VA has taken to address GAO’s appropriately define the work, activities, and resources necessary to recommendations, what aspects accomplish appeals reform implementation. Without following sound remain unaddressed, and risks these gaps pose for implementation. practices, it is unclear whether the schedule poses risks to successful implementation of appeals reform. For this statement, GAO reviewed VA’s updated plans, assessed VA’s • Address risk fully. GAO recommended that VA’s plan more fully address schedules against best practices, risks in implementing a new appeals process by, for example, testing all interviewed VA officials and reviewed information they provided about steps appeals options prior to full implementation. As of November 2018, VA took taken to implement GAO’s many steps to address risks, although opportunities exist to better assess recommendations. them. For example, although VA has used lessons learned from tests to update the implementation process, it has not fully tested all aspects nor has it developed mitigation strategies for all identified risks, such as veterans View GAO-19-272T. For more information, appealing to the Board at higher rates than expected. Until VA takes these contact Elizabeth Curda at (202) 512-7215 or remaining steps, it may not have sufficiently accounted for key risks in curdae@gao.gov. implementing the new process. United States Government Accountability Office Letter Letter Chairman Roe, Ranking Member Walz, and Members of the Committee: I appreciate the opportunity today to provide an update on the Department of Veterans Affairs’ (VA) plans for implementing a new disability appeals process while still attending to appeals under the current, or legacy, process. VA provides cash benefits to veterans for disabling conditions incurred in or aggravated by military service, paying about $72 billion to about 4.5 million veterans in fiscal year 2017. If veterans are dissatisfied with VA’s initial decision they can appeal—first to the Veterans Benefits Administration (VBA) and then, if not satisfied there, to the Board of Veterans’ Appeals (Board), a separate agency within VA. For appeals resolved in fiscal year 2017, veterans waited an average of approximately 3 years from the date they initiated their appeal to resolution by either VBA or the Board—and an average of 7 years for appeals resolved by the Board. Due in part to the challenges VA faces managing large workloads and deciding disability claims and appeals in a timely manner, in 2003 we designated VA disability compensation, along with other federal disability programs, as one of the government’s highest risk areas. 1 The Veterans Appeals Improvement and Modernization Act of 2017 (Act) makes changes to VA’s disability appeals process by replacing it with one that gives veterans various options either for further review by VBA or to bypass VBA and appeal directly to the Board. 2 These changes may generally take effect no earlier than February 2019, which is about 18 months from the date of enactment. The Act also built in flexibility for VA regarding this time frame by stating that most of these changes will not take effect until 30 days after the Secretary of Veterans Affairs certifies that the agency is prepared to carry out timely processing of appeals 1 Improving and modernizing federal disability programs is an area that we continue to monitor on our high-risk list. See GAO, High-Risk Series: Progress on Many High Risk Areas, While Substantial Efforts Needed on Others, GAO-17-317 (Washington, D.C.: Feb. 15, 2017). 2 Pub. L. No. 115-55, § 2, 131 Stat. 1105, 1105. Page 1 GAO-19-272T VA Disability Benefits under the new and legacy appeals process, in addition to giving VA the option of phasing in implementation of the new process at that time. 3 The Act further required VA to submit a comprehensive plan for implementing the new appeals process to the appropriate committees of Congress and GAO. 4 (VA submitted its plan to GAO on November 22, 2017.) The Act delineates 22 legally required elements—some with subparts—for this plan. In addition, the Act requires VA to provide progress reports to the appropriate committees of Congress and GAO at least every 90 days until the Act’s changes to the appeals process generally go into effect and then at least every 180 days after this date for 7 years. VA submitted progress reports in February, May, August, and November 2018. The Act also includes a provision for GAO to assess whether VA’s appeals plan comports with sound planning practices and identify any gaps in the plan. 5 In response, we have issued a series of reports and testimonies assessing VA’s plans. In our March 2018 report, we concluded that while VA’s November 2017 plan reflected aspects of sound planning, improvements in planning were still needed to ensure successful appeals reform. We recommended VA’s plan (1) address all legally required elements in the Act; (2) articulate how VA will monitor and assess the performance of appeals processes; (3) augment its project 3 Under the Act, the legal changes to VA’s appeals process will generally take effect on or after the later of (1) 540 days (approximately 18 months) after enactment, and (2) 30 days after the Secretary of Veterans Affairs submits to the appropriate committees of Congress (i) a certification that VA has the resources, personnel, office space, procedures, and IT required to carry out the new appeals system and to timely address appeals under the new appeals system as well as pending legacy appeals, and (ii) a summary of the expected performance outcomes used in making the certification with respect to legacy claims and a comparison of these expected outcomes with actual program performance with respect to the appeals under the legacy system (before the new system is implemented). Pub. L. No. 115-55, § 2(x)(1), 131 Stat. 1105, 1115. 4 The Act defines “appropriate committees of Congress” as the Committee on Veterans’ Affairs and the Committee on Appropriations of the Senate, and the Committee on Veterans’ Affairs and the Committee on Appropriations in the House of Representatives. 5 Pub. L. No. 115-55, § 3(c), 131 Stat. 1105, 1118. Page 2 GAO-19-272T VA Disability Benefits plan for implementation; and (4) address risk more fully. 6 VA agreed with our recommendations. 7 In a July 2018 testimony we concluded that VA had updated its plan and taken some steps to address aspects of these four recommendations, but further steps were needed. 8 My statement today addresses VA’s recent progress in implementing the four recommendations in our March 2018 report, what aspects of those recommendations VA has yet to address, and the risks these gaps pose for successful implementation of appeals reform. 9 For this statement, we reviewed VA’s most recent progress reports on its appeals reform plan, dated August and November 2018, and information we received from VA officials about steps taken to implement our March 2018 recommendations. We assessed VA’s schedules and supporting documentation against applicable best practices in GAO’s Schedule Assessment Guide. 10 We also interviewed VA officials and reviewed information related to VA’s progress in addressing related 6 GAO, VA Disability Benefits: Improved Planning Practices Would Better Ensure Successful Appeals Reform, GAO-18-352 (Washington, D.C.: Mar. 22, 2018). We also discussed our work and proposed recommendations in a January 2018 testimony. See GAO, VA Disability Benefits: Opportunities Exist to Better Ensure Successful Appeals Reform, GAO-18-349T (Washington, D.C.: Jan. 30, 2018). See also our report on VA’s appeals planning that pre-dated the Act: GAO, VA Disability Benefits: Additional Planning Would Enhance Efforts to Improve the Timeliness of Appeals Decisions, GAO-17-234 (Washington, D.C.: Mar. 23, 2017). 7 Subsequently, in April 2018 we designated two of our four recommendations—monitoring and assessing performance as well as addressing risks—as “priority recommendations” for VA to implement. Priority recommendations are open recommendations we believe warrant priority attention from heads of key departments and agencies. 8 GAO, VA Disability Benefits: Some Progress, but Further Steps Needed to Improve Appeals Reform Planning, GAO-18-661T (Washington, D.C.: July 24, 2018). 9 GAO-18-352. 10 GAO, GAO Schedule Assessment Guide: Best Practices for Project Schedules, GAO-16-89G (Washington, D.C.: December 2015). Page 3 GAO-19-272T VA Disability Benefits recommendations from work that we conducted prior to enactment of the Act. 11 The work upon which this statement is based was conducted in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background VA’s Current Disability VA’s process for deciding veterans’ eligibility for disability compensation Compensation Appeals begins when a veteran submits a claim to VA. 12 Staff in one of VBA’s 57 Process regional offices assist the veteran by gathering additional evidence, such as military and medical records, that is needed to evaluate the claim. Based on this evidence, VBA decides whether the veteran is entitled to compensation and, if so, how much. A veteran dissatisfied with the initial claim decision can generally appeal within 1 year from the date of the notification letter sent by VBA. Under the current appeals process (now referred to by VA as the legacy process), an appeal begins with the veteran filing a Notice of 11 We have been monitoring VA’s progress in addressing a related set of five recommendations from our 2017 report on VA’s appeals planning. See GAO, VA Disability Benefits: Additional Planning Would Enhance Efforts to Improve the Timeliness of Appeals Decisions, GAO-17-234 (Washington, D.C.: March 23, 2017). Specifically, we made five recommendations to improve VA’s ability to implement its proposed reform to the appeals process while addressing a growing appeals workload, with which VA agreed in principle. In summary, we recommended that VA develop: (1) a detailed workforce plan, (2) a complete schedule of information technology (IT) updates, (3) better estimates of future workloads and timeliness, (4) a robust plan for monitoring appeals reform, and (5) a strategy for assessing whether the new process improves veterans’ experiences over the current process. We also suggested that Congress require VA to pilot test appeals reform changes. As of December 2018, four of these recommendations remain open. We closed the third recommendation because VA developed better estimates of future workloads and timeliness. We also closed our suggestion to Congress because the Act authorizes VA to carry out programs to test any assumptions relied upon in developing its comprehensive plan and test the feasibility and advisability of any facet of the new appeals process. 12 For additional details about VA’s current and new appeals processes and the Act, see GAO-18-352. Page 4 GAO-19-272T VA Disability Benefits Disagreement. VBA then re-examines the case and generally issues a Statement of the Case that represents its decision. A veteran dissatisfied with VBA’s decision can file an appeal with the Board. In filing that appeal, the veteran can indicate whether a Board hearing is desired. Before the Board reviews the appeal, VBA prepares the file and certifies it as ready for Board review. If the veteran requests a hearing to present new evidence or arguments, the Board will hold a hearing by videoconference or at a local VBA regional office. The Board reviews the evidence and either issues a decision to grant or deny the veteran’s appeal or refers the appeal back to VBA for further work. VA’s New Appeals Process According to VA’s appeals plan, VA intends to implement the Act by February 2019, by replacing the current appeals process with a process offering veterans who are dissatisfied with VBA’s decision on their claim five options. Two of those options afford the veteran an opportunity for an additional review of VBA’s decision within VBA, and the other three options afford them the opportunity to bypass additional VBA review and appeal directly to the Board. Under the new appeals process, the two VBA options will be: 1. Request higher-level review: The veteran asks VBA to review its initial decision based on the same evidence but with a higher-level official reviewing and issuing a new decision. 2. File supplemental claim: The veteran provides additional evidence and files a supplemental claim with VBA for a new decision on the claim. The veteran can also request a VBA hearing. The three Board options will be: 3. Request Board review of existing record: The veteran appeals to the Board and asks it to review only the existing record without a hearing. 4. Request Board review of additional evidence, without a hearing. 5. Request Board review of additional evidence, with a hearing. In November 2017, VA initiated a test of the new VBA higher-level review and supplemental claim options. According to VA’s appeals plan, a purpose of this test—the Rapid Appeals Modernization Program (RAMP)—is to reduce legacy appeals by providing veterans with a chance for early resolution of their claims within VBA’s new process. Page 5 GAO-19-272T VA Disability Benefits Participation in RAMP is voluntary, but veterans must withdraw their pending legacy appeal to participate, according to VA’s appeals plan. In our March 2018 report, we found that VA’s November 2017 plan for VA Has Not Provided implementing a new disability appeals process while attending to appeals Complete Information under way in the current (legacy) process, addressed 17 of 22 elements required by the Act. 13 For the 5 remaining elements, we found that it on Four Elements in partially addressed 4 elements related to implementation monitoring, the Act productivity projecting, and workforce planning, and did not address 1 element related to identifying total resources. This element called for delineating the resources needed by VBA and the Board to implement the new appeals process and address legacy appeals. We recommended in March 2018 that VA address all 22 required elements in the Act in VA’s appeals plan to Congress—including delineating resources required for all VBA and Board appeals options— using sensitivity analyses and results from its test, RAMP, where appropriate and needed. 14 Since our March 2018 report, VA has taken some action on each of the five elements that we found were not fully addressed at that time. For example, VA added details related to projecting staff productivity, identifying total resources, as well as determining personnel requirements and productivity projections for processing appeals. For identifying total resources, VA added FTE information for other offices that help implement the appeals process and prepared a model to project resource needs. Although VA now addresses the 1 element related to projecting productivity, it only partially addresses 4 elements related to monitoring 13 We identified 22 required elements for VA’s comprehensive plan under section 3(a) and (b) of the Act. Specifically, subsection (a) contains 4 elements, and subsection (b) requires the appeals plan to address 18 elements. See GAO-18-352. 14 Sensitivity analysis—used in scenario planning to, for example, determine the resources needed for implementing a new process—is an analysis to determine how sensitive outcomes are to changes in assumptions, such as those used to determine resource needs. The assumptions that deserve the most attention should depend on the dominant benefit and cost elements and the areas of greatest uncertainty of the program or process being analyzed. See GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: Mar 2, 2009). Page 6 GAO-19-272T VA Disability Benefits implementation, workforce planning, and delineating the total resources. For example, as of November 2018, VA’s plan does not contain metrics for monitoring implementation. Moreover, for total resources, the updated plan does not delineate the total resources required by VBA and the Board, such as the resources necessary for information technology and training. We acknowledge that in some cases delineating total resources could prove challenging, such as delineating information technology resources for the legacy and new appeals processes. We also acknowledge that implementing corrective actions to fully address these 4 elements may be challenging within the next several weeks, but we continue to believe VA has an opportunity to further address these 4 elements as part of certifying the agency’s readiness prior to the full implementation of the new process. In our March 2018 report, we found gaps in VA’s planning for how it will VA Has Addressed monitor and assess performance of the new appeals process when it is Some Gaps in Its implemented. Specifically, we reported that the plan did not (1) establish timeliness goals for two of the three Board options (i.e., Board review of Plans to Monitor and additional evidence without a hearing and Board review of additional Assess Performance, evidence with a hearing); (2) articulate aspects of performance important for managing appeals, such as accuracy of decisions, veteran satisfaction though Further Steps with the process, or cost; (3) explain how the performance of the new Remain appeals process would be compared to that of the legacy process; or (4) explain how the agency would monitor relative workloads of, and resources devoted to, the new and legacy appeals processes. To address these gaps, we recommended that VA clearly articulate in its appeals plan how VA will monitor and assess the new appeals process compared to the legacy process, including specifying a balanced set of goals and measures—such as timeliness goals for all VBA appeals options and Board dockets, and measures of accuracy, veteran satisfaction, and cost—and related baseline data. Articulating a balanced set of goals that cover key aspects of managing appeals is important to avoid promoting skewed behaviors (e.g., favoring timeliness over accuracy) and to fully understanding performance. In its progress reports, VA addressed some but not all aspects of this recommendation (see table 1). Page 7 GAO-19-272T VA Disability Benefits Table 1: Key Steps Taken and Steps Remaining for Department of Veterans Affairs (VA) to Address GAO’s March 2018 Recommendation on Performance Measurement of Reform of Disability Appeals Process Aspects of GAO’s March 2018 Key steps remaining for VA to fully recommendation Key steps taken by VA since March 2018 address GAO’s recommendation Specify timeliness goals for all • In November 2018, the Board of Veterans • Develop timeliness metrics and goals for appeal options and measures for Appeals (Board) stated that it plans to two of the three Board options, and accuracy, veteran satisfaction and publish projected wait times for each new productivity/cost goals for the Board. cost (balanced measures) option after implementation. • Specify a complete set of balanced • In August 2018, the Veterans Benefits goals and measures for VBA and the Administration (VBA) developed Board to assess appeals performance, productivity/cost goals under the new including veteran satisfaction and, for process. the Board, accuracy goals. • In May 2018, VA reported that it is • Articulate in its appeals plan how VA will collecting data to inform development of use the Act’s and other metrics to balanced measures for all five new assess the relative performances of the appeals options (e.g., veteran satisfaction, new and legacy processes. accuracy, etc.). Monitor and assess the new • In August and November 2018, VA appeals process compared to the reported it plans to compare veterans’ legacy process experiences with the new and legacy appeals processes using satisfaction surveys. • As of May 2018, VA reported it had been developing sensitivity and other analyses to monitor and manage staff resources for both the new and legacy appeals processes. • VA has reported it was developing information technology capacity to produce and report metrics required under the Act.a Source: GAO analysis of VA’s appeals plan, progress reports, supporting documents, and interviews with VA officials. | GAO-19-272T a The Veterans Appeals Improvement and Modernization Act of 2017. VA has made progress in monitoring performance and addressing workload changes in its new and legacy appeals processes, but still lacks a complete set of balanced goals and measures. As we noted in our July 2018 testimony, VA has developed sensitivity models and other analyses to monitor and forecast future VBA and Board workloads, production, and staffing requirements to help VA manage the legacy and new appeals processes. 15 However, VBA and the Board have yet to specify a complete set of balanced goals for monitoring the performance of the new appeals processes. According to the November 2018 progress report, the Board 15 GAO-18-661T. Page 8 GAO-19-272T VA Disability Benefits plans to develop timeliness goals after VA fully implements the new appeals process. 16 Until VA fully develops a set of balanced goals and measures, the agency risks not fully understanding how well the reforms are performing. Regarding comparing the performance of the new and legacy appeals processes, VA has previously reported that the agency plans to implement the reporting requirements in section 5 of the Act. 17 This section requires VA to report performance measures related to, among other things, timeliness, productivity, and outcomes, without specifying whether or how VA should compare performance of the new versus legacy processes. In November 2018, VBA and Board officials told us they intend to use timeliness and productivity metrics from section 5 to compare the two processes. However, in its updated plans to date, VA has been reporting average timeliness of decisions made to date under RAMP—VA’s test of the two VBA options—without reporting the average time cases are pending. Moreover, VA has not been reporting timeliness data on both decisions and pending cases according to the month that they entered into RAMP, which present a more balanced indication of performance and trends. 18 In November 2018 VBA and Board officials told us they would 16 Moreover, by not establishing key goals, such as timeliness goals for all Board options, until after fully implementing the new appeals process, VA is missing an opportunity to more fully define its vision for what successful implementation would look like and what resources would be required to achieve that vision. 17 Section 5 of the Act requires VA to periodically publish on its website various metrics on the new and legacy processes. Pub. L. No. 115-55 § 5, 131 Stat. 1105, 1123. 18 We previously reported on the benefits and limitations of analyzing timeliness of a new process according to time of case completion versus time of case enrollment. In a prior review of the Integrated Disability Evaluation System (IDES), administered by the Department of Defense and VA, we found that analyzing IDES cases according to completion date resulted in shorter average processing times in the first year of IDES because it reflected those cases that were processed quickly. As such, organizing cases by enrollment date provided a better estimate of the processing times for the early IDES cases. However, this approach resulted in shorter processing times in the most recent full year of the program because only cases that finished quickly in that year could be analyzed. See figures 10 and 11 from GAO, Military Disability System: Improved Monitoring Needed to Better Track and Manage Performance, GAO-12-676 (Washington, D.C.: Aug. 28, 2012). Page 9 GAO-19-272T VA Disability Benefits consider reporting timeliness using a monthly cohort that reflects when appeals were filed. 19 VBA and Board officials also said they have taken steps to collect, through surveys, comparable information on veterans’ satisfaction with the new and legacy appeals processes. According to VBA and Board officials, they have pre-tested the surveys—which is considered a best practice by survey methodologists—and are coordinating the survey efforts with one another. VBA and Board officials also told us that the agency will report on accuracy and outcomes (grants and denials of claims) in the new process. However, they also stated that these measures would not provide a fair comparison with the legacy process because the Act eliminated several of the requirements formerly required in the legacy appeals administrative processes. 20 Although VA officials said they would develop a plan for comparing the performance of the two appeals processes after the new process is fully implemented, they did not indicate how soon they would do so. Developing such a plan would better position the agency to fully understand whether the new process is an improvement. Our March 2018 report identified elements of a high-quality and reliable VA Has Augmented implementation schedule that were missing from VA’s master schedule Its Master Schedule for appeals reform. Specifically, we reported that VA’s high-level master schedule—which the agency included with its November 2017 plan—did to a Limited Extent not (1) include all key activities; (2) show which activities must finish prior to the start of other activities, or the amount of time an activity could be delayed before the delay affects VA’s estimated implementation date; (3) reflect interim goals and milestones for monitoring implementation; or (4) assign resources for activities. We recommended that VA augment the master schedule for its appeals plan to reflect all activities—such as modifications to information technology systems—as well as assigned responsibilities, interdependencies, start and end dates for key activities for each 19 VBA and Board officials also noted that cases taking longer to process often reflect the lack of supporting evidence provided by veterans. 20 Further, Board officials stated that its current approach to quality review may not include sufficient cases to do a valid comparison of decision accuracy across the new and legacy processes, or among Board options. Page 10 GAO-19-272T VA Disability Benefits workgroup, and resources. These steps establish accountability and reduce overall risk of implementation failures. In response to our recommendation, the Board, VBA and other VA administrations made progress over time with developing and integrating underlying plans into the integrated master schedule (IMS) in spring and summer 2018. According to VA officials, VA set a baseline schedule for implementing appeals reform in response to the potential February 2019 implementation date established in the Act. 21 Since November 2017, VA’s plan and progress reports have stated that VA uses an agency-wide governance structure to coordinate implementation, and regularly uses the schedule as a management tool for monitoring progress on appeals reform. For example, the Board’s project manager meets regularly with those responsible for major activities to check progress, including weekly meetings with leadership, and identifies and corrects issues related to schedule execution. In October 2018, VA provided us with lower-level schedules and information that allowed us to conduct a more detailed assessment of VA’s IMS against applicable best practices criteria. 22 The six criteria we assessed lower-level schedules against were: • Capturing all activities: schedule should reflect all activities necessary to perform work to accomplish a project’s objective. • Sequencing activities: activities should be logically sequenced in the order they are to be carried out so that critical program dates can be met. • Assigning resources: schedule should reflect all resources necessary to complete work, verify whether resources will be available, and identify any constraints. • Verifying horizontal and vertical traceability: schedule should be rational and logically sequenced, account for interdependencies among activities, and provide a way to evaluate the current status (horizontal traceability). Also, the various levels of a schedule— summary, intermediate, and detailed—should be consistent with one 21 As previously noted, the Act built in flexibility for VA to phase in or continue preparing for certification beyond February 2019. 22 GAO-16-89G. Page 11 GAO-19-272T VA Disability Benefits another and enable different teams to work to the same schedule expectations (vertical traceability). • Updating the schedule using actual progress and logic: maintain and continually update the schedule to reflect a realistic forecast of start and end dates of activities. • Maintaining a baseline schedule: use original configuration of the program plan as a point of comparison for the current plan to manage scope, timeframes, and required resources. We found that, while VA has made progress with providing more detail, its master and underlying schedules only minimally met sound practices for project management. Specifically, as with our March 2018 assessment, we found that the schedule does not contain enough detail to manage the work or provide a realistic representation of the resources and time needed for this project. For example, the schedule did not contain a work breakdown structure that defines the work, activities, and resources necessary to accomplish implementation. Moreover, half of all the remaining activities are missing logic that shows which activities must finish prior to the start of other activities. In addition, the schedule contains an invalid critical path, meaning that the schedule does not present the amount of time that key activities could be delayed before such delays affect VA’s estimated implementation date. 23 Without a valid critical path, management cannot focus on activities that will detrimentally affect the key program milestones and deliveries if they slip. To address our March 2018 recommendation, VA would need to ensure that all activities are accounted for, that scheduled activities appear in the correct order, that resources are properly allocated, that all activities appear on the critical path, and that a schedule risk analysis accounts for all risks. We provide a more detailed explanation of our assessment results in appendix I. In addition, establishing an overly optimistic schedule can reduce capacity for carrying out a project and potentially create pressure to sacrifice the quality of work activities to meet deadlines. Moreover, many of VA’s 23 Further, the Board’s overall timeline for implementing its information technology system, Caseflow, lacks information clarifying key activities associated with this implementation. Specifically, although VA’s plan mentions that it is finalizing the algorithm for assigning appeals to judges for adjudication, there is no information that further describes this capability or its status. Page 12 GAO-19-272T VA Disability Benefits activities are slated to be concurrently completed just before implementation, posing a significant risk to implementing reform in February. For example, according to VA’s schedule, the agency needs to complete 117 activities after January 1, 2019. Further, other VA efforts to redesign or update key aspects of VA’s disability compensation process—including the Veterans Benefits Management System (VBMS)—were not driven by robust, comprehensive planning and did not achieve their schedule goals. 24 While VA intends to start full implementation in February, we do not know the extent to which the lack of a robust schedule poses risks to successful and smooth implementation. Even if taking corrective actions to address our findings may not be feasible before February, incorporating such lessons learned into future project planning could help VA improve its project scheduling capabilities. In our March 2018 report, we found that VA’s appeals plan could more VA Has Addressed fully assess key risks related to implementing the new appeals process. Many, but Not All Key In particular, we found that VA’s plan did not include testing of new Board options or clearly define how it would assess the RAMP test of the VBA- Risks to only options before implementing them more broadly. 25 Further, we Implementation reported that VA’s plan had not comprehensively reflected key risks because the agency had not established a complete and balanced set of goals and measures, which are a necessary pre-condition to effectively assessing risk. 26 24 GAO, Veterans Benefits Management System: Ongoing Development and Implementation Can Be Improved; Goals Are Needed to Promote Increased User Satisfaction, GAO-15-582 (Washington, D.C.: Sept. 1, 2015); Veterans’ Disability Benefits: Timely Processing Remains a Daunting Challenge, GAO-13-89 (Washington, D.C.: Dec. 21, 2012); and, VA Disability Compensation: Actions Needed to Address Hurdles Facing Program Modernization, GAO-12-846 (Washington, D.C.: Sept.10, 2012). 25 We previously reported on the benefits of testing appeals reform and the risks of not doing so, and recommended that Congress require VA to develop options for testing appeal reform prior to implementation. See GAO-17-234. The Act authorizes VA to carry out programs to test any assumptions relied upon in developing its comprehensive plan and test the feasibility and advisability of any facet of the new appeals process. 26 See GAO-18-352. A risk assessment is the identification and analysis of risks related to achieving the defined objectives. This assessment provides the basis for developing appropriate risk responses. See GAO, Standards for Internal Control in the Federal Government, GAO-14-704G (Washington, D.C.: September 2014). Page 13 GAO-19-272T VA Disability Benefits We recommended that VA ensure that the appeals plan more fully addresses risk associated with appeals reform by, for example, assessing risks against a balanced set of goals and measures, articulating success criteria and an assessment plan for RAMP, and testing or conducting sensitivity analyses of all five appeals options before fully implementing the new appeals process. In its progress reports, VA took many steps to address our recommendation, although key steps are remaining for VA to better assess risks associated with implementing appeals reform and managing appeals workloads in the legacy process (see table 2). Table 2: Key Steps Taken and Remaining Steps for Department of Veterans Affairs (VA) to Address GAO’s March 2018 Recommendation on Risks to Reform of Disability Appeals Process Aspects of GAO’s March 2018 Key steps remaining for VA to fully Recommendation Key steps taken by VA since March 2018 address GAO’s recommendation Test or conduct sensitivity analysis • Since May 2018, VA has been • VA has not conducted a full test of all for all aspects of the new appeals conducting a limited test of 3 options at aspects of the new appeals process. process the Board of Veterans’ Appeals • Although RAMP allowed (Board), and subsequently used results veterans an opportunity to appeal to update elements of the appeals directly to the Board as of May 2018, process. the Board did not begin adjudicating • Since May 2018, VA has developed these cases until October 2018. and began using sensitivity analyses to • Test of Board option was limited project budget needs and staffing by small scale, restricted selection of requirements. veterans, and limited time to conduct • VA has been testing both appeals test and assess results. options at the Veterans Benefits • Although VA’s August 2018 plan Administration (VBA) through the Rapid identified a risk that veterans may Appeals Modernization Program appeal to the Board at higher rates, (RAMP), and subsequently has used which could have implications for preliminary results to update elements timeliness and quality of decisions, of the appeals process. VA’s August and November plans do not identify a mitigation strategy. Define success criteria and articulate • As of November 2018, VA provided • VA lacks a comprehensive plan with how to assess test programs evidence of methodologies and data well-defined, measurable criteria for collection efforts for testing some, but fully assessing performance, and not all, aspects of reform evaluating final results of tests to inform • As of August 2018 VA has identified decision-making on new appeals lessons learned from tests and updated implementation. training, guidance, and forms needed for full implementation. • Since February 2018, VA has defined broad goals for tests related to how they will be used to monitor and assess new and legacy appeals. Page 14 GAO-19-272T VA Disability Benefits Aspects of GAO’s March 2018 Key steps remaining for VA to fully Recommendation Key steps taken by VA since March 2018 address GAO’s recommendation Assessing risks against a set of • Since November 2017, VA has • VA continues to lack a complete set of balanced goals and measures identified and continues to identify balanced goals and measures—for additional risks related to timeliness of example, with respect to veteran new VBA and Board options.a satisfaction for VBA and the Board and timeliness for two of the Board options—with which to assess risk. Source: GAO analysis of VA’s appeals plan, progress reports, supporting documents, and interviews with VA officials. | GAO-19-272T a In November 2018, the Board reported that it had executed, and the VBA is developing, an internal change management plan to facilitate successful implementation. Sound redesign and change management practices both suggest that tests be rigorously monitored and evaluated and that further roll-out occur only after an agency takes any needed corrective action and determines that the new process is achieving previously identified success criteria. 27 Until VA takes these remaining steps, it may not have comprehensively addressed key risks to better position the agency for successful implementation of appeals reform. In conclusion, VA is undertaking an ambitious effort to reform its disability appeals process—while onboarding hundreds of new staff and implementing new technology—that will affect the lives of hundreds of thousands of veterans with disabilities for years to come. Consistent with our prior recommendations, VA has made concrete progress to improve its planning for disability appeals reform while it attends to legacy appeals. Efforts such as resuming sensitivity analysis to monitor workloads and testing VBA and Board appeals options will provide useful information to guide VA through the uncertainty often associated with process change. However, VA has reported it plans to fully implement the new disability appeals process in February 2019 even though it has yet to fully address our recommendations. While fully implementing our recommendations prior to February 2019 may not be feasible, doing so would better position VA to ensure successful implementation. Nevertheless, VA should still work to increase clarity around its plans prior to fully implementing reform. 27 See GAO, Business Process Reengineering Assessment Guide GAO/AIMD-10.1.15 (Washington, D.C.: May 1997) and Data Act: Section 5 Pilot Design Issues Need to Be Addressed to Meet Goal of Reducing Recipient Reporting Burden GAO-16-438 (Washington, D.C.: April 19, 2016). Page 15 GAO-19-272T VA Disability Benefits Moreover, many of the principles of sound planning practices that informed our recommendations remain relevant during process change. By continuing to improve its approach to performance measurement, scheduling, and risk management, even after implementation, VA could better ensure that the new process meets veterans’ needs. Chairman Roe, Ranking Member Walz, and Members of the Committee, this concludes my prepared statement. I would be pleased to respond to any questions you may have at this time. Page 16 GAO-19-272T VA Disability Benefits For further information about this testimony, please contact Elizabeth H. GAO Contact and Curda at (202) 512-7215 or curdae@gao.gov. Contact points for our Staff Offices of Congressional Relations and Public Affairs may be found on the last page of this testimony. Other key contributors to this testimony Acknowledgments include James Whitcomb (Assistant Director), Juaná Collymore, Michele Grgich, Sara Pelton, and Rachel Pittenger. In addition, key support was provided by Susan Aschoff, Mark Bird, Alex Galuten, Jason Lee, Sheila R. McCoy, Almeta Spencer, and Walter Vance. Page 17 GAO-19-272T VA Disability Benefits Appendix I: Assessment of the Extent to Appendix I: Assessment of the Extent to Which VA Followed Aspects of Scheduling Leading Practices Which VA Followed Aspects of Scheduling Leading Practices For this testimony, we assessed the steps that the Department of Veterans Affairs (VA) has taken to address our March 2018 recommendations and what aspects remain unaddressed, including the extent to which VA is using sound practices for scheduling key projects. 1 In summary, we identified several areas where VA’s most recent schedule falls short of sound practices. Further incorporating sound practices into future project planning could help VA improve its project scheduling capabilities. We reviewed VA’s integrated master schedule (IMS) for the appeals reform effort and underlying sub-schedules to assess them against 6 of the 10 best practices, which we determined most relevant to our March 2018 recommendation that VA augment its master schedule for VA’s appeals plan to reflect all activities—such as modifications to information technology systems—as well as assigned responsibilities, interdependencies, start and end dates for key activities for each workgroup, and resources, to establish accountability and reduce the overall risk of implementation failures. Specifically, we analyzed the following related scheduling best practices: (1) Capturing all activities, (2) Sequencing all activities, (3) Assigning resources to all activities, (4) Verifying that the schedule can be traced vertically and horizontally, (5) Updating the schedule using actual progress and logic and (6) Maintaining a baseline schedule. We assessed VA’s lower-level schedules against these 6 best practices by: • Checking for specific problems that could hinder the schedule’s ability to respond to changes. For example, we: o Examined if there are any open-ended activities (i.e., activities with no predecessor and/or successors), o Searched for activities with poor logic: • For example, Start to Start successor only or Finish to Finish predecessor only which represent dangling logic, or 1 GAO, GAO Schedule Assessment Guide: Best Practices for Project Schedules GAO-16-89G (Washington, D.C.: Dec. 2015). Underlying these characteristics are 10 leading practices. These characteristics and leading practices were developed in 2012 based on our practices for creating a reliable cost estimate and in consultation with experts from the scheduling community. Page 18 GAO-19-272T VA Disability Benefits Appendix I: Assessment of the Extent to Which VA Followed Aspects of Scheduling Leading Practices • Logic on summary tasks rather than attached to detailed tasks (summary tasks are for organizing the schedule and should not drive the logic). o Looked for activities with constraints which keep the schedule rigid (e.g., start no earlier than, finish no later than, etc.), o Determined if activities were resource loaded—which helps to cost out the schedule—and examine whether resources are over- allocated or not available when needed, o Examined the schedule’s critical path to determine whether or not it was reliable and logical, o Examined schedule float and determined if it was reasonable, and o Examined whether the schedule was baselined, its status cycle, and what deviations there were from the original plan. We also determined if there were any actual start or finish dates recorded in the future and whether there was any broken logic between planned tasks. We also interviewed VA officials responsible for managing the schedule. We scored each scheduling leading practice on a five-point scale: “not met”, “minimally met”, “partially met”, “substantially met” and “fully met.” We determined the characteristic assessment rating by assigning each best practice rating a number and taking the average. Our resulting conclusions based on this assessment are as follows: • VA’s project schedule minimally meets the best practice of capturing all activities. The schedule does not have well-defined start and finish milestones and there is not a project work breakdown structure (WBS) or corresponding WBS dictionary to define the work for each WBS element. We were not able to independently verify contractor work or major handoffs and deliverables in the schedule. In addition, there were activities with duplicate names, which could make communication difficult between VA teams, particularly between team members who are responsible for updating and integrating multiple schedules. • VA’s project schedule minimally meets the best practice of sequencing activities. There are issues with missing dependencies, dangling activities, summary links, constraints and lags that affect the schedule meeting this best practice. Specifically, of the remaining activities, 55 percent have missing logic, over 12 percent are dangling, 42 percent have date constraints and 4 percent have leads assigned. When activities are not correctly linked, the program cannot Page 19 GAO-19-272T VA Disability Benefits Appendix I: Assessment of the Extent to Which VA Followed Aspects of Scheduling Leading Practices use the integrated master schedule (IMS) to identify disconnects or hidden opportunities and cannot otherwise promote efficiency and accuracy or control the program by comparing actual to planned progress. When this happens, the schedule will not allow a sufficient understanding of the program as a whole, and users of the schedule may lack confidence in the dates and the critical path. • VA’s project schedule minimally meets the best practice of assigning resources. While the schedule contains ‘Task Owner’ assignments, the Task Owner information has no effect on the durations or forecasted start and finish dates of detailed activities. Information on resource needs and availability in each work period assists the program office in forecasting the likelihood that activities will be completed as scheduled. If the current schedule does not allow insight into the current or projected allocation of resources, then the risk of the program’s slipping is significantly increased. • VA’s project schedule minimally meets the best practice of verifying the schedule is traceable horizontally and vertically. There was no evidence in the schedule of hand-offs within the schedule—that is givers and receivers are easily identifiable in the schedule. We were unable to determine the relationship between lower-lever activities in the project schedule and higher-level activities and milestones in the management briefs provided to us. Specifically, we could not map the activities in the briefs to activities in the schedule. This inconsistency also prevented the verification of dates between the project schedule and higher-level management documents, even with documents that were provided from the same month as the October schedule. Products and outcomes were not easily traced through the sequencing of effort in the project schedule. In both cases the schedule did not respond appropriately to “shocks”; that is, greatly increasing the durations of some activities to increase the overall time required to complete the project did not affect the dates of key milestones. The duration increase of each activity did not affect the overall time line because the activity in question had a constraint that would not allow the project to appropriately extend. • VA’s project schedule minimally meets the best practice of updating the schedule using progress and logic. Date anomalies, such as planned dates in the past or actual dates in the future, were found. The schedule was not current as of the date delivered to GAO. While officials report that they update the schedule regularly, a schedule narrative document does not accompany the schedule Page 20 GAO-19-272T VA Disability Benefits Appendix I: Assessment of the Extent to Which VA Followed Aspects of Scheduling Leading Practices update that would detail changes to the current schedule and describe information such as the status of key milestone dates, changes in network logic, and a description of the current critical path(s). • VA’s project schedule minimally meets the best practice of maintaining a baseline schedule. Officials said that the baseline schedule is the basis for performance measurement. But while baseline start and baseline finish dates were provided in the initial schedule, its activities were too high level, obfuscating the calculation of detail variances in subsequent schedules. There is also no evidence of a schedule basis document, which would include a general overview of the purpose of the schedule, other key basis information such as an overview of assumptions, rationale for durations specific to the CMR schedule, and required software settings. There is also no evidence of performance measuring. (103154) Page 21 GAO-19-272T VA Disability Benefits This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. The Government Accountability Office, the audit, evaluation, and investigative GAO’s Mission arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no cost is Obtaining Copies of through GAO’s website (https://www.gao.gov). Each weekday afternoon, GAO GAO Reports and posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to https://www.gao.gov Testimony and select “E-mail Updates.” Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, https://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Connect with GAO Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov. Contact: To Report Fraud, Website: https://www.gao.gov/fraudnet/fraudnet.htm Waste, and Abuse in Automated answering system: (800) 424-5454 or (202) 512-7700 Federal Programs Orice Williams Brown, Managing Director, WilliamsO@gao.gov, (202) 512-4400, Congressional U.S. Government Accountability Office, 441 G Street NW, Room 7125, Relations Washington, DC 20548 Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707 Strategic Planning and U.S. Government Accountability Office, 441 G Street NW, Room 7814, External Liaison Washington, DC 20548 Please Print on Recycled Paper.