United States Government Accountability Office Testimony Before the Committee on Veterans' Affairs, U.S. Senate ELECTRONIC HEALTH For Release on Delivery Expected at 3:30 p.m ET Wednesday, March 15, 2023 RECORD MODERNIZATION VA Needs to Address Change Management Challenges, User Satisfaction, and System Issues Statement of Carol C. Harris, Director, Information Technology and Cybersecurity GAO-23-106685 March 2023 ELECTRONIC HEALTH RECORD MODERNIZATION VA Needs to Address Change Management Highlights of GAO-23-106685, a testimony Challenges, User Satisfaction, and System Issues before the Committee on Veterans' Affairs, U.S. Senate Why GAO Did This Study What GAO Found VA uses the Veterans Health The Department of Veterans Affairs (VA) organizational change management Information Systems and Technology activities for the Electronic Health Record Modernization (EHRM) program were Architecture (VistA), which includes the partially consistent with seven leading practices and not consistent with one department's legacy EHR system, to leading practice (see table). provide health care to its patients. In June 2017, VA initiated the EHRM Extent to Which the Electronic Health Record Modernization (EHRM) Program's Activities program to replace VistA because it is Were Consistent with Organizational Change Management Leading Practices technically complex, costly to maintain, Leading practice GAO assessment and does not fully support the need to Developing a vision for change Partially consistent exchange health data with other Identifying and managing stakeholders Partially consistent organizations. Specifically, VA began Communicating effectively Partially consistent to acquire the same EHR system DOD Assessing the readiness for change Partially consistent was acquiring. VA has reported Increasing workforce skills and competencies Not consistent obligating about $9.42 billion on EHRM from fiscal year 2018 through the first Identifying and addressing potential barriers to change Partially consistent quarter of fiscal year 2023. Establishing targets and metrics for change Partially consistent Assessing the results of change Partially consistent GAO was asked to testify on its Source: GAO analysis of Department of Veterans Affairs (VA) data. | GAO-23-106685 recently completed review to determine the extent to which VA has (1) used Legend: Consistent – VA provided evidence that it conducted organizational change management activities mostly consistent with leading practices. Partially consistent – VA provided evidence that it organizational change management conducted organizational change management activities consistent with some of the leading practice strategies for the EHRM program criteria, but some key parts were not followed. Not consistent – VA did not provide sufficient evidence consistent with leading practices, (2) that it followed leading practices. assessed satisfaction with the new Until the program fully implements the eight leading practices for change system, and (3) identified and management, future deployments are at risk of continuing change management addressed EHR system issues. GAO challenges. These challenges hinder effective use of the new electronic health identified leading change management record (EHR) system, impede users' knowledge of new workflows, and limit the practices and evaluated VA's activities utility of system improvements. against these practices. It also reviewed results of surveys that VA Most users have expressed dissatisfaction with the new system. VA's 2021 and conducted to determine users' 2022 surveys showed that users were not satisfied with the system's satisfaction with the new EHR, performance or training. About 6 percent (120 of 2,066) of users agreed that the conducted interviews with selected system enabled quality care. In addition, about 4 percent (92 of 2,074) of users users, and interviewed officials on user agreed that the system made them as efficient as possible. Further, VA has not satisfaction goals. Further, GAO established targets (i.e., goals) to assess user satisfaction. Until it does so, VA analyzed system trouble ticket data lacks a basis for determining when satisfaction has sufficiently improved for the and compared them to VA's service system to be deployed at additional sites. Such a basis helps ensure that the level agreement with its contractor. system is not deployed prematurely, which could risk patients' safety. What GAO Recommends VA did not adequately identify and address system issues. Specifically, VA did GAO made 10 recommendations to VA not ensure that trouble tickets for the new EHR system were resolved within to address change management, user timeliness goals. It subsequently worked with the contractor to reduce the satisfaction, system trouble ticket, and number of tickets that were over 45 days old. Nevertheless, the overall number of independent operational assessment open tickets has steadily increased since 2020. Accordingly, it is critical that deficiencies. VA concurred with the system issues be resolved in a timely manner. Additionally, although VA has recommendations. assessed the system's performance at two sites, as of January 2023, it had not View GAO-23-106685. For more information, conducted an independent operational assessment. Without such an contact Carol C. Harris at (202) 512-4456 or independent assessment, VA will be limited in its ability to (1) validate that the harriscc@gao.gov. system is operationally suitable and effective, and (2) identify, track, and resolve key operational issues. United States Government Accountability Office Letter Letter Chairman Tester, Ranking Member Moran, and Members of the Committee: I am pleased to be here today to discuss the Department of Veterans Affairs' (VA) Electronic Health Record Modernization (EHRM) program. As you know, the use of IT is crucial to helping VA effectively serve the nation's veterans. Specifically, VA uses the Veterans Health Information Systems and Technology Architecture (VistA) to manage health care to its patients, which contains the department's electronic health record (EHR). 1 In June 2017, VA initiated the EHRM program to replace VistA because it is technically complex, costly to maintain, and does not fully support the department's need to exchange EHRs with other organizations, such as the Department of Defense (DOD) and private health care providers. As a result, VA began to acquire the same Oracle Cerner EHR system DOD had selected. VA has reported obligating about $9.42 billion on EHRM from fiscal year 2018 through the first quarter of fiscal year 2023. My statement today is based on a recently completed review to determine the extent to which VA has (1) employed organizational change management strategies for the EHRM program consistent with leading practices, (2) assessed users' satisfaction with the new system, and (3) identified and addressed EHR system issues. 2 Appendix I provides information on our objectives, scope, and methodology. The work on which this statement is based was conducted in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. 1An EHR is a collection of information about the health of an individual and the care provided to that individual, such as patient demographics, medications, and past medical history. 2On March 10, 2023, we provided a final briefing on the results of our review to the House and Senate Committees on Appropriations, Subcommittees on Military Construction, Veterans' Affairs, and Related Agencies; House and Senate Committees on Veterans' Affairs; Rep. Jim Banks; and Rep. Susie Lee. We plan to publish the briefing in a report. Page 1 GAO-23-106685 VA's IT systems provide capabilities to establish and maintain EHRs that Background health care providers and other clinical staff use to view patient information in inpatient, outpatient, and long-term care settings. The department's health information system-VistA-serves an essential role in helping the department to fulfill its health care delivery mission. However, VistA is technically complex: it is comprised of about 170 clinical, financial, and administrative applications that support health care delivery at more than 1,600 medical facilities. In addition, VA has approximately 130 versions of the system department-wide. VA is in the process of replacing VistA because it has been in operation for more than 30 years, and as previously mentioned, is costly to maintain, and does not fully support VA's need to electronically exchange health records with other organizations, such as DOD. Toward this end, VA established the EHRM program and contracted with Oracle Cerner to acquire Millennium (the core EHR system) and HealtheIntent (a cloud- based software platform that aggregates health data from multiple data sources to create a longitudinal patient record). 3 VA's contract also includes requirements for Oracle Cerner to: • conduct reviews and assessments of medical facilities to determine facility needs prior to deployment (e.g., technology infrastructure); • provide services, including project management, change management, training, and testing; and • host and deploy EHRM across the VA enterprise. The EHRM Integration Office (EHRM IO) is the organization within VA that is responsible for planning and implementing the EHRM program. 4 Deployment Schedule for Initially, VA planned to deploy the new system at sites in stages based on the New EHR System their geographical location over a 10-year period, through 2028. In October 2020, VA first deployed the new EHR at the Mann-Grandstaff VA 3A cloud-based service can allow an agency to only pay for the IT services used, when executed effectively. 4Theoffice was previously referred to as the Office of Electronic Health Record Modernization. Page 2 GAO-23-106685 Medical Center and planned to deploy it to other sites. 5 However, in March 2021, VA identified issues with the initial deployment, which led to a strategic review of the program. The strategic review identified eight challenge areas for EHRM, as well as plans and progress towards addressing those challenges. 6 After the review, VA deployed the new system to the following locations in 2022: • Jonathan M. Wainwright Memorial VA Medical Center (Walla Walla) in March 2022, • VA Central Ohio Health Care System (Columbus) in April 2022, and • Roseburg VA Health Care System and VA Southern Oregon Rehabilitation Center and Clinics (White City) in June 2022. In June 2022, VA announced that it would be pausing future deployments of the system until 2023 to allow time for improvements to the system. Subsequently, in October 2022, VA delayed deployments until June 2023 to address technical and other system performance issues. EHRM Costs VA contracted with the Institute for Defense Analysis to provide an independent cost estimate for the program. In September 2022, the Institute reported that the EHRM life cycle cost estimate was $49.8 billion: • $32.7 billion for a 13-year implementation phase and 5These sites are within the Veterans Health Administration's (VHA) Veterans Integrated Services Network 20 (VISN 20) and VISN 10. VHA is divided into areas called Veterans Integrated Services Networks (VISNs). There are currently 18 VISNs throughout VHA based on geographic location. VISNs provide oversight and guidance to the VA Medical Centers and VA Health Care Systems within their area and are sometimes called a "network." VISN 20 includes medical centers and community-based outpatient clinics in the states of Alaska, Washington, Oregon, most of Idaho, and one county each in Montana and California. VISN 10 serves veterans in the Ohio, Indiana, and Michigan areas. 6VA summarized the results of its strategic review in the Electronic Health Record Comprehensive Lessons Learned report. Department of Veterans Affairs, Electronic Health Record Comprehensive Lessons Learned (Washington, D.C.: July 2021). The eight challenge areas described in the report are improving the veteran experience, ensuring patient safety, providing extended training to the frontline employees, building confidence at VA sites, implementing organizational and program improvements, making governance effective, improving operational efficiencies, and centralizing data management for workers and veterans. Page 3 GAO-23-106685 • $17.1 billion in sustainment costs for the following 15 years. As previously mentioned, VA has reported obligating about $9.42 billion on EHRM from fiscal year 2018 through the first quarter of fiscal year 2023. This includes three areas: • the EHR contract ($4.49 billion), • IT infrastructure ($2.61 billion), and • program management ($882 million). In addition, VA reported obligating about $1.27 billion and $170 million on the program from the Veterans Health Administration (VHA) and the Office of Information and Technology (OIT), respectively. GAO Has Reported on VA In 2015, we designated VA health care as a high-risk area for the federal Health Care and EHRM government, in part due to its IT challenges. 7 In addition, we have previously reported on the EHRM program: • In June 2020, we reported on the process for configuring the department's new EHR system. 8 We noted that VA's decision-making procedures were generally effective, but the department did not always ensure key stakeholder involvement. We recommended (and VA concurred) that VA ensure the involvement of all relevant deployment site stakeholders in the EHR system configuration decision process. The department has begun to improve subject matter expert identification and involvement, but this type of involvement needs to continue until different stages of modernization unfold. As such, our recommendation remains not fully implemented as of February 2023. • In February 2021, we reported that VA had made progress toward deploying the new EHR system by making configuration decisions, 7VA's IT issues were highlighted in our 2015 high-risk report and subsequent high-risk reports. See GAO, High-Risk Series: An Update, GAO-15-290 (Washington, D.C.: Feb. 11, 2015); GAO, High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others, GAO-17-317 (Washington, D.C.: Feb. 15, 2017); GAO, High- Risk Series: Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas, GAO-19-157SP (Washington, D.C.: Mar. 6, 2019); and High-Risk Series: Dedicated Leadership Needed to Address Limited Progress in Most High-Risk Areas, GAO-21-119SP (Washington, D.C.: Mar. 2, 2021). 8GAO, Electronic Health Records: Ongoing Stakeholder Involvement Needed in the Department of Veterans Affairs' Modernization Effort, GAO-20-473 (Washington, D.C.: June 5, 2020). Page 4 GAO-23-106685 developing capabilities and interfaces, completing testing events, and deploying the system at the first site in October 2020. 9 However, we noted that the department was at risk of developing a system that may not perform as intended or could negatively impact the likelihood of successful adoption by users if critical and high severity test findings (that could result in system failure) were not resolved prior to future deployments. 10 We made two recommendations (and VA concurred) in February 2021, including that VA postpone deployment of the new EHR system at planned sites until any resulting critical and high severity test findings are appropriately addressed. VA stated that it planned to continue to test and appropriately adjudicate all critical and high severity test findings prior to future deployments. We will continue to monitor VA's actions to implement our recommendation as the department makes additional system deployments. • In February 2022, we reported that our work and VA's analyses indicated challenges with the quality of transferred data and with how the new EHR system worked for some users. 11 For example, VA identified errors in allergy, medication, and immunization data, which raise patient safety concerns. We recommended that VA establish and use performance measures and goals to ensure the quality of transferred data. The department concurred with our recommendation and began to take steps to address it. However, those steps had not yet been completed as of February 2023. 9GAO, Electronic Health Records: VA Has Made Progress in Preparing for New System, but Subsequent Test Findings Will Need to Be Addressed, GAO-21-224 (Washington, D.C.: Feb. 11, 2021). 10A critical test finding results in the failure of the complete software system. A high severity test finding results in the failure of the complete software system; however, there are acceptable workarounds. 11GAO, Electronic Health Records: VA Needs to Address Data Management Challenges for New System, GAO-22-103718 (Washington, D.C.: Feb. 1, 2022). Page 5 GAO-23-106685 According to federal guidance and other leading practices, change VA's Organizational management practices are intended to apply an organized and structured Change Management framework to the often chaotic and perplexing world of organizational change. 12 Effective change management techniques help managers to Activities Were plan, organize, and negotiate successful changes in the organization. The Partially Consistent objective of managing organizational change is to maximize the likelihood of successfully implementing change quickly and with reduced risk. with Leading Leading practices for change management activities include: (1) Practices developing a vision for change, (2) identifying and managing stakeholders, (3) effectively communicating, (4) assessing the readiness for change, (5) increasing workforce skills and competencies, (6) identifying and addressing potential barriers to change, (7) establishing targets and metrics for change, and (8) assessing the results of change. As shown in table 1, VA's organizational change management activities for the EHRM program were partially consistent with seven of the leading practices and not consistent with one leading practice. 12ProjectManagement Institute, Inc., Managing Change in Organizations: A Practice Guide, (Newtown Square, PA: 2013); Office of Personnel Management (OPM), Migration Planning Guidance Information Documents, Change Management Best Practices, October 2011; GAO, Business Process Reengineering Assessment Guide-Version 3, (GAO/AIMD-10.1.15); Prosci, The Prosci ADKAR® Model, A Goal Oriented Change Management Model to Guide Individual and Organizational Change; ISACA, COBIT 2019 Framework, 2019. ADKAR® is a registered trademark of Prosci, Inc. Page 6 GAO-23-106685 Table 1: Extent to Which the Electronic Health Record Modernization (EHRM) Program's Activities Were Consistent with Organizational Change Management Leading Practices Leading practice GAO assessment Developing a vision for change Partially consistent Identifying and managing stakeholders Partially consistent Communicating effectively Partially consistent Assessing the readiness for change Partially consistent Increasing workforce skills and competencies Not consistent Identifying and addressing potential barriers to change Partially consistent Establishing targets and metrics for change Partially consistent Assessing the results of change Partially consistent Legend: Consistent – VA provided evidence that it conducted organizational change management activities mostly consistent with leading practices. Partially consistent – VA provided evidence that it conducted organizational change management activities consistent with some of the leading practice criteria, but some key parts were not followed. Not consistent – VA did not provide sufficient evidence that it followed leading practices. Source: GAO analysis of Department of Veterans Affairs (VA) data. | GAO-23-106685 Practice 1: Developing a This practice includes: Vision for Change • identifying the compelling need for change and benefits of the desired (Partially Consistent) change that can motivate stakeholders to accept and willingly participate to make the change successful, and • understanding the business context and developing strategies to define the change approach through a formalized methodology. The department developed a vision to have a comprehensive EHR accessible across VA, DOD, and community care providers to enhance the quality of health care. To realize the vision, VA plans to modernize its EHR system with a commercial solution to improve the delivery of quality health care to veterans, enhance the provider experience, and promote interoperability. In addition to a new EHR system, VA's vision includes implementing standardized clinical practice workflow processes for use in the new system. However, VA has not established a VA-driven strategy for change. A VHA commissioned report from April 2021 noted the need for a VA-driven change management strategy to formalize the structure and people capabilities to support the readiness of end users and drive adoption. According to EHRM program officials, following the strategic review, VA Page 7 GAO-23-106685 had articulated the need for change for VA health care, established change champions at Veterans Integrated Services Networks (VISNs) and VA medical center sites, and augmented super users, among other activities. Also, VA had hired additional government staff to move towards VA-led change management. However, as of January 2023, it had not provided documentation of a VA-driven change management strategy. Practice 2: Identifying and This practice includes: Managing Stakeholders • identifying stakeholders, which are individuals, groups, departments, (Partially Consistent) and organizations that have a direct interest in the change effort and will be directly affected by and/or have influence over the change effort; and • obtaining stakeholder buy in by assessing the anticipated impact of the change. Given their power to sustain or derail a change initiative, stakeholders and their concerns should be identified and understood. The program identified stakeholder groups, created a stakeholder communication strategy and plan, and conducted numerous workshops at the national and local level for the purpose of engaging, identifying, and analyzing stakeholders. However, as we reported in June 2020, 13 VA did not always effectively communicate information to stakeholders, including medical facility clinicians and staff to ensure relevant representation at local workshop meetings. As a result, local workshops did not always include all relevant stakeholders. As of February 2022, the EHRM program office had updated local workshop agendas to identify VA stakeholders recommended to participate in each session. However, users continued to share examples where they reported that the program did not fully identify stakeholders for local workshops. For example, one user said it was difficult to determine the correct person that should attend the local workshops. Another user said they were not involved in many important meetings, which contributed to issues with the new system in that department. Because not all users were included in relevant local workshops, the program had not fully anticipated the impact of change to users and had not fully understood stakeholders and their concerns. In addition, in February 2022 we reported that the department did not have a stakeholder register 13GAO-20-473. Page 8 GAO-23-106685 to identify and engage key stakeholders for the EHRM program and recommended that they develop such a tool. 14 EHRM IO officials said that in August 2022 they began conducting workshops with Directors from future implementation sites to focus on site stakeholder engagement. If VA continues to focus on site stakeholder engagement, this should better position the department to effectively identify and manage stakeholders, while addressing our open recommendation. Practice 3: This practice includes: Communicating Effectively • communicating the what, when, why, and how of the change (Partially Consistent) frequently, and in a targeted and compelling manner; and • sustaining change through ongoing communication, consultation, and representation of stakeholders. The program defined a communication plan to engage with stakeholders involved with the EHR system implementation and deployment. The program used various methods to communicate with stakeholders, and documented over 5,000 completed communications between December 2018 and October 2022. These communications targeted enterprise, national, internal, and external site level staff. Examples of these communications included executive leadership briefings, kickoff meetings for initial deployment locations, training coordination activities, and site specific plans. However, users of the system indicated that information on system changes and the status of trouble tickets were not effectively communicated after initial system deployment. For example, a user stated that an update to the system caused changes to the user's workflows without being properly notified prior to the update. Further, in November 2020, the program identified a risk that a communication plan had not been established to inform VA end users of changes, major incident management, upgrades, and package releases. As of July 2022, this risk was still open and a communication plan for changes in sustainment had not been finalized. 14GAO-22-103718. Page 9 GAO-23-106685 In October 2022, EHRM program officials said that rather than develop the sustainment communication plan they were communicating through weekly user updates. They also had established an EHRM Sustainment Resource Center for all VA staff to access information on EHR changes, upgrades, and downtime events. EHRM program officials also said that there was communication of system downtimes and scheduled maintenance. The program planned to begin sending regular communications summarizing critical updates and establish a team to create a plan for communicating changes in sustainment. However, even with these activities, documentation of feedback from sites indicated the need to distribute more frequent updates on change request and system downtimes. Practice 4: Assessing the This practice includes: Readiness for Change • measuring the state of readiness using periodic checkpoints, analysis, (Partially Consistent) and metrics; and • resolving any potential problems in a timely fashion. The program assessed its readiness for change by conducting change readiness questionnaires (CRQ) to serve as a baseline assessment across the initial deployment sites and to allow a tailored change effort to address gaps. According to the EHRM Integration Office, CRQs measure staff readiness for change and are administered twice prior to deployment and once post-deployment. According to the program office, as of January 2023, VA had conducted 55 CRQs at 28 deployment sites. However, VA received response rates between 17 percent and 23 percent from the CRQs for the first three deployment sites. Specifically, in response to mid-deployment CRQ surveys, about 23 percent (297 of 1,320) of Mann-Grandstaff staff participated, 17 percent (296 of 1,737) of Central Ohio (Columbus) staff participated, and 21 percent (186 of 898) of Jonathan M. Wainwright (Walla Walla) staff participated. Therefore, VA had limited responses to assess readiness for change. In addition, results from the CRQs indicated that users were not ready for the change and the program did not have assurance that it had resolved potential problems in a timely fashion: • results from the Mann-Grandstaff VA Medical Center indicated low scores related to training and practicing in the new environment, and Page 10 GAO-23-106685 • results from the Jonathan M. Wainwright Memorial (Walla Walla) and the Central Ohio (Columbus) VA Medical Centers showed that staff had indicated low scores for the knowledge of how to change and the ability to implement the change on a day-to-day basis. Program officials said that CRQs are an indicator of readiness, but low results would not prevent deployment because the program was taking actions to provide user support. However, the program did not conduct another assessment before deploying the system to ensure their actions in response to concerns were effective and readiness was achieved. Practice 5: Increasing This practice includes empowering stakeholders with the knowledge for Workforce Skills and how to successfully change and gain the full benefits from the change by training them in the new processes, skills, and competencies needed Competencies (Not throughout the transition. Consistent) The program outlined goals and objectives of training, site preparation steps, key milestones for training, the process for scheduling courses and registering staff, and a process for continuous training improvement. However, training has been a weakness for the program: • training was a noted concern for users at Mann-Grandstaff in CRQs; • training was identified as an area for improvement in the Secretary's Strategic Review as the review noted that employees felt inadequately trained for their responsibilities which translated into operational errors; and • in July 2021, VA's Office of Inspector General identified numerous issues with training, including that users reported there was insufficient time for training, limitations with the training domain, challenges with user role assignments, and gaps in training support. In response to a September 2022 VA survey, most users (87 percent - 1,803 of 2,071) disagreed or strongly disagreed that their initial training prepared them well to use the EHR and that their ongoing EHR training/education was helpful and effective. 15 Further, our structured interviews corroborated users' views regarding training. Specifically, 47 of 15In response to the statement, my initial training prepared me well to use the EHR, 5 percent (102 of 2,071) of users agreed, 8 percent (166 of 2,071) were indifferent, and 87 percent (1,803 of 2,071) disagreed or strongly disagreed. In response to the statement, my ongoing EHR training/education is helpful and effective, 11 percent (217 of 2,063) of users agreed, 21 percent (432 of 2,063) were indifferent, and 69 percent (1,414 of 2,063) disagreed or strongly disagreed. This does not sum to 100 due to rounding. Page 11 GAO-23-106685 63 users disagreed or strongly disagreed that training was effective in preparing them to use the new system. 16 For example, users said that trainers were familiar with the system but were not familiar with VA's workflows and processes. EHRM program officials acknowledged that training did not meet users' expectations and effectively support the transition because the contractor- provided training focused on using the system. However, users needed additional training and support for learning new workflow processes simultaneously. They said the program took a number of actions to address training issues, including adding additional clinical experts to support onsite training and increased use of a hands-on practice environment (i.e., sandbox). According to the program, VA planned to conduct additional user adoption strategies to reinforce training such as more robust provider champion and super user networks, show and tell demonstrations, workflow adoption activities, and coaching support during and after go-live. EHRM program officials said they were working with VHA and Veterans Benefits Administration (VBA) partners to ensure that context on VA policy, process, and operations was provided by appropriate subject matter experts. In addition, in May 2022, the department notified Oracle Cerner about its concerns with training and in September 2022, Oracle Cerner announced that it would work with an outside entity to make the training more efficient, applicable, and useful for caregivers. EHRM program officials also noted that in the future, the program planned to conduct training earlier in the deployment timeline to allow time for additional workflow adoption activities and had initiated a plan to pilot the handoff of sustainment training to VHA. To address the lack of familiarity with VA workflows and processes, VA noted it has worked with Oracle Cerner to define additional change management activities. Nevertheless, VA has not yet demonstrated if these actions to increase workforce skills and competencies have been effective. Practice 6: Identifying and This practice includes: Addressing Potential • taking steps to identify and understand potential resistance barriers or Barriers to Change roadblocks throughout the change efforts; (Partially Consistent) 16Of 63 users, 47 disagreed or strongly disagreed that they had been provided effective training on how to use the new EHR system, six neither agreed nor disagreed, and 10 agreed. Page 12 GAO-23-106685 • taking actions to address barriers that might derail change efforts when they arise; and • examining daily activities of impacted groups experiencing change to identify and understand legitimate causes of resistance caused by design, execution, or implementation issues. Legitimate issues should spur re-evaluation of the solution design of the project. The program identified activities to regularly monitor resistance to change. In addition, VA conducted site visits and change readiness questionnaires, among other things, to gather feedback and propose actions or recommendations to address feedback. Oracle Cerner and VHA also conducted change impact assessments to identify the level of complexity and effect of new workflows. Specifically, according to EHRM program officials, the program initiated an evaluation of high-risk workflows to optimize them and conducted a sprint to review the configuration and workflows in key areas to address users' experience. In addition, the Secretary's Strategic Review identified barriers that needed to be addressed and, as of January 2023, VA had completed 45 of 69 actions identified in the review, and 24 were in progress. VA planned to complete these action items by October 2024. Nevertheless, the EHRM program did not always adequately take action to address barriers such as user issues. For example, although the program tracked user issues and encouraged users to submit tickets to the help desk, many users (43 of 63) we interviewed said that they were dissatisfied or very dissatisfied with obtaining resolution to problems with the new EHR system. 17 Several users said that it was 6 months or even a year before they heard back about a ticket. Multiple users said that the help desk closed tickets without satisfactory resolution. Practice 7: Establishing This practice includes: Metrics and Targets for • establishing measurement systems and targets to measure the Change (Partially adoption of the change, and Consistent) • establishing measurement systems and targets to measure the resulting outcomes of the change. 17Of 63 respondents, 43 said they disagreed or strongly disagreed that the help desk provided adequate resolution to problems they encountered with the new EHR system, 10 neither agreed nor disagreed, and four agreed that the help desk provided adequate resolution to problems they encountered with the new EHR system. Six respondents said they had no basis to judge. Page 13 GAO-23-106685 The program planned a number of key activities for measuring the adoption of change. For example, the EHRM program proposed metrics such as the amount of time spent in the EHR system and the number of patients seen in an ambulatory setting. The plans also included a post- deployment survey of users aimed at measuring users' perspective on their ability to use the new system. In this post-deployment survey, a score of 68 of 100 is an average score. Additionally, VA conducted an EHRM system user satisfaction survey. The program also identified metrics for performance of the new system such as measuring veteran experience, health care operations, workforce support, and quality and safety. However, beyond the average score, VA had not fully established targets to measure the adoption of the change. In addition, the department has not yet fully established goals or key performance indicators for the performance metrics. Further, VA did not have a plan that outlined the metrics, including agreed upon targets, to measure the results of the change. VA reported in November 2022 that it was continuing to refine functional and technical quality standards to define success, including metrics to define access to care and clinical operational efficiency but did not provide a timeline for when it would be final. Practice 8: Assessing the This practice includes: Results of Change • measuring adoption of the change and obtaining feedback from (Partially Consistent) stakeholders to help determine how successful the change was, • measuring and reporting established metrics to demonstrate resulting outcomes of the change, and • taking action needed to ensure that the change is reinforced and sustained. To measure adoption, the EHRM program collected data, such as amount of time spent in the EHR system and the number of patients seen in an ambulatory setting. In addition, as previously mentioned, the program has conducted a number of post-deployment surveys since deployment of the system at Mann-Grandstaff VA Medical Center in October 2020. Further, the program has been tracking performance metrics such as veteran experience, health care operations, workforce support, and quality and safety since initial deployment in October 2020. Page 14 GAO-23-106685 However, VA has not fully identified specific targets. In addition, users shared examples of concerns about their productivity using the new system and veterans' access to care. For example, several users told us that their departments were at about 50-80 percent of pre-go live volumes. One user said that workload that was managed by 15 staff prior to go live now required additional staff. Another user said that previously, they could process referrals within a week, but now it took them longer and there were over 8,000 referrals in the queue. Further, several users said that patient safety issues had increased. In addition, the program had not demonstrated that it had taken action needed to ensure that the change has been reinforced and sustained. For example, a March 2021 survey aimed at measuring Mann-Grandstaff users' perspective on their ability to use the new system noted that 82 percent of users agreed or strongly agreed that the new EHR was cumbersome to use, and 84 percent agreed or strongly agreed that the new EHR was unnecessarily complex. VA's Organizational The program's organizational change management activities were not Change Management fully consistent with leading practices for a variety of reasons. First, VA did not have a VA-driven strategy for how its efforts would supplement the Activities Were Not Fully contractor-led change management activities for deployment. According Consistent with Leading to EHRM program officials, the contractor's change management Practices for Various activities focused on the activities required to deploy the new system. Reasons However, these activities did not address user challenges with transitioning to new workflow processes. Further, EHRM officials noted that the program office had experienced transition in change management leadership and vacancies in their change management staffing. This limited the resources available for coordinating and implementing change management activities. Until the program implements all eight of the leading practices for change management, future deployments could be at risk of similar change management challenges. This could hinder users' ability to effectively use the system, impede their knowledge of new workflows, and limit the utility of system improvements. Further, the results of post-deployment questionnaires demonstrate the need for improvements in organizational change management activities. Specifically, according to VA-reported data, users provided feedback that Page 15 GAO-23-106685 was below average regarding their abilities to use the new EHR system. 18 Based on the program's research, a score of 68 out of 100 was considered average and scores below 68 were below average (see table 2). Table 2: Department of Veterans Affairs Electronic Health Record (EHR) Modernization Program Post-deployment Feedback on New EHR System Site Date Average summed system usability scale score Mann-Grandstaff VA Medical Center February - March 2021 24.38 Jonathan M. Wainwright (Walla Walla) May - June 2022 32.33 Central Ohio (Columbus) July 2022 24.14 Roseburg July - August 2022 23.19 Southern Oregon (White City) August 2022 24.72 Source: GAO analysis of Department of Veterans Affairs reported data. | GAO-23-106685 In our interviews, users expressed concern about staff morale and burnout. One user reported working 60 hours a week and trying not to drown in carrying out duties because completing chart reviews, which used to take 15-30 minutes using the old system, was now taking hours or even days. Other users said that providers were volunteering their time, and one user said this was because tasks took 10-15 percent more time to complete. Finally, users noted that staff in their department had resigned, specifically due to problems with the new EHR system. Additional details about users' satisfaction with the new system are discussed in the next section of this statement. Accordingly, we recently made seven recommendations to VA regarding change management activities. 19 Specifically, we recommended that the Secretary of VA: • document a VA-specific change management strategy to formalize its approach to drive user adoption; 18The response rates to these questionnaires ranged from 12 percent to 22 percent. 19These recommendations were conveyed in our March 10, 2023, briefing to Congressional committees and members and will be published in a report. Page 16 GAO-23-106685 • ensure that the department's planned improvements to communication of system changes meets users' needs for the frequency of the updates provided; • take steps to improve change readiness scores prior to future deployments; • ensure steps taken by the EHRM program and Oracle Cerner to increase workforce skills and competencies through improved training and related change management activities have been effective; • address users' barriers to change, by ensuring planned completion of all actions identified in the Secretary's Strategic Review; • develop a plan, including a timeline, for establishing (1) targets for measuring the adoption of changes and (2) metrics and targets to measure the resulting outcomes of the change; and • measure and report on outcomes of the change and take actions to support users' ability to use the system to reinforce and sustain the change. VA concurred with our recommendations. GAO and federal IT guidance recognize the importance of defining Users Generally program goals and related performance targets and using such targets to Expressed assess progress in achieving the goals. 20 Also, leading practices identify continuous customer feedback as a crucial element of IT project success, Dissatisfaction with from project conception through sustainment. Particularly for IT programs the New System, but like EHRM, where development activities are ongoing, customer (i.e., end user) perspectives and insights can be solicited through various methods. VA Has Not Such methods include interviews and satisfaction surveys, to validate or Established Goals raise questions about the project's implementation. Further, leading practices emphasize that periodic user satisfaction data be proactively used to improve performance and demonstrate the level of satisfaction the project is delivering. Measuring user satisfaction with the system is essential for monitoring progress towards pre-established goals or targets 20GAO, Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity, GAO-04-394G (Washington, D.C.: March 2004); Executive Office of the President, Office of Management and Budget, Evaluating Information Technology Investments, A Practical Guide (November 1995), Office of Management and Budget, Preparation, Submission, and Execution of the Budget, OMB Circular No. A-11 (August 2022), and General Services Administration, Modernization and Migration Management (M3) Playbook, accessed Oct. 20, 2022, https://www.ussm.gov/m3. Page 17 GAO-23-106685 and allows programs to understand whether users' operational needs have been met. VA Has Taken Steps to VA has taken steps to obtain feedback on the performance and Obtain Feedback; Users implementation of EHRM. Specifically, in September 2022, VA conducted a survey of users from two regions, VISN 10 and VISN 20, where the new Generally Expressed EHR system had been deployed. In addition, VHA conducted another Dissatisfaction with the survey in September 2021 and September 2022 to assess Mann- New System Grandstaff employees' perceptions of the implementation of the new EHR. In September 2022, VA worked with a contractor to conduct a user satisfaction survey to determine user satisfaction with the Oracle Cerner system. In December 2022, the contractor reported on VA's results in comparison to other health care systems. The results of the survey indicated that users were not satisfied with the performance of the new system or the training for the new system. For example, about 6 percent (120 of 2,066) of users agreed that the system enabled quality care. 21 In addition, about 4 percent (92 of 2,074) of users agreed that the system made them as efficient as possible. 22 In addition, fewer VA users reported that they agreed that the system enabled them to deliver high-quality care when compared to DOD and other health care systems. For example, about 23 percent (1,000 of 4,432) of DOD users agreed that the system enabled quality care. In response to the survey of Mann-Grandstaff users' morale in September 2021, most users noted that as a result of the new EHR implementation, their morale, job satisfaction, and level of burnout had worsened (see table 3). 23 21Inresponse to the statement, the EHR enables me to deliver high-quality care, 6 percent (120 of 2,066) users agreed, 15 percent (306 of 2,066) were indifferent, and 79 percent (1,640 of 2,066) disagreed or strongly disagreed. 22In response to the statement, the EHR makes me as efficient as possible, 4 percent (92 of 2,074) users agreed, 6 percent (130 of 2,074) were indifferent, and 89 percent (1,852 of 2,074) disagreed or strongly disagreed. 23About 56 percent (833 of approximately 1,500) recipients responded to this survey. Page 18 GAO-23-106685 Table 3: Department of Veterans Affairs Veterans Health Administration Survey Feedback on New Electronic Health Record (EHR) System, as of September 2021 Improved Not changed Worsened Number Percent Number Percent Number Percent Total As a result of the EHR 6 0.7% 133 16.0% 691 83.3% 830 implementation, my morale has: As a result of the EHR 6 0.7% 173 20.8% 652 78.5% 831 implementation, my job satisfaction has: As a result of the EHR 4 0.5% 154 18.6% 670 80.9% 828 implementation, my level of burnout has: Source: GAO analysis of Department of Veterans Affairs' data. | GAO-23-106685 In September 2022, most users still noted that as a result of the new EHR implementation, their morale, job satisfaction, and level of burnout had worsened (see table 4). 24 Table 4: Department of Veterans Affairs Veterans Health Administration Survey Feedback on New Electronic Health Record (EHR) System, as of September 2022 Improved Not changed Worsened Number Percent Number Percent Number Percent Total As a result of the EHR 91 12.3% 119 16% 532 71.7% 742 implementation, my morale has: As a result of the EHR 90 12.1% 142 19.2% 509 68.7% 741 implementation, my job satisfaction has: As a result of the EHR 87 11.8% 123 16.7% 528 71.5% 738 implementation, my level of burnout has: Source: GAO analysis of Department of Veterans Affairs' data. | GAO-23-106685 When asked to rate the improvement in the EHR since they began using the new medical record, of 741 respondents, 231 (31 percent) said no improvement, 372 (50 percent) said minimal improvement, 49 (7 percent) said moderate improvement, and 89 (12 percent) said great improvement. 24About 52 percent (742 of approximately 1,440) recipients responded to this survey. Page 19 GAO-23-106685 Similarly, our interviews from the first three deployment sites indicated that users were not satisfied with the new system. Specifically, 51 of 63 users said that they disagreed or strongly disagreed that overall they were satisfied with the new EHR system. In addition, 48 of 63 users said they disagreed or strongly disagreed that the new EHR system met the expectations they had prior to and during go-live. Table 5 provides the results from our interviews regarding user satisfaction of the new system. Table 5: Users Responses to Statements on New Electronic Health Record (EHR) System Strongly Agree Neither Disagree Strongly No basis Total agree agree nor disagree to judge disagree The new EHR system meets the 2 9 3 23 25 1 63 expectations I had prior to and during go- live. The new EHR system is available when I 1 18 11 29 4 0 63 need it. Health data was migrated accurately from 0 11 6 22 21 3 63 the old EHR system to the new EHR system. Compared to the old system the new EHR 1 1 6 21 33 1 63 system requires fewer steps to accomplish what I need to do. Overall, I am satisfied with the new EHR 1 6 5 31 20 0 63 system. Source: GAO analysis of interviews with Department of Veterans Affairs officials. | GAO-23-106685 Further, Mann-Grandstaff users' responses to structured interview questions only minimally indicated improved satisfaction or changes in the perceptions of the effect of the new EHR system on productivity or quality of care from our interviews in 2021 to our interviews in 2022. Specifically, as shown in table 6, in 2021, most users (23 of 26) said they disagreed or strongly disagreed that overall they were satisfied with the new EHR system. In 2022, most users (18 of 23) said they disagreed or strongly disagreed that they were satisfied with the new EHR system (see table 6). Page 20 GAO-23-106685 Table 6: Mann-Grandstaff User Satisfaction with New Electronic Health Record (EHR) System in 2021 and 2022 Do you agree or disagree with the following statement based on your Mann-Grandstaff 2021 Mann-Grandstaff 2022 current experience using the new EHR system? (Out of 26) (Out of 23) Overall, I am satisfied with the new EHR system. Strongly agree 0 0 Agree 0 1 Neither agree nor disagree 3 4 Disagree 6 9 Strongly disagree 17 9 Source: GAO analysis of interviews with Department of Veterans Affairs officials. | GAO-23-106685 VA Has Not Established VA has not established targets (i.e., goals) to assess user satisfaction. Goals for User Satisfaction EHRM provided several reasons for why the program had not established specific goals for user satisfaction for the system: • In February 2022, EHRM program officials stated there was an opportunity for additional metrics such as user satisfaction targets in the future. • In October 2022, EHRM program officials stated they were focused on addressing technical changes to improve the system usability before establishing targets or goals for user satisfaction. Nevertheless, until VA establishes goals for user satisfaction, the department will be limited in its ability to objectively measure progress toward improving EHRM users' satisfaction with the system. The department will also lack a basis for determining when satisfaction has improved. Such a basis would help ensure that the system is not prematurely deployed to additional sites, which could risk patients' safety. Accordingly, we recently recommended that the Secretary of VA establish user satisfaction targets (i.e., goals) and demonstrate improvement toward meeting those targets prior to future system deployments. 25 VA agreed with this recommendation. 25These recommendations were conveyed in our March 10, 2023, briefing to Congressional committees and members and will be published in a report. Page 21 GAO-23-106685 Efforts to identify and address system issues can be supported by VA Did Not activities such as resolving trouble tickets quickly and conducting an Adequately Identify independent operational assessment. VA did not adequately identify and address system issues. Specifically, VA did not ensure that trouble tickets and Address System for the new EHR system were resolved within timeliness goals, but Issues subsequently worked with the contractor to reduce the number of tickets that were over 45 days old. Additionally, although VA has assessed the system for user performance at two sites, as of January 2023, VA had not conducted an operational assessment to evaluate if the new EHR system satisfies the intended use and user needs in the operational environment. VA and Its EHR System VA's contract with Oracle Cerner addressed the support and resolution of Contractor Have Worked trouble tickets during and after implementation of the new EHR system. Based on impact and urgency, each ticket is assigned a priority of critical, to Improve Trouble Ticket high, medium, or low. 26 Resolution According to a service level agreement (SLA) between VA and Oracle Cerner, resolution timeliness goals varied depending on the ticket priority levels as follows: • Critical tickets: 100 percent of trouble tickets resolved or mitigated through VA approved mitigation strategy within 5 hours and closed within 24 hours. 27 • High tickets: 90 percent of trouble tickets resolved within 16 hours and no single ticket exceeds 64 hours. 26Critical - A patient safety condition exists or greater than 25 percent of concurrent users across a medical center are unable to process transactions or access managed solutions critical to their ability to conduct daily business; and no bypass or alternative is available. High - When (15-25 percent) of concurrent users across a VAMC and associated facilities are unable to process transactions or access managed solutions required to conduct daily business or a component of managed software required to compete a crucial workflow is non-functional for more than one user and no bypass or alternative is available. Medium - A component, minor solution, or procedure is down, unusable, or difficult to use but, no immediate impact on service delivery, financial, or patient care. Critical and high problems that have an acceptable workaround, or bypass available will be assigned as a moderate incident. Low - A component, procedure or personal application (not critical to Client) is unusable. No impact to business, single Incident failure, and an acceptable workaround, alternative, or bypass is available. 27A ticket is considered 'resolved' when Cerner places the ticket in a 'Client Action' status for the client to approve / confirm the issue is addressed. A ticket is considered 'completely resolved' when VA has approved and confirmed that a trouble ticket placed in "Client Action" has been fully addressed. 'Completely resolved' and 'closed' are used interchangeably. In the trouble ticket data, 'closed' is a ticket which has been resolved and cannot be reopened. Page 22 GAO-23-106685 • Medium tickets: 80 percent of trouble tickets resolved within 4 business days and no single ticket exceeds 60 calendar days. • Low tickets: 80 percent of trouble tickets resolved within 6 business days and no single ticket exceeds 60 calendar days. However, VA determined that during the 25 month period from December 2020 to December 2022, Oracle Cerner did not meet the SLA established for the resolution of system trouble tickets. Specifically, Oracle Cerner did not meet the SLA for: • critical severity trouble tickets for 4 of the 25 months • high severity trouble tickets for 15 of the 25 months • medium severity trouble tickets for 21 of the 25 months, and • low severity trouble tickets for 24 of the 25 months. To address a higher-than-expected volume of tickets that were not addressed within 60 calendar days or less, in August 2022, Oracle Cerner developed a 120-day plan to reduce the number of open tickets that were 45 days or older. Oracle Cerner developed its plan in response to a VA letter of concern regarding the new EHR system's performance. As of January 2023, Oracle Cerner had reduced the number of tickets that were 45 days or older from 714 to 108. 28 Nevertheless, as of December 2022, VA had over 1,400 open tickets, which was more than the number of tickets at the end of 2020 and 2021. Figure 1 depicts the number of open trouble tickets per month from October 2020 to December 2022. 28According to Cerner's plan some incidents could be converted to change requests, if appropriate. Page 23 GAO-23-106685 Figure 1: Department of Veterans Affairs Electronic Health Record Modernization Open Trouble Tickets per Month Oracle Cerner officials provided explanations for the difficulties with meeting the SLA. • VA's IT systems are more complex than DOD's, which contributes to a larger number of trouble tickets. • Oracle Cerner relies on the local informatics staff to help triage the tickets, but some VA sites had little informatics support, which increases the burden on the Oracle Cerner help desk. 29 • Oracle Cerner needed to apply additional staffing resources to address the problem. Additionally, according to VA's strategic review status report, the department recognized that its capacity to resolve the volume of tickets at 29Health Informatics is a multidisciplinary and integrative field that focuses on health information management and information technology in support of health care. The field of health informatics draws from computer, cognitive, and social sciences for the development, change management, implementation, configuration, deployment and evaluation of systems that manage health information. Page 24 GAO-23-106685 the five deployment sites was overwhelmed due to, among other things, an insufficient number of subject matter experts. Until the program resolves trouble tickets according to established time frames, users' system issues will not be resolved in a timely manner. Additionally, there is a risk that VA will not be able to address users' system issues effectively going forward, particularly when larger sites go live. Accordingly, we recently recommended that the Secretary of VA ensure that future system trouble tickets are resolved within established timeliness goals. 30 VA concurred with our recommendation. VA Has Not Conducted According to leading practices for software verification and validation, a Independent Operational product should be evaluated to determine whether it satisfies the intended use and user needs in the operational environment. 31 An operational Assessments to Validate assessment is an evaluation of operational effectiveness and operational That the System Meets suitability made by an independent operational test activity with user User Needs support as required. 32 The EHRM program's master test plan from May 2021 described plans to execute an independent post-production validation and operational assessment to assess the degree to which the new EHR met the users' needs in their daily operational use in the production environment. According to the program's test plan, the purpose of the operational assessment was to evaluate the system's efficiency, effectiveness, usability, user satisfaction, and training. However, VA has not conducted an operational assessment and, as of January 2023, did not plan to do so. EHRM program officials said that they did not plan to execute an independent operational assessment because such an assessment would be duplicative to existing post-go-live evaluations and change assessment surveys, and disruptive to site operations. Further, the EHRM Master Test Plan was updated to remove the requirement for an operational assessment. 30These recommendations were conveyed in our March 10, 2023, briefing to Congressional committees and members and will be published in a report. 31IEEE Standards Association, IEEE Standard for System, Software, and Hardware Verification and Validation (IEEE Std. 1012-2016), (September 2017). 32Defense Acquisition University Glossary, accessed September 1, 2022, https://www.dau.edu/acquipedia/pages/articlecontent.aspx?itemid=46. Page 25 GAO-23-106685 In July 2022, officials from VHA and the EHRM program office conducted a post-go-live study at the Columbus deployment site. These officials observed slow system response, system errors, user interface issues, and inefficient workflows that affected the end user experience. In addition to these observations, the study report stated that the scope of the assessment was limited and recommended further usability assessments. Following the July 2022 study, VA conducted a review focused on standardization, usability, and safety issues at the five deployment sites. 33 The study team reviewed 300 issues and prioritized 30 to address that were related to patient safety. Additionally, according to EHRM program officials, in September 2022 they visited the Columbus deployment site to obtain feedback from users on high-risk workflows. Program officials said they also conducted an assessment at two sites in an effort to improve system performance. However, because these assessments were not conducted independent of the program, they lack the objective evaluation and analysis characteristic of an independent operational assessment. Until an independent operational assessment of the new EHR system is conducted, VA will be limited in its ability to validate that the system is operationally suitable and effective, and to identify, track, and resolve key operational issues. An operational assessment, particularly if conducted by an independent entity, would enable the department to systematically catalog, report on, and track resolution of assessment findings with greater rigor, transparency, and accountability. Accordingly, we recently recommended that the Secretary of VA reinstitute plans to conduct an independent operational assessment to evaluate the suitability and effectiveness of the new EHR system for users in the operational environment. 34 VA agreed with this recommendation. In summary, the program's organizational change management activities were not fully consistent with eight leading practices. These practices are especially important given that VA's transition to the new EHR was challenging for users at the initial deployment sites. In addition, VA undertook several efforts to assess user satisfaction with the new system, 33VA referred to this review as the EHRM Sprint Project. The Sprint Project work streams included VHA EHR governance processes, medical order issues, clinical episode review team review and assessment actions, and collaborative readiness. 34Theserecommendations were conveyed in our March 10, 2023, briefing to Congressional committees and members and will be published in a report. Page 26 GAO-23-106685 but results indicated that users were dissatisfied with the system. Further, VA had not established targets or goals for user satisfaction. Consequently, it is not evident what basis the department will use to determine when satisfaction has sufficiently improved to support a decision to deploy the system at additional sites. Such a basis is critically important to ensuring that systems not be deployed prematurely and pose unnecessary risks to patient health and safety. Finally, VA did not ensure that system issues had been addressed within established timeliness goals nor has it conducted an independent operational assessment, which could be beneficial in validating that the system satisfies user needs in the operational environment. We will continue to assess these issues and to track VA's progress in response to our recommendations. Chairman Tester, Ranking Member Moran, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have. If you or your staff have any questions about this testimony, please GAO Contact and contact Carol C. Harris, Director, Information Technology and Staff Cybersecurity, at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found Acknowledgments on the last page of this statement. GAO staff who made key contributions to this testimony are Mark Bird (Assistant Director), Merry Woo (Analyst- in-Charge), Tim Barry, Chris Businsky, Quintin Dorsey, Rebecca Eyler, Ash Harper, Igor Koshelev, Christy Ley, Monica Perez-Nelson, Rachael Scott, Eric Trout, Walter Vance, Adam Vodraska, and Charles Youman. Page 27 GAO-23-106685 Appendix I: Objectives, Scope, and Appendix I: Objectives, Scope, and Methodology Methodology The review upon which this testimony was based culminated in a briefing to Congressional committees and members. 1 The objectives of our review were to determine the extent to which: 1. VA has employed organizational change management strategies for the EHRM program consistent with leading practices, 2. VA has assessed satisfaction with the new system, and 3. VA has identified and addressed EHR system issues. To address the first objective, we conducted a literature search for organizational change management leading practices. 2 We identified leading change management practices that are applicable to organizational transitions, such as VA's EHR system modernization. We then evaluated VA's activities against these practices by examining program plans for organizational change management and discussing the program's approach with cognizant EHRM program officials. To assess whether the program's activities were aligned with its planned approach and leading practices, we reviewed EHRM change management documentation, such as wave deployment plans, stakeholder communication strategy and plan, change impact analyses, site deployment and work plans, and change readiness questionnaire reports. We supplemented our analysis with examples from interviews with users from the Mann-Grandstaff VA Medical Center, Jonathan M. Wainwright VA Medical Center (Walla Walla), and VA Central Ohio Health Care System (Columbus), the three locations where the new system was first deployed. To address the second objective, we obtained and reviewed results of surveys that VA conducted to determine users' satisfaction with the new 1On March 10, 2023, we provided a briefing on the results of our review to the House and Senate Committees on Appropriations, Subcommittees on Military Construction, Veterans' Affairs, and Related Agencies; House and Senate Committees on Veterans' Affairs; Rep. Jim Banks; and Rep. Susie Lee. We plan to publish the briefing in a report. 2Project Management Institute, Inc., Managing Change in Organizations: A Practice Guide (Newtown Square, Pa.: 2013); Office of Personnel Management, Migration Planning Guidance Information Documents, Change Management Best Practices (Oct. 7, 2011); GAO, Business Process Reengineering Assessment Guide, version 3, GAO/AIMD-10.1.15 (Washington, D.C.: May 1997); ISACA, COBIT 2019 Framework (2019); and Prosci, The Prosci ADKAR® Model, A Goal Oriented Change Management Model to Guide Individual and Organizational Change, accessed Feb. 21, 2021, https://www.prosci.com/methodology/adkar. ADKAR® is a registered trademark of Prosci, Inc. Page 28 GAO-23-106685 Appendix I: Objectives, Scope, and Methodology EHR, including a survey conducted by VHA to assess Mann-Grandstaff employees' perceptions of the implementation of the Oracle Cerner EHR and post-deployment system usability surveys conducted by the EHRM program office. We obtained documentation regarding the department's administration of its user satisfaction surveys to determine that the data were sufficiently reliable for our purposes. We met with EHRM program officials and VHA officials to discuss whether the department had established any goals for user satisfaction. We conducted structured interviews with selected users from the Mann- Grandstaff VA Medical Center, Jonathan M. Wainwright VA Medical Center (Walla Walla), and VA Central Ohio Health Care System (Columbus), the three locations where the new system was first deployed. Specifically, we conducted structured interviews with 63 users at these three locations between April and August 2022. The methodology for selecting interviewees was as follows: we received a list of Mann-Grandstaff VA Medical Center employees who have been involved with national EHR councils. First, we conducted two pre-test interviews with leadership staff and made minor revisions to our structured interview instrument. We then selected one user from each of the 16 departments represented among the councils. For departments that had multiple users involved in the national councils, a user was randomly selected. In addition, two users were selected based on recommendations from the Mann-Grandstaff Medical Center leadership. Finally, an additional six users were selected based on recommendations from interviewees for a total of 26 interviews between April 2021 and June 2021. Following these interviews, we conducted additional interviews with 23 of the same users between April 2022 and June 2022. While the users' responses cannot be generalized to the entire population of EHR users at the initial deployment site, they represent a broad range of user roles and clinical areas at the sites. Following interviews with Mann-Grandstaff VA Medical Center, we conducted structured interviews with selected EHR users from the Jonathan M. Wainwright Memorial VA Medical Center (Walla Walla) and VA Central Ohio Health Care System Center (Columbus). We conducted 40 interviews in total, 19 from Walla Walla and 21 from Columbus between June 2022 and August 2022. The methodology for selecting interviewees at these two locations was as follows: we requested and received a list of representatives from a variety of clinical areas from both sites. We then interviewed the chief of staff at Page 29 GAO-23-106685 Appendix I: Objectives, Scope, and Methodology each location. In addition, we selected 18 user representatives from Walla Walla and randomly selected 20 users from the list of user representatives from Columbus, excluding those who were not obvious users of the system. While these users' responses cannot be generalized to the entire population of EHR users at these deployment sites, they represent a broad range of user roles and clinical areas at the sites. To address the third objective, we obtained data on system trouble tickets from October 2020 to December 2022. We analyzed VA's data on the contractor's performance meeting time frames established in the service level agreement (SLA) for the contractor to address system trouble tickets. We also obtained a summary of monthly reports from Oracle Cerner to VA on trouble ticket resolution with respect to the SLA. We also analyzed the trouble ticket data for trends in the number of open tickets at the end of each month. We assessed the reliability of the trouble ticket data by reviewing it for obvious errors and missing data and interviewed responsible officials about any discrepancies in the data. We determined the data to be sufficiently reliable for the purposes of our briefing. We also obtained documentation of the EHRM program's testing activities, including test plans and results. We then analyzed the plans, as well as test activities that had already been completed, to determine whether they constituted an independent operational assessment. We supplemented our analyses for our objectives by interviewing relevant VA officials, including the EHRM IO Executive Director, Functional Champion, and Deputy Chief Information Officer. We conducted our performance audit from February 2021 through March 2023 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Page 30 GAO-23-106685 This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. The Government Accountability Office, the audit, evaluation, and investigative GAO's Mission arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no cost is Obtaining Copies of through our website. Each weekday afternoon, GAO posts on its website newly GAO Reports and released reports, testimony, and correspondence. You can also subscribe to GAO's email updates to receive notification of newly posted products. Testimony Order by Phone The price of each GAO publication reflects GAO's actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO's website, https://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Connect with GAO Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov. Contact FraudNet: To Report Fraud, Website: https://www.gao.gov/about/what-gao-does/fraudnet Waste, and Abuse in Automated answering system: (800) 424-5454 or (202) 512-7700 Federal Programs A. Nicole Clowers, Managing Director, ClowersA@gao.gov, (202) 512-4400, U.S. Congressional Government Accountability Office, 441 G Street NW, Room 7125, Washington, Relations DC 20548 Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Stephen J. Sanford, Managing Director, spel@gao.gov, (202) 512-4707 Strategic Planning and U.S. Government Accountability Office, 441 G Street NW, Room 7814, External Liaison Washington, DC 20548 Please Print on Recycled Paper.