inne AN EVALUATION 0] | | RMP REVIEW CRITERIA Final Report on Contract HSM 1 10-72-329 MEDICAL CARE AND EDUCATION FOUNDATION, INC. BOSTON, MASSACHUSETTS a i Sy es z 2B AN EVALUATION OF THE RMP REVIEW CRITERIA Final Report on Contract HSM 110-72-329 Medical Care and Education Foundation, Inc. Boston, Massachusetts June 1973 foe id ie ES EE 4 aye necig This report was prepared in partial fulfillment of Health Services and Mental Health Administration Contract HSM 110-72-329, "An Evaluation of ; the RMP Review Driteria." 2 Harold W. Keairnes, MD Project Birector : Betty Bradshaw, MBA Project Manager “ Allan L. Willey, MBA 4 Project Consultant 2 | Project Staff = Kdward Miller, AB 23 Patricia Kress, AB es Maureen Fitzgerald, MA . Maureen Vachss, AB e Patricia Miller, AB EE a iii TABLE OF CONTENTS ee Heel “ PAGE Acknowledgements vii Executive Summary ix 2 Chapter I The Setting for the Evaluation 1 Chapter ITI Approach to the Evaluation and of Reports of Component Activities 19 ~ A. Statistical Studies of RMP oh Review Criteria Scores 23 . B. Reported Usefulness of the RMP os Review Criteria Scores 29 is C. Values Expressed in Descriptions of Local RMPS 43 (Ss D. Summary of Conferences with RMPS Staff 53 (3 S Chapter III Evaluation Findings 61 Chapter IV - Recommendations 69 Appendix A RMP Review Criteria - 87 Appendix B Description and Scope of Work — . 95 for Contract Lee § % ACKNOWLEDGEMENTS The collaboration and support of the Project Officer, Joseph de la Puente, contributed significantly to the completion of this contract. Much of the work could not have been done without his support and the support of Roland Peterson, Director of the Office of Planning and Evaluation and Robert Walkington, Chief, Evaluation Branch. Ms. Judy Silsbee of the Division of Operations and Development provided perspectives that contributed to the analysis of the findings in terms of the needs of RMPS. Throughout - this contract, RMPS staff, at all levels throughout the organization, were supportive and helpful. Fred R. Bookstein, Harvard University, consulted in the design of the statistical studies, supervised the computer analysis, and provided the primary interpretation of the findings. Consultations with James R. Miller, III, Stanford University and Alan Sheldon, Harvard University served to provide guidance in technical aspects of worth assessment procedures and conference techniques. Robert W. Murphy, Executive Director of the Medical Care and Education Foundation, Inc. was a ready resource for identifying the concerns of RMP coordinators and for helping the project director maintain a practical perspective in this work. Patricia S. Keairnes reviewed the manuscript at multiple stages in its development and made many substantive suggestions. vii EXECUTIVE SUMMARY OF FINAL REPORT ON AN EVALUATION OF THE RMP REVIEW CRITERIA Contract No. HSM 110-72-329 ix we Mea take LN 3 rd Regional Medical Programs are a.federally funded program that provides grants to fifty-six local Regional Medical Programs "to improve generally the quality and enhance the capacity of the health manpower and facilities available to the Nation and to improve health services ‘for persons residing in areas with limited services" by promoting regional cooperative arrangements among medical schools, research institutions, and hospitals of research and training and regional linkages among health care institutions and providers that will make available to their patients the latest advances in the prevention, diagnosis, treatment and rehabilitation of heart disease, cancer, stroke, kidney disease and other related diseases. Because the legislation provides such a broad mandate, all applications for grant support are subjected to a formal review and approval process. The three year program grant application undergoes a site visit evaluation, Regional Medical Program Review Committee consideration, and National Advisory Council action. Annual continuation grant applications are reviewed by the Staff Anniversary Review Panel (SARP). Supplemental grant applications are reviewed by SARP and the National Advisory Council. To lend uniformity to the review process, a set of evaluation criteria were developed and introduced into the review process in 1971. In 1972, after the Regional Medical Program Review Criteria had been in use for approximately one year, a decision was made that an independent contractor should evaluate them and their contribution to the review process. The Scope of Work for that contract called for an evaluation of previous experience and recommendations regarding their continued use. This is a summary of the final report from that contract. Description of the RMP Review Criteria The RMP Review Criteria consist of three major criteria © groupings ,twenty individual criteria and more than one hundred questions. The questions under each individual criteria are intended to clarify the meaning of the criteria and to indicate the basis upon which a numerical] score can be assigned. The numerical score reflects the reviewers’ evaluation of a local regional medical program against the individual criteria. The numerical scores for individual criteria are summed to obtain scores for the major criteria and for all criteria. Weights assigned to each criteria specify their relative importance. Regional Medical Programs are classified as "A", "B" or "C" regions on the basis of their average weighted total scores. The Review Criteria play a role in the review process at several different levels. They are a general guide for the preparation of the historical profile in the staff briefing document, a summary report prepared by staff in the Division of Operations and Development to accompany the submitted triennial or anniversary review grant appli- xi cation through the review process. Site visit reports related to the review of triennial grant applications are supposed to be formatted according to the Review Criteria and are expected to summarize all the relevant information in regards to each individual criteria. The Review Committee for triennial grant applications and the Staff Anniversary Review Panel for anniversary review grant applications are required to score the programs under review against the Review Criteria and to include the scores with their reports and recommenda- tions. The scores may influence funding decisions by the Director of RMPS although no funding award may exceed that approved by the National Advisory Council. The notification of award and the advice letter sent at the conclusion of the review process to each applicant program summarizes the findings and recommendations , not uncommonly , in the language and conceptual framework of the Review Criteria. Description of the Evaluation Contract As the staff of Regional Medical Programs Service saw the RMP Review Criteria in use, they felt that significant benefits had accrued to the program, but that further improvements in the criteria and the ways in which they were used might lead to even greater benefits. They, therefore, prepared a Scope of Work for an evaluation contract which specified two purposes: 1. To assess the internal validity of the Review Criteria used in rating Regional Medical Programs, and 2. To identify the perceptions of those utilizing the criteria in the National review process as to their effectiveness. The Scope of Work detailed a series of specific questions addressed to those purposes and called for a final comprehensive report that would include "recommendations and suggestions (and basis thereof) for modifying the criteria, their application in the present review process, and that overall review process itself." Contract HSM 110-72-329 to evaluate the RMP Review Criteria was awarded to the Medical Care and Education Foundation, Inc., in June, 1972. Based on the Scope of Work for the contract, the technical proposal submitted by MCEF and negotiations between the project office and the project director, three specific evaluation tasks were undertaken: 1. Statistical studies related to internal validity, 2. Interviews with ten selected people participating in the review process at different levels, and 3. Conferences with selected RMPS staff. These specific tasks were supplemented by analyses of various documents , xij toon Beeenere sec ti Rd 23 a { = ay merges oe Tey eek Ree informal interviews with RMPS staff, and formal and informal consul ta- tions with various experts. Findings from the Evaluation Contract The major findings resulting from the evaluation contract can best be reported in relation. to the two purposes specified in the Scope of Work. -- To assess internal validity The evaluation of the internal validity of the Review Criteria consisted of a series of analyses of scores prepared by individual members of the Review Committee and the Staff Anniversary Review Panel, the only individuals who were required to evaluate local Regional Medical Programs by rating them against the Review Criteria. Each member of the two groups was given a rating form and asked to score all programs considered during a particular review cycle on that form. The scoring consisted of assigning a numerical value of 1, 2, 3, 4 or 5 for each of the twenty individual criteria to indicate the reviewer's judgement as to the program's quality. "5" indicates highest quality; "1" indicates lowest quality. The values are assigned independently by each reviewer after the presentation and discussion of a program had been completed and the recommendations had been specified. The scores of all reviewers were then summed with and without weighting -- weighting points are assigned to indicate the relative importance of each criteria -- and the summed scores were reported with the Review Committee or Staff Anniversary Review Panel recommendations . The major questions to be addressed by these analyses were: 1. To what extent is there variance in total scores, scores on major criteria groupings, and scores on individual criteria among programs classified as "A", "B" and "C" regions. 2. To what extent do the RMP Review Criteria constitute a scale or measure? 3. To what extent is there variance in scores by the same observer under different conditions or by different observers under the same conditions. The analyses indicated that there were statistically signifi- cant differences in the raw and weighted total scores and the scores on major criteria groupings among programs classified as "A", "B" and "C" regions. Because programs were classified on the basis of their total scores and because the correlation between scores on the major criteria groupings and the total scores exceeded 0.9, this finding was not unexpected. . setss xiii An analysis of the scores for the individual criteria showed exceptionally high inter-criteria correlations. When the correlation matrix was subjected to principal components factor analysis, only one factor was identified. If the major criteria groupings had been valid, there should have been three factors. The fact that only one factor was identified indicated, according to the consulting statistician, that the Review Criteria did not constitute a scale or measure that was in any way superior to a single rating on Overall Quality. Scoring on the individual criteria did not, in his opinion, contribute to an identification of those specific qualities which differentiate between good and poor RMPs. More specifically, a good program gets good scores on all criteria and a poor program gets poor scores on all criteria. The analyses also showed exceptionally high inter-observer correlations. There was no significant bias of the primary reviewers , as a class, nor of any individual reviewer. The absence of significant bias was shrouded in such an extraordinary degree of agreement among the scores of all the reviewers for each program that independence of judgement appeared to be absent. Stated another way, the scores for a_given program were so uniform as to suggest that they ha been, at least tacitly, agreed to before the reviewers completed their ratings. In the absence of scalarity and independent peer judgement, the assessment of internal validity becomes meaningless. The internal consistency of the scores and the consistency among different observers indicated that the Review Criteria scores express the consensus of the review groups as to the quality of the programs rather than the convergence of independent opinion. -- To identify perceived effectiveness in assessing comparative quality -- Conferences and interviews were used to identify the perceived effectiveness of the Review Criteria in assessing comparative quality. Five of the six specific questions addressed by these activities were concerned with the relative importance of major criteria groups and individual criteria, the availability of information and the rationale for adding and deleting criteria. The sixth question related to perceived improvements in the review process that had resulted from use of the criteria. A conference that was held to answer the first five questions failed to achieve that objective because, under the conditions of the conference, selected RMPS staff manifested an extraordinatory diversity of opinion on the subject. Although they agreed on the importance of the three major criteria groupings: Performance, Process, Program Proposal --, they were unable to cluster the specific questions from: the Review Criteria into an agreed uponset of individual criteria categories. As a result, there was no basis for describing the relative xiv wa Cod i oe an peereey Sit vase u es Ss ‘ek. “aed os aah, eS 0 “a A Ee is ie at « aS ERA IE) 3 es Pv aressl Paves importance of criteria or a rationale for adding or deleting criteria. The announced phase-out plans for RMP made it unfeasible to pursue further activities designed to develop a basis for answering those questions. There were, in that conference as well as in another con- ference with selected RMPS staff and in ten interviews with selected participants in the review process, many criticisn6 regarding the tack of appropriate information for evaluating programs against the existing criteria. The second conference with RMPS staff and the ten interviews indicated that the Review Criteria had been a very effective manage- ment tool for achieving certain specified purposes within the operation of Regional Medical Programs Service, especially within the review process. The Review Criteria were developed and introduced into the review process over a one year period spanning the latter part of 1970 and the first part of 1971. This occurred during the transition to the Triennial Award and Developmental Component grant application. procedure from the original project grant application procedure, a period of fiscal cut-backs in the total RMPS program and the intro- duction of the modifications in RMPS program called for by the RMP renewal legislation of 1970. The evidence from the conferences and interviews suggested that RMPS had, at least, four specific purposes for the development and introduction of the Review Criteria into the review process: 1. To facilitate a change in the review process from a project-oriented to a program-oriented approach ; 2. To provide additional structure to the technical assistance provided by RMPS to local programs to foster better program planning, development and operation; 3. To establish a mechanism for preserving continuity in the review process in the presence of changing personnel on the various review bodies; and 4. To serve as part of the rationale for selective funding decisions made by the Director of RMPS. . It was reported that the RMP Review Criteria had made signi- ficant contributions to the achievement of these purpdses -- all of which were achieved -- and that these contributions resulted primarily from the framework that the Review Criteria provided for the organization of information, discussion, reports, recommendations and advice letters. XV The participants in the interviews and conferences, while reporting the effectiveness of the Review Criteria as a management tool, identified many weaknesses that would need to be resolved if the criteria continued to be used. These included criticisms of: 1) the excessive number of criteria, overlap of criteria and absence of precise definitions; 2) the artificiality of the scoring process, and 3) the excessive number of questions, their redundancy, and their lack of explicitness. These criticisms did not, however, detract from their support of continued use of the Review Criteria in the review process and their strong endorsement of efforts to correct the identified weaknesses. Recommendations The evaluation of the RMP Review Criteria, although reported jn the findings, assumes a more coherent form in the recommendations. The seventeen recommendations are based on findings, criticisms and comments as reported in the three specific studies, the reports of the two conferences with RMPS staff and additional data derived from analyses of RMP-related documents, informal interviews and consulta- tions. The full rationale for each of the recommendations is reported in Chapter I y. All recommendations are submitted on the premises that Regional Medical Programs will continue to operate as a federal program and that the Review Criteria, in modified form, will continue to be used by RMPS as an adjunct to future RMP activities. The first six recommendations related to the purposes for which the RMP Review Criteria should be developed and utilized in the future. Recommendation #1: The Review Criteria should be used to develop and maintain a balanced perspective in the review process on performance, process, and program proposal. Comment: The review process, according to the findings, has been transformed from an emphasis on discrete projects to an emphasis on programs as whole units. Unfortunately, this has resulted in a focus on "accreditation" with little attention, particularly in the rating procedure, to past performance or program proposal. Recommendation #2: The Review Criteria should continue to be used as a device for promoting continuity in the review process. Comment: The Review Criteria were reported to be helpful in promoting continuity in the review process. There, obviously, is a continuing need — as long as there continues to be changes in personnel involved in that process. Xvi ayer ay ee} od aaa aa a Arai SVG cm agyt ee fet ah Recommendation #3: The RMP Review Criteria should be used to assure consistency of evaluative judgements within all levels of the review process and among different review cycles. Comment: An unexpected diversity of opinion and viewpoint expressed by staff members from different units within RMPS at one of the con- ferences from this contract has been identified and reported. To a lesser, extent, the same diversity was illustrated by the inter- views with members of the National Advisory Council, Review Committee, and Staff Anniversary Review Panel. Although diversity is probably essential to the development of exciting and innovative programs such as Regional Medical Programs, it can create problems in review and evaluation processes. Review and evaluation implies a single set of expectations uniformly applied to all applicants. Evaluative judgements, ideally, should be based on explicit standards that are not dependent upon the composition of the review body, the particular interests of individuals, or changing emphases over time. The Review Criteria, if the appropriate procedures are developed, can be used to assure consistency of evaluative judgements through the application of management principles to the review process. Recommendation #4: The RMP Review Criteria should continue to serve as part of the rationale for selective funding decisions. Comment: Selective funding is based, in part, on the use of incentives to promote the development and operation of high quality local Regional Medical Programs. The use of the Review Criteria, according to reports, encouraged more consensus as to which programs were of "high" quality and of "low" quality and, therefore, provided a useful rationale for selective funding decisions related to the use of incentives. Recommendation #5: The RMP Review Criteria should be used to provide a consistent framework for local program planning, development and operations. Comment: The Guidelines for Regional Medical Programs plus various publications, statements and letters have served as the primary frame- work for local program planning, development and operations. As RMPS staff began to be more active in providing technical assistance to local programs, they found, according to reports, that the Review Criteria were being used by some local programs as program guides. Although the RMP Review Criteria were designed primarily for the review process, it can be assumed that more programs will have good scores and higher quality programs when they consciously use the criteria as program guides. To be used as such program guides, the Review Criteria should be modified in their content and widely publicized in official RMPS documents. xvii Recommendation #6: The Review Criteria should be used to facilitate the development of information that can be used in RMPS program evaluation and in the justification of Regional Medical rograms to the Congress, the Administration and the pu lic. Comment: In early 1973, according to RMPS staff, many local Regional Medical Programs were making significant improvements in the planning, development and operation of their programs and some, as indicated by high Review Criteria scores and "A" ratings, were judged to be of high quality. In spite of these favorable evaluations reported out of the review process, recommendations to terminate the Regional Medical Programs were made by the Administration. The recommendation, as reported on p. 383 of the Appendix to the Budget Proposal for Fiscal Year 1974, includes the following statement: "... there is little evidence that on a nationwide basis the RMP's have materially affected the health care delivery system..." . That statement implies that the Administration had objectives related to changes in the "health care delivery system" that were not addressed by the evidence presented. Modifications of the Review Criteria suggested in subsequent recommenda- tions might have been addressed earlier if the Review Criteria had been considered as part of the program evaluation and program justification -effort of RMPS. The next three recommendations deal with changes in the content and design of the Review Criteria and are concerned with: A. Compatibility of content with Administration objectives for Regional Medical Programs; B. Broader participation in the development of the content; and . 'C. Technical changes in design to eliminate criticisms regarding: ; 1. Excess focus on "accreditation", 2. Excess number of criteria, 3. Confusion with the numerical scoring process, and 4. Weaknesses of questions. Recommendation #7: The content of the Review Criteria should be related whenever possible and appropriate, to s ecific guidelines and objectives to be established by the Administration for the egional Medical Programs at the National level on © | the basis of the legislation and the legislative proceedings. xviii — (4 3 4 Recommendation #8: The content of the Review Criteria should be modified under the leadership of RMPS staff with the participation of National Advisory Council members, Review Committee members, RMP coordinators and staff, and Regional Advisory Group members. Recommendation #9: The design of the Review Criteria should be significantly modified as follows: a. The major criteria should be retitled Achievements , Organizational Processes and Proposed Plans; b. The individual criteria should be reduced in number from twenty to twelve or less; c. The numerical values of 1, 2, 3, 4, and 5 should be accompanied by statements that indicate the precise meaning of each numerical value for each ‘ndividual criterion and each major criteria grouping; and d. The questions should be reduced in number and should be translated into a form that can be answered by Present/Absent or that calls for specific qualitative descriptions or quantitative measures. The last eight recommendations are based on the prior recommendations and call for significant modification in the operations of RMPS, especially those operations related to the review process. In these recommendations, the terminology of Levels is introduced.. The proposed Level One Criterion is Overall Quality. The proposed Level Two Criteria are Achievements, Organizational Processes, and Proposed Plan. The Level Three Criteria correspond to the present individual criteria and are suggested in Table 4-1. The Level Four Descriptors are proposed as the replacement for the specific questions of the present Review Criteria Examples are shown in Table 4-2. Recommendation #10: The Review Criteria should be included in the Guidelines for Regional Medical Programs and/or in an RMPS Regulations and Procedures Manual so that local programs understand their importance for local program planning, development and operation. A complete description of how the Review Criteria are used by RMPS in the review process and their technical assistance activities should also be provided in the same way. xix - Recommendation #11: RMPS staff in the Office of Systems Management should be responsible for the design of the Level Four Descriptors in lieu of the specific questions, for the redesign of the grant_application so that the information appropriate to these descriptors is available, for the analysis of the information related to these descriptors so that absolute and/or comparative standards can be developed, and for the preparation through the Management Information System of standardized information related to the escriptors for use in the review process. Recommendation #12: RMPS staff in the Division of Operations and Development should be .responsible for the design of the Level Three Criteria with the participation of the National Advisory Council, the Review Committee and local RMPs and for preparing the scores with accompanying statements of justification for each individual criterion for the staff briefing document. Recommendation #13: Site visit teams should study the Level Three Criteria scores and statements of justification as prepared by the Division of Operations and Development and the Level Four _ Descriptor information as prepared by the Office of System Management and should describe, in their site visit report, their evaluation of the submitted scores, justification, and data. Their report should include modifications of the - Submitted Level Three Criteria scores if indicated by the site visit findings. Recommendation ¥#14: The Review Committee and Staff Anniversary Review Panel should study the submitted scores, ustifications and data including modifications from the site visit team and should include evaluative comments in their report, but they shou repare scores With statements 0 ustification only for the Three Level Iwo Criteria. Because there is no evidence of independent judgement occuring in the present pattern of Review Criteria scoring, it is recommended that cations reflect the consensus the scores and their justifi he of the group and that scorin by individual members of t group be eliminated. The scores and justification on the Level Two criteria should be included in their report and recommendations. XX “a Tee ts! BS ee ae Recommendation #15 The National Advisory Council should study the scores and statements of justification on Level Two and Three Criteria in staff briefing document, site visit report, Review Committee and Staff Anniversary Review Panel reports and should include evaluative comments in their report. In addition, it is recommended that the National Advisory Council identify by consensus a score and an accompanying statement of justification that reflects their evaluation of the programs on Overall Quality and include this with their report and recommendations. Recommendation #16: RMPS staff in the Office for Planning and Evaluation should monitor the scores, justifications, data and evaluative comments reported by the Office of Systems Management, Division of Operations and Development, site visit teams, Review Committee, Staff Anniversary Review Panel and National Advisory Council and should analyze them for consistency of evaluative judgement among the review bodies, for perceived adequacy of the submitted information, and for redundant and/or missing criteria. Recommendation #17: The Review Criteria data, scores and statements of justification from each of the review bodies should be submitted to the local Regional Medical Programs as a verification procedure and as a further stimulus to use of the criteria in local program planning, development and operations. Conclusions: The belief that the RMP Review Criteria had made significant contributions to the operations, specifically the review procedures , of RMPS was confirmed by this evaluation. Continued use of the Review Criteria, however, should be accompanied by significant improvements in their content and design and by alteration of the ways in which they are used. Seventeen specific recommendations regarding the purposes of the Review Criteria and their content, design and appli- ‘cation have been presented. These recommendations should lead to improvements in the effectiveness and acceptability of the Review Criteria in the review process, in the merging of local program efforts with national guidelines and objectives, and in the development of mechanisms for improving the management of the review process. xxi my “| ie] CHAPTER I a THE SETTING FOR THE EVALUATION wits pa io a4 4 Regional Medical Programs were authorized by PL 89-239, the. "Heart Disease, Cancer, and Stroke Amendments of 1965". The legislation authorized the Surgeon General of the United States to grant funds to area-wide organizations for the planning and opera- tion of local Regional Medical Programs for the following purposes: a) Through grants, to encourage and assist in the establish- ment of Regional Cooperative arrangements among medical schools, research institutions, and hospitals for research and training (including continuing education) and for related demonstrations of patient care in the fields of heart disease, cancer, stroke and related diseases; b) To afford to the medical profession and the medical institutions of the Nation, through such cooperative arrangements , the opportunity of making available to their patients the latest advances in the diagnosis and treatment of these diseases; and c) By these means, to improve generally the health manpower and facilities available to the Nation, and to accomplish ‘these ends without interfering with the patterns, or the methods of financing, of patient care or professional practice, or with the administration of hospitals, and in cooperation with practicing physicians, medical center officials, hospital administrators, and representatives from appropriate voluntary health agencies. Fifty-six local Regional Medical Programs serving the entire United.States and the Commonwealth of Puerto Rico have received grants under this legislation. Each grant application funded by Regional Medical Programs Service has had the approval of a local advisory group and the National Advisory Council. At the National level, National Advisory Council action and recommendations are preceded by site visits and Regional Medical Program Review Committee considera- tion of grant applications for three year periods and grant appli- cations from new local organizations aspiring to operate a regional medical program. Annual continuation grant applications that fall within the National Advisory Council approved plan and budget are reviewed and approved by the Staff Anniversary Review Panel without further consideration unless there is a request for supplemental funds or a change in program. To facilitate some needed changes in the review process, a set of Review Criteria were developed and introduced into the review process of Regional Medical Programs Service (RMPS) as a routine procedure in the summer of 1971. The development of the RMP Review Criteria began in the summer of 1970 in response to the perceived need by RMPS staff to facilitate a transition from a project oriented to a program-oriented review process, to provide a more structured framework for technical assistance to Tocal programs than existed at that time, to establish a mechanism for preserving continuity in the review process with turnover. of personne? on the various review bodies, and to serve as part of the rationale for selective funding decisions by the Director of RMPS. The initial set of criteria were developed and pre-tested on eighteen site visits in the fall of 1970. The semi-final set of criteria were developed in a crash effort lasting three to four weeks in February, 1971, to meet administrative requirements of the Health Services and Mental Health Administration (HSMHA). Minor modifications were de in the spring of 1971 so that they could be used on a routine -basis. The RMP Review Criteria consist of twenty individual criteria organized into three major groupings. The relative importance of each of the criteria, both major and minor, are_ specified in weightings--shown below in parentheses--which total to one hundred. The information to be used in evaluatina a local Regional Medical Program. against these criteria is specified in a series of questions under each of the twenty criteria -- see Appendix A. A. Performance (40) 1. Goals, Objectives, and Priorities (8) 2. Accomplishments and Implementations (15) 3. Continued Support (10) 4, Minority Interests (7) B. Process (35) ‘1. Coordinator (10) 2. Core Staff (3) 3. Regional Advisory Group (5) 4. Grantee Organization (2) 5, Participation (3) 6. Local Planning (3) | . 7. Assessment of Needs and Resources (3) 8. Management (3) 9, Evaluation (3) eofcuts SEAS foo a 4 C. Program Proposal (25) Action Plan (5) lt ° 2, Dissemination of Knowledge (2) 3. Utilization Manpower and Facilities (4) 4, Improvement of Care (4) 5, Short-Term Payoff (3) 6. Regionalization (4) 7. Other Funding (3) The Review Criteria play a role in the review process at several different levels. They serve as a general guide for the pre- paration of the historical profile in the staff briefing document, a summary report prepared by the staff in the Division of Operations and Development to accompany the submitted triennial or anniversary review grant application through the review process. Site visit reports related to the review of triennial grant applications are supposed to be formatted according to the Review Criteria and are expected to summarize all the relevant information in regards to each individual criteria. The Review Committee for triennial grant applications are required to score the programs under review against the Review Criteria and to include the scores with their reports and recommendations. In this scoring process, the members of these two review groups are asked to rate the programs under consideration by assigning a numerical value of 1,2,3,4 or 5 for each of the twenty individual criteria to indicate the reviewer's judgement as to the program.'s quality. The scores are then summed, averaged and reported with and without weighting to the National Advisory Council. On the basis of the official scores from the Review Committee and the Staff Anniversary Review Panel, local programs were classified as Ha", "B", or "C" regions with "A" regions being the most highly regarded. The scores and the classification may influence funding decisions by the Director or RMPS although no funding award may exceed that approved by the National Advisory Council. The notification of award and the advice letter sent at the conclusion of the review process to each applicant program summarizes the findings and recommendations, not uncommonly, in the. language and conceptual framework of the Review, Criteria. . Description of the Evaluation Contract As the staff of Regional Medical Programs saw the RMP Review Criteria in use during the period 1971-1972, they felt that significant benefits had accrued to the program, but that further improvements in the criteria and the ways in which they were used might lead to even greater benefits. They, therefore, prepared a Scope of Work for an evaluation contract which specified two purposes: 1. To assess the internal validity of the Review Criteria used in rating Regional Medical Programs, and 2. To identify the perceptions of those utilizing the criteria in the National review process as to their effectiveness. The Scope of Work - see Appendix B - detailed a series of specific questions addressed to those purposes and called for a final comprehensive report that would include "recommendations and suggestions (and basis thereof) for modifying the criteria, their application in the present review process, and the overall review process itself. Contract HSM 110-72-329 to evaluate the RMP Review Criteria was awarded to the Medical Care and Education Foundation, Inc. in June 1972. Based on the Scope of Work for the contract, the techni- cal proposal submitted by MCEF and negotiations between the project office and the project director, three specific evaluation tasks ~were undertaken: 1. Statistical studies related to internal validity, 2. Interview with ten selected people participating in the review process at different levels, and 3. Conferences with selected RMPS staff. These specific tasks were supplemented by analyses of various documents, informal interviews with RMPS staff, and formal and informal consultations with various experts. ~ Some | ~ modifications were negotiated because of the scheduled phase-out plans of RMPS - see Appendix B. Although the essential content of the evaluation contract was specified in the Scope of Work, the work was approached by the - contractor within the context that existed at the time that the Review Criteria were developed and formally introduced into the review process. This context included the history of the legislation. for Regional Medical Programs, especially the renewal legislation of 1970, and the history of the review process within Regional Medical Programs, especially the changes that were introduced in the fall of 1970.- The work was approached with the knowledge that explicit standards for the evaluation of program performance were uncommon perhaps non-existent, within the Department of Health, Education and 6 mee ae faa be vs “sy sy cae) ss wo es Gi | GEES Welfare at that time and that this attempt to develop such standards was complicated by the changes occuring within Regional Medical Programs in regards to the purposes for which RMP had been established and to the procedures for review approval and funding of local programs. Legislative History Ordinarily legislation is introduced into the House of Representatives and/or the Senate of the United States. The legisia- tion is then referred to a committee for hearings and a report. The conmittee report is introduced on the floor of each congressional body and the legislation is then discussed, possibly amended, and approved or disapproved. After the legislation is approved by both the House and the Senate, it is referred to a conference committee which resolves any differences and reports the final version of the legislation back to both bodies for final approval. The reports of the committees plus the discussion on the floor provide the basis for interpreting the language of the final legislation. The legislative history of Regional Medical Programs began in 1965 in the 89th Congress with H.R. 3140 and S. 596 " A Bill to Amend the Public Health Service Act to Assist in Combating Heart Disease, Cancer,-Stroke and Other Major Diseases." It continued with renewal legislation, hearings, reports and discussion all of which form the basis for the legislative history. The original legislation for Regional Medical Programs was stimulated by the findings of the Report of the President's Commission on Heart Disease, Cancer and Stroke, issued in 1964. The Commission, made up of distinguished physicians, scientists, and informed citizens, established eight subcommittees and a method of operation which called for: "1. The collection of information from agencies, groups, and institutions concerned with these diseases.... 2. The holding of hearings at which expert witnesses... presented their views.... 3. The preparation of the report and its recommendations..."! The Commission Report, which led directly to introduction of the legislation authorizing Regional Medical Programs , focused attention on a number of long-term trends including the development of a national biomedical research community of unprecedented size and gradual move- ment towards attempts at regionalization of health resources. The Report recommended the establishment of a national network of regional heart disease, cancer, and stroke centers for clinical investiga- tion, teaching and patient care in universities, hospitals, and 1 Report to the President: A National Program to Conquer Heart Disease, Cancer and Stroke, Vol. 1, December 1964, U.S. Government: Printing Office, Washington, D.C. - pp 89-90 research institutes across the country. It also recommended a series of diagnostic and treatment stations in communities which would develop a close relationship with the regional centers. The third major recommendation was that a broad and flexible program of grant support be undertaken to stimulate the formation of medical complexes whereby university medical schools, hospitals, and other health care and research agencies and institutions work in concert. Legislation in response to the Commission Report was introduced in the Senate as S. 596 by Senator Lister Hill of Alabama. The Senate version as reported came closer to the recommendations of the Commission Report than did the later House version. As written in the Report of the Senate Committee on Labor and Public Welfare on S. 596: "This legislation would implement recommendations of the President's Commission. The primary thrust of this bill is to provide for the planning, establish- ment, and operation of regionally coordinated medical complexes for heart disease, cancer and stroke, and other major diseases which will link together medical centers, categorical research centers, and diagnostic and treatment stations located in community hospitals or other health facilities". ’ In the House, the bill was introduced as H.R. 3140 by Rep. Oren Harris of Arkansas. The Committee on Interstate and Foreign Commerce held extensive hearings during which it became apparent that some groups did not favor the concept of regional medical complexes. While testimony favorable to the legislation was submitted on behalf of the American Heart Association, American Cancer Society, and the Association of American Medical Colleges, testimony in opposition was submitted by the American Medical Association and several State medical societies. Because of the substantive objections of some of the witnesses, the bill was modified substantially in the House committee. As discussed by Rep. Harris in the floor debate: "There were great fears that there would be a major Government medical program set up with clinics, categori- cal centers, administrative centers, hospitals, and so forth operated by the Government. So we decided that instead of calling these by the term ‘complexes ,' which had developed an image of that kind, we would refer to them jn the bill as ‘programs’. The bill provides for programs 2Report of the Committee on Labor and Public Welfare, U.S. Senate, to accompany $. 596, U.S. Government Printing Office, Washington, D.C., 1965, p. 9. ‘ep Fe eae x dee sit inact wok ee a] i utilizing existing medical centers, hospitals and institutions. We provide for cooperative arrangements whereby medical schools in cooperation with clinical centers in the area and with the hospitals in the area, and other health activities, shall set up an advisory local committee. That advisory local committee will decide.'"3 Rep. Ancher Nelson of Minnesota, a member of the Committee on Interstate and Foreign Commerce, reflected the view of organized medicine that the bill should have had more discussion and been postponed for a year, during House debate on the bill: "T agreed that more time would be useful to allow the discussion to run its course and supported a motion to defer action until next session of the Congress. This did not prevail because the White House cannot let Congress do its work in an orderly fashion these days and apparently jt was ready to settle for anything contuing the words, heart, stroke and cancer. Despite misgivings on the part of many of its members, the committee settled down to write some legislation which could meet the objections and stil] make a start in the direction indicated in the original charge of the President's Commission. The result is the bill before you today, which came from the committee with full support." Representative Nelson's remarks to the House, as recorded in the Congressional Record, provide a good summary of changes made in the proposed legislation by the Committee on Interstate and Foreign Commerce and provide a clear statement of Congressional intent, an intent which was subsequently modified by legislative amendments and program experience: "The changes are many and they are not mere clari- fications or exercises in semantics. They change an amorphous mass of objectives into a recognizable program which deals with units and controls thoroughly understood by those who must work with them. The bill now talks in terms of medical programs, put together by existing institutions under the eye of a local advisory board. It talks about cooperation and not coordination. The former means voluntary involvement and the latter infers an imposed plan. It talks of hospitals and not of diagnostic and treatment stations. The latter is an entity not familiar to practitioners, but we can all visualize a hospital and have a definite idea what it does, what it looks Tike and who runs it..." 3congressional Record: House of Representatives , Eighty-Ninth Congress, ‘First Session, Volume III, Part 22, U.S. Government Printing Office, Washington, 1965 p. 24123, Sept. 1965. 4ibid., p. 24126 “What did the committee substitute and what changes were made? The report on the bill accurately states: ‘Numerous changes were made in the introduced bill by the committee designed generally to better define the scope of the program and to clarify the intent so as to guarantee that the legislation will accomplish its purpose without interfering with the patterns or methods of dinancing of patient care or professional practice or with the administra- tion of hospitals.’ "This statement alone indicates the magnitude of the changes and the fact that the legislation as introduced was miles off the mark. Here are the specific changes: "First: Regional medical complexes were mentioned earlier in my statement. No one could even now define what they are or how they would operate. The committee substituted the term ‘regional medical program’. At the same time, all authority to use funds for new construction, including replacement of existing buildings was removed. These are referred to in the report as primarily semantic changes. Do not believe it. They move the specter of huge, new autonomous institutions which receive their funds directly from the Government and quickly dominate every phase of medical practice and hospital practice in the fields of heart, stroke and cancer. "Second: The original legislation allowed for expansion for other major diseases. The committee restricted the scope of this legislation to related diseases. That too is something more than a refinement. We have no idea that plans devised by the various States will be the ultimate answer in conquering the three diseases named. This is experimental. It cannot guarantee success in the war on heart disease, stroke and cancer. It will do well if; from the many medical programs devised, we discover one or two which have real promise. There is little reason to leave room for expansion into other fields. “Third: The term ‘cooperative’ was substituted for 'coordinated' wherever the latter appeared. This helps to remove the prospect of domination of the program by one large institution. A program can be beautifully coordinated if all the power is concentrated. at the head. What we are striving for here can only work if all elements participate through cooperative arrangements. 10 we] : cS Lege ay i aa a ER cain Esbyicd SEIT bene ey Bed eb (heresy Wes "Fourth: Grants will be used for planning, conducting feasibility studies and operating pilot projects for the establishment of regional medical programs of research, training and demonstration activities. "Fifth: Diagnostic and treatment stations have been eliminated. The bill now speaks of hospitals which participate in the program. This also demons trates the basic character of the changes made in committee. Now the bill refers to the local hospital participating in a cooperative program. We can explain to anyone what a hospital is by merely mentioning its name. Any citizen has a definite idea of how a hospital operates, what it looks like, what kind of people service it, and who runs it. The fight against heart, stroke, and cancer will come to the local patient through the people he knows and trusts and through an-institution with which he is thoroughly familiar. "Sixth: This legislation provides for advisory councils at both the local and national level and in each case the council must recommend a program before it can be imple- mented or funded. Of these two, however, the Council at the local. level is by far the more important. First of all, its membership is important. It must include practicing physicians, medical center officials, hospital administrators, medical society representatives, voluntary health agency personnel, as well as people from other organizations concerned with the program. But even the best council will not guarantee a sound program if that program is set up before the council is organized and has a chance to act upon it. For this reason the bill provides that the advisory council must be organized and must pass upon the local programs before it may be considered by the Surgeon General. This should guarantee that the plan worked out in the State will not be lopsided, concentrating too heavily upon one area of activity or placing too mych authority or responsibility with any one institution." Although Rep. Nelson described the House bill as including revisions which "change an amorphous mass of objectives into a recogniz~ able program which deals with units and controls thoroughly understood by those who must work them," it may be asked whether the new emphasis on voluntary arrangements: and local cooperation did not make the bill even more flexible and subject to varying interpretations. In either case, the Senate finally accepted the House version of the bill, with passage as Public Law 89-239 in October of 1965. Stbid., pp. 24126-7 1 jn 1968, renewal legislation was introduced in the Conaress. The report? of the Committee on Interstate and Foreign Commerce sub- mitted to the House of Representatives on June 10, 1968 indicated that the development of cooperative programs under the control of local advisory groups had resulted not only in specific accomplishments, but also in greater understanding of what was needed at the local level and how it could be achieved: "During the hearings and from materials supplied for the records, the conmittee received many descriptions of specific developments under Regional Medical Programs which serve to illustrate the nature of the emerging program, the diversity of approaches being developed, the effectiveness of regional cooperative arrangements in bringing together in common effort the health resources of the region, and the accomplishments already underway in making possible the wider application of the advances of medical sciences in these fields. All of these descriptions represent progress in the initial stages of the development in operational regional medical programs." "In many respects, the legislation presents an answer to the problems expressed in the saying that 'what is every- body's business is nobody's business’. There are a sub- stantial number of individuals, groups and institutions throughout the United States who are concerned with the prevention of heart disease, cancer, stroke and related diseases, and the treatment of patients suffering from these diseases. Until the enactment of this legislation, there were few mechanisms through which coordinated programs could be established relating the activities of the various individuals, groups and institutions concerned with these diseases to activities of the others. This legislation provides a means whereby the individuals, groups, and institutions concerned with these diseases are encouraged to plan their activities jointly so as to coordinate them, as well as to plan projects that otherwise might never get off the ground for lack of sponsor. In this fashion, everybody's business, in the fields of heart disease, stroke, cancer and related diseases become the business of a specific group, which plans and operates programs having the overall objective of reducing morbidity and mortality from these diseases." SReport of the Committee on Interstate and Foreign Commerce, House of Representatives to Accompany H.R. 15758, U.S. Government Printing Office, Washington, 1968, pp. 11-12 12 +. ha Ca are fot fa 3 “4 4 : fe Eg a The Administration, however, did not appear to be as pleased with the achievements of Regional Medical Programs as was the Congress. This was reflected, in part, by the Administration recommendation for a two year rather than a five year extension. The renewal legislation was passed in 1968 with the signi- ficant changes confined to authorization for multi-program services, that is, project involving more than one local Regional Medical Program and the allocation of 1% of appropriation for evaluation. During the next two years, almost all of the local programs became operational, but the Administration continued to manifest dissatis- ‘faction with the overall progress of the program. In 1970 this took the form of budgetary restrictions and legislative proposals that would remove the categorical mandate and merge the RMP with Comprehensive Health Planning, the National Center for Health Services Research and Development, and the National Center for Health Statistics. Congress, however, continued in its support of RMP and proposed in S. 3355, an extension which preserved the categorical emphasis and extended it to include kidney disease. The testimony’ of Senator Edward M. Kennedy of Massachusetts at the hearings in 1970 on the renewal legislation epitomized those supportive attitudes: "Today, thanks in large part to the success of RMP, we now have far more doctors and health organizations working together cooperatively than anyone might have expected a few years ago. I believe that we must continue in our efforts to expand the program and. increase its funding... Because of its involvement with the medical schools and the universities, I believe that RMP holds great promise for influencing the health delivery system in a beneficial way. Further, because it has encouraged the programs to develop as conceived by the indigenous medical forces in a State, or a metropolitan area, it holds equally high promise for successfully regionalizing the health care system after years of relatively unproductive efforts towards this purpose." The legislation reported to the Senate as S. 3355 and sub- sequently passed as PL 91-515, October 30, 1970 reflected an accommoda- tion to the views of both the Administration and the Congress. This accommodation to both a categorical disease and medical care system approach is clearly evident in the revised statement of purpose: 7Hearings before the Subcommittee on Health of the Committee of Labor and Public Welfare, U.S. Senate, Ninety-first Congress, Second Session on S. 3355, S. 3443 and related bills, Part I. U.S. GOP, Wash. 1970, pp. 13 126-7 "(a) through grants and contracts, to encourage and assist in the establishment of regional cooperative arrangements among medical schools, research institutions, and hospitals or research and training (including continuing education), for medical data exchange, and for (related) demonstrations of patient care in the fields of heart disease, cancer, stroke and kidney disease, and other related diseases; (b) to afford to the medical profession and the medical institutions of the Nation, through such cooperative arrange- ments, the opportunity of making available to their patients the latest advances in the (diagnosis and treatment of these diseases) prevention, diagnosis, and treatment and rehabilitation of persons suffering from these diseases; (c) to promote and foster regional linkages among health care institutions and providers so as to strengthen and improve primary care and the relationship between specialized and primary care; and - (d) by these means, to improve generally the quality and enhance_the capacity of the health manpower and facilities available to the Nation (,) and to improve health services for persons residing in areas with limited health services, and to accomplish these ends without interfering with the patterns, or the methods of financing of patient care or professional practice, or with the administration of hospitals, and in cooperation with practicing physicians, medical center officials, hospital administrators, and representatives” from appropriate voluntary health agencies."8 Additional changes reflected an accomodation to the divergent points of view about increased community control and increased central direction. Briefly stated, these changes: 1. Broadened the specified membership and obligations of the local advisory groups " to insure adequate community orientation." 2, Provided for "consideration of the application -- for each operational grant -- by each public and nonprofit agency or organization which has developed a comprehensive regional, metro- politan area, or other local area plan referred to in Section 314 (b) covering any area ‘in which the regional medical program for which the application is made will be located.” 3, Authorized the Secretary to facilitate interregional cooperation, and develop improved national capability for delivery of health services, to assist in meeting the costs of special projects for improving or developing new means for the delivery of health services concerned with the diseases with which this title is concerned, and to support research, studies, investigations, training, Bpublic Law 91-515-, October 30, 1970, "Heart Disease, Cancer and Stroke Amendments of 1970". 14 ied 4 en C3 ¢ nl GH iy oy rye ney ash. bad SS “oe? and demonstrations designed to maximize the utilization of manpower jin the delivery of health services. The Regional Medical Program legislation of 1970, in many respects, marked the end of a five year legislative history. At its inception, Regional Medical Programs were identified as a cooperative effort in regionalization of research, training and demonstration activities designed to improve the care of patients suffering from heart disease, cancer and stroke. After five years of operational experience and additional consideration by the Congress in 1968 and 1970, the legislation began to specify balanced working relationships between categorical disease care and the rest of the medical care system and between national and local organizations concerned with improving the health manpower and facilities availabte to the Nation. The successful negotiations of the divergent attitudes and approaches, as apparent in the renewal legislation of 1970, was a mixed blessing. The changes provided the opportunity for the development of a more balanced approach to meeting the medical care needs of the Nation and of communities through regional activities, but obfuscated exactly what achievements were to be expected and who was in control of the program. In the absence of specific objectives or guidelines for Regional Medical Programs from HEW or HSMHA, RMPS staff, as they began the development of the Review Criteria in the summer of 1970, were forced to use a set of imprecise, overlapping and, often, confusing set of directions abstracted from the 1970 legislation and the President's Health Message of 1971. History of the Review Process The legislation for Regional Medical Program specified that the activities would be carried out “through grants" to "public or nonprofit private agencies and institutions" with certain specified characteristics. Two types of grants were specified: (1) Grants for planning and (2) Grants for Establishment and Operation. Grants could be funded only if they had the approval of the local advisory group and of the National Advisory Council. The composition of both groups was specified in the legislation. At the national level, only those grant proposals which had received approval from the local-now referred to as "regional" advisory group were accepted for possible funding. These grant proposals were entered into a review process patterned after that of the National Institute of Health. This included: 1. RMPS staff review for completeness and preparations of - special analyses and reports called "staff briefing documents" for all grant proposals. wok 15 2. Site visits of experts, peers and public at the request of RMPS staff. 3. Panel reviews by experts in selected fields, e.g. heart disease, at the request of RMPS staff. 4. Review Committee consideration and recommendations based on technical considerations for all grant proposals. 5. National Advisory Council consideration and recommendations based on policy and program con- sideration for all grant proposals. The Director of RMPS could fund only those grant proposals approved by the National Advisory Council. Although the National Advisory Council usually followed the recommendations of the Review Committee, they were not bound by the Review Committee recommendations. The National Advisory Council had the final and only decision as to which grants could be funded. The Director of RMPS was obligated to fund grants approved by the National Advisory Council, but he could fund at a level less than that approved if there were limited funds or an apparent need to reallocate funds between grants or grantees. From 1967, the first year that grants for establishment and operation of regional medical programs were received, until 1970, each component project of a grant application was considered as a separate grant. Operational grant applications consisting of one or more separate projects were submitted by some or all of the fifty-plus local regional medical programs at deadlines for review cycles scheduled three or four times a year. RMPS staff, in 1969, and the Federal Assistance Streamlining Task Force (FAST), in 1970, identified the fact that the project- , oriented review resulted in the excessive work Toad for the review process, significant variations in total funding levels among the regions, and an inability to develop cohesive and logical programs at the local level. This led to recommendations for significant changes in the review process. The review process was changed for applications received on or after August 1, 1970. The new review process restricted local programs to a single operational grant application each year on a specified date and made provision for a triennial award which allowed for much greater local program autonomy within an approved program plan. Previously a local regional medical program could support and carry out only those specific activities which had been approved by the National Advisory Council. Under the new procedure, a local regional medical program could carry out any activity which could be funded within the approved funding level. A triennial award included a developmental component which was money specifically set aside for the development of short term activities within the three year plan. In addition, a program receiving a triennial award was allowed to reallocate funds to any short term or long term activities judged by the local advisory group to 16 isd a meet the program's objectives as specified in the approved three year plan. Reviewing a three year program plan whose approval reduces central control over funded activities is a significantly different task from reviewing discrete projects. Not only would the various review groups have to switch from project-oriented to program- oriented review, but they would have to become concerned with the composition and decision-making ability of the local advisory groups so as to "insure adequate community orientation" as specified in the new legislation. Helping the Review Committee, the National Advisory Council and the site visit terms to make the transition was an important issue for RMPS staff in thé fall of 1970. Conclusion The RMP Review Criteria have been an integral part of the Regional Medical Program review process at the national level since 1971. They were developed and introduced during the latter part of 1970 and the early part of 1971, a period of transition in the purpose and methods of operation as indicated in the renewal . legislation of 1970 and a period of transition in the review process. The request of RMPS for an evaluation of the Review Criteria called for an assessment of the adequacy and effectiveness of the Review Criteria within the review process and for recommendations that might increase their usefulness in the future operation of Regional Medical Programs. 7 3 “i a 24 CHAPTER II APPROACH TO THE EVALUATION AND REPORTS OF COMPONENT ACTIVITIES 19 eet] 2 he otha gt hs] Be Aba, eer edged Ck A contract such as this one calls for close co laboration between the Project Officer and the Project Director in order to assure that the needs of the agency will be met. Negotiations between the Project Officer and the Project Director as to the selection of specific work activities and the level of quality expected is an important part of the process. According to the Scope of Work for the contract, the first phase called for a statistical analysis to determine the internal validity of the Review Criteria as measures: the second phase called for an opinion survey of those using the criteria to determine their adequacy and effectiveness in assuming the comparative quality of local RMPs; the third phase called for the preparation of a comprehensive report with findings, recommendations and suggestions. The agreed-upon specific activities were: 1) statistical studies related to "internal validity” 2) interviews with ten selected users 3) conferences with selected RMPS staff The results of these activities are reported in the following sub- sections of this chapter. A. Statistical Studies of RMP Review Criteria Scores B. Reported Usefulness of the RMP Review Criteria in the Review Psocess from Interviews with Ten Users C. vapues Expressed by Ten Users in Descriptions of Local RMPs D. Summary of Conferences with RMPS Staff on RMP Review Criteria The general theme to be found in the reports is that, in spite of many technical and methodologic weaknesses, the RMP Review Criteria were ~ found to be very helpful to the operation of RMPS at a time of great need for a new framework for the review process and for guiding local programs in the confusing task of carrying out the -changing intent of the RMP legisla- tion. The information contained in these reports plus the results of analysis of various document, informal interviews with RMPS staff and formal and informal consultations with. various experts from the substance of the evaluation and provide the rationale for the recommendations reported in- Chapter IV. The studies reported in this chapter are merely a prelude to the formal evaluation and recommendations to follow. ae 21 wae) waned a3) Car Oey fo oe A. By STATISTICAL STUDIES OF RMP REVIEW CRITERIA SCORES 23 Members of the Review Committee and the Staff Anniversary Review Panel routinely score all programs that they review against the RMP Review Criteria. Each member submits a set of scores for each program that he reviews. All the scores for an individual program are averaged and reported as official scores to the National Advisory Council and the Director of RMPS. The computer data file with the scores of all individuals for all programs reviewed in the review cycles of November 1971, February 1972, and June 1972 was subjected to a series of statistical studies designed to answer the questions posed in the Scope of Work - See Appendix B. Of the 590 individual scores contained in the file, only 277 were selected for analysis. They were selected on the basis of completeness, that is, scores recorded for all twenty categories, and on the basis that these five or more complete scores for an individual program. For many analyses » - the regions were categorized into "A", "RB" and "C" regions on the basis of information provided by RMPS. A complete report can be found in the Second Quarterly Report from the contract. Findings: 1. The average raw scores of "A", "B" and "C" regions are significantly different from each other. TEST: Analysis of variance (F 25.67, p. 0.001) 2, The average weighted scores of "A" "B" and "C" regions are significantly different from each other. TEST: Analysis of variance (F 696.6, p. 0.001) 3. There is a consensus among reviewers regarding "A", "B" and "C" regions. TEST: Principal Component Factor Analyses of Regions Centered Scores - Only one factor identified. EXPLANATION: When the scores of each reviewer are expressed as deviations from the mean score for each region and subjected to principal components factor’ analysis, only one factor was identi fied. This means that all reviewers followed a single pattern in scoring regions, j.e. there was consensus. 4. The scores of no individual reviewer reflects an unusual range. TEST - A: Bartlett's Chi-Square Test on Means Scores and Standard Devi ation of Individual Review (X2 4.6116, p= 70.5 ae 25 TEST - B: Principal Components Factor Analysis of Review Scores - Only one factor identified. EXPLANATION: When the scores of each reviewer are expressed as deviations from his mean score for all regions and subjected to principal components factor analysis, only one factor was identified. This means that no reviewer or group of reviewers had a range or patterns of variation that differed from the other reviewers. No primary reviewer has a greater tendency than other primary reviewers to pull the scores of other primary reviewers to a modal value. Primary reviewers scores in general, are slightly higher than the mean's scores. TEST: Bartlett's Chi-Square Test on Mean Scores and the Standard Deviation of Regions Grouped by Primary Reviewer. (X° - 13.21., p. 0.212) The same statements recorded in 1-5 above can be made for subset scores related to the criteria on performance, process and program proposal. The correlation coefficients between overall weighted scores and the weighted scores for performance, process, and program proposal are 0.945, 0.948, and 0.938 respectively. There are strong positive associations between overall weighted scores and the scores for all individual criteria except minority interests. TEST: Principal Components Factors Analysis. EXPLANATION: A quotation from the report of the statistician describes the strength of the relationships and the significance of the finding. ‘Our first hint that the data is simplistic comes when we examine the full correlation matrix on the twenty items. The entries in this matrix are astoundingly high. If we exclude item four from the analysis (a decision analytically corroborated later), every correlation but (8.3) is larger than .400. For some variables, such as #18, every corre- lation is larger than .500. This consistency strongly suggests that (except for item four) differences in the face significance of items are not being reflected in the scoring. 26 Tey ed - fea. cry 2 a . ead fret f \ ; rt Lester cael In the raw principal-component analysis of this matrix, column one of the printoutis a column of weights by which we multiply the separate items (after they are standardized to the same variance) before summing to form the best scale. This data analyst has never before seen a set of factor loadings so large and homogeneous. With the exception of #4, all are larger than .700. Here is a scale which does not correlate less than .7, and occasionally as much as .85, with each . of nineteen items. The “latent root" at the bottom of the column is the total squared correlation of this optimal scale with the separate items. Its maximum is 20, achievable only if our items are all identical. We are nearly two-thirds of the way to that ultimately redundant situation.” 9, The relationship between site visit findings as reflected in site visit reports and the scores for selected criteria was not examined because the site visit reports were not provided by RMPS. Conclusions: Although some of the findings were predictable, e.g. the signi- ficant differences in scores of "A", "B" and "C" regions. most of the findings were surprising. Ordinarily studies of peer judgement procedures show very poor correlation between scores of different judges, but that was not the case in this study. The correlations between scores of different judges was extraordinarily high. Similarly there is usually considerable variation in the correlation of individual item.scores and total scores, but that, again, was not the case. There were extraordinarily high corelations of individual criteria scores with each other and with total scores. These unexpected findings, although appearing to show consistent judgement or extreme internal reliability to the uncritical reader, actually indicated two very serious flaws in the Review Criteria as measures. First, the extraordinarily high correlations between reviewers suggests that there was agreement between the reviewers as to what the _ scores should be. Each reviewer did indeed, score the programs independently and there was no formal discussion as to what the scores should be, but the scoring was done after a full presentation and discussion had been completed and recommendations were made. The high correlations between the scores of the various reviewers reflected the fact of group consensus as to the quality of the program, and indicates that_group judgement rather than individual judgement determined the scores. 27 Second, the high correlations between individual criteria scores and total scores led to the identification of only one factor when the scores were subjected to factor analysis. The separation of the individual criteria into three major categories, if it was a valid separation, should have led to the identification of three factors. Ordinarily one might have expected that more than three factors would have been identified. The fact that only one factor was identified revealed that the Review Criteria function as a unidimensional measure, that is, a good program is good on everything and a pocr program is bad on everything. This failure to identify more than one factor suggests that the scores on individual criteria were probably based on an " a priori” judgement of overall quality and unconsciously adjusted so that the total scores reflected that overall judgement. The only exception was the scoring on the criteria related to minority interests. Significant changes in the scoring procedure would be needed to introduce, in an operational sense, independent peer judgement and independent judgement on each criteria. Only when independent judgement occurés will statistical studies indicate the degree of reliability. validity, precision, objectivity and constancy that is present. better solution, when orf considers all the problems reported in the literature on the evaluation of independent judgemental procedures , would be to transform the process from subjective to objective measures related to the criteria. A second choice solution would be to provide precise definitions for each of the scale points used in the scoring. 28 Flat Komen fee eet Sey Cnet lo s Sees dale! s I é SO Foe xn ah he Lyte. md ee ee ma ad Le B. REPORTED USEFULNESS OF THE RMP REVIEW CRITERIA IN THE REVIEW PROCESS FROM INTERVIEWS WITH TEN USERS 29 esd LS fi ye a aD Determining "the perceptions of those utilizing the (RMP Review) criteria in the national review process as to their effectiveness" is one of the expressed purposes of the contract to evaluate the RMP review Criteria. A related question from the Scope of Work for the contract is, "Nave the review criteria and their utilization resulted in a significant improvement, either substantially or in terms of credibility, in the decision-making process?" This is the report on one of two activities designed to obtain information sufficient to answer that question. The first activity was a conference held in two sessions with RMPS staff who are involved in various uses of the Review Criteria. The results of that conference are reported in Section D - "Summary of Conferences with RMPS staff on RMP - Review Criteria." The second activity consisted of a series of interviews with ten users of the Review Criteria whose opinions of and insights into their uses was considered! important by the Project Officer and his advisors. These users included the Director of RMPS, three members of the Staff Anniversary Review Panel, three members of the Review Committee and three members of the National Advisory Council as follows: Director RMPS Dr. Harold Margulies Staff Anniversary Review Panel Mr. Richard Russell Acting Chief Western Operations Branch Division of Operations and Development Dr. Edward Hinman Director Division of Professional and Technical Development Mr. Cleveland Chambliss Director _ Division of Operations and Development Review Committee Dr. J.E. Kralewski Assistant Professor and Director Division of Health Administration University of Colorado Medical Center Dr. William G. Thurman Professor and Chairman . Department of Pediatrics - University of Virginia 31 Dr. Alexander Schmidt Dean Abraham LincoIn School of Medicine University of Illinois National Advisory Council Dr. Clark Millikan Consultant in Neurology Mayo Clinic Dr. Anthony L. Kamaroff Beth Israel Hospital Boston, Massachusetts Mr. C. Robert Ogden President and General Counsel North Coast Life Insurance Company The interviews were basically open-ended, guided interviews with a section of structured, closed-ended questions related to the relative usefulness of the RMP Review Criteria Scores in decision-making in comparison with the following items. 1. Grant Application Program File | Staff Briefing Document Site Visit Report Review Committee Report Primary Reviewer's Presentation ~s OO om FSF WD HW Secondary Reviewer's Presentation 8. Discussion Each respondent was asked to rate each of the sources of information, including the Review Criteria Scores as Very Important, Important, Uncertain, - Not Important or No Value to decision-making in the review process. This is a report on that rating exercise, all. discussion and comment recordéd during the rating exercise, and any general comments made during the _. interview related to the usefulness of the Review Criteria. . It has been suggested that the Review Criteria scores should not be treated as sources of information. Only the National Advisory Council 32 are wh oe I £ id 2 eed eas! Seger ~ wv 2 23 4 rr SST kate ee aa and the Director of RMPS receiv2 any Review Criteria scores and that is only in the form of total scores. The results of the rating exercise clearly indicate that all respondents were aware of the Review Criteria and the scoring procedure and that they were not constrained in their evaluation by any reference to the Review Criteria scores as a "source of information”. Table of Definitions 1. Grant Application - The one year and/or three year plan of operation submitted by the local RMP. 2. Program File - Previous grant applications, site visit reports, advice letters and other written information which serves as a background to the grant applications under consideration. 3. Staff Briefing Document - A summary and analysis of the important points out of the Grant Application and of current developments with the “RMP prepared by the Operations Officer from Division of Operations and Development assigned to the local RMP. * 4, Site Visit Report - The report prepared by the chairman of the site visit team at the conclusion of the site visit. Such teams consist of members of the National Advisory Council, RMPS staff, Consultants — and a chairman from the Review Committee. ar 5. “Review Committee Report - The analysis of the local RMP's program and grant application and recommendations on program plan and funding” level for the next one year and/or three year period. mn 6. Review Committee Scores - The average of the scores prepared by =. each member of the Review Committee on each-of the 20 criteria — in the Review Criteria. This includes both average raw scores and average weighted scores. ., 7. Primary Reviewer's Presentation - The oral presentation of the primary - Yeviewer, usually the chairman of the site visit team as to his analysis of the local RMP's progress and grant application, the fundings of the site visit and the recommendations of the site visit team on program plan and funding level. 8. Secondary Reviewer's Presentation - The oral presentations of the second- ary reviewer based on analysis of grant application, staff briefing document and site visit report usually without benefit of partici-. pation in the site visit. Ce Pes 9. Discussion - Any discussion in committee over and beyond the oral presentations of the primary and secondary reviewers. . 9° wanes 33 TABLE 2B-1 Availability of Various Information Sources to Nembers of SARP, Review Committee, National Advisory Council and Director RMPS. X = Supplied ; specifically SARP REVIEW COMMITTEE COUNCIL beforehand wn -~ fs a 2 2 @ = Available but ae ~ > « = not specifically Y a ex ae an © & supplied ud Wy FE Se wo a rm 2 oe = wn oe — ard ~ LJ WwW om Oo.” H | & | = 2 | sh | & os | = | 2 -| & NA= Hot Available as 7 = = > Pate 5 i tu 5 tus vet (or applicable) ec ° = Lad i a a mee mz me . ” a. 2 1. Grant Application X ® X X X « X X e a 2: Program File e e e e e e e e e 3. Staff Briefing: Document X X xX X X X X X X 4. Site Visit Report : NA NA x fx X x x X x 5. Primary Reviewer's Presentation NA NA NAC NA X Xx NA NA NA 6. Secondary Reviewer's Presentation | NA NA X X NA X NA NA NA 7. Discussion X X X X X X X X Xx 8. Review Committee Report NA NA NA. NA NA NA X X x 7. 9..,Review, Committee Scores. —|_—NA NA | NA NA NA na |X X x 34 eect ER | ge, rrr fam EH PETA eng Tet i 3 * Le gd Ves ae Jet fem 4 hee oe < Ae ee: Tents ca i z se ce The Table of Definitions describes all of the sources of information referred to in the interviews. Table 2B-1 indicates, in part, the availability of the various sources of information to each of the four classes of respondents. The following brief descriptions indicate how the various groups carry out their review function. Staff Anniversary Review Panel: This is composed entirely of senior RMPS staff. A local RMP under review is presented by the Operations Officer assigned to the program. The members of SARP base their discussions and recommendations on this presentation, the grant application, the staff briefing document and, if necessary, the program . ile. Review Committee: This is composed of health professionals selected on the basis of their technical competence in various areas related to RMP operations. The RMP under review is presented by the primary reviewer who, not uncommonly, was chairman of the site visit team. Corroborations or additional findings are furnished by a secondary reviewer who was not on the site. visit team. The discussion and recommendations are based on the primary and secondary reviewers presentation, the grant application, the staff briefing document, site visit report and, if necessary, program file - usually presented by the Operations Officer assigned to the program. National Advisory Council: This council is composed of those categories of people designated in the legislation plus other experts and laymen who are chosen to preserve the intent of Congress as specified in the legislation. A local RMP under review is presented _ by a primary reviewer who, in most cases, was a member of the site visit team. Corroboration and additional findings are presented by a secondary reviewer. The discussion and recommendations are based on the grant application, staff briefing document, site visit report, Review Committee report and recommendations and the Review Criteria score - a total score only - submitted by the Review Committee. Director of Regional Medical Program Service: The Director is constrained in his discussions by the recommendations of the National — Advisory Council, but, within those constraints, he has the opportunity to make selection funding decisions, i.e. to fund at less than the National Advisory Council approved funding level and to provide specific direction for future program development. He usually is present at the deliberations of the National Advisory Council and has all sources of information plus the counsel of RMPS staff to guide him in his actions. 35 Importance of Various Infor Decision-Making in RMPS Review Proces TABLE 2B - 2 mation Sources to s -~ All Respondents Percent of Responses oe) ~ » . <= fon co = Information Sources > 8 2 ps ol Z Number of oo o| oO &s} £2 5 Responses > 2 a wv ou o ke EY =)2] | = —_ Grant Application 75 12 12 99 8 Program File 38} 12 | 12 | 25 | 12 | 99 8. Staff Briefing Documents 62 12 25 99 8 Site Visit Review Committee Revi C itt Sere 20 | 40 | 30 | 10 100 10 Primary Reviewer's " g Presentation 15 25 100 . Secondary Reviewer's ) Presentation Ve 120 | 26 38 12 | 99 8 Discussion 25 | 50 | 12. 12 | 99 8 TOTAL NUMBER OF RESPONDENTS ........cceeecee rece sere receeeeneees 10 36 peo fee i Saget o_ a? i 3 S Oa FINDINGS AND DISCUSSION - General Opinions on the Review Criteria and Process The general opinions of the respondents on the review criteria _ and the review process provided a background for interpretation of their - attitudes toward the various information sources. Positive statements focused on the idea that the criteria provided a consistent framework through which a wide range of components measuring the strengths and weaknesses of an RMP can be considered. They felt that such a uniform approach was needed to create a basis for comparison between RMPs. In addition, they felt the existence of the criteria in written form gave the local RMPs a better idea of how they would be reviewed and a better framework to use in their own planning and evaluation. Those respondents who felt positively about the criteria generally agreed there were some areas of weakness as they stand now, but that they do provide a good overall approach to evaluation. Reservations on the review criteria centered in two areas. First, respondents felt the weakness of the present definition of what an RMP should do made it generally difficult to evaluate the organization. They stated that no specific goals had been laid out for RMPs so that detailed outcome measures could not be established. They felt that the existence of the review criteria had not solved this problem, but had merely exemplified that it did exist and that until a new definition could be worked out, effective review was impossible. Second, some felt that the numeric analysis, particularly the weighting system, applied to the criteria was artificial or at least needed reexamination. Others disputed the idea that an RMP should be measured numerically at all. They questioned whether such a distillation of details really reflects the significant features of an RMP. General Opinions on the Importance of the Various Information Sources to Decision-Making in Review Process : On the basis of the analysis of all responses, the following observations on the relative importance of various information sources can be made. The respondents thought the primary reviewer's presenta- tion was the most important source. Between these two extremes, the information sources fell into two groups in terms of perceived importance: the "more significant” groups and the "less significant" group. In the "more significant" group were the grant application, the site visit report, the staff briefing documents, and the review committee report. Respondents placed review committee scores, along with discussion and ‘the program file, into the "less significant" group of sources. These observations were discussed further below. Table 2B-1 presents the interview results upon which these observations were made. 37 The primary reviewer's presentation was generally viewed as the most important source of information in evaluating an RMP. Of the eight individuals who discussed the primary reviewer's presentation, six said it was very important, and two said it was important. However, while saying this, several respondents insisted on discussing inherent weaknesses in this source of information. These stemmed chiefly from the feeling that, through a primary reviewer technique, a great deal of significance became attached to the opinions and presentation of a single person. Two respondents thought that, if the reviewer was particularly impressed by the RMP and made a very positive presentation, the importance of the rest of the committee procedure is lessened (Hinman, Russell). The fact that some reviewers made more effective presentations than others with more influence on the group was also pointed out (Russell). Another respondent felt that, because the presentation was oral rather than written, not all the collected information was communicated in some instances (Therman). In general, - respondents thought that the quality of this source of information was dependent on the amount of effort the reviewer had put into the project and his personal attitude toward the region. Despite these weaknesses, committee and council members apparently relied heavily on the primary reviewer's presentation for an analysis of the region that they do not have time to do themselves. Several res- pondents discussed features of committee and council procedures which helped explain the situation. In most cases, the primary reviewer is the only individual who was well informed on the RMP under review. "T think the weakness of the Review Committee or (Advisory) Council is that most of the people don't do their homework. It (written material) hasn't been read by the time we've arrived (Thurman)". Primary reviewers seemed to have more extensive and detailed information than other committee and council members. "If you're not the primary reviewer you get briefer documents...Review Committee recommendations is what it amounts to, and those are accepted somewhat non-critical ly by members of the NAC who say ‘well, we're relying on the entire process’ (Ogden)". The fact that the primary reviewer's presentation was thought to be most important should also be examined in relation to the fact that the secondary reviewer's presentation was viewed as the least important source of information. There was a range of opinion on this item, but, of the eight who addressed the subject, only one found it very important (Chambliss) and one found it important. Six people chose the lower three ranks to label the secondary reviewer's presenta- tion. Some of the respondents expressed the feeling that these presenta- tions were useful only in terms of being a double check on the primary reviewer's presentation (Russell) and gained importance only where there was an obvious difference of opinion (Millikan). The remaining documents fell less easily into a. clear position within a rank ordering according to importance, but tended to cluster into groups of “itfore significant" and "less significant" sources. 38 E Lt Coty oF! Sy acreage” “3 3 Included in the "more significant" group were: the site visit report, review committee report, grant application and staff briefing documents. All of these documents had a higher percentage of responses in the very important category and higher cumulative percentages in the very important and important categories. A higher degree of consensus on importance was also expressed concern- ing documents in the "more significant" group than those in the "less significant" group. The grant application was regarded as a source which can vary greatly in terms of usefulness, depending on the quality of the document. "In certain instances where things are beautifully put together and told (in terms of) exactly how they are going or how they have gone off, the grant application can be very important. There are other instances, that I was personally involved in, where I see the grant application was of very little value other than as a kind of reference document from which one could begin to ask some questions (Millikan)." "It's not because it's necessarily factual, (everybody makes a grant look better than it is), but it gives. you a feel for the entire program and unknowingly it points out a great many of the weaknesses of the program as well as the strengths of the program (Thurman) ". Use of the grant application also depends on the individual's role in the review process. For primary reviewers or members of the site visit teams, the document became more important (Hinman). Another respondent indicated, "As a member of the NAC, I never see a grant application (Kamaroff)". A great deal of time was required to study a grant application thoroughly, (Russell) and this tends to limit the number of reviewers who used it effectively. When not studying the particular RMP in detail, the grant application could become mixed with the staff briefing documents (Margulies). One respondent particu- larly noted the value of the RAG report within the application: "We look in the RAG report to see how they have addressed the issues that we raised in the previous review or whether they have really addressed the, or whether we are getting a snow job (Russell) ." Staff priefing documents were seen as a source for developing a historical perspective on RMPS (Millikan, Thurman, Russell). Eight. respondents discussed the documents which were prepared by individuals who were "uniformly involved almost all the time” with the RMP and have collected a large volume of information. However, one respondent added that information can be presented in a biased manner (Thurman) . Documents do vary considerably, the best being those which related what the RMP has done without trying to convey any opinion as to whether the RMP was good or bad. : a Site visit reports were labeled very important by four people and important by three people, but few respondents gave explanations for 39 these opinions. One respondent (Kamaroff) added the fact that site visit reports became particularly important to committee or council review when a member of the committee or council was on the site visit team. Respondents also commented that the national review _ criteria served to encourage greater consistency and clarity of format in the site visit reports. The site visit itself was obviously one of the key steps in the review process; however, one respondent (Ogden) expressed the concern that more preparation by team members for the site visit is needed. He felt that site visits could become highly controlled by the RMP staff, not allowing the team to explore weaknesses or other specific areas which must be covered under the guidelines. The respondent thought that the team should meet beforehand, discuss the program, and divide up different areas of investigation and reporting to insure that the team examined all necessary aspects of the RMP. Another respondent added that small informal group meetings between RMPS staff and the local RMP could often be relied on to be more informative than the site visit because of these weaknesses in the site visit procedure. Review Committee reports were cited as either important or very important by the five respondents who discussed the source. One respondent (Millikan) particularly noted that the report was nearly always automatically accepted as valid by the NAC. He felt that the council assumed that the Review Committee had done a careful job and seldom reversed its recommendations. Only when new information was introduced after review, when information from the site visit had not been weighed by the committee, or where a matter of policy or philosophy was involved would the council consider changing the recommendation. The "less significant" group of sources are: the discussion, the review criteria scores and program files. Respondents felt discussion becomes important only when various other sources of information were in disagreement. At SARP, NRC and NAC meetings, when the information sources presented do not make similar recommendations for the RMP under review, discussion on differing opinions became significant. Two respondents cited discussion as "very important" and four said it was "important". There was a variety of opinion on the importance of the program file. Again the program file, like the grant application was used more by some individuals more than others in the review cycle because of the study time involved. Its analysis was most often undertaken by _staff members. ‘Two individuals labeled review committee scores “very important” and four seemed to think they were important”. However » in discussion a number of reservations were voiced on the scoring design and procedure. 40 cla yakorsy Es eetiel * One respondent voiced strong opinions that RMP evaluation could not "be reduced to numbers" (Schmidt). Another said there were failures in the weighting scheme which called for reexamination (Ogden). Another respondent said committee members are reluctant to score programs when they don't have strong comparative information or site visit experience (Kralewski). Other sources of information, which were not directly probed by the interviewer, were described by the respondents as being important. Several said they used the Management Reporting and Evaluation System (MRES) to some extent. One added that because the summaries were in the form of computer printouts they were difficult to handle (Russell). A variety of less formal sources of information were thought to be important to different degrees depending on circumstances. Accumulated personal knowledge and peripheral information picked up while working "on various tasks, often unrelated to RMP, were picked out as being influential factors (Thurman; Russell). Respondents also noted small meetings and seminars as being means by which information was directly obtained from an RMP (Margulies, Hinman) . CONCLUSIONS On the basis of the analysis of the interviews, certain common attitudes toward different information sources became clear. Review Committee Scores were not generally thought to be as important as several other sources of information; they fell into a group of “less significant" information sources. Other sources in this group are discussion and program files. The primary reviewer's presentation seemed to be regarded as the most important. The grant application, the site visit report, the staff briefing documents, and the review committee report also were thought to be more important than scoring. The respondents view the secondary reviewer's presentation as the least important source. However, there was also a-strong feeling that the significance of an information source dependent on its quality is highly variable. The grant application and staff briefing documents were particularly singled out to be sources where this is true. , Other variables were mentioned which make the use of documents dependent on specific circumstances of the review cycle. For example, if there were discrepencies between the recommendations suggested in what were generally the “most significant" sources of information, the "less significant" sources become much more valuable. Conversely, if there was strong agreement on recommendations for the RMP to begin with, the “less significant" sources dropped from concern. Clearly: there were variables, particularly the quality of the information sources:or | details of the specific situation, which prevent rigid rules on the degree. of usefulness .from being applicable. LR 4] oie ‘s . C. VALUES EXPRESSED BY TEN USERS IN DESCRIPTIONS OF LOCAL RNPS ie LTT Cy wena roger™ Perea tas pT 3 a aye pokey The RMP Review Criteria were designed to assess the comparative quality of Regional Medical Programs. In the contract for the evaluation of the RMP Review Criteria, the contractor was asked to "determine the adequacy and effectiveness of the Review Criteria in assessing the comparative quality of Regional Medical Programs as perceived by those utilizing them." In this process, he was asked to address his efforts towards answering a series of questions,=but he was not limited to those questions. In the approach to this assignment, the Review Criteria were assumed to represent a series of values which could be used to describe the quality of a local RMP. It was further assumed that the Review Criteria would be adequate and effective for that purpose only if they coincided with the values that those involved jn the review process used in making judgements during the review process. In the ideal situation, all users would have iden- tical values and that set of values would correspond to those of the Review Criteria. Ordinarily one would expect differences among the users and between the users and the Review Criteria. The degree of correspondence or lack of correspondence between the value sets of the users and that of the Review Criteria is an index of the ade- quacy and effectiveness of the Review Criteria in assessing the comparative quality of Regional Medical Programs as specified above. Three sources of information were developed in an attempt to respond to this item in the contract: 1. Analysis of the Review Criteria in a worth assess- ment model ; 2. Conferences with RMPS staff - see Section D; and 3. Interviews with ten selected people participating in the review process at different levels. This is the report on the interviews with ten selected respondents. The general design of the interviews and a description of the respondents has been presented in Section B. The data from the interviews for this analysis has been drawn from the answers to the following open-ended lead questions: 1. Describe, in terms of accomplishments, a specific RMP that has done well. , 2. Describe, in terms of accomplishments , a specific RMP that has not done well. 3. What kinds of things can a good RMP achieve that a poor one cannot? ore 45 12. 13. 14. 1. g. V7. 19. 20. TABLE 2C - 1 Number and Percent of Respondents Making Statements in Twenty Categories of Concern Categories of Concern An RMP should be involved and work closely with the communities in the region (Ex. through subregionalization, outreach, etc. An RMP should be involved in activities dealing with categorical diseases. An RMP should be involved in activities dealing with training and. education of all types of health manpower. The key health interests should be accurately represented on the RAG. An RMP should serve as a forum to bring to- gether al] the key people in the health care system.. An RMP should serve as a helper and advisor to other health planning agencies, especially CHP. The RAG should actively establish the program's goals and priorities. An RMP should be involved in pro- ducing meaningful relationships between providers and consumers. An RMP should be a flexible organi- zation which can serve as a catalyst for action in a variety of settings. An RMP should have an-effective evaluation mechanism. An RMP should have a strong management team. RMP should concern itself with deter- mining the needs of the region, and developing goals and objectives to meet these needs. A = RMP should sponsor activities which will be able to support themselves after the - withdrawal of RMP funds. A s An RMP should be involved with activities dealing with patient care appraisal. The coordinator of an RMP should direct the program towards achieving the long- range goals and major objectives that the program has established. An RMP should have a capable and competent staff. An RMP should develop close relationships and open communication with RMPS. An RMP should be actively involved in — addressing the interests of minority groups. - An RMP should be involved in data collection. De The RAG should determine which specific activities should be initiated in the region. 46. Number Respondents. Percent 80 70 70 50 50 50 50 40 40 40 40 30 30 30 30 30 20 20 20 10 “an eens ny am, oy ed Soe ged sacs fae LEAL? gaecesg Low tis Bead Wa ASRS See SOS een he aren ir eae TELS SSR By _— a Awe The verbatim answers to these questions were trans formed into a series of statements. The statements from the answers to all the questions from all respondents were then categorized according to the concerns that were being expressed. As a verification procedure, a list of the categories of concern was sent to each respondent. Each was asked to sort the categories into quartiles by order of relative importance to local RMPs. All data was subjected to qualitative and non-parametric statistical analysis. FINDINGS Twenty categories of concern were found to encompass all the statements made in response to all three questions - see Table 2C-1. Seven categories included statements by 50% or more of the respondents; sixteen categories by 30% or more. The total of 188 statements reflected an average of 2.35 statements per respondent per category. Because a chi-square analysis showed no relationship between the average number of statements per respondent per category and the number of respondents citing each category, a decision was made to base all subsequent analyses only on the number of respondents citing each category. Ten respondents participated in the interviews; only seven participated in the verification procedures. Of the seven categories including statements by 50% or more of the ten respondents, five exceed this 50% level for the seven respondents. Table 2C-2 shows that the proportion of respondents in the verification procedures citing each category concerned was similar to that of all ten respondents. Table 2C-2 indicates that there was no strong relationship between the number of respondents citing a category of concern in the original interviews and the number who specified that the category belonged in the first or first and second quartile of importance. Although not displayed, the data shows a similar lack of relationship between the data on importance and the number of statements in each category. Discussion The discrepancies between the results of the original interviews and the verification procedures were unexpected. Careful re-reading of the original interview suggested that the context in which the respondents described the qualities of good, poor and ideal RMP's might help to explain those discrepancies. This re-readina 47 TABLE 2 c - 2 - Number of Respondents Making Statements in Each - pd Category of Concern and Ranking Categories in = Quartiles of Importance . - NU. OF RESPONDENTS : Categories Interview Verification* Importance Importance of Concern Sample Sample Quartile Quartile o I 1a Il vt 1 8 5 4 7 hi 2 7 5 0 0 7 2 8 4 5 4 0 2 fa 5 5 3 2 4 ‘ 6 5 4 1 3 a ee 7 5 2 4 6 ey 8 4 3 1 3 Ee 9 4 2 4 7 f 10 4 3 4 5 wu " 4 2 1 3 he 12 3 3 4 6 : FA 13 3 2 0 4 a4 14 3 2 0 1 15. 3 2 5 6 ® ; 3 nn 17 2 0. 0. 1 | on 18 2 2 0 2 E 19 2 2 0 7] 0 2° oe ae: 3 (nyo (10) () 0) (7) (3 *Only seven respondents returned the verification questionnaire. 48 Soa “7 vt “4 also re-emphasized the fact that the ten people selected because of their participation in the RMP review process interpreted the questions about "accomplishments" in a manner quite different from that intended by the interviewer. The interviewer asked the respondents to think of RMPs that had done well, that had not done well and that were ideal and to describe them in terms of "accomplishments". The intent was to obtain descriptions of the impact that the three types of RMPs had had on the medical care system of the regions that they served. Analysis of the complete text of all interviews revealed that the respondents chose to describe the RMPS in terms of: 1. Capability - the organizational structure of the RMP and the perceived competence of its staff and regional advisory group 2. Direction - the-type of activities undertaken. by the program, and 3. Charisma - the characteristics of the leadership evidenced by the program and its activities. A Regional Medical Program, according to the respondents , may be said to have capability if it has: 1. A Regional Advisory Group which is broadly representa- tive of key health interests in the regions, which actively establishes the program's goals and objectives, and which determines which ‘specific activities should be initiated -- 2. A coordinator who has assembled a capable and competent staff, who directs the program towards achieving the long-range goals and major objectives that the program has established, and who has an effective management and evaluation mechanism -- 3. A program which serves as a forum to bring together all the key people in the health care system and as a helper and advisor to other health planning agencies. The direction of a Regional Medical Program can be considered satisfactory, according to the respondents, if its activities are: 1. Concerned with meeting the needs of the region -- Concerned with long term funding of activities -- Actively addressing the interests of minority groups =- > WwW Involved with. activities dealing with patient care, especially those that will improve the health status -of the population in regards to the categorical diseases -~ 49 5. Involved in activities dealing with training and education of all types of health manpower. a A regional medical program, according to. the respondents, has charisma if it is: 1. Responsive 2. Flexible o Aggressive ALE 3 4. Facilitative, that is, work with and through others 5. Expressive of leadership, that is, effective in getting the health care system to solve the problems cm that need to be solved a gone 6. Catalytic as so grrr, poet 7. Regional The following descriptions of the same RMP by two different respondents illustrate the importance of the directional and character components to the identification of a good program: Rea "There were some good (programs) who were categorically Bi oriented, but who have not done well since the alteration 7 of the mission statement. Others who weren't very good categorically came back strong with that alteration. I can think of several that I would consider to be good RMPs. For instance, let's take RMP-A. There are lot's of things wrong with RMP-A, but, on the basis of looking at their initial premise which was that Regional Medical Programs should try to do something about the quality of care for the majority of citizens in a state...,. they moved very smoothly. They were particularly strong in heart disease because the coordinator was a heart disease man but he also was able to have enough vision to think about cancer and . stroke and kidney disease. He's also been able to make - an alteration in reference to thinking about delivery of health care patterns in that area. In talking about what they've accomplished, I'd say that, when you talk about 2 RMP in Region A, you're talking about something that [ all the physicians would recognize and that much of the hee enoeeer: Weve th 3. va et ae Ci er + Cita population would know about because of the tremendous = publicity they've had for various projects. The Regional ( Medical Program existed as an entity that tried to relate aad the person who's receiving care to the providers of care 50 Net nadet nt no As wien gS eg yor al fed aad =a and I think that's one thing that has been successful jn some RMPs and RMP-A is a good example. | think it has established quality of care particularly in heart disease, almost as much in stroke, not so much in cancer, but part of that relates to the capabilities of provi- ders in the area, not necessarily the weaknesses in the RMP. And lastly, I think that they have detected | weaknesses on the base of significant data and have done their best to encourage programs to cure the weaknesses. And I guess, for me, that's a very succinct summary of what I think RMP is all about." "I'1l] talk about what has fascinated me to a very great extent in RMP-A and that is that they have tried to work through and with the hundred or so hospitals in the state in a way to upgrade hospital care and to improve their training programs so that better health care could be delivered throughout the state as a whole. They have worked in terms of developing a very competent and capable staff. They've always had strong, competent, enlightened leadership. They have been able to make changes in their program direction, I think, in a very favorable way. They've shown evidences of really trying to respond to the needs throughout the state. They have not concentrated in the large population centers of the state, but have tried to move their activities out to the far reaches of the state and to the smal] community hospitals and to develop training programs for a wide range of physicians, not just specialists in the (major metropolitan area)." The quoted descriptions illustrate the emphasis on the qualities of "direction and charisma", an emphasis that was more pervasive in the original interviews than in the structured analysis of “categories of concern". It is an emphasis that was almost totally eliminated in the verification procedure. The fact that "capability" was the dominant concern on the verification procedure may have resulted from the design of the pro- cedure. The original interviews allowed the respondents to talk ~ about those qualities -- although the questions were asked in terms of "accomplishments" -- that they felt were important. They were able to speak as individuals to an interviewer trained in guided and supportive interview techniques. The verification pro- cedure, however, was a highly structured, closed-ended mail-telephone survey. It-is hypothesized that the structured instrument presented | an examination-like situation which caused the respondents to answer . not just as individual, but as individuals whose answers were going to be compared with those of others. Knowing that the respondents were all part of a review process that focussed on "accreditation", 5] i.e. assessing the capability of an RMP -- see Section D -- one can assume that they were trying to give answers that others participating in the review process would give. Why there should be discrepancies between the concerns of the respondents as members of review bodies and their concerns as individuals raises questions that go beyond the scope of this report, but it can be suggested that the current emphasis in the review process on "accreditation" or "capability" reflects a state in the development of the review process which differentiates between “satisfactory” and “unsatisfactory” programs. As the review process continues to develop, one could anticipate attempts to differentiate between "superior" and "eatisfactory" programs that would consider the qualities of "direction" and "charisma". The evidence suggests that considerable effort would be required to further modify the review process to be concerned with "accomplishments". Conclusion Interviews with ten selected participants in the review process plus a verification procedure completed by seven of them provided a basis for identifying three general qualities that can be used to evaluate local Regional Medical Programs: 1. Capability 2. Directéon 3. Charisma The fact that there were marked discrepancies in the emphasis between the interviews and the verification was construed to suggest that there is a possibility that the review process will evolve from its present concern with "accreditation" as a device to differentiate "satisfactory" from "unsatisfactory" programs to concern with differentiating "superior" . from "satisfactory" programs. Evaluating programs in the review process in terms of the "accomplishments" probably will continue to be deferred until there are substantive improvements in the concepts of "accomplishments" and in the information base for describing them. 52 od aeresprne 1 ot agar. “ vod ep eeinm aia inet 3 eee ta ae isa) pli C costo fete er gonna sound Eee Quy aL eld Pe sg VES) D. pT eo : > mo SUMMARY OF CONFERENCES WITH RMPS STAFF ON RMP REVIEW CRITERIA 53 SARE ay are ina! ame SUMMARY OF CONFERENCES WITH RMPS STAFF ON RMP REVIEW CRITERIA February 27, 1973 INTRODUCTION Selected RMPS staff were convened in two separate conference sessions to answer the following general questions: 1. How could the process of RMP Review Criteria development be improved? >. How could or should the use of the RMP Review Criteria be altered? 3. How could or should the content of the RMP Review Criteria be improved? These lead questions were designed to promote a thorough evaluation of the entire Review Criteria process by this group of RMPS staff. This is a summary of the major points, conclusions and recommenda- tions flowing from each of the sessions. SESSION I Development Process Although the first efforts to develop review criteria for use “in the review and approval of local programs was undertaken in the summer of 1970, accelerated development was undertaken in the early spring of 1971 in response to HSMHA's criticism of RMPS' selective funding program. HSMHA, in essence, felt that there was no systematic basis for decisions made under the proposed selective funding program. In an effort to overcome this objection in time to influence the administration of a major budget reduction becoming operative at that time, a staff of eight or nine people began and completed the task of developing the RMP Review Criteria in a period of two weeks. The effort, however, was not sufficient at that time for that purpose. Subsequent modification of the criteria and the development of a complete process for evaluating and scoring local RMPs against the criteria was successful in leading to approval by HSMHA of RMPS’ selective funding approach. The development process was carried out by RMPS staff alone on the basis of the RMP legislation, the RMP mission statement, the President's Health Message of 1971, the advice of the National Advisory Council and the accumulated experience of the RMPS staff. The major criticisms, both of which resulted from the very short time interval over which the criteria were developed, related to the absence of input from other than the senior staff of RMPS and from the coordinators and other local program staff. Recommendations for future development of revised review criteria called for the participation of many more members of RMPS staff and of staff and regional advisory group members from local programs over a sufficient time period so that the criteria better reflect the concerns of all involved in the RMP. It was also recommended that the criteria should focus on those items that tend to discriminate between good and bad RMPs and should specify the objective data to be used in making or contributing to the evaluation of the programs against the criteria. Finally, it was recommended that the Review Committee and the National Advisory Council, the primary users of the results of such a review against criteria outside of RMPS staff, should be intimately involved in any future development of criteria and/or scoring procedure. Utilization of Review Criteria The Review Criteria were reported to be helpful to the review process, but not necessarily easy to use. The Review Criteria formed a structure for the preparation of staff briefing documents and for the site visits that was found to be useful to varying degrees largely determined by the general quality of the work done by the staff person of the site visit chairman. When the information in the staff briefing document and/or the site visit report was well organized in relation to the Review Criteria, the users of those documents were quite content; if not well organized and concise, the users tended to be dissatisfied. The subjective nature of most of the criteria, the absence of precise definitions, and the general lack of objective data related to the criteria were significant deterrants to any uniform benefit to the structuring of information in the review process. The fact that selected individuals were able to use the criteria in a helpful way suggested that improvements in the design of the criteria might lead to a more uniform benefit. The participants in this session felt that the local RMPs had not been unhappy with the criteria per se, but that they were uncom- fortable with the rating system which, in general, they did not under- stand. They, the local RMPs, tended to ignore the fact that the rating system was a device for comparing the 56 programs with each other and to make the erroneous assumption that somehow the scores of ratings were equated to dollar funding. The criteria were acceptable as evaluation concepts or guidelines, but the scores were unacceptable as an evaluation measure. Their confusion was further compounded when they learned that an RMP with a reasonable rating might receive less money than an RMP with a low rating. They failed to understand that selective funding based on Review Criteria scores was not a mechanism for ironing out what might be historical inequities. 56 oer eee. roms: Saat pep caney oe ro ee arya TE) presen tees Lo oe ae a sitet Most of the objections focused on the use of the Review Criteria for the development of scores and ratings. Many of the people who were asked to score programs were reported to object to the lack of precise definitions and to the lack of precise or complete information on which to base their judgements. It was reported that some low scores resulted from the absence of information rather than from any deficiency in the programs. No specific recommendations for changes in the utilization of the criteria were made, but most of the respondents hoped that any revisions in the criteria would make them easier to use, primarily, in relation to the obtaining and organizing of information and, secondarily, to the scoring and rating. Content of Review Criteria When the Review Criteria were originally developed, it was the reported intention of RMPS to periodically modify and/or update them. Such an intent was based both on periodic changes in RMPS policy and on increasing insight into those factors associated with the success of a local RMP. The participants felt that modification of the Review Criteria to reflect changes in policy might be innappropriate, but that the content of the Review Criteria should somehow be able to accomodate such policy changes without changing the fundamental precepts and definitions on which the criteria were developed. They felt that the Review Criteria was not a proper way to communicate changes in RMPS program policy. The content of the Review Criteria was judged to be oriented more towards the structure and organization of an RMP than towards its accomplishments. The criteria tended to focus on where an RMP was rather than where it was going or what it had done, but that focus -was based on a rather unstructured and unvalidated model. Although there was some interest expressed in developing more accomplishments or outcomes-oriented criteria, this was thought to be extraordinarily difficult. The major recommendation for changing the content of the Review Criteria related to those changes needed to make them more useful to the users. It was suggested that some means be established to monitor the problems which users of the Review Criteria experience so that such problems can be addressed in any revisions. Such revisions would include changes in content as well as changes in definitions and specifications of necessary data. Summar In general, it was reported that the review criteria worked relatively well, but that certain improvements would be desirable. The participants felt that the Review Criteria was a more rational, 57 but limited basis for evaluation judgements than previously existed and that those involved in the review process had demonstrated a willingness to work with criteria. More acceptance of the use of the Review Criteria could probably be anticipated if they were revised by the processes specified towards the resolution of the problems identified. SESSION I] Development Process The participants stated that the review process was directed towards the determination of whether or not the local goals and objectives of the program were being met and how well they related to national goals and objectives. They also stated that the precipitous development of the RMP Review Criteria in a thirty day time period by a group of 10-12 RMPS senior staff was in response to the desire of RMPS staff to administer a major cut in funding by selective reduction of funds for local RMPs rather than an across-the-board reduction. The content of the Review Criteria was thought to reflect the perception of the RMPS staff as to the essential elements of a local RMP. The fact that no performance objectives for RMP as a whole had ever been articulated in the Review Criteria as they were developed. Later modification of the Review Criteria to deal with performance against national objectives as they were specified had been anticipated. The following recommendations were made: 1. Members of the steering committee of coordinators , coordinators of local RMPs, and/or evaluation staff of local RMPs should be involved in any future revisions of the Review Criteria. 2. RMPS should contract with a management consultant firm to provide unbiased guidance to the development of the Review Criteria from the management point of view. 3. Adequate time and effort of talented staff should be invested in the development of a conceptual model of the ideal local RMP and of criteria related to that model. 4. Future development of the Review Criteria should provide for major input from the users, e.g. DOD staff, Review Committee, National Advisory Council. 5. Future development of the Review Criteria should deal clearly with the interface between national goals and local programatic goals. 58 co hed prirat nen ab nite seer “at mika re i } i i a4 ELS ee See ore et aa & se ast 6. Future development of the Review Criteria should include precise objectives for their use. For: example, if the Review Criteria are to be used as guides for local programs in improving their performance, then they should be process-oriented, but, if they are to be used in an award-penalty system, then they should be output-oriented. 7. Involvement of both RMP and non-RMP related people in future developments should minimize any unreasonable biases in the Review Criteria. Utilization of Review Criteria The participants chose to discuss the utilization of the Review Criteria and their content primarily in terms of the development process. A series of questions were raised: . 1. How does the use of the Review Criteria influence the process by which they are developed? 2. How does the development of the Review Criteria take into account existing or changing national policies; goals and objectives? 3. What are the Review Criteria trying to measure? 4. What future uses of the Review Criteria can be anticipated? In the ensuing discussion, it was felt that the use of the review criteria had and should influence the process by which they are developed. The group pointed out that "questions" will influence "answers". As a result, the nature of the questions was felt to be just as important as the portent of the answers. The participants felt that the relation between national RMP objectives, the Review Criteria and the measures were not clear. They further felt that the specific objectives for the uses of the measures were sufficiently objective. These were thought to be problems antici- pated in the original development process because of the brief develop- ment period; they were also problems which were intended to be solved by revisions in the Review Criteria which did not occur. The present Review Criteria are primarily a device for accrediting local programs, not for measuring their accomplishments and/or the outsome of their activities. The following recommendations were made: 1. Local RMPs should be evaluated on the basis of their -accomplishments in relation to national goals and objectives. 59 2. Because the Review Criteria are used primarily by RMPS and its committees for the comparison of local RMPs, the development of the Review Criteria should be done primarily by RMPS personnel. Such Review Criteria should then become an integram part of the RMPS management process in relation to local programs and should not be restricted to the review process alone. 3. Measurements or scoring against Review Criteria should be done primarily by RMPS staff, but should be subjected to review by the local RMPs. . 4. The translation of objectives and criteria into measures should involve both RMP and non-RMP personnel. Summar The present Review Criteria are useful in the review process as an accreditation device. More ‘intensive and sophisticated development of the Review Criteria would increase both their usefulness to the review process and the scope of their use. 60 cad we ve “4 . 4 ee ava ote med Ee yt eles aoe eo rnd cc tee menay a geedoe 4 ot, Get ” a C23 Woes 4 on tat Bog tot rane wh ree spy ded id sii Gees (elas “= ay a CHAPTER III EVALUATION FINDINGS ‘61 This evaluation of the RMP Review Criteria was conducted at the request of the Regional Medical Programs Services and was designed to obtain information which would "include, but not necessarily be limited to answering" the specific question raised in the Scope of Work - see Appendix B. That information has, in large measure, already been presented in the history of the legis- lation, the history of the review process, and the reports of component activities. The evaluation, however, requires that the information be organized and reported in a manner which responds to the manifest concerns of RMPS. , Findings Regarding "Internal Validity" The first area of concern was the "internal validity of the Review Criteria" and the contractor was instructed to "assess the manner in which the established Review Criteria are, in their application, sensitive tools in adequately and objectively assessing the effectiveness of Regional Medical Programs for the purposes of Regional Medical Programs Service review at the National level by both the Regional Medical Program Review Committee and the Staff Anniversary Review Panel." The statistical analyses relative to this concern and all the questions specified in the Scope of Work have been reported on pages 23-28 of this document and in the Second Quarterly Report from this contract. Those analyses were based on scores prepared by individual members of the Review Committee and the Staff Anniversary Review Panel. They indicated that there were statistically significant differences in the raw and weighted total scores and the scores on major criteria groupings among programs classified as "A", "B" and "C" regions. Because programs were classified on the basis of their total scores and because the correlation between scores on the major criteria groupings and the total scores exceeded 0.9, this finding was not unexpected. An analysis of the scores for the individual criteria showed exceptionally high inter-criteria correlation. When this correlation matrix was subjected to principal components factor analysis, only one factor was identified. If the major criteria groupings had been valid, there would have been three factors. The fact that only one factor was identified indicated, according to the consulting statistician, that the Review Criteria did not constitute a scale or measure that was in any way superior to a single rating on Overall Quality. Scoring on the individual criteria did not, in his opinion, contribute to an identification of those specific qualities which differentiate between good and poor RMPs. More specifically, a good program gets good scores on all criteria and a poor program gets poor scores on all criteria. 63 The analyses also showed exceptionally high inter-observer correlations. There was nosignificant bias of the primary reviewers, as a class, nor of any individual reviewer. The absence of significant bias of any significance was shrouded in such an extraordinary degree of agreement among the scores of all the reviewers for each program that independenceof judgement appeared to be absent.. Stated another way, the scores for a given program were SO uniform as to suggest that they had been, at least tacitly, agreed to before thes reviewers completed their ratings. In the absence of scalarity and independent peer judgement, the assessment of internal validity becomes meaningless. The internal consistency of the scores and the consistency among different observers indicated that the Review Criteria scores express the consensus of the review groups as to the quality of the programs rather than the convergence of independent opinions. These conclusions to the effect that the Review Criteria are a very weak tool for adequately and objectively assessing the effectiveness of Regional Medical Programs was substantiated by criticisms and recommendations for improvements reported in the conferences with RMPS staff, pp. 53 - 60 and the interviews, pp. 29-41 . These criticisms can be summarized as follows: 1. The scoring against the Review Criteria is too subjective. 2. Adequate information for making subjective judge- ments against the criteria is not uniformly. available. 3. The scoring does not contribute significantly to decision making in the review process. Many of the recommendations from RMPS staff called for making the scoring process more objective and for improving the availability of appropriate objective information. None of the weaknesses identified during this evaluation, however, appeared to have disillusioned the users who, uniformly anticipated further use of the Review Criteria and who, therefore, hoped that the weaknesses would be eliminated. Findings Regarding "Effectiveness" The second area of concern to RMPS was "perceptions of those utilizing the criteria in the national review process as to their effectiveness". The contractor was instructed to ‘determine the adequacy and effectiveness of the Review Criteria in assessing the comparative quality of Regional Medical Programs as perceived by those utilizing them (e.g. Review Committee members, Regional fA He Ty eed ec fa he ak ponent Staal ed z E 5 yy GST hese! Sata Sot Er : Sete F 4 3 =o € 7 ei Cree 3 Medical Programs staff). The information obtained was to be aimed at, but not necessarily limited to answering a set of six specific questions. Five of the six specific questions concerned the relative importance of major criteria groupings and individual criteria, the availability of information and the rationale for adding or deleting criteria. The sixth question related to perceived improvements in the review process that had resulted from use of the Review Criteria. No information that would answer the first five questions has yet been reported because of an unexpected occurrence. ‘Following principles of worth assessment as described in the book, Professional Decision Making by James R. Miller, III, and as articulated in consultations with Dr. Miller, a conference to answer these questions was held with selected RMPS staff. The objective of the conference was to identify those qualities that the participants, as a group, felt were important in evaluating the .local RMPs. The intent was to compare the identified qualities with the Review Criteria so that the five questions could be answered. The agenda of the conference called for the participants: 1. To iridividually sort the questions contained in the Review Criteria into clusters of similar or related questions and to title those clusters -~- 2. To compare and discuss the clusters and titles of individual participants -- 3. To develop, as a group, a single set of cluster titles --. 4. To individually sort the questions into clusters under the agreed upon titles. The agenda was excessively long for the scheduled time of the conference and, therefore, the objectives could not be achieved, ‘but a totally unexpected phenomenon that occurred during the comparison and discussion segment suggested that days rather than hours would have been needed to develop a consensus on cluster titles. The phenomenon was two-fold: a. Marked differences of opinion about qualities that were relevant to the evaluation of RMPs--differences both among individuals and among groups of individuals from different Divisions and Offices of RMPS; and b. An unwillingness of the participants under the circum- stances of this conference to negotiate their differences and come to consensus on cluster titles. 65 An analysis of the questions clusters prepared at the start of the conference by each individual failed to show any consistent clustering patterns and gave further evidence of the diversity of opinion apparent in the discussion. Because the development of a consensus would require considerably more effort on the part of the participants then they would be willing to provide due to the announced phase-out plans for RMP, this approach to answering the five questions was — discontinued. Subsequent conferences -- see pp 53-60 and analysis of interviews -- see pp. 29-41 suggested that the major criteria groupings were satisfactory in concept, that is that Regional Medical Programs should be evaluated on their Performance (Achievements) , Process (Organizational Capability) and Proposed Program (Proposed Plan), but that significant modifications in the individual criteria and their. clustering would be needed to.make that concept operative. Many felt that, in spite of the major criteria titles, the Review Criteria tended to focus, at least in terms of how they were applied, on Process in an "accreditation" concept: "The criteria tend to Focus on where an RMP is rather than where it is going or what it as done.’ Specific criticisms were levelled at the individual criteria, the scoring process and the detailed questions. Individual Criteria: RMPS staff were particularly concerned about the number of criteria and the overlap between them. The _. interviews identified concern’ about their lack of precision and RMPS staff reported that site visitors, in particular, did not always have a clear understanding of what was intended by each individual criterion title. Scoring Process: Numerical scoring was disliked by many of those interviewed because, according to the impressions of the interviewer, they felt that the numbers did not allow them . to express their judgements in a way with which they were comfortable. They apparently felt that numerical scoring was an artificial process. They reported that the assign- ment of numerical values was a subjective process that, as currently being practiced, required each reviewer to compare the program under consideration with other programs with which he was familiar. They apparently would have preferred some Trechanism for getting around the need for making comparisons and would have preferred to have reliable information uniformly available upon which they would base their judgements. 66 tne! reed f ee Bs este ae wal 3 renee cee hit fre seen int Dilan a or one: pate "3 i ‘nota porte ke. Poa f acs Questions: The questions were criticized for their number, their redundancy, their lack of explicitness and the inavail- ability of relevant information. In spite of these criticisms, RMPS staff and selected members of the National Advisory Council, Review Committee and Staff Anniversary Review Panel felt that the Review Criteria had been an effective management tool for the following purposes: 1. To facilitate a change in the review process from a project-oriented to a program-oriented approach; 2. To provide additional structure for technical assist- ance in program development provided to local programs by RMPS; 3. To establish mechanism for preserving continuity in the review process in the presence of turnover of - personnel on the various review bodies; and 4. To serve as a partial basis or rationale for selective funding decisions made by the Director of RMPS. The Review Criteria contributed to or facilitated the change in the review process from a project-oriented to a program- oriented approach by providing a framework for organizing and/or soliciting information. Although many of those involved in the review process were quite critical of the requirement for scoring programs agains the criteria, that requirement probably forced them to pay more attention to the details of the criteria than would have occurred if there had been no such requirement. The scoring exercise, although of low repute among the users, was probably an essential ingredient to the facilitation of the changes in the review process. The Review Criteria, according to RMPS staff, began to serve as the structure for advice letters to local RMPS in 1971. Prior to that time, local programs received only a notification of award, a notice which may or may not have provided an assessment of their strengths and weaknesses as identified in the review process and, therefore, varied considerably as a useful guide for future program development. As the review process shifted to the program-oriented approach, RMPS began to play a more supportive role in local program development. This took the form of reporting to the local programs those areas within the framework of the Review Criteria in which they had been judged to be doing poorly and advising them to make specific improvements. Because the Review Criteria were. publicized and formed the framework for the advice letter, local programs quickly began to understand how they would be reviewed in the future and, as a consequence, they had a better framework for 67 for their own planning and program development. Although no specific information was available to evaluate the Review Criteria as a mechanism for preserving continuity jn the review process, RMPS staff, in informal conversations, did report that the criteria had been helpful in this area. The basis for selective funding decisionsmade by the Director - of RMPS has not been explicitly stated in a public document, but it appears, on the basis of conversations with RMPS staff and relevant - BMPS memos, that the decisions are based on the concepts of equity, incentives, and other unidentified factors. The concept of equity impTies that there should be an allocation of funds whicn Is re ated to parity of funding on a per capita basis and to the existing quantity of available resources and magnitude of need within regions. These components of the concept of equity plus other still to be identified components have not been transformed into a funding formula; they are, however, considered in selective funding decisions. The concept of incentives, that is, programs that do well should be rewarded and those that do poorly should not be rewarded, also contributes to the selective funding decisions. The Review Criteria contribute to the implementation of the concept of incentives in selective funding decisions in the following manner. On the basis of Review Criteria scores, each local program receives the designation of " "ype, "Cc", with "A" being the highest. Almost all "A" regions are funded at 100% of the award level approved by the National Advisory Council. Only a few of the "c" regions are funded at the 100% level. This is based on the premise that an "A" program is more accountable and, therefore, better able to get full value from their funds than a "C" program. The variations of funding levels within each class of programs reflects the influence of the concept of equity as well as variations in amount of money available for awards during different parts of the fiscal year. Conclusions The evaluation findings indicate that the Review Criteria have been a very effective "management tool" within the operations of Regional Medical Programs Service, but that considerable improvements in their design, content and application are needed to make them. function as an adequate and objective measure that effectively assesses the "comparative quality" of loca Regional Medical Programs. 68 a perme: Satan i) Seer See: ua a eg sree u & i: % nw a4 Pat we zy] CHAPTER IV RECOMMENDAT IONS 69 _ . The evaluation of the RMP Review Criteria, although already a reported in the evaluation findings, assumes a more coherent form in By the recommendations. The seventeen recommendations are based on the findings, criticisms and comments as reported in the three specific 833 studies, the reports of the two conferences with RMPS staff and addi- ie tional data derived from analyses of RMP-related documents, informal interviews and consultations. In many ways, the findings presented as a rationale for the recommendations are more understandable than the findings presented as information related to the purposes and questions posed in the Scope of Work. The rationale for the recom- mendations, therefore, serves as a partial reiteration of previously reported information and as a clarification of the concepts underlying the approach to the evaluation. The recommendations fall into three general groupings. The first six recommendations outline some uses for the Review Criteria in 1) the review process, 2) selective funding decisions, 3) local RMP program planning, development and operations, and 4) RMP program evaluation. Recommendations #7-9 describe suggested methods for improving the content of the Review Criteria and call for some specific modifications in their design. The last eight recommendations deal with specific changes in the ways in which the Review Criteria are developed and utiltzed tn RMPS operations. ns All the recommendations are made on the premises that Regional ie Medical Programs will continue to operate as a federal program and that © = the Review Criteria, in modified form, will continue to be used by RMPS a as an adjunct to future RMP activities. The announced phase-out plans e for Regional Medical Programs and their subsequent reversal makes the Pe - first premise tenuous. The second premise, although conditioned on the first, is reasoned in terms of the support for the Review Criteria ge evoked from RMPS staff and various participants in the review process a during this evaluation effort. Should either or both of these premises prove to be false, the reconmendations should be reviewed for their applicability to ather federal health programs that choose to develop explicit criteria for use in review and approval decisions and in other aspects of program operations. - Recommendation #1: The Review Criteria should be used to develop and maintain a balanced perspective in the review process on performance, process, and program proposal. The RMP Review Criteria are organized into three major groupings: 1) Performance, 2) Process, and 3) Program Plan. These groupings have weights of 40, 35 and 25 respectively and imply that they follow that order of relative importance. 7 Comments from the conferences and interviews and the analysis of findings from the interviews and the verification study indicate at: 1. The review process focusses on "accreditation" and emphasizes criterta related to "process." 2. The Review Criteria, in spite of their design, emphasize "Process" over "Performance" and "Program Proposal.” Although the statistical studies indicated that scoring against the Review Criteria resulted in a rating against "Overall Quality" and not in measurements involving a series of different qualities, it can. be assumed that “Overall Quality” is really a proxy for those qualities that describe the capability of an RMP, i.e. "Process", in terms of its organizational structure and perceived competence of its staff and regional advisory group. Many of the participants in the evaluation activities expressed dissatisfaction with the focus on "accreditation" and advocated the balanced approach that underlay the original design of the Review Criteria. Specifically they advocated an increased emphasis on "achievements" or "performance", but felt that such an emphasis would require the develop- ment of better concepts and information for making judgements in this area. The difficulties in focussing on "Program Plan" were described as the result of the absence of any National objectives or guidelines - against which a local program plan could be judged. In no case, however, did the participants withhold support for the balanced concepts in the Review Criteria related to Performance, Process and Program Plan and they generally supported further efforts to make those balanced concepts operative. Recommendation #2: The Review Criteria should continue to be used as a device for promoting continuity in the review process. The evaluation findings indicated that the Review Critéria had been effective in promoting continuity in the review process. As long as there continue to be changes of personnel in the review process, there will be a need for the new people to know the bases for decision- making that have been established. It was reported that some RMPS staff and site visit chairmen had used the Review Criteria quite effectively to indoctrinate new members of the site visit teams in the information to be collected on the site visits. It was also reported that the Review Criteria had been used to facilitate the incorporation of new members into the Review Committee and the National Advisory Council. Such uses, obviously, can and should be continued. eT ss 72 corners aes Nae fei alt ned pena tm, I oy Coe chery wecremr ond Oa Es gene ey aa eee Saat at ated aa ge ee, sageey Fea ead A Fe ‘'s i Recommendation #3: The Review Criteria should be used to assure consistenc of evaluative judgements within all levels of the review process and among different review cycles. The conference with selected RMPS staff to identify the relative importance of the various criteria and the interviews and verification study with ten participants in the review process suggested that there was considerable possibility for variations in the bases for evaluative judgements and decision-making among the different individuals and groups involved in the review process. Although the present use of the Review Criteria may mitigate against this possibility, more active . efforts to prevent it and, thereby, assure consistency in the review process should be undertaken. Diversity, while essential to the develop- ment of innovative programs, is anathema to review and evaluation. Review and evaluation implies a single set of expectations uniformly applied to all those undergoing those types of judgements. Evaluative judgements , ideally, should be based on explicit. standards whose application should not be dependent upon the composition of the review bodies, the particu-. lar interests of individuals, or changing emphases over time: that have not been reflected in well-publicized formal policy modifications. The Review Criteria, properly used, can function both as the explicit standards for judgement and as the basis for evaluating the consistency of the performance of groups making those judgements. Recommendation #4: The Review Criteria should continue to serve as part of the rationale for selective funding decisions. RMPS has been and will continue to be faced with the need for making budgetary decisions within the funding levels approved by the National Advisory Council. These decisions will be required when the total funds approved for award by the National Advisory Council exceed the total funds available and/or when there are unexpected reductions in the funds available. The budgetary decisions may call for uniform percentage reductions in awards to all programs or selective reductions in awards to some programs. Historical ly, RMPS has tried to use the selective funding approach. Initially, their selective funding decisions were found unacceptable by HSMHA because there was no explicit basis for such decisions. Subsequently, selective funding decisions were allowed, in part, on the basis of program classifications derived from Review Criteria scores. The full basis for current selective funding decisions has not been published, but the concept of incentives does appear to play a role in these decisions. The concept of incentives means that programs that receive an "A" classification as a result of their review criteria Scores are more likely to receive 100% funding than those classified as "C" regions. As currently practiced, this. rationale does encourage programs to try to meet the standards implicit in the Review Criteria, but the objections to the scoring process, the 73 reported weaknesses of the criteria and the potential for variability of application opens the use of this rationale for selective-funding decisions ' to criticism. The fact that the users of the Review Criteria in the Review Committee and Staff Anniversary Review Panel reported their contributions to the development of consensus as to which programs were of "htgh" quality and of "Jow" quality suggest that the classification as "A", "B", and "C" may be more valid than the objections would indicate, but further use of the Review Criteria in selective funding decisions should be dependent upon eliminating the bases for the objections. It can be anticipated that further use of the classtfication of programs based on Review Criteria scores, particularly tf the bases for the objections can be eliminated, will promote the development and operatton of high qualtty local Regional Medical Programs. Recommendation #5: . The Review Criteria should be used to provide a consistent framework for local program planning, development and operations. The fundamental unit of organization within Regional Medical Programs is the local Regional Medical Program. . The organization which receives the planning and/or operational grant award is charged with the responsibility of organizing a regional advisory group that develops the goals and objectives for the program and approves all grant applica~ tions. They are also charged with the responsibility of assuring that any proposed activities have been reviewed by the appropriate Compre- hensive Health Planning Area-wide Agencies for compatibility with any local plans for health care systems development. The local autonomy implicit in the local organization of Regional Medical Programs and the requirements for local approval and/or review belies the fact that there are some uniform purposes specified in the legislation for Regional Medical Programs. To date, the processes related to local autonomy have received more emphasis than achievements related to National objectives, but there is an increasing tendency to shift the emphasis. Local programs , thus’far, have been allowed to develop and operate within the loose framework supplied by Guidelines for Regional Medical Programs Paps various publications, statements and jetters. The promulgation by RMPS of grant programs for Emergency Medical : Services and Area Health Education Centers signified the transition towards a greater degree of central direction. The fact, as reported in the conferences, that some local Regional Medical Programs, with and without the prodding of RMPS staff have begun to use the RMP Review Criteria to guide their program planning, | development and operations ,suggests that the Review Criteria may serve a role in guiding local programs towards the achievement of National objectives . Although the summary of the RMPS staff conference indicated that the . Review Criteria are an inappropriate mechanism for communicating changes in RMPS policy, modifications in the Review Criteria that emphsize the. 74 ores ed ay eary faenerens Soy 3 Seanad tyes a : : a oy importance of RMPS policy and National objectives for RMP in local program activities might lead to greater compiance with them. If widely publicized, the Review .Criterta could serve more uniformly as a guide to Jocal programs and would probably have the effect not only of developing more programs that score well against the criteria, hut also of developing more programs that make substative contributions to the achievement of National objectives for Regional Medical Programs. Recommendation #6: The Review Criteria should be used to facilitate the development of information that can be used in RMPS program evaluation and in the justification of Regional Me ical Programs to the Congress, the Administration and the public. : Regional Medical Programs have been criticized by the Administra- tion for their failure to make significant changes in the health care system of the Nation: "there is little evidence that on a nationwide basis the RMP's have materially affected the health care delivery system... That criticism implies that the Administration had objectives related to changes in the "health care delivery system" that were not addressed by the avatlable evidence. —— In contrast to the judgements of the Administration are the evaluations of local RMPs developed through the review process. Many local programs, according to RMPS staff and the reports from the review process, were making significant improvements in the planning, development and operations of their programs and some, as indicated by their high Review Criteria scores and ratings, were judged to be of high quality. The discrepancies between the evaluation of the Administration and that of RMPS could be accounted for in a number of ways, but accounting for those discrepancies is not nearly as important as eliminating them. Modifications of the Review Criteria to emphasize the importance of National objectives and utilizing the Review Criteria as a tool for developing information to be presented to the Administration, the Congress and the public that would document achievements related to those objectives seem indicated. Such modifications might have been considered earlier if the Review Criteria had been considered as part of the program evaluation and program justification effort of RMPS and not just as a device for evaluating local programs in the review process. Recommendation #7: The content of the Review Criteria should be related, whenever possible and appropriate, to specific guidelines and objectives : to be established by the Administration for the Regional edical Programs at the Nationa : legislation and the legislative proceedings. - The absence of specific National guidelines and objectives for Regional Medical Programs from the Review Criteria reflects a reluctance 1 Budget Proposal for Fiscal Year 1974, Appendix, p. 383 75 within the Department of Health, Education and Welfare to establish performance objectives for its component programs. As long as guidelines and objectives for Regional Medical Programs are deduced hy RMPS staff from the legislation and Presidential Health Messages ~- the process currently followed according to RMPS staff -- there remains the opportunity for DHEW to second-guess the staff.and, after-the-fact, outline a set of objectives for evaluating the program that, in fact, were never available to guide the program in its activities. Establishing reasonable objectives for social programs is extraordinarily difficult. Delegating the responsibility to lower levels in the administrative hierarchy is a common ploy for the development of objectives and one that implfes acceptance of those objectives by the higher levels. Whether the objectives are established at a higher level or a lower level is not the issue. Regardless of where they are established, the objectives should be explicitly, stated and accepted at all levels and should adequately reflect the intent of the Congress as recorded in the legislation and the legislative proceedings. The objectives need not be all-inclusive. They should specify the standards against which cost-effectiveness studies can be done, but they need not limit the range of benefits identified in cost-benefit studies. According to RMPS staff, DHEW or HSMHA has not, in the past, provided Regional Medical Programs with performance objectives. Neither has RMPS established and publicized such objectives. The Review Criteria are the nearest proxy to performance objectives, but “they fall short. The RMPS Mission Statement is a similarly imprecise proxy for performance objectives. If and when National objectives are established for Regional Medical Programs, relating the Review Criteria to them should significantly increase the probability that they will be achieved. Recommendation #8: The content of the Review Criteria should be modified under the leadership of RMPS staff with the participation of National Advisory Council! members, Review Committee members , RMP coordinators and staff, and Regional Advisory Group members . The development of Review Criteria can be a tedious, time- consuming and expensive process. Their initial development was done by a small group of senior RMPS staff. . Sedected RMPS staff in conference indicated that the original methods of development was undesirable and that future developments of the Review Criteria should allow for broad participation of National Advisory Counci] members, Review Committee members and RMP coordinators, staff and Regional Advisory Group members . They did, however, indicate that future development of the criteria should continue to be under the direction of RMPS staff with advice of expert consultants and that it was of sufficient importance so that cost factors should be disregarded. 46 wad oe a ad "dae fea oe sy p iE crores fest: ers Te a! Bas tae OF 3 > A iw cet bes i fol Ga Si The question-clustering conference with RMPS staff and a consultation with James R. Miller, III, an authority in this field, made it quite clear that such broad participation would complicate the process of developing the Review Criteria and would not necessarily increase their appropriateness or acceptability. Nevertheless, broad participation was so strongly recommended that, regardless of any anticipated problems, it should be carried out. A suggested approach to this recommendation that include facets of subsequent recommendations arepresented later in this chapter. Recommendation #9: The design of the Review Criteria should be significantly modified as follows: a. The major criteria should be retitled Achievements, Organizational Processes and Proposed Plans; b. The individual criteria should be reduced in number from twenty to twelve or less; c. The numerical values of 1, 2, 3, 4 and 5 should be accompanied by statements that indicate the precise meaning of each numerical value for each individual criterion; and d. The questions should be reduced in number and should be translated into a form that can be answered by Present/Absent or that calls for specific qualitative descriptions or quantitative measures. The RMP Review Criteria consist of twenty individual criteria organized into three major criteria groupings: Performance, Process, and Proposed Program. Neither the individual criteria nor the major criteria groupings have definitions. The intended, but unexpressed definitions of the individual criteria are supposedly expressed in the specific questions, but the fact that those questions could not be re-sorted into groupings under those titles by RMPS staff suggests that the questions do not function adequately as definitions. This recommendation responds to a series of criticisms identified by the evaluation studies, but, in addition,is based on the concept of the Review Criteria as a hierarchy of related values. Level One of the hierarchy is a value entitled: Overall Quality. Level Two, according to this recommendation, is a set:of values entitled Achievements , Organizational Processes, and Proposed Plans. Level Three calls for three sets of values, not to exceed a total of twelve individual criteria, that indicate the components of the three criteria at Level Two. Level Four specifies the related objective information that should be used in making evaluative judgements against Level Three criteria. Table 4-1 is a sample of how such a hierarchy might appear. This recommendation, however, is concerned with detailed changes in design that go beyond the simple development of a hierarchy. Table 4-2 illustrates most of these changes as they relate to the Level Two criterion of Achievements. 77 VEL ONE yverall Quality TABLE 4-1 Sample RMP Review Criteria To Illustrate Four Level Hierarchy LEVEL TWO LEVEL THREE Regionalization of Health Care System Resolution of Identi- Achievements fied Deficiencies in Delivery System Achievement of National Objectives for Delivery System Coordinator Staff Regional Advisory Group Organizational Processes Management Planning Evaluation Related to Identified Deficiencies in Delivery System Related to National Proposed Plan Objectives for Delivery Sys tem Util izes Concepts of Regionalization LEVEL FOUR List regional networks by type List subregional networks by type and population served List deficiencies List activities by deficiencies List impact data by deficiencies List activities by objective List impact data by objective Rit TABLE 4-2 Sample Criteria Definitions, Scoring Definitions and Descriptors Related to Second Level Criterion of Achievements LEVEL TWO Achievements: The degree to which funds expended by the Jocal RMP have resulted in identifiable improvements in the health care system of the region. Scoring Definition: l- No identifiable improvements 2- Few identifiable improvements 3- Few identifiable improvements in high priority areas 4- Many identifiable improve- ments in high priority areas 5- More identifiable improve- ments in high priority areas than might be expected from the level of funds expended LEVEL THREE Regionalization: The degree to which regionalization of the health care system of the region has been attained. Scoring Definitions: 1- No identifiable regionaliza- tion 2- Sub-regional or regional collaborative efforts estab- lished 3- Sub-regional or regional networks established for supportive services to health care system, @.g. continuing education 4- Sub-regional or regional networks established for health care delivery in one or wore cateyorical disease areas 5- Sub-reaional or reaional networks established for comprehensive health care delivery including the categorical diseases, preventive, primary, emergency and rehabilita- tive care Deficiencies: The degree to which significant deficiencies in the health care delivery system of the region have been resolved. Scoring Deficiencies: l- wveficiencies not identified Z2- No apparent resolution of identified deficiencies 3- Partial or complete resolu- tion of identified deficiencies of low priority 4- Partial or complete resolu- tion of identified deficiencies of high priority 5- All significant deficiencies resolved that might indicate resolution of the deficiencies National Objectives: The degree to which National objectives for improvements in the health care delivery system have been achieved within the region. Scoring Deficiencies: . T- No activities related to National objectives - 2- No identifiable achievements related to National objectives 3- Partial or complete achievements of some National objectives in one or more sub-regions 4- Partial or complete achievement of some National objectives in entire region 5- Partial or complete achievement of all National objectives in- entire region LEVEL FOUR Regionalization Descriptors T. List of functional regional networks by type of support service, categorical disease, or elements of comprehensive services 2, List of functional sub-regional networks by type and size of population served Deficiency Descriptors: T. List deficiencies in health care system of region and their priority, preferably as supplied by Comprehensive Health Planning Areawide and Statewide Agencies or as supplied by tne Regional Advisory Group and reported in the last submitted triennial plan 2: For each deficiency, list the activities undertaken since the Jast triennial plan was approved that purport to resolve. the deficiency . For each deficiency, summarize any reported data.or descriptions National Objectives Descriptors: T. For each objective, list activities undertaken in last grant period that. purport to achieve that objective . . For each objective, summarize any reported data or descriptions that might indicate partial or complete achievements of that objective -Implementation of this recommendation should make it possible to eliminate some of the reported criticisms including 1) excess focus on "accreditation", 2) excess number of criteria and associated problems, 3) confusion with numerical scoring process, and 4) weaknesses of the specific questions. Recommendation #10: The Review Criteria should be ineluded in the Guidelines for Regional Medical Programs and/or in an egulations and Procedures Manual so that local programs understand their importance for local program planning, development and operation. A complete description of how the Review Criteria are used by RMPS in the review process and their technical assistance activities should also be provided in the same way. Evaluation systems can have a profound effect on program. erformance when the evaluation standards, the evaluation methods and the uses of the evaluation findings are clearly understood by the managers and personnel of the programs. Publishing that information in official documents ts one of the first steps leading towards such understanding. then the information about the evaluation system is promulgated through official documents with visible evidence that the evaluation systems are being used, the evaluation standards then begin to function as program guides. Only when they function as program guides will programs stand any reasonable chance of receiving high ratings when evaluated against the standards. Recommendation #11: RMPS staff in the Office of Systems Management, should be responsible for the design of the Level Four Descriptors in. lieu of the specific questions, for the redesign of the grant application so that the information appropriate to these descriptors is available, for the analysisof the information, | related to these descriptors So" at absolute and/or comparative standards can be developed, and for the preparation throu h the M ent Information System of standardized in ormation related anagem to che descriptors for use in the review process. Lack of information was one of the oft-repeated criticisms of the Review Criterfta. It was also one of the factors cited in the Administrations recommendation that Regional Medical Programs be phased out. Obtaining, analyzing and presenting information, however, is no trivial task and, therefore, should be guided by the precepts of relevancy. At the present time, the Office for Systems Management processes certain information from the grant applications in a management information system. Unfortunately, this system is not closely linked efther to the Review Criteria or to the implied ee unpublicized, objectives of the” edical Programs. Administration for Regional This recommendation, which assumes that the previously recom- mended alterations in the Review Criteria will be made, 7s based on the 80 ay? eae) 63 t : ow bint ed Sa] fomraeh, ea } = 4 SRS m3 sy 4 set a existing responsibilities of the Office of Systems Management for the management information system. It calls for that Office to take the primary respons aaa for developing the appropriate Level Four escriptors as a replacement for the specific questions of the Review Criteria and to obtain, analyze and present that information 1) for the review process, 2) for program justification to Congress, Admin- istration and the public, and 3) for other management purposes as needed. If closely tied to the modified Review Criteria, the information related to the Level Four Descriptors should eliminate the expressed criticism about lack of information. Recommendation #12: RMPS staff in the Division of Operations and Development should be responsible for the design of the Level Three triteria with the participation of the National Advisory Council], the Review Committee and local RMPs and for preparing the scores with accompanying statements of justification for each indivi- dual criterion for the staff briefing document. The staff of the Division of Operations and Development provides the primary interface between RMPS and local RMPs. They have a significant responsibility for helping local RMPs to plan, develop and operate their programs at a high level of quality and effectiveness. This recommendation is based on the following assumptions: 1. The staff of DOD has the best first-hand knowledge of those factors that indicate a high level of quality and effectiveness ; 2. The factors. that indicate a high level of quality and effectiveness should be translated into Level Three Criteria; 3. Review Criteria, if used as program guides, should be used. as guides for technical assistance to local programs by DOD staff; . 4. Scoring programs against Level Three Criteria by DOD staff will emphasize to the staff and to the local RMPs that: the criteria are to be used as program guides; and 5. Scoring program against Level Three Criteria by DOD staff can serve as an internal evaluation of the effectiveness of their technical assistance activities. Recommendation #13: Site visit teams should study the Level Three Criteria - scores and statements of justification as prepared by the Division of Operations and Development and the Level Four © Descriptor information as prepared by the Office of Systems — Management and should describe, in their site visit report, their evaluation of the submitted scores, justifications, © and data. Their report should include modifications of the submitted Level Three Criteria scores if indicated by the site visit findings. 81 Site visit teams have a great amount of work to accomplish in a very short time. ‘Accomplishing that work can be facilitated by providing them with readily understandable information related to Level Four Descriptors and with scores and statements of justification for Level Three Criteria in the staff briefing document. Their task, by being transformed into a veriffcation procedure» should become considerably easier. Their evaluation of the submitted Review Criteria information and scores should make weaknesses in the Review Criteria, the Level Four Descriptor information and the evaluative judgements of DOD staff readily apparent, and, therefore, amenable to corrective action. Recommendation #14; The Review Committee and Staff Annivyersar Review Panel should study the submitted scores, justifications and data including modifications from the site visit team and should include evaluative comments in’their report, but they should repare scores with statements of justification only for = three Level Two criteria. Because there is no evidence of independent judgement occurring in the present patterns of Review Criteria scoring, it is recommended that the scores and their justifications reflect the consensus of the group and that scoring by individual members of the group be eliminated. The scores and justification on the Level Two Criteria should be included in their report and recommendations. The RMP Review Committee and the Staff Anniversary Review Panel have had the sole responsibility for scoring programs against the Review Criteria. These recommendations call for different groups to score programs at different levels: Number of Criteria Level Criteria Review Group Level One Criterion (1) . National Advisory " — Council Level Two Criteria , (3) Review Committee and Staff Anniversary Review Panel Level Three Criteria (12) Staff of Division of Operations and Develop- ment; verified by site visit teams Level Four Descriptors (2) Staff of Office of Systems Management This means that, in the future, the Review Committee and Staff Anniversary. Review Panel would have responsibility for scoring programs against the three pro osed Second Level Criteria: Achievements, Organiza~- tional Processes ard Proposed Plan -- a significant reduction in work load from the present required scoring against twenty criteria. This reduction in work load would make it possible for their scoring process to be modified in a way that would resolve the following criticism and adverse evaluation findings: ve wy ee 8 1. The scoring process fails to function as independent peer judgement; a 2. The scoring against twenty individual criteria functions as a rating against a single criteria of Overall Quality and not as a true measure; 3. The scoring process depends on information that varies considerably in its availability; and 4. The scoring process ts dependent upon comparative experience with many programs. The proposed modifications in the scoring process will: 1. Eliminate the pretense of independent peer judgement by calling for scoring by concensus; 2. Increase the likelihood of rating against more than one criteria by reducing the number of required scores to three and, thereby,allowing the members to focus on the differences between the criteria; 3. Assure that the appropriate information is available by calling for an evaluation of the submitted information and for the preparation of scores with statements of justifica- tions by concensus, a procedure that is intended to eliminate the aifferential understanding and weighting of information that occurs with individual scoring without required statements of justifications; and 4. Allow the comparative experience of the group to influence the scoring and, thereby, mitigate the effect of variations in the experience of individual members in reviewing multiple programs . Although the scoring by the Review Committee and the Staff Anniversary Review Panel may be influenced by the descriptive data submitted from the Office for Systems Management and the Level Three scores submitted by the staff of DOD with or without modifications by a site visit team, the scoring should be based primarily on the judgement of the members. If there are discrepancies between Level Two scores and Level Three scores, it can be assumed that corrective actions need to be taken in terms of the Level Three Criteria or the scoring process within DOD. Recommendation #15; The National Advisory Council] should study the scores and statements of justification on Level Two and Three Criteria in staff briefing document, site visit report, Review Committee and Staff Anniversary Review Panel reports and should include evaluative comments in their report. In addition, it is recom- mended that the National Advisory Council, identify, by concensus, ing statement of justification that reflects a score and an accompanyi “their evaluation of the programs on UveralT Quality ana imc u i with their report and recommendations. 83 The National Advisory Council's role in Regional Medical Programs is specified in the legislation. In carrying out that role, they must approve all grant applications and recommend a funding level before funds can be awarded. “At the present time, they make no formal evaluation judgements other than those recorded in their recommendations for approval and funding level. Because of the use of Review Criteria scores in the classification of programs for selective funding decisions by the Director of RMPS, a formal rating from the National Advisory Council on Overall Quality seems appropriate. This rating would serve as additional guidance from the group designated hy the legislation as responsible for approved funding levels to the Director of RMPS in his selective funding decisions. This rating would also serve as a check on evaluative judgements against Level Two and Three Criteria. Recommendation #16: RMPS staff in the Office for Planning and Evaluation should monitor the scores, justifications, data and evaluative comments reported by the Office of Systems Management, Division of Operations and Development, site visit teams, Review Committee, Staff Anniversary Review Panel and National Advisory Council] and should analyze them for consistency of evaluative judgement among the review bodies, for perceived adequacy of the submitted information, and for redundant and/or missing criteria. The evaluation studies suggest considerable diversity of judgemental values among individuals and groups of individuals involved in the review process and between the same individuals under varying conditions. At present, there are no methods for assuring consistency within the review process and it is possible for RMPS staff; through their evaluative judgements , to.direct programs towards goals that are not considered important by the Review Committee and the National Advisory Council. Centralization of responsibility for evaluative judgement is one approach to the elimination of diversity, but one that is not feasible in a program that is the size of Regional Medical Programs. This recommendation proposes a management systems approach that would assure consistency by identifying for the management of RMPS when consistency is not present. The necessary — management actions, when consistency is found to be absent, could include modifications of the criteria, education of personnel, and/or other reasonable tactics. Analysis of the Review Criteria scores according to methods described in Professional Decision Making by James R. Miller, III, will not only make it possible to 1 entity redundant and missing criteria, but also will make it possible to develop weights for Level Two and Three criteria scores by a rational method. Recommendation #17: The Review Criteria data, scores and statements of justification from each of the review bodies should be submitted to the local Regional Medical Progra ; the local Re ional Medical Programs as_a operations. we oem Ss ; w = Se yas) oq sr, aD Tet Ce =e, ala This recommendation is designed to promote "maximum feasible understanding" of the common:purposes for RMP and Tocal RMPs. Secrecy hreeds suspicion; openness engenders trust. This recommendation symbolizes the openness necessary to foster the collaborative working relationship between RMPS and local RMPs: that is advocated as the first step in success- ful regionalization at the local level. Throughout these recommendations is the assumption that the Review Criteria will be modified and will continue to be used. The experience of this evaluation leads to a suggested sequency of activities that would, at a reasonable cost and effort, result in the recommended modifications in content and design. Modifications in RMPS operations related to the Review Criteria and the review process, as proposed in these recommendations, are the prerogative of RMPS management to accept or reject, but any such decisions are contingent upon many unique factors beyond the scope of this study. The procedures for modifying the content and design, however, are not only independent of many of those factors, but also generally applicable to Regional Medical Programs and other federal programs that desire to develop review criteria: Suggested Activities for Modifying the Content and Desig Review Criteria 1. Review, modifications and acceptance of criterion definition and scoring definitions for Level One Criteria: Overall Quality -- by National Advisory Council based on draft document prepared by Staff Anniversary Review Panel and consultants. 8 2. Review, modification and acceptance of criteria definitions and scoring definitions for Level Two Criteria: Achievements , Organiza- tional Processes, and Proposed Plan by RMP Review Committee based on draft document prepared by Staff Anniversary Review Panel and consultants. 3. Publication and distribution of Level One and Two criteria, criteria definitions and scoring definitions to RMPS staff, coordinators and Regional Advisory Group chairmen for review and comment. 4. Modification, finalization and approval of Level One and Two criteria, criteria definitions and scoring definitions by National Advisory Council, RNP Review Committee and RMPS Director. - 5. Review, modification and acceptance of Level Three criteria titles, criteria definitions and scoring definitions by Steering Committee of Coordinators based on draft prepared by staff of Division of Operations and Development and consultants. The Level Three criteria titles should relate directly to the Level Two criteria definitions and scoring definitions. Qe 6. Publication and‘distribution of Level Three criteria, criteria definitions and scoring definitions to RMPS staff, coordinators and Regional Advisory Group chairmen for review and comment. 7. Modification, finalization and approval of Level Three criteria, criteria definitions and scoring definitions by National Advisory Council, RMP Review Committee, Steering Committee of Coordinators and RMPS Dtrector. 8. Design of Level Four Descriptors by staff of Office of Systems Management with consultants. 9. Review, modification and acceptance of Level Four Descriptors by Staff Anniversary Review Panel. 10. Publication and distribution of Level Four Descriptors to RMPS staff, coordinators and Regional Advisory Group chairmen for review and conment. 11. Modification, finalization and approval of Level Four Descriptors by Nattonal Advisory Council, RMP Review Conmittee, Steering Committee of Coordinators, and Director RMPS. 12. Redesign and pretest of grant application and management information system by staff of Office of Systems Managements. 13. Publication and distribution of redesigned grant application and data to be generated by management information system to RMPS staff, coordinators and Regional Advisory Group chairmen for review and comment. 14. Modification, finalization and approval of redesigned grant application and data to be generated by management information system by National Advisory Council], RMP Review Committee, Steering Committee of Coordinators and Department of Health, Education and Welfare. 15. Implementation of redesigned Review Criteria, grant application and management information system according to procedures developed by RMPS management. This sequence of activities should be able to be completed in less than six months and should result in an acceptable set of Review Criteria. The need for further revisions should be detected by the recommended activities of the Office for Planning and Evaluation and such revisions should be made on an annual basis. Conclusions The seventeen recommendations that flow from this evaluation | of the RMP Revtew Criteria would result in the resolution of most of the critictsms and adverse evaluation findings. They cal] for significant changes in the content, design and application of the Review Criteria -- changes that would influence the review process as well as many of the technical assistance, information and evaluation activities of RMPS. The ultimate objective of these recommendations is to facilitate the achievement of National objectives and the resolution of regional needs for improvements in the health care delivery system. w & my. i tli f w &e alae ke APPENDIX A RMP REVIEW CRITERIA 87. ray Cas ey wee per Pee yt vent Gale ropes my POTTY Cy PY eek VhOny Ge? ce December 28, 1971 RMP REVIEW CRITERIA A. PERFORMANCE (40) GOALS, OBJECTIVES, AND PRIORITIES (8) 1. a. b. Cc. Have these been developed and explicitly stated? Are they understood and accepted by the health providers and institutions of the Region? Where appropriate, were community and consumer groups also consulted in their formulation? Have they generally been followed in the funding of opera- tional activities? Do they reflect short-term, specific objectives and priori- ties as well aS long-range goals? Do they reflect regional needs and problems and realistically take into account available resources? ACCOMPLISHMENTS AND IMPLEMENTATIONS (15) Have core activities resulted in substantive program ac- complishments and stimulated worthwhile activities? Have successful activities been replicated and extended throughout the Region? : Have any original and unique ideas, programs or techniques. been generated? Have activities led to a wider application of new knowledge and techniques? Have they had any demonstrable effect on moderating costs? Have they resulted in any material increase in the availa- bility and accessibility of care through better utilization of manpower and the like? Have they significantly improved the quality of care? Are other health groups aware of and using the data, expertise, etc., available through RMP? Do physicians and other provider groups and institutions look to RMP for technical and professional assistance, consultation and information? If so, does or will such assistance be concerned with quality of care standards, peer review mechanisms, and the like? CONTINUED SUPPORT: (10) a. b. Is there a policy, actively pursued, aimed at developing other sources of funding for successful RMP activities? Have successful activities in fact been continued within the regular health care financing system after the withdrawal of RMP support? 89. MINORITY INTERESTS (7) Do the goals, objectives, and priorities specifically deal with improving health care delivery for underserved minorities? How have the RMP activities contributed to significantly increasing the accessibility of primary health care services to underserved minorities in urban and rural areas? How have the RMP activities significantly improved the quality of primary and specialized health services delivered to minority populations; and, have these services been developed with appropriate linkages and referrals among in- patient, out-patient, extended care, and home health services? Have any RMP-supported activities resulted in attracting and training members of minority groups in health occupations? Is this area included in next year's activities? " What steps have been taken by the RMP to assure that minority patients and professionals have equal access to RMP-supported activities? Are minority providers and consumers adequately represented on the Regional Advisory Group and corollary committee structure; and do they actively participate in the deliber- ations? . Does the core staff include minority professional and supportive employees and does it reflect an adequate consideration “of Equal Employment Opportunity? Do organizations, community groups, and institutions which deal primarily with improving health services for minority populations work closely with the RMP core staff? Do they actively participate in RMP activities? What surveys and studies have been done to assess the health needs, problems, and utilization of services of minority groups? B. PROCESS (35) 1. a. b. c. d. ‘a. COORDINATOR (10): Has the coordinator provided strong leadership? Has he developed program direction and cohesion and established an effectively functioning core staff? Does he relaté and work well with the RAG? Does he have an effective deputy in name or fact? CORE STAFF (3) Does core staff reflect a broad range of professional and discipline competence and possess adequate administrative and management capability? 90. 9 Contd aad G ghee date 3 cr RPE Ki veoedey OD cE CORE STAFF (3) (continued) b. Are most core staff essentially full-time? c. Is there an adequate central! core staff (as opposed to institutional components)? REGIONAL ADVISORY GROUP (5) a. Are all key health interests, institutions, and groups within the region adequately represented on the RAG (and corollary planning committee structure)? b. -Does the RAG meet as a whole at least 3 or 4 times annually? c. Are meetings well attended? d. Are consumers adequately represented on-the RAG and corollary committee structure? Do they actively participate in the deliberations? e. Is the RAG playing an active role in setting program policies, establishing objectives and priorities, and providing overall guidance and direction of core staff activities? f. Does the RAG have an executive committee to provide more frequent administrative program guidance to.the coordinator and core staff? 'g. Is that committee also fairly representative? GRANTEE ORGANIZATION (2) a. Does the grantee organization provide adequate administrative and other support to the RMP? b. Does it permit sufficient freedom and flexibility, especially insofar as the RAG's policy-making role is concerned? — PARTICIPATION (3) a. Are the key health interests, institutions, and groups actively participating in the program? -b. Does it appear to have been captured or co-opted by a major interest? c. Is the Region's political and economic power complex involved? LOCAL PLANNING (3) a. Has RMP in conjunction with CHP helped develop effective local planning groups? u b. Is there early involvement of these local planning groups in the development of program proposals? . ee, c. Are there adequate mechanisms for obtaining substantive CHP review and comment? ae i 91. C. ASSESSMENT OF NEEDS AND RESOURCES (3) Is there a systematic continuing identification of needs, problems, and resources? — Does this involve an assessment and analysis based on data? Are identified needs and problems being translated into the Region's evolving plans and priorities? Are they also reflected in the scope and nature of its emerging core and operational activities? MANAGEMENT (3) a. b. c. Are core activities well coordinated? Is there regular, systematic and adequate monitoring of projects, contracts, and other activities by specifically assigned core staff? Are periodic progress and financial reports required? EVALUATION (3) Is there a full-time evaluation director and staff? Does evaluation consist of more than mere progress reporting? Is there feedback on progress and evaluation results to program decisions and modifications; specifically have unsuccessful or ineffective activities been promptly phased out? PROGRAM PROPOSAL (25) 1. a. b. Cc. d. e. f. g. DISSEMINATION OF KNOWLEDGE (2) a. ACTION PLAN (5) Have priorities been established? Are they congruent with national goals and objectives, including strengthening of services to underserved areas? Do the activities proposed by the Region relate to its stated priorities, objectives and needs? Are the plan and the proposed activities realistic in view of resources available and Region's past performance? Can the intended results be quantified to any significant degree? Have methods: for reporting accomplishments and assessing results been proposed? - Are priorities periodically reviewed and updated? ede Have provider groups or institutions that will benefit been targeted? 92. ad on 7 pietie win vo é at 7 pod 1 lad Fey Why Seed Es Gs Pak DISSEMINATION OF KNOWLEDGE (2) (continued) b. oO ho Have the knowledge, skills, and techniques to be disseminated been identified; are they ready for widespread implementation? Are the health education and research institutions of the Region actively involved? Is better care to more people likely to result? Are they likely to moderate the costs of care? Are they directed to widely applicable and currently practical techniques. rather than care or rare conditions of highly specialized, Tow volume services? UTILIZATION MANPOWER AND FACILITIES (4) a. Will existing community health facilities be more fully or effectively utilized? oe Is it likely productivity of physicians and other health manpower will be increased? Is utilization of allied health personnel, either new kinds or combinations of. existing kinds, anticipated? Is this an identified priority area; if so, is it proportion- ately reflected in this aspect of their overall program? Will presently underserved areas or populations benefit significantly as a result? , IMPROVEMENT OF CARE (4) Have RMP or other studies (1) indicated the extent to which ambulatory care might be expanded or (2) identified problem areas (e.g., geographic, institutional) in this regard? Will current or proposed activities expand it? Are communications, transportation services and the like being exploited so that diagnosis and treatment on an out- patient basis is possible? . Have problems of access to care and continuity of care been identified by RMP or others? Will current or proposed activities strengthen primary care and relationships between specialized and primary care? Will they lead to improved access to primary care and health services for persons residing in areas presently underserved? Are health maintenance and disease prevention components included in current or proposed activities? ; If so; are they realistic in view of present knowledge, state-or-the-art,-and other factors? . . SHORT-TERM PAYOFF (3) a. Is it reasonable to expect that the operational activities proposed will increase the availability of and access to services, enhance the quality of care and/or moderate its costs, within the next 2-3 years? 93. b. c. a. SHORT-TERM PAYOFF (3) (continued) Is the feedback needed to document actual or prospective pay-offs provided? Is it reasonable to expect that RMP support can be withdrawn successfully within 3 years? REGIONALIZATION (4) Are the plan and activities proposed aimed at assisting multiple provider groups and institutions (as opposed to groups or institutionssingly)? Is greater sharing of facilities, manpower and other resources envisaged? Will existing resources and services that are especially scarce and/or expensive, be extended and made available to a larger area and population than presently? Will new linkages be established (or existing ones strengthened) among health providers and institutions? Is the concept of progressive patient care (e.g., OP clinics, hospitals, ECF’s, home health services) reflected? OTHER FUNDING (3) aw Is there evidence the Region has or will attract funds other -than RMP? If not, has it attempted to do so? Will other funds, (private, local, state, or Federal) be available for the activities proposed? Conversely, will the activities contribute financially or otherwise to other significant Federally-funded or locally- supported health programs? 94. APPENDIX B DESCRIPTION AND SCOPE OF WORK FOR CONTRACT HSM 110-72-329 TO EVALUATE THE RMP REVIEW CRITERIA Sot ay aed. pea : be wrest | CRS r. Shen Chasis areeeh oy ar aa? 95. Det it ‘ \ and “A ARTICLE I DESCRIPTION AND SCOPE OF WORK A. The purposes of this study are (1) to assess the internal validity (e.g., repeatability, descrimination) of the Review Criteria used in rating Regional Medical Programs; and (2) the perceptions of those utilizing the criteria in the national review process as to their effectiveness. In performance of this contract, the Contractor shall: 1. Assess the manner in which the established Review Criteria are, - n their application, sensitive tools in the adequately and objectively assessing the effectiveness of Regional Medical Programs for the purposes of Regional Medical Program Service review at the national level by both the Regional Medical Pro- gram Review Committee and the Staff Anniversary Review Panel. This assessment should include, but not necessarily be limited to answering the following questions: (a) Are the average raw scores of "A," "By" and "C" Regions significantly different from each other? ("A" Regions are the most highly regarded while "C" Regions are the least highly regarded.) (b) Are the average weighted scores of "A," "B," and "C" Regions significantly different from each other? ("A" Re- gions are the most highly regarded while "C" Regions are the least highly regarded. ) _(c) Is there a consensus among reviewers regarding "A," "B," and "C" regions? (Use of standard statistical methods such as analysis of variance to test pertinent hypotheses. ) (d) Are there individual reviewers whose scores reflect an unusual range? (e) Are there particular primary reviewers who tend to "pull" ‘the scores of the other reviewers towards a model value? (f) What are the answers to questions (a) - (e) considering . each broad grouping of criteria (i.e., performance, process, and program proposal) as well as the overall or total criteria? (g) What is the relationship between the scores for each of these broad groupings of criteria and the overall. scores? (h) How do the weighted mean scores based upon the individual criteria compare with the. single overall assessment scores? *(i) What is the relationship between site visit findings as reflected in site visit reports (and summarized at the meeting) and the scores for selected criteria (e.g., coordi- nator, regionalization)? *Question not addressed by contract activities by agreement ‘with Project officer : a « (j) What is the relationship between scores and the percent- nm ages of requests recommended for award? mi 2. Determine the adequacy and effectiveness of the Review Criteria a in assessing the comparative quality of Regional Medical Programs a as perceived by those utilizing them (e.g., Review Conmittee ce members, Regional Medical Programs staff). The information ob- tained should be aimed at, but not necessarily limited to answer- Cy ing the following questions: be (a) What is the relative importance of each of the major criteria = groupings? Be (b) Which individual criteria are the most important and on credible,--that is, should be given the greatest weight bf in assessing the comparative quality of regions? Bs (c) For those criteria singled out as most important is the e data and information made available generally adequate to i make judgments? If not, what kinds of data would be desir- able and/or required? g pu (d) What additional criteria, if any, should be included; which ‘ if any of present ones might be eliminated? rn (e) Of those suggested for deletion, what is the basis there- 2 fore - (1) Unimportant, not too relevant? 4 (2) Difficult to judge, intrinisically or because of the “ unavailability of data? Bs tv (f) Have the review criteria and their utilization resulted in a significant improvement, either substantially or in terms of credibility, in the decision-making process? *