EVALUATION STRATEGY Agency/Program - HSMHA/RMPS Evaluation Strategy Current Objectives: RMPS has three major objectives in FY73 insofar as program evaluation is concerned. These are: (1) To materially improve the processes by which RMPS (a) evaluation strategy and plans are developed, (b) evaluation activities (both contract and in-house) are implemented and monitored, and (c) evaluation findings are utilized. To this RMPS proposes to establish a small, high-level, in-house evaluation group. (See Project Summary No. 11.) (2) To launch and tentatively assess the viability of a program of "challenge" contracts (or grants) aimed specifically at the 56 RMPs and looking towards their increased contribution in evaluating the effectiveness of the program. This activity would account for close to 50% of the earmarked evaluation funds that is anticipated will be available to RMPS next year. (See Project Summary No. 1.) (3) To undertake an assessment of RMP decentralization, which has ' become an increasingly central part of RMPS' program strategy over the past 12-18 months. (See Project Summary No. 2.) . Strategy Statement: As implied by objective 1 above, RMPS does not at this time have a comprehensive, well-developed evaluation strategy. It seeks to develop such during the next 8-10 months. The strategy finally developed and agreed to will be presented to HSMHA and HEW prior to or as a part of the RMPS FY74 Evaluation Plan. Development of RMPS' long-term evaluation strategy will involve at least three groups. The principal one will be the smail, high-level, in-house group already referred to. The others are the Ad Hoc RMP Evaluation Group, which has been in existence for over a year now and consists of RMP coordinators and evaluation directors, and the National Advisory Council. While no comprehensive, long-term evaluation strategy can be presented at this time, certain tentative, guiding principles are. These principles, enumerated below, (1) have served as a touchstone in the development of the specific evaluation proposals for FY73 and (2) will provide an initial point of departure of frame of reference in the development of a far more cohesive as well as comprehensive strategy. (1) Closely relate RMPS evaluation efforts to major program priorities, activities, and management needs. (2) (3) (4) (5) (6) Seek to achieve a good "balance"! or "mix'' between (a) program content (e.g., kidney disease) and process (e.g., decentralization) priorities and (b) continuing (e.g., educational programs) and new (e.g., emergency medical services) program activities. Anticipate new major RMP program thrusts (e.g., community-based educational programs) in order to initially incorporate minimal but common evaluation indices and data into such activities required for subsequent retrospective evaluation efforts. Develop output measures relevant to the evaluation of the impact of the overall program in terms of RMP legislative objectives (e.g., regionalization) activities and concerns (e.g., resource development, improved utilization of manpower). Promote cooperative evaluation endeavors (e.g., interaction among selected HSMHA programs at the local level) between RMPS and other HSMHA programs, especially NCHSR&D. Improve the effectiveness of the evaluation efforts of the 56 RMPs; this includes (a) encouragement of appropriate multi- regional evaluations and (b) broader dissemination of evaluation findings, proven techniques and methodologies, etc. that have general applicability. . . , . Zs Consideration and development of a long-term RMPS evaluation strategy should take into account both the potential strengths of RMP and the particular problems it faces. From an evaluation standpoint, RMP possesses at least two major strengths. (1) (2) The 56 RMPs constitute a significant evaluation resource. Nearly all possess a full-time evaluation director; these individuals and their staffs collectively represent a broad, high-level evaluation capability; and the yearly regional evaluation invest- ment currently approximates $4 million. RMPs at their best present a unique blending and linking of academic talent and provider orientation. This represents both a real capacity and capability for evaluation and a good, practical setting. There are a number of problems als¢. These include: (1) (2) The very nature of the program. RMP is broad in scope, aims at systems change and institutional reform, and frequently operates in a helping or facilitative fashion. The program's major thrusts as reflected by the specific activities and objectives of the 56 RMPs change over time. Indices and data for measuring program accomplishments and impact generally are ill defined or not readily available. (3) RPS evaluation staff capability is quite limited in terms of numbers, technical competence, and discipline mix. Thus, the evaluation strategy which RMPS will be developing should, among other things, aim at capitalizing on the strengths noted in order to resolve or reduce the problems cited. PROJECT SUMMARY - NO. 1 Agency/Program - HSMHA/RMPS. Project Title: Program of "Challenge" Contracts (Grants) to RMPs for Evaluation Method: Contract/Grant Dates: 7/1/72 - 6/30/73 Background and Discussion As noted in the Strategy Statement, the 56 RMPs constitute a significant evaluation resource; one that is characterized by a rather unique blending of academic and provider orientation and talent. This program would seek to exploit that resource and unique characteristic on a trial basis. What is proposed is essentially as follows: * RMPS would set aside about $250,000 of its FY73 earmarked evaluation funds for this purpose. “ * These funds would be available only to selected RMPs for modest evaluation studies and projects -- that is, not to exceed $50,000 or 12 months as a general rule. * Proposals would be solicited in certain targeted areas. The target areas will be limited to 4 or 5 priority areas. Initial discussions have identified the following potential target areas: (1) Impact of affiliation with University Health Science Centers on the quality of care provided in community hospitals. (2) Utilization of new categories of health personnel - effects on cost, quality of care, and availability of service. (3) Training programs for coronary care nurses -- evaluation of problems encountered in an occupational area with high job turnover. (4) Impact of new information technology on activities of physicians. (5) Effectiveness of RMPs' role in providing technical assistance (both directly and as a "broker" in identifying resources needed). £ ‘ «Program of "Challenge" Contracts (Grants) to RMPs for Evaluation (continued) @ (6) Impact of continuing education courses on actual practice behavior of physicians. (7) Methods used by RMPs in determining major areas of program emphasis. (8) Effectiveness of RMPs in serving as a facilitator or change agent in developing new programs or activities within a region (HMOs, Emergency Medical Systems, etc.). Final selection of target areas will be made following further discussion within HSMHA and the Ad hoc RMP Committee on Evaluation. * Proposed studies or projects would have to have general applicability. * Multiregional proposals would be actively encouraged. * Detailed guidelines, procedures, and requirements governing this program would be developed beforehand for HSMHA and HEW concurrence. * Announcement of the program, including submission dates for pro- posals, would be made by mid-summer. * Review, approval, and funding processes would involve active 9 participation of HSMHA/OPPE, HEW/OASPE, and RMP staff as welF as RMPS. * Once awarded, individual contracts under this program would be handled in the same manner and be subject to the same reporting and other requirements as all others funded from earmarked evaluation funds. PROJECT SUMMARY - No. 2 AGENCY /PROGRAM - HSMHA/RMPS PROJECT Project Title: Method: Date Start: Completion Date: Assessment of RMP Contract April 1, 1972 March 31, 1973 Decentralization ° PROBLEM Need for Evaluation: There has been a considerable degree of decentralization of decisionmaking authority in RMP within the past 12-18 months. Specifically, the responsibility for determining (1) the technical and professional merit or adequacy of particular proposals and (2) which of the "collective and competing" proposals judged to be technically satisfactory actually will be supported within the total funds available to a Region, essentially has been transferred to the 56 individual RMPs by RMPS, the headquarters program. The latter, however, continues to establish (1) broad program objectives and priorities and general policies and procedures, and (2) annual funding levels for the individual RMPs, but based upon its review and assessment of each program (as opposed to specific project proposals). Decentralization to the RMPs is one key element of RMPS' overall strategy. As such, it has elicited considerable interest and attention by HSMHA recently. One of the overarching issues identified in the course of developing the FY74-78 Program Plans was RMP decentralization. (Attached is a draft of the subject position paper prepared by RMPS. It is supplied in lieu of a supplemental Project Background Statement.) Because decentralization has only begun to fully emerge as a major and critical element of RMP operations within a relatively recent time span, no attempt to assess it systematically and thoroughly has been made to date. In view of its centrality to RMPS' strategy, and HSMHA's growing interest in the RMP "model" of decentralization and its possible implications for other programs, strongly Suggests that such an assessment would be highly desirable. Objectives of Evaluation: This assessment could be aimed at obtaining answers to the following questions: 1. What more specifically has been the extent and nature of RMP decentralization to date in terms of - - Z (a) the intentions and implementing actions of RMPS including its Review Committee and Council? (b) the perceptions of and actual exercise of the expanded authority by the 56 RMPs, including their RAGs? 2. What significant disparities if any exist between RMPS intentions and implementing actions on the one hand and the RMPs' perceptions and utilization of these decentralized authorities; and why? Page 2 - Assessment of RMP Decentralization 3. How has the decentralized authority been exercised by the RMPS in terms of - (a) congruity with their explicit priorities and objectives? (b) extent of responsiveness to national RMP priorities (e.g. emergency medical services)? (c) categorical emphasis? _ 4, Has decentralization led to any important administrative efficiencies ' and/or greater effectiveness insofar as the review and funding processes of the RMPs and RMPS? 5. Have there been any corollary developments (e.g., developmental component, earmarks) that have directly and significatly abetted or-impeded decen- tralization? 6. What are the policy, operational, and other implications of RMP decentrali- zation for the management of that program; and for other HSMHA grant programs? EVALUATION Methodology: A specific and detailed methodology still has to be worked out. Tentative plans call for a three-phased study. 1. The first, of about 2-3 months, would be aimed at assessing RMPS' “intentions and analyzing its implementing actions; a very preliminary assessment of the perceptions and use made by RMPs of the decentralized authority; and tailoring the second phase accordingly. 2. The second phase, of about 6-7 months, would be the survey proper, including the collection and analysis of data and field studies. 3. The last, if about 2-3 months, would be aimed at determining and sharpening the implications of RMP decentralization. It would entail small group interactions involving selected RMP, key RMPS, and perhaps certain HSMHA staff. A number of different techniques would be required. Content and other analysis of both RMPS and RMP documentation (e.g., policy statements, application materials, RAG minutes), participant observation at RAG, Review Committee, and Council meetings, and market surveying would be among these. Z The more intensive field study portion would no doubt need to be limited to a relatively small number (5-8) of fairly "representative" RMPs. Previous Work: None to speak of. Use: This, of course, would be dependent in the final analysis on the findings and their implications. Potentially these well might be some very significant uses and/or actions flowing from this assessment. For example, if it were found that decentralization was resulting in more rapid and substantively t ' Page 2 - Assessment of RMP Decentralization responsive than that from specific preprogrammed, operational projects, and that there were other corollary advantages (e.g., shorter turn-around time), RMPS very probably would seek to expand or increase decentralization of programming to the RMPs. Certain shortcomings or problems uncovered should lead to modifications in present arrangements. > These leaves aside any possible implications for other HSMHA programs, and how those might be used. DRAFT/4-21-72 RMP DECENTRALIZATION Webster's defines decentralization as the "Dispersion or distribution of functions and powers from a central authority to regional and local authorities.'' In RMP terms decentralization in essence involves and aims at transferring primary responsibility for project review and specific funding decisions from the HSMHA/RMPS (or Federal) level to the 56 RMPs (or the local level). . RMPS as the central authority will continue to (1) set and disseminate broad rere objectives and priorities, policies and procedures, and re- porting and other administrative requirements and (2) determine annual funding levels for the individual RMPs based upon a program review and assessment of each. The RMPs in turn will be the local authorities for determining (1) the technical and professional adequacy of activities it proposes for support and (2) which of those proposed activities it actually will undertake to support within the funds awarded it. Backeround and Description Two points should be noted initially with respect to RMP decentraliza- tion. First, the initial legislation mandated a degree of local program autonomy. It prescribed a Regional Advisory Group, broadly representative A orn of provider groups, institutions, and interests, who’concurrence was required before any operational request(s) could be submitted to RMPS for review and possible; the RAG was given what in effect is a local veto. Second, the program (or core) component portion of the total award to | RMPs (as opposed to the projects component), always has had less specificity to it. Thus, Regions-have had more latitude in the use of program component funds, which have accounted for 40-45% of their total awards for a number of years now, This latitude has in recent years been increasingly exer- cised by then. There were two principal factors that prompted the greater decentrali- - zation which has taken place over the past 12-18 months. One was the decision, prompted by necessity and other factors, to change from a project to a program focus in the review of applications at the national level by the RMP Review Committee and Council. The other, which reinforced that decision and accelerated its implementation, was the FAST recommendation made in mid-1970 that "Project review and funding responsibility be de- centralized directly to each Regional Medical Program as soon as it meets established criteria." Several subsequent developments -~- developments that on the gne hand flowed from the decision to decentralize and on the other wore critical ingredients of that process -- included: (1) The development of an RMP Mission Statement in June, 1970 that outlined in broad terms the nature of the RMP mechanism, and what their evolving mission had become; (2) the. development of a ‘set of related RMP Review Criteria as a basis for assessing individual regional programs; (3) the institution of a rating system utilizing those criteria for ranking RMPs; and (4) inauguration in FY72 of a strong selective funding policy that provides (a) greater fund increases to RMPs that have demonstrated outstanding maturity and potential whose proposals are overall most nearly congruent with the RMP mission and national priorities and (b) greater technical assistance to those RMPs that are doing least well aimed at bringing them up to a comparable level. Operationally the effect has been to give the best RMPs, the so-called "A Regions, considerable programming flexibility and latitude. Such 8 p & ) 2 Regions, within the level of their annual awards, may initiate new activities within prior RMPS concurrence with a very few exceptions, such as end-stage kidney disease treatment activities, projects involving experimentation with human subjects, or entailing construction costs in excess of $25,000. | It is hoped that by the end of FY73 this same degree of flexibility and latitude can be extended to all (or nearly all) RMPs. Before that time, one additional important step will have been completed vis-a-vis all the Regions. Namely, verification of the adequacy of their own review and funding processes in terms of RMPS established minimum standards and requirements , will have been completed. This will insure that RMPs in exercising the programming latitude being extended to them are in fact adequately assessing proposals from a technical standpoint, applying their own priority ond ranking systems, and their RAGs are appropriately invoived in the decisionmaking and funding process. G& 5-1-72 Discussion Decentralization traditionally has been positively judged in terms of (1) enhancement of the quality o£ decisions made, implementing actions, and results obtained and/or (2) greater administrative efficiency. It has, con- versely, been negatively assessed in terms of lessened (1) programmatic responsiveness and (2) administrative safeguards. RMP decentralization has its particular program context and specific aspects. It is neither total as in the case of revenue sharing nor merely adminis- trative as in the case of dispersion of grant-awarding authority from head- quarters to regional offices within the Federal structure. In programmatic, short-run terms RMP decentralization very probably must be judged in the following specific terms. owe (1) Whether, on the one hand, the 56 RMPs continue to be responsive to major national program priorities. (2) While, on the other hand, the RMPs exhibit greater freedom and flexibility in adapting those to the particular conditions and cir- cumstances, needs,.and resources of the local areas. (3) Enhanced administrative effectiveness and efficiencies at both the local and national levels. It is still too early to conclusively assess the relative success (or failure) of RMP decentralization in the above or any other terms. For RMP decentrali- zation was initiated less than one year ago and has yet to take full effect. Moreover, a systematic. assessment will be required. (Such is contemplated, RMPS' proposed FY73 Evaluation Plan includes a major assessment of decentrali- zation.) /3 Early evidence suggests that is, hoxever. Rs have, for example, responded quite well to two recent national prioriti es, HMO development and EMS improve- ment. In the first six months of IY72, ___ of the 56 RMPs directly engaged. in or indirectly assisted with IMO development efforts. Such activities included (to be supplicd). By and large these RMP efforts did not require additional funding or specific RMPS action. Most recently, 36 RMPs have responded to the EMS priority with over 50 proposals. The parallel BHME effort to implement an AHEC program along the lines of the Carnegie Comnission Report and RMP efforts to extend and expand upon existing activities in the direction of community-based educational programs perhaps will provide some interesting comparisons of the extent to which the RMP mechanism is adaptive to local conditions and circumstances, needs, and resources. of In administrative terms, decentralization is allowing RMPS to reduce the amount of paper pushing considerably at the national level. This is turn is permitting RMPS ‘staff to provide far more in the way of management and technical assistance to the RMPs. FAST is now conducting a follow-up survey of the implementation of its recommendations, including decentralization. That survey should provide some fairly specific data and information on the extent of comparable reductions in paperwork within the RMPs themselves. Two things already have been noted as a result of the verification visits made to date. The turn-around time for many local applications can and is being reduced significantly; and far closer monitoring and surveillance of project activities is possible through the 56 RMPs than RMPS staff. One major issue in RMP decentralization is the degree to which formal earmarks will be the administrative means and corollary for giving expression to national priorities. For to rely largely upon formal earmarks for this purpose runs some grave risks. The development of a local decisionmaking process and institution, the RMP and its Regional Advisory Group, requires not only time but continued exercise of thedecisionmaking function. The effect of earmarks is to reduce and erode that function. ' i Thus, great care needs to be taken so that (1) the portion of the total RMP grant funds earmarked at any one time for specific purposes is limited (e.g., not in excess of 25%) and (2) individual earmarks are sufficiently large (c.g., $5 million) to allow all Regions interested in engaging in a specified program activity to have some assurance that meritorious proposals from a large number of RMPs (as opposed to only a few) can be funded. Another important issue is that of the placement and ultimate decisionmaking authority at the local level in the RMP scheme of decentralization. It is RMPS' position that final responsiblilty for determining the scope, nature, direction, and pace of the overall program at the local level must reside téy wide the RAG rather than the grantee, institution or organization. Only the former is representative of all the key health proups, institutions, and interests of a Region. The grantee, be it a university or medical society, represents only a single institution or interest; its role and responsibility is essentially that of an administrative agent for the RMP and not policy 7 setting. A major problem and concern is that of long-range planning and priority- setting at the local level. RMPs given their strong provider orientation . us and links, cannot hope to abrogate into themselves this responsibility. This must in the present scheme of things be the function of CHP, especially the areawide agencies. To date, however, CHPs generally have not progressed to the point where local needs and priorities have been specifically identi- fied and well articulated. Thus, exercise of the decentralized RMP decisionmaking authority has not been as well focused or targeted as it could and should be because of the absence of well-developed local priorities by the larger community. Implications for Other HSMHA Programs RMP decentralization would appear to have certain implicat ions for other HSMHA programs, including possible application. This is true even though experience to date is limited and nof systematic, thorough assessment has been undertaken as yet. First, it should be pointed out that the RMP "model" has limited applicability. It requires that there be a local decisionmaking mechanism or institution created or in place to which authority can be decentralized. Thus, the RMP "model" of decentralization, and the lessons to be learned from it, might have applicability to CHP or the State health authority concepts. It is not, however, directly applicable to HSMHA‘s many project-type grant programs <3 such as Family Planning, cen, and 314(d). It also is quite possible that its direct applicability does not extend to certain R&D-type experiments such as EHSDS. 8 Where applicable, however, it well may be that not to decentralize significant decisionmaking and administrative functions and responsibilities to the local mechanism that has been established is to fail to exploit fully the Federal investment that has been made in creating a management and implementing capacity. And if one Federal aim is to create local decisionmaking capabilities, the increasing exercise of that function is necessary. Responsible decentralization does require a heightened concern with the adequacy of local decisionmaking processes, Thus, there is a need to devise and apply means aimed at insuring that the process is a sound and satisfactory one. For unless this is done, there can be little basis for assuming, as decentralization does, that outcomes in terms of local decisions and actions will be satisfactory. ce The choice in the final analysis is not the stark one of centrasization versus decentralization. Rather it is the degree of one and the other. The alter- natiyes to a significant degree of decentralization were summarized many a years ago by David Liléenthal. They include: * Fewer citizens participating in governmental administration. * Less and less community responsibility. * The self-fulfilling prophecy that "good administration" is only possible A from Washigton. * An ever wider gulf between local communities and national govemment, between citizens and their vital concerns. The above are from an RMP standpoint unnecessary as well as unacceptable ’ alternatives. PROJECI SUMMARY No. 3 @ 2:02.00: HSMHA/RMPS PROJECT Project Title: Method: Date Start: Completion Date Evaluation Tools for Kidney Contracts- September 1972 September 1973 "Life Plan" Program Consultants-In House PROBLEM Need for Evaluation: RMPS intends to initiate a "Life Plan" program to organize integral systems of delivery of dialysis and transplantation services. The primary goal is to develop a minimum of 50 tertiary treatment centers in the next 5 years so that all eligible patients will have access to these life saving measures. The second objective is to prevent duplication and under utilization of established resources. It is expected that the patient care reimbursement mechanism will pro- vide the payment for direct patient services once the resources are established. It is essential therefore that the Regional Medical Programs Service soon have available a system to evaluate the effectiveness of this Program. Objectives of Evaluation: Develop an efficient system of retrospective evaluation of "Life Plan" supported activities in order to assess the impact of such an investment. | a ION M Je thodology: Petnoco ony 1. Development of appropriate criteria of evaluation including such factors as change in amount of hospitalization required; change in days of disability, extended years of life, degree of rehabilitation, relief from welfare need; cost, containment; contribution to the national economy; degree of resource utilization. Development of a system of collection and analyses of needed data including model report forms, standard definition of terms, time schedule of reports. Recommendations for an efficient national communication system to coordinate the activities of the life plan for measurement of quantity and quality of medical care delivered. A . . . Development of a system of evaluation of performance including analysis of sources of payment and costs. Recommendations for the development of an efficient peer review mechanism including the recommended criteria for determining level of continuing Federal support of grantees. Development of a final report - The contractors will develop a final report after periodic review of the above listed tangible products by a Kidney Disease Advisory Committee established to advise the Regional Medical Programs Service on the administration of the "Life Plan" program. Page 2 - Evaluation Tools for Kidney "Life Plan" Program Previous Work: Throughout the last 5 years of kidney disease activity under the aegis of the "Kidney Disease Control Program," patients with end-stage disease were served through the following progressive mechanisms: 1. A series of grants (10) for "Center Hemodialysis" in ten centers throughout the nation. These grants served more than 500 patients in centers through- out the nation. Survival rates were 90 percent the first year, 80 percent the second, and subsequent ten percent decreases each suceeding year. Quality of life was not measured at the time, but center schedules seldom permitted normal activities for these patients, making gainful employment practically impossible. 2. Subsequent efforts included improved evaluation criteria for 12 Home Dialysis Programs throughout the nation. These home dialysis programs were funded through contracts. The evaluation criteria, in addition to survival (which remained the same and in some areas better than the "Center" experience), state of health as a measure of quality of life was introduced. Seventy percent of the patients treated with this modality were able to resume some of their normal activities, being depicted in the "able to work" category. The patient remained at home and the cost of treatment was reduced by at least 50 percent. 3. Transplantation, by now, has provided a new and practical option to patients with end-stage uremia. Thus, a patient's management can be tailey-made to his state of health and level of activity. A patient can, for instance, be cannulated for dialysis, be tissue-typed to wait for a transplant, and simultaneously trained to go home or to a "low overhead" facility. A “Life Plan" for the treatment of patients with end-stage renal disease has now been proposed. This program would provide an optional combination of options to all Americans who suffer from end-stage renal disease and meet the medical criteria for maintenance therapy. Information available in the progress of these programs would be extremely useful in designing the evaluation scheme for new programs. Use: A vital investment of this nature must not proceed in a vacuum, without a reproducible and efficient method of assessment. The plan is to develop an "End-Stage Renal Disease Center' for every four million residents throughout the United States. That would be a total of 50 centers. If a reproducible system for continued retrospective evaluation is developed and accepted by those who would be making local, State and national decisions regarding these programs, it will be successfully accepted as a useful, practical method for making decisions. PROJECT SUMMARY - No. 4 AGENCY/PROGRAM: HSMHA/RMPS PROJECT Project Title: Method: Date Start: Completion Date: Evaluation Tools for Contractors- September 1972 September 1973 Emergency Medical Services Consultants- Activities In House PROBLEM Need for Evaluation: Emergency Medical Service programs are to be supported in a multiplicity of sites and modalities. RMPs are particularly interested in providing improved emergency services in response to their original mission towards the reduction of death and disability due to heart disease and stroke. A common denominator must be developed toward the evaluation of these programs. To this effect, we must determine what is the most significant information necessary for cont inued evaluation of these programs, the method of collection, and pertinent decisions that would be made as a result of favorable and unfavor- able results. The information determined to be most significant should be consistent with evaluation indices developed by HSMHA program authorities. Objectives of Evaluation: To provide decisionmakers and retrospective evaluators with information amenable to the effective evaluation of Emergency Medical Systems, with particular emphasis on cardiac and cerebrovascular trauma. EVALUATION Methodology: Toward the objective of this project, it is proposed that: 1. A group of RMPS staff representatives of the various divisions and offices to work in concert with an ad hoc advisory group including pertinent HSMHA program authorities be instituted toward directing this evaluation effort. 2. The group proceed expeditiously to identify acceptable and reproducible evaluation parameters, and test their acceptability with EMS program directors and decisionmakers. 3. Test the reproducibility of parameters identified. 4. As often as applicable, decide -beforehand the type of action that could possibly ensue favorable and/or unfavorable results. 5. Develop a system for updating and enhancing the evaluation mechanism. 6. Produce indicated materials for continued implementation of the evaluation systems in each EMS program, e.g., workbooks, manuals, forms, etc. Previous Work: Not applicable Page 2 - Evaluation Tools for Emergency Medical Services Activities 1. Use: Implementation of this’ project would yield the following products: An evaluation tool amenable to comparative evaluation of various EMS program modalities. This would yield information regarding the char- acteristics of effective programs, particularly in terms of cost effectiveness, reduction of mortality, and the prevention of disability. A sensitive and timely tool that would provide indications for = continuing and/or curtailing activities and/or the use of specialized techniques. This would make effective program components available to the largest number of people as soon as possible, and stop ineffective and/or high-risk modalities from continued use as soon as possible, thus curtailing waste and/or risk, as soon as possible, protecting the largest number of people, We would be able to assess ways in which RMPs can effectively contribute towards the reduction of death and disability due to specific categorical diseases, in this case heart and stroke, as they proceed towards advanced implementation stages of their program. PROJECT SUMMARY - NO. 5 Agency/Program - HSMHA/RMPS Project Title: Evaluation of RMP Site Visits > Method: Principally contract but also some in-house and consultant Dates: 10/1/72 - 3/31/73 Problem Need and Background: Site visits are important, perhaps the single most important element in the RMP triennial Review Process. That Process, in turn, culminates in the comparative ranking and classifying of the RMPs and leads to decisions as to their level of funding. At three-year intervals each RMP prepares an overall application which outlines its program and requests funding support for the next three years operation. This triennial request is reviewed by the RMP Review Committee and National Advisory Council, and two major actions occur. One, the RMP is rated and classified as either an A, B, or C region and a determination of the appropriate level of funding for each of the next three years is made. After the receipt of the application, but prior to the meetings of either the Review Committee or the Council, a site visit to the RMP is conducted. The team conducting the visit reviews current progress of the RMP, and prepares a report which evaluates the region and recommends its ranking and level of funding. This report is then presented by a site team member at the Review Committee meeting and subsequently at the Council meeting, and forms the basis of discussion and in nearly all instances is decisive in determining the final outcome both with regard to ranking and funding levels for the next three years. The site visit thus is the principal tool used in judging the progress of the region and as such has a number of separate purposes, It allows the site team and indirectly the review groups to gather information which cannot adequately be described in writing, to assess the people involved in the RMP, to determine the interrelationships between people and organizations, to view the physical facilities available, to resolve problem areas which have arisen previously and to provide a high level of technical assistance or policy advice to the Region. ~ Objective: The principal purpose of this project would be to evaluate or assess the quality of the current site visit procedure in both its structural and process aspects and indirectly to determine the validity in the gross sense of the input -- information, judgments, and recom- mendations -- of site visits to the RMP Review Process. In undertaking this project all aspects of the site visit process would be studied and evaluated. Major aspects of a site visit include the following: a. Decision to conduct a site visit (this type of site visit is normally triggered by the receipt of the triennial application). Evaluation of RMP Site Visits (continued) @ b. Selection of the site visit team. c. Distribution of pre-visit materials to the site visitors. d. Discussions between the staff contact person and the RMP concerning the conduct of the site visit and areas of particular concern. ° e. Discussion or meeting of the site visitors by themselves prior to the start of the actual visit (usually called the Executive Session). f. The site visit proper. g. Executive session of the site visit where major recommendations and courses of action are decided. h. Feedback session with the RMP coordinator and a request for supplying of any additional information which will be needed. i. Preparation of report. j. Presentation of the site visit report to the Review Committee and National Advisory Council. oe © Evaluation Methodology: Detailed methodology will have to be developed by the contractor and because of the difficulties in developing a methodology which will adequately deal with the problems of validity and replica- bility of site visit results a key factor in determining the successful bidder will be the experience and capabilities in developing an appropri- ate methodology. It is likely that the methodology would include: 1. Contractor personnel as observers in a number of site visits and in observing the review process generally. 2. Obtaining perceptions of RMPS through interviews and question- naires and validating those. 3. Obtaining through interviews the perceptions of site visitors and RMPS staff. 4. Comparative analysis of other types of site visits (e.g., accreditation visits, research project site visits, such as those conducted at NIH, and annual progress review conducted by foundations. Previous Work: None to our knowledge dealing specifically with site visits. However, the work done by sociologists and political scientists Evaluation of RMP Site Visits (continued) in small group decisionmaking and spécifically that body of research dealing with decisionmaking in the judicial process. (Murphy's work with Supreme Court decisionmaking and the body of literature dealing with the decisionmaking process in the petit jury would be applicable.) Use: Findings and recommendations from this contract would enable RMPS to modify and strengthen present site visit procedures so as to improve their quality and validity and remove or minimize significant short- comings. In addition, because the site visit is widely used throughout government health agencies, the results of this study could have major impact on a number of other programs. PROJECT SUMMARY - NO. 6 Agency/Program - HSMHA/RMPS Project Title: Development of an Evaluation Manual for RMPs Method: Contract Dates: August 1972 to June 1973 Problem Need: While the 56 RMPs collectively are making a substantial annual investment in evaluation and have a broad, high-level staff capability in that area, most individual RMPs have quite limited staff and other resources for evaluation. This limitation on the resources available results in the inability of any single program to afford the range of expertise necessary for a complete assessment of various projects and programs which make up a single RMP. Consequently, many evaluation efforts tend to be restricted to the type of questions a specific individual or staff feels confident to ask and answer. Unfortunately, this disciplinary narrowness may lead one to ask the wrong questions or to be unaware of the appropriateness of concepts, techniques, or data collection strategy available from other disciplines. Objectives: Project would have several major objectives. The most important of these would be the development of a working manual or set of guidelines which would aid project directors and evaluators in (1) deciding appropriate questions to ask; (2) determining the kinds of information which would best answer these questions; (3) determining appropriate sources for assembling data; (4) assessing analytical tech- niques or presentation methods which might be most useful for evaluating specific program or projects; and (5) determining the appropriateness per se of a given evaluation technique in a specific case. A second major objective of the project would be to develop this working manual in a format which could be used as a training instrument with individual RMPs or in more general regional or national conferences dealing with evaluation. . / Evaluation Methodology: It would differ from previous efforts in two significant ways. First, the attempt would not be to take a specific activity (i.e., continuing education programs) and design an evaluation strategy. Rather, it would attempt to define general questions applicable to a variety of programs or projects necessary to determine impact. For example, the criteria established by RMPS which govern the site visit Development of an Evaluation Manual for RMPs (Continued) evaluation are general questions about involvement of various groups, the administrative process, the effectiveness and the efficiency of the program, etc. Starting at this general level, one might then ask more specific questions, the answers to which provide evidence for the more general. It would be at this more specific level that the guidelines would become valuable as suggestions for data collection, involvement of appropriate individuals and utilization of available, but unknown to the program staff, approaches. Second, rather than isolated discussions of the contributions potentially available from various disciplines in the general area of program evaluation, the purpose would be to discuss and illustrate the potential contribution to the answering of specific types of questions. Thus, one might suggest a particular technique would be appropriate in one case but not in another. Methodology used in developing the manual which serves the functions described above would be developed by the contractor. It is likely that the following steps would be taken: a. The creation of a task force representing RMPS and evaluation directors and coordinators from the RMPs to develop a pre- liminary definition of the type of objectives to be realized from action programs or component projects and the more specific questions which can be used to examine the degree of realization of these objectives. b. A survey aimed at RMPs evaluation directors and the evajuation directors of other health programs to determine the type of techniques and methodologies now being used. c. A thorough review of the literature in all relevant disciplines. d. A series of working meetings of the task force at which drafts will be discussed and modified. e. Sample application of the techniques in several regions. Previous Work Use The study by Arthur D. Little, Inc. on the RMP, the recent in-house survey of RMPs evaluation resources, and the two-year ISS contract activity which terminates this June. / Project will have several major uses: (1) to assist a given RMP with its limited resources in bringing to bear the variety of evaluation techniques which may be appropriate for the solution of a given problem; (2) equipping the evaluation director with enough knowledge concerning techniques outside his particular disciplinary specialty to allow him to pursue consultant and other resources with maximum effectiveness; and (3) the provision of a training tool which can improve the overall effectiveness of evaluation within RMPs. @® PROJECT SUMMARY - NO. 7 Agency/Program - HSMHA/RMPS Project Title: Evaluation of RMP Technical Assistance Activities Method: Combination of contract and in-house Dates: 10/1/72 - 3/31/73 Problem Need: Provision of technical assistance and consultation, broadly defined, to RMPs is viewed as an increasingly important element of RMPS staff activity. This is particularly true for the ''C"' Regions -- that is, those which have demonstrated less progress and strength comparatively -- where the RMPS program is one of providing greater technical assistance as opposed to larger infusions of grant funds. Technical assistance is both general and professional and is provided primarily by the Division of Operations and Development and the Division 2 of Professional and Technical Development respectively. ae Objectives: The principal purposes of this project would be (1) to determine the kinds and extent of technical assistance now provided by RMPS staff to RMPs, (2) its usefulness and quality from the Regions' standpoint, and (3) staffs! perceptions of RMP needs or demands in this regard. Evaluation Methodology: This project would entail a close working relationship between the contractor and a small group of selected RMPS staff responsi- ble for its in-house aspects. The contractor's principle roles would be in (1) developing jointly with staff the framework within which this assessment would be made, (2) independently corroborating the Regions! perceptions of the kinds, magnitude, quality, and usefulness of technical assistance rendered by RMPS, (3) validating what it is that RMPs actually is providing and staffs' perceptions of Regions' needs, and (4) making such recommendations as indicated by its findings as to how specifically RMPS technical assistance functions and activities might be improved. Previous Work: None. This project, however, may well be able to draw upon and should be coordinated with Nos. 4, RMP Evaluation Needs ,and 6, Evaluation of Site Visits. the character and mix of and in time RMPS' staffing for technical Use: The results of this effort very probably would be used to modify 9 assistance. PROJECT SUMMARY - No. 8 an HSMHA/RMPS | OJECT Project Title: Method: Date Start: Completion Date: RMP Evaluation Consultants- July 1972 July 1973 Clearinghouse In House » PROBLEM Need for Evaluation: Evaluation activities throughout the 56 RMPs in the United States continue to grow both in terms of amount of work being done and its scope. As RMPs throughout the country proceed to implement their plans, and new opportunities for improving the quality and accessibility of health services appear in the horizon, the need for evaluation of the work being conducted and methods of making decisions becomes more imperative. It is important that these local evaluation efforts attain some visibility for the benefit of all 56 RMPs. It is proposed, therefore, that a program to catalog, disseminate, and comment on the applicability and generalizations of these efforts be instituted. Objectives of Evaluation: The objective of this effort would be to desseminate periodically pertinent information regarding evaluation activities being conducted throughout RMPs in the United States. EVALUATION of 12200: Toward the objective of this project, it is proposed that: 1. A group of RMPS staff representatives of the divisions and offices are to work in concert with selected members of the RMPS Ad Hoc Evaluation Group to delineate objectives, characteristics, and guidelines of the clearninghouse program. 2. RMPS staff be assigned to implement the program. 3. At least two meetings of the developmental staff and consultants group should take place in order to assess: a. The effectiveness of the work being conducted. b. Determine new directions. c. Identify indications for continued activity and/or characteristics of the system that should be terminated, Previous Work: Not applicable Use: Implementation of this project will yield the following products: 1. Evaluation tool amenable to comparative assessment of the various evaluation efforts being conducted through RMPs throughout the country. This would yield information regarding the characteristics regarding effective programs, © particularly in terms of how the results of evaluation are implemented. 2 Timely instrument for avoiding duplication of effort and/or identifying gaps in evaluation programs throughout RMPs. , 3. The identification of technical support in specific areas of evaluation towards PROJECT SUMMARY - No. 9 @ AGENCY/PROGRAM: HSMHA/RMPS PROJECT Project Title: Method: Date Start: Completion Date; Evaluation of Consumer Contracts- January 1973 December 1973 Participation in RMP Consultants- Deve lopment In House PROBLEM Need for Evaluation: In the development of consortia of health planners and providers, several RMPs have been developing methods that facilitate the participation of consumers in the development of improved health delivery systems. It is important to determine the characteristics of useful, positive, and effective consumer intervention. It is important that we categorize and evaluate the various avenues of consumer participation within the confines of specific RMPs. Objectives of Evaluation: To assess the various patterns of consumer partici- pation within the Regional Medical Programs. EVALUATION & Methodology: 1. Develop a small number of practical, efficient, and reproducible measures of effective participation. 2. Select an indicated number of RMPs where consumer participation has been encouraged. 3. Identify and categorize the salient patterns of consumer participation towards the development of improved health delivery systems in the RMPs selected. 4, Evaluate the various patterns of participation in terms of their effectiveness. 5. Among RMPs not initially selected, identify an indicated number of RMPs where consumer participation has mot been particularly encouraged. 6. Compare the assets and liabilities of RMPs where consumer participation has been encouraged that were initially selected with the assets and liabilities of the RMPs where consumer participation has not been particularly encouraged and were selected through 5. above. Previous Work: Several important studies regarding participation of health consumers in the development of plans and policies have been supported through HSMHA. Most recently, HSMHA's Office of Planning and Evaluation has been supporting a study being conducted by Community Change, Inc., addressed to consumer participation in the administrative processees in various levels of Page 2 - Evaluation of Consumer Participation in RMP Development HSMHA service projects. An important ingredient of this evaluation study would be the utilization of intermediate reports made by the latter con- tractor. The scope of work for the above contract includes the following definitions which should be pertinent to the development of the scope of work of the present RMP-oriented study: , Consumer - A recipient or person who is eligible to receive services from HSMHA service projects. Participation - Involvement in the administration of health programs designed to service the population from which the consumer comes in any of the following ways: 1. Membership on an advisory board or Council. 2. Membership on a governing board or Council that is concerned with policy development both in the preplanning and program developmental stages. 3. Holding a position in the administrative hierarchy such as the “director of non-professional employees." It is recognized that this category involves a consumer who has in a sense become a provider by virtue of his professional employment. However, it was also felt that such a consumer had reached’ the 'pinnacle' of consumer participation. “7? Official Consultant to a Program - Although a consultant is also a paid employee the same rationale for inclusion in the definition holds as to that of number 3 above. Advocate - Either a patient or consumer advocate. The definitions stated above may possibly be adapted in a form consistent with this RMP-oriented evaluation. For example, the definition of consumer may be addressed to "service from RMPs' implementation projects" instead of "HSMHA Service Projects." The documentation regarding consumer participation throughout the RMPs is presently available. RMPS is well informed through the plans submitted for review, as well as many of the reports depicting their activities, e.g., East Los Angeles and New Jersey. / Use: HSMHA policy and guidelines display the position that consumer partici- pation is a favorable’ ingredient in the development of improved health systems. While this may be so, it is important to test this hypothesis. In addition, it would be very useful to identify the characteristics of useful and effective consumer participation as well as how much it costs to obtain it, maintain it, and apply its product. @ PROJECT SUMMARY - NO. 10 . a oa L: eo Sa oo HSMHA/RMPS Project Title: RMP Involvement in Helping Launch and Support New Federal Initiatives: HMOs Method: Contract - Dates: 1/1/73 - 12/31/74 Problem Need: There is considerable anecdotal evidence that RMPs have assisted other program efforts at the State, regional, and local levels. There has been no systematic assessment of the importance, extent, and character of such RMP assistance. Objectives: The major objective of this project would be to assess the importance, measure the extent, and define the characteristics of RMP assistance in helping to launch and support HMOs specifically in 1971-72. A secondary objective would be to identify what seem to be _: -tho-key variables in explaining why certain RMPs have been more helpful than others in this regard. “Evaluation Methodology: Three somewhat distinct aspects or phases are envisaged. - The first would be development and/or elaboration of the methodology itself. This would include getting some rough notion of what in fact has been done by RMPs to assist with HMO development, determining the more specific parameters of that assistance, developing the appropriate survey techniques, questionnaires, etc., needed, and selecting the RMPs (5-6) and HMO sites (20-25) to be looked at specifically. This could _.he done largely by reviewing documentation in the RMPS and HMOS files _ interviewing selected key staff of those programs. The second wed be an on-site examination of the RMPs and HMOs selected. This very possibly would utilize survey techniques and questionnaires. The third would be a compilation and analysis of the data and information obtained leading to a final report of findings and conclusions. Previous Work: None to speak of. It is possible, however, that the in-house work done with respect to RMP-CHP relationships might be of son: help in defining categories or kinds of assistance and support. Uses: This project would serve at least two uses. For one thing, it could provide RMPS, HSMHA, and HEW with a better, more valid indication of the reality (or unreality) of RMP as a helping organization, whether in fact it can be expected to play a significant role as the local Title: RMP Involvement in Helping Launch and Support New Federal Initiatives: HMOs agents for helping launch and support new Federal initiatives. From a program management standpoint, it also may provide some helpful clues as to the comparative value of differing kinds of assistance. PROJECT SUMMARY - NO. 11 Agency/Program - HSMHA/RMPS Project Title: RMPS Evaluation Committee Method: Essentially in-house, with possible occasional use of con- sultants Dates: July 1, 1973 - continuing Problem Need: Evaluation has not had the degree of critical, continuing atten- tion of RMPS' top program and management staff that really is required not only in order to (1) develop a meaningful comprehensive, long-term evaluation strategy but (2) to insure that specific evaluation activities (both contract and in-house) are well planned, conducted, and monitored and (3) to implement evaluation findings and results as fully and quickly as appropriate. The establishment of such an RMPS Evaluation Committee is viewed as a major first step in correcting this situation. Objectives: The overall objective is to establish such a funétioning group. Tentative plans call for it to include representation srom: (1) Immediate Office of the Director -- Very probably the Deputy Director, RMPS, who would serve as Chairman. (2) Each of the two operating divisions, Division of Operations and Development and Division of Professional and Technical Development (3) Office of Systems Management -- Very probably the Director. (4) Office of Planning and Evaluation -- Initially both the Director and the Chief of the Evaluation Branch, who would serve as the Executive Secretary. (5) Other organizational units of RMPS (e.g., FMO) on an ad hoc or as required basis., It is intended that this committee meet regularly, about once a month, with the first meeting planned for mid-June. It is intended, moreover, that the comnittee will function as a working group as well as con- sidering and taking action, subject to the approval of the Director, on specific items. The Office of Planning and Evaluation will be responsible for providing staff support and services. The first and top priority item for the Committee's "agenda! will be consideration and development of a proposed comprehensive, long-term RMPS Evaluation Comnittee (continued) RMP evaluation strategy. Other items which it will be concerned with on a continuing basis include: , (1) (2) (3) (4) (5) Assist in developing more detailed specifications for proposed contract evaluation activities and studies included in the approved FY73 Evaluation Plan. This would be done largely through small or working groups involving other RMPS staff. A special priority effort would be the elaboration and implementa- tion of the proposed Program of Challenge Contracts to RMPs for Evaluation. Consideration of findings and results of completed evaluation activities and studies (e.g., Regionalization of Coronary Care) with a view to developing and recommending appropriate imple- mentation actions to the Director of RMPS. Periodic monitoring of ongoing evaluation activities and studies (e.g., Evaluation of Heart Guidelines) for progress. Development of appropriate mechanisms and procedures for co- ordinating and relating its efforts to the Ad Hoc RMP Evaluation Group. Identification of major evaluation issues or problems where pos- sible action by the Director or policy consideration by the Council appears to be indicated. ° PROJECT SUMMARY - No. 12 Agency/Program - HSMHA/RMPS Project Title: Development of a Technical and Management Assistance Program Methods: In-house and consultants Dates: July '72 - July '73 Problem and Background There is a need to strengthen the evaluation capability both at the RMPS level and at the level of the 56 RMPs. One method of doing this will be through the development of a technical and management assistance program which will maximize the effectiveness of the individual RMPS evaluation efforts; (2) encourage collaborative and cooperative activities among the evaluation components of the separate RMPs; (3) provide a method of focusing the activities of the individual units on major areas requiring evaluation; and (4) assist in focusing the evaluation activities of RMPS to insure that they are responsive to the needs identified at the regional level. This would include assistance in updating and modifying both the short-range and long-range RMPS evaluation strategy. we Methodology While the detailed methodology for accomplishing the overall purpose is still being developed, certain portions of it have been identified. The contract proposals for the development of an evaluation manual and con- commitant training program and for an RMP evaluation clearinghouse are major portions of the overall project development. Other components which have already been identified include the development of a roster of consultants with specific skills in evaluation which could be drawn upon by the individual RMPs. Such a roster would be developed from evaluation personnel existing in the individual RMPs and through the identification of consultants outside of RMP who have specific competencies. Other activities would include the development of training courses, the preparation of case studies, and the development of a method of exchanging particularly pertinent evalu- ation studies and reports developed within the RMPs and elsewhere. PROJECT SUMMARY - NO. 13 Agency/Program - HSMHA/RMPS Project _ Title: Fund Turnover Method: In-house Dates: 9/72 - 2/73 Problem and Background One of the major strategies of RMPS is that project support should be in the nature of seed money to allow the planning for an implementation of projects which, if successful, will be subsequently supported by local funds. The reasons for this policy strategy include: (1) the belief that where there is local involvement and commitment, projects will be more successful; (2) the constant turnover and thus provision of "free money" will allow a given level of funding to produce a higher level of activity within the regions; (3) this policy will encourage local self-sufficiency; and (4) the turnover effect provides for the availability of free money which can be used to respond to changing national and local priorities and to emergent needs. ie Currently information is not available which indicates how successful RMPS has been in implementing this strategy of fund turnover and the use of RMP projects as an initial stimulant for local support. A pilot study conducted in 1971 indicated that in a number of regions projects were being continued for an inordinately long length of time. It is now felt desirable to extend this pilot study to determine on a national basis the success of this turn- over concept, to determine problems which it has engendered, and to make recommendations concerning any necessary changes in operating policies or procedure. A random sample of all projects funding during the first three years of RMPS will be chosen and a detailed analysis will be made. Major areas of interest will include: 1. length of committed funding 2. genesis of proposal - , 3. degree of local commitment 4. cost per service unit provided 5. degree of local funding -- amount of increased local support over period of RMP support 6. RMP evaluation as to success of project 7. ability to apply costs to alternate beneficiary FUNDING SUMMARY PROPOSED EVALUATION PROJECTS AND STAFF ACTIVITIES Project Title First Priority: 1. 2. 3. 4, De Program of "Challenge" Contracts to RMPs for Evaluation Assessment of RMP Decentralization Evaluation Tools for Kidney Disease "Life Plan" Activities Evaluation Tools for EMS Activities Evaluation of RMP Site Visits Second Priority: 6. 7. 10. RMP Evaluation Manual Evaluation of RMP Technical Assistance Activities RMP Evaluation Clearinghouse Evaluation of Consumer Participation in RMP Develop- ment RMP Involvement in Helping Launch and Support New Federal Initiatives: HMOs Staff Activities: ll. 12. 13. RMPS Evaluation Committee Development of a Technical and Management Assistance Program Fund Turnover ~ TOTAL Estimate $250,000 200,000 75,000 100,000 75,000 50,000 65,000 af 15,000 50,000 75,000 $955,000 NOTE: Most of these estimates are little more than rough guesstimates at this juncture. @ RELATIONSHIP BETWEEN PROGRAM PRIORITIES AND EVALUATION ACTIVITIES : Budget Proposed Past Program Allo- Evaluation Evaluation Evalu- Priorities cations Projects Efforts ations Process Decentralization cose 2,9 12 . h Turnover of Funds . 13 Selective Funding (and Techni- . cal Assistance Does 5,7 e Program 1) Innovation and Improving Health Care System Emergency Medical Services 4.5 4 Quality of Care Standards -- d Outpatient and Ambulatory Care 15.5 . Rural Health Delivery -- , oe @ Manpower Development and | Utilization Continuing Education 12.3 New Skills 19.4 New Kinds of Health Personnel 2.2 Commmity-Based Education Programs 4.0 3) Regionalization and Institu- tional Linkages g,f,c Kidney 8.0 3 7 4) Cooperative Relationships with Other Health Programs 10 5) Other oer 6,8 , a,b 1. Challenge grants will be directed at several program priorities which cannot be specified until final determination of target areas is made. @ Implementation, No. 11, will affect all evaluation activities. PAST EVALUATION ACTIVITIES UTILIZING EARMARKED FUNDS Information Support System Short-term Training for Evaluators Effects of Coronary Care Activities on Regionalization Evaluation of Heart Guidelines Validating RMPS Review Criteria Measures and Methods for Assessing "Facilitation" Evaluation of Dial Access Library Services Development of Reporting System for Evaluation/Compatible with MIS