I THE COOPERATIVE SHARING DF AUDIOVISUAL MATERIALS IN MEDICAL SCHOOLS ...a network approach a case study from the National Medical Audiovisual Center • National Library of Medicine THE COOPERATIVE SHARINC OF AUDIOVISUAL MATERIALS IN MEDICAL SCHOOLS ...a network approach The Project in Cooperative Teaching is supported by a contract from the Office of Audiovisual Educational Development, Bureau of Health Resources Development, and the National Medical Audiovisual Center, National Library of Medicine, and a grant from the National Fund for Medical Education. The interpretations and conclusions contained in this report are those of the authors and do not necessarily reflect policies of the U.S. Department of Health, Education, and Welfare. U.S. DEPARTMENT OF HEALTH, EDUCATION, AND WELFARE PUBLIC HEALTH SERVICE NATIONAL INSTITUTES OF HEALTH NATIONAL LIBRARY OF MEDICINE NATIONAL MEDICAL AUDIOVISUAL CENTER ATLANTA, GEORGIA 30333 FEBRUARY 1974 COMMITTEE MEMBERSHIP Morton A. Stenchever, M.D., Chairman of the Committee Thomas C. Brown, Ph.D., Educational Consultant, University of Utah Ronald A. Chez, M.D., NICHD (National Institute of Health) Theodore Fainstat, M.D., Ph.D., University of Kansas Gerald R. Holzman, M.D., Michigan State University Wayne L. Johnson, M.D., State University of New York, Buffalo John V. Kelly, M.D., University of Pennsylvania Robert M. Kretzschmar, M.D., University of Iowa James A. Merrill, M.D., University of Oklahoma J. George Moore, M.D., University of California at Los Angeles Ben M. Peckham, M.D., Ph.D., University of Wisconsin David W. J. Reid, M.D., University of Alberta Marvin Rulin, M.D., University of Pittsburgh John J. Sciarra, M.D., Ph.D., University of Minnesota Joseph C. Scott, Jr., M.D., University of Nebraska William T. Stickley, Ph.D., Case Western Reserve University Luther Talbert, M.D., University of North Carolina Preston Lea Wilds, M.D., Medical College of Georgia PREFACE The last decade has witnessed many concerted efforts to improve the quality of education in the health sciences. Some of these efforts have involved groups of schools working toward common objectives and developing instructional programs to achieve these objectives. Similar activities have also been organized through professional discipline- oriented societies such as the American Physiological Society, American Association for Cancer Education, American Society of Hematology, and the American Academy of Orthopaedic Surgery. These efforts have ranged from the locating and peer reviewing of available material to developing comprehensive instruction programs for an entiie discipline. The following publication documents the successes and failures of a comprehensive effort by the Steering Committee for Cooperative Teaching, Association of Professors of Gynecology and Obstetrics in (1) developing a cooperative teaching network among the Gynecology-Obstetrics Departments in United States and Canadian medical colleges, and (2) implementing educational techniques to improve instruction. This project was built on the premise that for a viable cooperative teaching network to succeed, the first need is the development of a diffusion network by which current ideas. materials, and information on audiovisual material could be shared. Among the major initial tasks were (1) developing a teaching bulletin, (2) peer-reviewing available material, (3) disseminating selected audiovisual material, and (4) evaluating audiovisual material and teaching strategies. Approximately two years were spent developing the teaching network and this effort comprises the first portion of this report. The second portion of this report describes a model for the development, evaluation, and sharing of new educational materials. The report describes how a consensus for a standard curriculum in Obstetrics and Gynecology was obtained and how production of materials in these areas is being coordinated among Departments of Obstetrics and Gynecology. It is hoped that you find this report of value. Hopefully, other groups can learn from the mistakes and successes of this pioneer effort by the Steering Committee for Cooperative Teaching Association of Professors of Gynecology and Obstetrics. Richard A. Lasco, Ph.D. Project Officer in ACKNOWLEDGEMENTS The Steering Committee for Cooperative Teaching wishes to express its sincere thanks to Ms. Deberah Bullias, Viola Higgins, and Carol McBride for typing this report. In addition, we deeply appreciate the statistical studies and editing efforts of Mr. Stanley D. Pace and Ms. Dolores Thompson. IV TABLE OF CONTENTS CHAPTER PAGE 1 The Germination of an Idea ....................... 1 2 Early Sharing of Audiovisual Materials.................. 9 3 Assessment Versus Evaluation: The Politics of Educational Research and Educational Innovation ....................... 13 4 The Impact of Innovations in Obstetrics and Gynecology Under- graduate Teaching Programs ....................... 17 5 Faculty Reactions to the APGO Teaching Aids Package......... 25 6 Student Reactions to the APGO Teaching Aids Package......... 35 7 Intraorganizational Cooperation and the Thrust in New Directions . . 43 8 Conclusions ............................... 49 BIBLIOGRAPHY 53 v LIST OF TABLES TABLE TITLE PAGE 1. Units in Teaching Aids Package ....................... 10 2. Steering Committee for Cooperative Teaching Workshops ........... 15 3. Changes in Clerkship Characteristics of Undergraduate Teaching Programs in Obstetrics and Gynecology, 1970-71 and 1972-73........ 21 4. Changes in Evaluation Procedures Used on an Undergraduate Clinical Clerkship in Obstetrics and Gynecology, 1970-71 and 1972-73............................................ 22 5. Changes in Hardware (Machine) Holdings Used in Undergraduate Teaching Programs in Obstetrics and Gynecology, 1970-71 and 1972-73............................................ 23 6. Changes in Software (Teaching Aids) Holdings Used in Under- graduate Teaching Programs in Obstetrics and Gynecology, 1970-71 and 1972-73........................................ 24 7. Faculty Time Spent in Instruction with Undergraduate Medical Students in Obstetrics and Gynecology, 1971-72 and 1972-73........ 26 8. Comparison by Media Type of Faculty Responses About Familiarity with Units in Teaching Aids Package, 1971-72 and 1972-73.......... 27 9. Comparison by Media Type of Faculty Responses About Avail- ability for Student Usage of Units in Teaching Aids Package, 1971-72 and 1972-73 .................................. 28 10. Comparison by Media Type of Faculty Responses About Requiring Students to Use Units in the Teaching Aids Package, 1971-72 and 1972-73............................................ 29 11. Comparison by Media Type of Faculty Responses About Units in Teaching Aids Package Helping Develop General Knowledge of Obstetrics and Gynecology, 1971-72 and 1972-73................ 29 12. Comparison by Media Type of Faculty Responses About Units in Teaching Aids Package Helping Answer Questions of Residents and Staff, 1971-72 and 1972-73 .............................. 30 13. Comparison by Media Type of Faculty Responses About Units in Teaching Aids Package Helping Answer Questions on a Formal Examination, 1971-72 and 1972-73.......................... 31 14. Comparison by Media Type of Faculty Responses About Appro- priateness of the Format of Units in the Teaching Aids Package for the Conveyance of the Content, 1971-72 and 1972-73 ............ 32 15. Comparison of Faculty Responses to Need for Future Teaching Aids Packages and Willingness to Purchase New Materials, 1971-72 and 1972-73............................................ 32 16. Comparison of Faculty Responses About Overall Rating of the Teaching Aids Package, 1971-72 and 1972-73 .................. 33 17. Comparison by Media Type of Student Responses About Using Units in Teaching Aids Package, 1971-72 and 1972-73.............. 37 vi LIST OF TABLES (CONTINUED) TABLE TITLE PAGE 18. Comparison by Media Type of Student Responses About Their Required Usage of Units in Teaching Aids Package, 1971-72 and 1972-73............................................ 37 19. Comparison by Media Type of Student Responses About Usefulness of Units in Teaching Aids Package, in Developing General Knowledge in Obstetrics and Gynecology 1971-72 and 1972-73 ...... 38 20. Comparison by Media Type of Student Responses About Units in Teaching Aids Package Helping Answer Questions of Residents and Staff, 1971-72 and 1972-73 .............................. 39 21. Comparison by Media Type of Student Responses About Units in Teaching Aids Package Helping Answer Questions on a Formal Examination, 1971-72 and 1972-73.......................... 39 22. Comparison by Media Type of Student Responses About Appropriateness of the Format of Units in the Teaching Aids Package for the Conveyance of the Content, 1971-72 and 1972-73 .... 40 23. Comparison of Student Responses About Overall Rating of the Teaching Aids Package, 1971-72 and 1972-73 .................. 41 24. Comparison of Student Responses About Need for Future Teaching Aids Packages........................................ 41 Vll CHAPTER 1 THE GERMINATION OF AN IDEA Developments in technology often lead to the development of new institutions designed to assist society in gaining the greatest benefits from scientific progress. The complexities of the social, economic, political, and educational structures of the North American society are at least partially the direct result of new information. As new information becomes available, societal institutions, especially those for higher learning, tend to create structures for identifying new information and to develop methods for disseminating and using it. Such structures, which may assume a variety of forms, subject new information to analysis through empirical and experimental research models/6,/ One such effort in medical education involving the discipline of obstetrics and gynecology concerns a multi-institutional cooperative networking approach. Before reviewing activities which led to the development and implementation of this multi-institutional cooperative teaching effort, it seems appropriate to review the history of medical education and identify technological advances which provided the force for such an effort. Even before recorded history, man felt the need for some one or more persons in the group to care for the sick or injured. It was stated by McGlothlin that the first division of labor, in a professional sense, occurred when the societal structure of ancient man required two different men to serve the society in two areas of concern which control man's fate ( i.e., the spiritual concern and the medical concern).(27) European Antecedents Medical education, although rooted in the tradition of the Greek physician Hippocrates, became formalized near the beginning of the Middle Ages with the establishment of the early societies or guilds of scholars and students. The earliest medical school was the University of Salerno, Italy, founded during the 11th century. It prospered as the center of medical education for several hundred years, taking its place beside the University of Paris (Theology) and the University of Bologna (Canon Law). (18) This triumvirate of separately located institutions formed the base from which Western education emerged/6,1 In Europe in the 16th century, most physicians received their education exclusively in the universities. Schools of the greatest repute during that period were Italian. Next in rank were the French and then the Germans. However, institutions located outside of Italy were well attended/^ The universities in these various locales were divided into different faculties, generally after the nationalities of the faculty and students, leading to so-called "nations." Rectors were chosen, one from each nation represented, and a "College of Rectors" was formed. The Rectors negotiated with state officials and maintained a power which they did not lose until the close of the 16th century. As an example, the German nation, especially those from poor classes who were known as traveling scholars, had obtained their foundations for university training in Latin schools founded during that period in numerous places. Bands of students united together, and during their travels the worst barbarities were an everyday occurrence. These traveling scholars became formally organized and, like the apprentices of the artisans, were governed by their elder comrades. After obtaining rudimentary knowledge necessary for entry into the universities, roving students entered into the major institutions in their appropriate nations. Famous teachers were received at these institutions with great ceremony. Professors and students stood in more friendly relations than probably exist at the present day. Thus, the attachment of the students to their professors was secured throughout their life. Many professors were physicians of the various princes, who constituted in some cases much of the student body of these institutions. They instilled in the students the love of science which strengthened and sustained both the professors and the students. They satisfied their thirst for science and their efforts after truth and knowledge, often at the neglect of their physical needs. These early medical colleges, then, created the forces for promoting the sciences and arts and constituted a unique combination of the poor and the rich for such advances/^ Until the close of the 16th century, Italy served as the model for educational arrangements. Italy yielded precedence to the Netherlands and to France, whose universities at Linden, Paris, and Montpellier then became the goal of all those who desired to receive the highest level of training in these departments of science. The method of instruction everywhere was the same with the exception of the universities of the Netherlands. In most medical schools of the time, the teacher gave simple theoretical lectures and dictated appropriate prescriptions. The ordinary language of the colleges was still Latin until 1688, when one of the great lecturers of the time ventured to deliver lectures in German. There were medical colleges where professors examined patients in front of students, made the diagnosis in the presence of students, and determined treatments to be provided. However, most medical colleges of the time lectured only on theoretical medicine. Student influence on the curriculum during this period dictated such policies. However, the seeds of change whereby faculty would set the curriculum had their greatest impetus from the Netherlands/^/ In England, from the middle of the 17th century until 1868, students were required to reside in a college or hall; however, clinical instruction was given in hospitals. In 1697, an anatomical theater was first erected in "Surgeons Hall" in Edinburgh, Scotland. However, occupation with practical anatomy was regarded by many physicians as a business unworthy of them, and a separation between the "ordinary physician" who practiced surgery and the "higher physicians" who considered surgery beneath their dignity existed/4,/ Learned societies which had been formed earlier provided great input into the medical schools at these and later times, especially as they supplemented the instruction at the universities. Many practicing physicians who belonged to these societies lectured at the universities through the 17th century. These physicians were not simply savants but were very frequently zealous investigators and contributed much to the higher learning of the period. The changes which were occurring in medical education during this period had great impact on the structure of medical education in the United States and Canada/4,/ 2 Medical Education in the United States and Canada to 1910 Although the American college was based upon English models, the establishment of the first United States medical college at the College of Philadelphia in 1765 was strongly influenced by the University of Edinburgh, Scotland/5,/ The first faculty members at the University of Pennsylvania were John Morgan and William Shippen, who had received advanced training in medicine at Edinburgh. It should be noted that Benjamin Franklin, who contributed so much to the development of American education, was much involved in advising Morgan as a young student in the selection of the University of Edinburgh and subsequently assisted in raising money for the medical college at Philadelphia/42y It was over sixty years later that the first Canadian medical college was established at Montreal, Quebec, Canada (McGill University Faculty of Medicine, founded in \829).( 14) Two years after the establishment of the Medical College at the College of Philadelphia, Kings College (later named Columbia University) in New York became the second medical college founded in the colonies. Dr. Samuel Clossy, a graduate of Trinity College, Dublin, Ireland, had been lecturing in anatomy at Kings College since 1763, and he joined with Dr. Samuel Bard, a graduate of Edinburgh, and four other physicians in founding the medical college, thereby bringing into early colonial medical education the influence of another European institution/42,/ Harvard established a medical college in 1783, and medical colleges were established at Dartmouth in 1798 and at Yale in 1813. Early Canadian medical colleges were McGill University Faculty of Medicine, established in 1829; University of Toronto Faculty of Medicine, 1843: University of Montreal Faculty of Medicine, 1843; Laval University Faculty of Medicine, 1852; and Queens University Faculty of Medicine, 1854. These medical colleges represented both French and English models of medical education/74,/ These colleges were closely associated with their parent colleges — an association which undoubtedly facilitated communication and interaction between the medical college and other departments and colleges of the parent institution/^ Although some medical colleges were established during the early period of American higher education, the training of the physician was generally accomplished under an apprenticeship system. Brubacher and Rudy reported that: The would-be physician started making himself useful by washing bottles. later mixing drugs, and, perhaps at a still later stage, progressing to such routine matters as blood-letting. By being present in the doctor's office and accompanying him on calls, he picked up much of the lore in diagnosis and therapy/#,/ The period of North American expansion between the years 1800 and 1860 also was a period of great expansion of institutions of higher learning/./,/ The effect on medical education can be observed in the statement made by Abraham Flexner: A harmful precedent was established with the foundation early in the 19th Century at Baltimore of a proprietary school, the so-called Medical Department of the so-called University of Maryland. Before that, a college of medicine had been a branch growing out of the living university trunk. This organic connection guaranteed certain standards and ideals. . . which medical education could, as experience proved, ill afford to forego/74,/ Medical education experienced a period of simultaneous expansion and deterioration during this period as evidenced by the e^'.'.blish.r.cr.'i of about 450 such proprietary schools in the United States and Canada. The mode of instruction consisted of a lecture 3 hall and a practitioner or two. "it soon became obvious that establishing a proprietary medical school was a lucrative enterprise."(29) As a reaction to the trend, the American Medical Association was founded in 1874 by concerned physicians. The organization was conceived of as an instrument of reform/42,/ The task of the newly created medical association was quite difficult. Charles Eliot, president of Harvard from 1869 to 1909, attempted to raise standards at the Harvard Medical School in 1870 by proposing to the Medical School that five-minute oral quizzes, used as final examinations, be replaced by written examinations. He was told that written examinations were impossible for the Medical School as a majority of students could not write well enough/75,/ It was reported that the state of medical education at colleges was so low that: "... at Bowdoin,. . . undergraduates there looked down on medical students on its campus with a disdain and were not above playing pranks on them."(8) During the period 1860 to 1880, several colleges, notably Columbia University and the University of Michigan, introduced the experimental method and laboratory facilities to the colleges of medicine in an attempt to raise standards/29,/ Similar thrusts undoubtedly were occurring in Canada as medical colleges were being established at Dalhousie University (1868) and the University of Manitoba (1883) J14) The Effects of the Flexner Report: 1910-1950 The establishment in 1876 of Johns Hopkins University, which was patterned after German universities of the times, ushered into the United States' system of higher education a rebirth of university-oriented medical education, which had floundered shortly after the end of the Colonial Period/5,/ Flexner was influenced by the medical program established at Johns Hopkins University, and his timely but overdue report on the state of medical education in 1910 reflected this influence.^74^ The Flexner Report stimulated in the United States a reorganization of medical education which led to the demise of most proprietary schools of medicine and an integration of medical colleges with parent institutions by the middle 1920's.( 23) A new era in medical education was the major result of the Flexner Report. Medical colleges either were sponsored by a parent university or quickly associated with a nearby university in order to attempt to bring about a medical education program that was more comprehensive in its coverage of the basic medical sciencesY29,/ In assessing the impact of the Flexner Report, Pusey stated: "It was his [Flexner's] conviction that the teaching of medicine had to be taken away from proprietary schools, wedded to practice, and strongly implanted in universities."(31) Githens reported that the objectives of medical education, following the Flexner Report, were to: . . . produce physicians having backgrounds in all that was known of human biology and competent in all aspects of the art and science of medicine . . . This produced the most rigid curriculum imaginable in essentially all medical schools for about forty years. It also produced faculty attitudes directed toward covering all the materials in all the fields of medicine to certify that the graduate would be a reliable physician for the publicY7f5^ The trend in medical education from the 1910's to the 1940's was then to stress basic medical sciences during the first two years of medical education, clinical medicine during the second two years; and provide supervised practice through an internship program the year after graduation from medical college. During World War II, international problems brought about technological and scientific advancement, which 4 continued at an increased rate after the end of the war. However, the basic structure of medical education remained unchanged.f2J,i Forces for the Reorganization of Curriculum: 1950—1970 The idea that improvement in curriculum was a major need in medical colleges was behind Miller's provocative article, "Adventures in Pedagogy.""(28) A very negative rebuttal to such a notion was made by Lymen in "Disaster in Pedagogy."(25) Forces to focus the attention of medical college faculties on the problem of seeking alternative means to the "Flexner Model" of medical education became apparent during this period. The establishment of centers for research in medical education in a number of institutions, initiated in 1956 by Ham at Case Western Reserve and Miller at the University of Illinois, focused attention on the needs and characteristics of medical students, faculty, and curriculum./^,/ A comprehensive study of medical colleges attempting radical departures from traditional curriculum patterns was reported by Lee.(23) The Association of American Medical Colleges, through a variety of efforts closely associated with the thrust noted above, culminated in a formal workshop on the medical school curriculum/7,/ While the historical perspective of events during this period offers insight into the manner in which change became a way of life in medical colleges, it seems best to focus on the impact of educational technology as a force leading to the development of a "networking approach." In addition, the results of experimentation contrasting traditional teaching practices with mediated learning programs seem important in placing the idea of networking in an intelligible perspective. The Educational Media Movement Advocates for improving medical education have viewed technological advances as the means for creating more effective and efficient learning programs for medical students. Past impacts of individualized medical educational programs using multimedia methodologies in both the basic science and clinical science curriculums have been reported. In 1964, West and Stickley completed an evaluation of the effectiveness of using eight millimeter motion pictures supplemented with written programmed instructional materials in pharmacology, compared with traditionally operated laboratory learning experiences. Scores on a post-test between the two groups of students indicated that students who used the self-instructional mediated programs earned higher test scores than the control group who completed the traditional laboratory program. However, West and Stickley concluded that the cost of producing the eight millimeter motion pictures and programmed materials was quite expensive when compared to the cost of materials used in the traditional laboratory program.(39) Use of a self-instructional program in biochemistry to be used as a precourse supplement, reported by Christensen in 1965, showed favorable results in knowledge gains by students./77,/ It was concluded in another study by Lloyd that students were extremely receptive to self-instructional programs involving clinical problem solving, particularly programs providing "delayed feedback."(24) A teaching-machine program and a series of lectures in electrocardiography were compared by Andersen, Owen, Hall, and Smart, and the results indicated no significant difference in effectiveness or efficiency between the two methods.(2) It was concluded by Azneer, Kesler, and Cacamo that a 5 self-teaching program incorporating taped lectures and filmstrips was superior to the conventional textbook approach in the learning of diabetic acidosis.(3) Mulvihill used self-teaching, audiovisual programs as a supplement to the curriculum at Dartmouth Medical School and concluded that they were effective for this purpose. He indicated that utilization by the students was excellent, and comments received from faculty and students were quite favorable Y50,/ A significant study by Feldman in 1969 indicated that medical-student mode of performance improved as a result of student utilization of self-instructional programsY7 J,/ In 1971, McCarthy assessed student perceptions of an individualized self-instructional unit in general surgery and concluded that all students involved in the study reported "favorable" or "very favorable" comments about these materials.(26) The effectiveness of teaching-machine programs compared to formal lectures in the curriculum of clinical medicine was studied by Sweet and Doyle in 1971. This study reported that students considered the teaching-machine method superior to a formal \ec\.\xxe.(38) During 1973, Guyton completed a multimedia, self-teaching package study involving senior medical students at the University of Utah. Twelve self-instructional media units were selected for senior-student self-study uses during a preceptorship program in practicing physicians' offices throughout Utah. Instructional units within the package covered topics in physical medicine and rehabilitation, pediatrics, emergency medicine, ENT, obstetrics, internal medicine, and human sexual behavior. Of the twelve items randomly selected from over 400 media programs identified and obtained, three units were from the Association of Professors of Gynecology and Obstetrics Teaching Aids Package. An experimental and a control group of senior medical students were selected; both groups were administered a pretest, post-test, and delayed retention tests of both knowledge and attitudes. Students in the experimental group did not differ significantly from students in the control group on the knowledge pretest. However, the experimental group performed significantly better (.05 level) on both the post-test and the delayed retention knowledge tests. The pretest, post-test, and delayed retention attitudinal tests concerning media format indicated little difference between the experimental and the control groups, except for their reaction to 16 mm. films with supplemental monograph materials. Guyton concluded that students generally had positive attitudes towards the programs in the self-teaching package and especially preferred visual media with auditory supplementation. He also concluded that programs such as 16 mm. films/supplemental monograph and audio cassette/2 x 2 slides were among the most popular. However, these programs were considerably more expensive than other programs. He noted that faculty must be cognizant of the fact that there are wide differences among student attitudes regarding the media formats and that these personal preferences are dependent upon multiple criteria.(17) These studies indicate that students generally hold favorable attitudes toward the use of self-instructional materials. Reports concerning knowledge gains with self-instructional materials versus other teaching techniques — lectures, etc. — are conflicting and a case can be made for either side of the question. In all probability, the attitudes of faculty toward self-instructional materials influence student reactions to such materials in the efforts they might expend in using those that are made available to them. While not documented, this hypothesis may account for conflicting evidence regarding student knowledge gains using various teaching techniques. The educational media movement began to take shape in obstetrics and gynecology with the publication of Wild's and Zachert's programmed case study approach in Lysaught's Programmed Instruction in Medical Education in 1964(40) and of Russell's study in the use of audiovisual aids in 1966.(33) In January 1969, Chez reported replacement of faculty lectures with "audiovisual booths for the presentation of 6 fundamental information and data, individual tutoring . . "(10) To these reports were added Hunter's experience, which indicated extensive use of audiovisual materials in learning carrels at the University of Washington. Seattle/27,/ and Stenchever's concern with the need for students to have modality options in their learning programs.(36)These stimuli led to the efforts of a small group of dedicated individuals from various institutions to work together for cooperative teaching purposes. Two major outcomes resulted between 1970 and 1971. Rulin and Chez disseminated the "Pittsburgh Package" in early 1970 to fifty-five departments of obstetrics and gynecology in medical schoo\s,(32) and the Steering Committee for Cooperative Teaching, Association of Professors of Gynecology and Obstetrics, disseminated a Teaching Aids Package to 129 departments of obstetrics and gynecology in the United States and Canada the following year.(37) 7 CHAPTER 2 EARLY SHARING OF AUDIOVISUAL MATERIALS For the purpose of discussing cooperative efforts in the use of media in teaching programs, an Ad Hoc Committee was founded during July 1969. Representatives of departments of obstetrics and gynecology from eight universities met at Case Western Reserve University. Four objectives were identified by the Committee. They were: 1. The sharing of evaluation data concerning the learning effectiveness of teaching materials among the members of the Ad Hoc Committee. 2. The establishment of a Newsletter which would delineate the efforts of the Ad Hoc Committee and describe progress in the field of self-instructional materials. The Newsletter would be sent to department chairmen of obstetrics and gynecology at medical schools throughout the United States and Canada. 3. The identification of contact persons in all colleges in the United States and Canada who would be interested in information and products produced by the Committee. 4. The establishment of cooperative ventures in the identification, development, and testing of instructional materials and an agreement to meet in the future to review these effortsY75,/ Construction of the Network for Dissemination of the APGO Teaching Aids Package The construction of the network went through a number of phases. In the first phase, eight professors gathered together at Case Western Reserve University on July 9 and 10, 1969.* They formed the Ad Hoc Committee for cooperative teaching and wrote the objectives listed above.(37) Communication among the Ad Hoc Committee members continued through telephone conversations and sporadic interchanges as various members gathered in small groups at national conferences during 1969 and early 1970. By that time, sufficient interest had been generated by previous activities so that a nucleus of individuals from a number of institutions throughout the United States and Canada had expressed interest in the efforts of the Ad Hoc Committee. In January 1970, the Committee's first Newsletter was published and included contributions from Committee members concerning the use of learning materials in their departmental teaching programs. The Committee then met in February 1970 during the annual conference of the Association of Professors of Gynecology and Obstetrics (APGO). It was at this time that the Steering Committee for Cooperative Teaching, *Funds for this meeting were partially provided for by the Carnegie Corporation of New York. 9 Association of Professors of Gynecology and Obstetrics, was established and additional members were added. Newsletters were published throughout 1970 and early 1971 and included Steering Committee membership review of audiovisual materials and devices. A working philosophy had been established by the Steering Committee to utilize those modalities which did not require extensive investment in hardware. The determination was made to enable the Steering Committee to put together a sampler package of audiovisual materials which could be used by all institutions in the United States and Canada. Throughout 1970, Committee members continued to refine materials they were developing, continued to identify excellent materials produced by both commercial and noncommercial producers in obstetrics and gynecology, and approached and obtained in February 1971 support from the Lister Hill National Center for Biomedical Communications, National Library of Medicine, to fund a Cooperative Teaching Project Center located at the University of Utah. Concurrent with these activities, additional institutions expressed interest in the activities of the Steering Committee. During the period February 1970 to February 1971, largely through the efforts of the individual Steering Committee members, personal contacts with other faculty members, and its Newsletter, 66 institutions expressed an interest in participating in the Committee activities. During March of 1971, nonparticipating departments were invited by means of a postcard survey to join the network. By June 1971, 116 medical college departments of obstetrics and gynecology were included in the network. Since that date, 14 additional colleges have been added to bring the total number of colleges participating in the Cooperative Teaching Network to 130. During that period, the title of the Newsletter was changed to the Cooperative Teaching Bulletin, published by the Steering Committee for Cooperative Teaching, Association of Professors of Gynecology and Obstetrics. Three previous Newsletters had been sent out. However, with the establishment of the Cooperative Teaching Project Center, this communication device was put on a quarterly basis and sent to all medical school deans, department chairmen of obstetrics and gynecology, and faculty contact persons at institutions in the network. The Teaching Aids Package selected at the February 1971 meeting of the Steering Committee consisted of 22 teaching aids in obstetrics and gynecology. The Steering Committee felt the aids were a sample of material relevant for undergraduate medical students and were technically excellent with regard to mode of presentation, ease of use by students, and quality of production. Included in the Teaching Aids Package were five workbooks, one syllabus, three audiotapes, six audiotapes with slides, two programmed case studies, and five motion pictures. Table 1 is a listing of items included in the Teaching Aids Package. TABLE 1. UNITS IN TEACHING AIDS PACKAGE Type Aid Title, Authors) and Institution Where Developed Programmed Textbook Application of Gynecologic Oncology by Wilds, P. L., and Zachert, V., Medical College of Georgia Programmed Textbook Essentials of Gynecologic Oncology by Wilds, P. L., and Zachert, V., Medical College of Georgia Workbook Labor: A Workbook in Obstetrics and Gynecology by Stenchever, M. A., and Kitay, D. Z., Case Western Reserve University 10 Workbook Human Sexual Behavior: A Workbook in Reproductive Biology by Stenchever, M. A. Testing by Stickley, W. T., Case Western Reserve University Programmed Workbook Workbook in Obstetrics and Gynecology by Moore, J. G., University of California at Los Angeles, as modified by Wilds, P. L., Medical College of Georgia Programmed Case Abdominal Enlargement With Ophelia Zop by Kelly, J. V., University of Pennsylvania Programmed Case Premature Delivery With Mrs. Esmeralda Burp by Kelly, J. V., University of Pennsylvania Syllabus Syllabus, for Use With the Carter Model by Carter, J., Indiana University Tape-Slide Program Human Sexual Response by Chez. R. A., University of Pittsburgh Tape-Slide Program Maternal Physiology by Chez, R. A., University of Pittsburgh Tape-Slide Program Infertility by Chez, R. A., University of Pittsburgh Tape-Slide Program Endocrinology of Pregnancy by Chez, R. A.. University of Pittsburgh Tape-Slide Program Contraception by Chez, R. A., University of Pittsburgh Tape-Slide Program Pelvic Mass by Johnson, W. L., Indiana University Audiotape Program Antenatal Care With Agent 38-24-34 by Kelly, J. V., University of Pennsylvania Audiotape Program Difficult Labor With Mrs. Jer C. Bounce by Kelly, J. V., University of Pennsylvania Audiotape Program Postpartum Hemorrhage With Gina Lollipop by Kelly, J. V., University of Pennsylvania Motion Picture Film Postpartum Hemorrhage by Ortho Pharmaceuticals Motion Picture Film The Female Pelvic Examination by Hunter, C. A., Jr., Indiana University Motion Picture Film Normal Delivery by Hunter, C. A., Jr., Indiana University Motion Picture Film Normal Labor by Hunter, C. A., Jr.. Indiana University Motion Picture Film Prolongation of Labor Due to Uterine Dysfunction by Hunter, C. A., Jr., Indiana University These materials were seen as samples of various modalities which could be used for instructional purposes. In addition, materials had previously been approved by content 11 experts on the Steering Committee as effective, efficient learning materials. All items had been shared among the members of the Steering Committee and tested on their students, and Committee members found these materials useful in their departmental teaching programs. Decentralized Centralization During May 1971, the Steering Committee met and agreed to a districting plan of the United States and Canada based on geographical lines. Each member of the Committee became a locally available consultant to medical college departments in his area. The National Medical Audiovisual Center agreed to accept copies of materials from producers, package the materials, and mail materials to network participants. In addition to the Cooperative Teaching Bulletin, the link which formed the strongest bond for the network was the dissemination of the Teaching Aids Package, During July and August of 1971, materials were mailed to 116 medical school contact persons. During the period August 1971 to May 1973, additional copies of the Teaching Aids Package were sent to 13 new United States medical school departments of obstetrics and gynecology and to nine institutions located outside of the United States boundary. Five of these were sent to a medical college in Canada, which placed the Teaching Aids Package in all of its teaching hospitals; one was sent to the Pan American Conference for Medical Education in Bogota, Colombia: three packages were sent overseas to medical schools in other nations for demonstration and teaching purposes. One United States medical school obtained two additional packages. In all, 139 Teaching Aids Packages were disseminated. Two sample packages were displayed, one at the Lister Hill National Center for Biomedical Communications and one at the National Medical Audiovisual Center. These centers are located in different geographical areas, but both are parts of the National Library of Medicine. The dedicated efforts of the small group °f persons who initiated the Ad Hoc Committee to share instructional materials grew to capture the attention of medical educators in obstetrics and gynecology in all medical schools in the United States and Canada and four medical education units outside of the North American continent. This is evidenced by the number of institutions who presently have the Teaching Aids Package and by the number of persons who have asked to be placed on the mailing list to receive the Cooperative Teaching Bulletin which now goes to over 50Q persons. The results of the dissemination provide insight into the feasibility of such an effort and its impact on undergraduate obstetrics and gynecology teaching and learning. 12 CHAPTER 3 ASSESSMENT VERSUS EVALUATION: THE POLITICS OF EDUCATIONAL RESEARCH AND EDUCATIONAL INNOVATION A distinction between assessment and evaluation needs to be made in order to identify the conceptual framework which formed the initial base for the approach taken by the Steering Committee for Cooperative Teaching in assessing the impact of the Teaching Aids Package on medical college departments of obstetrics and gynecology in the United States and Canada. Evaluation There exists in medical education a strong desire for evaluationY7^7y New approaches have been proposed which would measure the outcomes of education in terms of changes in student performance on a scientifically based prescription. These approaches would attempt to guarantee the product produced by an educational system. That is, professors would have to face up to the demand for accountability to guarantee student learning as a prime goal.(5) The need for closely controlled experimental approaches is important. Assessment A necessary first step in assessment is the collection of empirical data and its analysis. Assessment has been defined as "the determination over time of the outcomes of education in light of different contingencies."/22,/ Assessment, then, serves two purposes: (a) the politics of using information for decision-making (accountability) and (b) research (empirical). Some have charged that "assessment is really politics masquerading as research.'Y79y Educational programs, then, usually serve the interests of competing groups. A more pristine second component of the assessment process can be found in the word "research," which develops the evidence concerning the efficacy of any approach. Confusion exists when decision makers equate monitoring of a project with evaluation of a project. As stated by House, "auditors end up trying to evaluate and evaluators end up attempting to audit programs.''(19) Such concepts lead to a determination of the rationale for any assessment program. Is it evaluation for evaluation's sake, or evaluation for control, or evaluation for aiding decision-making? The selection of the question sets the tone and assessment approach for a program. 13 Assessment Approach Due to the diversity of the distribution of the Teaching Aids Package, the Steering Committee recognized the futility of attempting to develop a strictly experimental evaluation design. As virtually all departments of obstetrics and gynecology in medical schools in the United States and Canada were a part of the project, it was impossible to establish adequate control groups for comparative purposes. When planning for survey procedures, the Steering Committee decided that the assessment program should attempt to determine the impact of the teaching materials distributed by the Steering Committee on departmental teaching programs. Parameters for the assessment program were developedY57,/ Means were developed to monitor teaching programs existing in departments prior to the distribution of the Teaching Aids Package and to later reappraise teaching programs and compare these data to note changes. The attitudes of faculty about the materials in the Teaching Aids Package were assessed over a two-year period to determine if materials would remain viable. A group of students from a number of institutions in each district was surveyed over a two-year period to insure student opportunity to react to materials. In view of the above, The Steering Committee attempted to gather data, analyze it, and use it to aid decision making concerning directions and approaches appropriate to meeting the needs of network participants. It was recognized that in order for the network to remain viable, Steering Committee efforts had to be sensitive and responsive to actual problems faced by a large group of faculty from many institutions. Assessment Techniques During the 1971 annual meeting of the Association of Professors of Gynecology and Obstetrics, the Steering Committee met and the basic formulation of the assessment design as outlined above was determined. During the following seven months, three instruments were developed. These were: (a) the "Faculty Assessment of the Teaching Aids Package" form, (b) the "Student Assessment of the Teaching Aids Package" form, and (c) the "Teaching Program Assessment" form. Evaluation, Teaching and Evaluation Workshop Following the development of instruments to assess program parameters and faculty/student reactions to the Teaching Aids Package, a number of techniques to gather uata were used — mailings, telephone survey techniques, and district workshops. The most effective means to gather data proved to be district evaluation workshops. The location, dates, and number of participants who attended district workshops are shown in Table 2. At the workshops, emphasis was placed on obtaining survey data from the faculty representatives attending. In addition, the groups focused on the sharing of ideas and techniques for improving instruction. Educational consultants were secured to add professional educational input to participant discussions. A recent consequence of the workshops has been the development and testing through cooperative efforts of new instructional materials produced by workshop participants. A number of products are being produced and tested at the present time for possible use throughout the United States and Canada via the existing network capabilities. 14 TABLE 2 STEERING COMMITTEE FOR COOPERATIVE TEACHING WORKSHOPS Location Dates Number Attending University of California at Los Angeles Stanford University University of Minnesota University of Indiana Lister Hill National Center for Biomedical Communication National Library of Medicine University of Utah University of Pennsylvania University of Pittsburgh University of Chicago Michigan State University University of Oklahoma December 10, 1971 June 14-17, 1972 November 21-22, 1972 November 30, 1972 December 1. 1972 December 8-9, 1972 December 11, 1972 January 26. 1973 May 7, 1973 June 4-6, 1973 June 27,1973 12 15 13 14 9 15 15 10 13 18 Total Participants 143 Data concerning faculty reactions were collected for the academic years 1971-1972 and 1972-1973. These data will be presented later. However, faculty reactions to the Teaching Aids Package in 1971 with specific data were reported by the Steering Committee.(35) Major conclusions reached from that study were: 1. A majority of faculty members responding to the questionnaire were senior members at participating institutions. This indicated a high level of faculty concern with new teaching methods in general and with the Teaching Aids Package in particular. 2. Most respondents felt that materials in the Teaching Aids Package were helpful to students in acquiring necessary knowledge about obstetrics and gynecology. 3. Students were being given an opportunity to use the materials; however, they were not required to do so. Students had available to them some alternatives in educational efforts. 4. Materials appeared to be useful in fulfilling teaching objectives, as little negative criticism was presented and a large majority of respondents indicated that more teaching material should be produced and made available. A majority of the faculty state that they would spend departmental funds to purchase additional materials, and, indeed, some departments had purchased additional items fiom the original Teaching Aids Package to expand their use in teaching programs. 15 5. Many faculty made suggestions as to content areas for which materials were not included in the original Teaching Aids Package and for which a need existed. 6. Modality preferences of faculty members were noted by the Steering Committee. This confirmed an original bias of the Steering Committee that different faculty members would prefer different modalities depending upon their local circumstances/55,/ To date, information about undergraduate teaching programs in departments of obstetrics and gynecology, faculty changes over time, and student reactions to the Teaching Aids Package have not been reported. 16 CHAPTER 4 THE IMPACT OF INNOVATIONS IN OBSTETRICS AND GYNECOLOGY UNDERGRADUATE TEACHING PROGRAMS One means of determining the impact of an educational innovation is to focus on output results caused by the stimulus provided. The Steering Committee was cognizant of the difficulty of attempting to associate causally educational changes with the Teaching Aids Package dissemination over a three-year period (1970-71 to 1972-73). It was evident that educational incentives from efforts by other groups as well as spontaneously created programs, could evoke educational improvement. Results have been reported by Rulin and Chez from the "Pittsburgh Package," distributed to 55 medical schools prior to the dissemination of the Teaching Aids Package.(32) A five-year study of teaching programs by Johnson also may have influenced changes. (See Appendix A.) In addition, during the dissemination and implementation period, the American Association of Obstetricians and Gynecologists Foundation, Incorporated (AAOG) through the Kennedy Foundation Curriculum Committee, published a Guide to a Basic Curriculum in Obstetrics and Gynecology. This publication listed 40 terminal objectives and a number of enabling objectives. It included case studies representing the basic concepts determined by that Committee as the basis for an undergraduate obstetrical and gynecological teaching program./20,/ These efforts, fully embraced by the Steering Committee membership, certainly influenced the educational changes which were noted by the Steering Committee over the study period. In addition, a distribution of instructional materials by the Council on Residency Education in Obstetrics and Gynecology (CREOG), the publication of the motion picture catalog by the American College of Obstetrics and Gynecology (ACOG), and the efforts of the Southern Medical Schools Consortium in developing and disseminating self-instructional materials, probably contributed to the changes noted. Purpose and Methodology It was the purpose of the teaching programs assessment study to monitor the impact of the Teaching Aids Package on teaching programs in participating departments of obstetrics and gynecology. As noted, other groups undoubtedly influenced some of the changes found. Unfortunately, there is no realistic manner in which the impact can be measured accurately and credit appropriately given to each group involved based upon each group's contribution. Even with such problems, it was determined that the information was worth gathering and reporting and that the efforts of the Steering Committee in its dissemination contributed in some way to changes found. Data were collected for the academic year 1970-71 prior to the period when the Teaching Aids Package was distributed. A second assessment of teaching programs was made during the 1972-73 academic year. Therefore, the two sets of data represented a three-year study. 17 Instruments Used and Returns Obtained During the period June 1971 to October 1971, the Steering Committee devised a methodology for collecting information regarding teaching programs in the participating institutions. This assessment occurred concurrently with the dissemination of the Teaching Aids Package. Contact persons were asked to respond for the year previous to its arrival. The Steering Committee was interested in four descriptive areas concerning teaching programs in obstetrics and gynecology. The first area determined characteristics of the clerkship with respect to students and student/faculty contact hours; the second was a description of the evaluation instruments used in each department; the third was a list of the hardware holdings within the department for educational purposes; and the fourth was a list of the software or other educational media held and used by the departments. During the second year of the project, this instrument was revised somewhat but still sought the same information. Assessment Results During the 1970-71 academic years, 97 departments reported about their teaching programs. At that time, there were 116 medical schools in the Cooperative Teaching Network so the return rate was 84 percent. During the 1972-73 academic year, 95 departments reported on their teaching programs. Although the network had grown to 130 schools, 14 of the schools were new to the network and did not have students, thereby reducing the number of teaching program assessment possibilities to 116, which represented an 82 percent return rate. Statistical Treatment of Data As noted earlier in Chapter 3, the nature of the teaching program study as well as information collected from faculty and students (see Chapters 5 and 6) was (a) to gather baseline descriptive information about undergraduate learning programs in obstetrics and gynecology through empirical research with an emphasis on assessing faculty and student reactions to the Teaching Aids Package, and (b) to assist the Steering Committee in making decisions about activities they should undertake to enhance educational innovations and to further the use of the newer educational technology for departments in the network. During the study period, it was not feasible or appropriate, given the "state of the art" in departments of obstetrics and gynecology in all medical colleges in the United States and Canada, to attempt an experimental-control group research program. The approach taken by the Steering Committee was to demonstrate to faculty, and hopefully to some students, a variety of mediated learning strategies. Each unit in the Teaching Aids Package was independent of all other units. A post-test for all units was not available and, in all probability, would not be valid if one were available since a "Sampler Package" is very different from an integrated instructional package based on specific objectives. All data reported are descriptive in nature. In addition, as assessments occurred over time, the purpose was empirical descriptive research. While percentage figures and, in some cases, means were determined, assessment of change over time under varying conditions could best be determined using the Chi-square statistic. This statistical technique allows data to be cast in a matrix where each cell can be analyzed to determine the direction of changes noted, especially when statistical significance using the 18 Chi-square is found.(41) Both the .05 and the .01 levels of significance were considered as criteria for data analyzed. The .05 level means there exists a five percent possibility that changes observed are due to chance or uncontrolled influences. Therefore, changes reaching the .05 level had a statistical confidence level of 95 percent, and the changes reaching the .01 level had a statistical confidence level of 99 percent/47,1 It should be noted that analyzed data which did not reach the .05 or .01 levels using the Chi-square statistic could be predictive of trends or changes occurring. However, a minimum level was set at the .05 level as it is generally used as the criterion point in educational research/47,/ Clerkship Characteristics Over a three-year period, several changes were noted in clerkship characteristics. These changes are shown in Table 3. A Chi-square statistic was used to determine differences between data for academic years 1970-71 and 1972-73. A significant Chi-square for scheduled student-faculty contact hours per week was found. The direction indicated that faculty were spending significantly more time with students during the second assessment period. Changes in Evaluation Procedures Changes were also found in evaluation procedures used by departments. Data are shown in Table 4. A significant Chi-square was found for departments using written essays as a measuring instrument. The direction for this difference was that departments, as a whole, were using fewer written essay questions in their evaluation program. Another significant Chi-square was found for departments using rating sheets for clinical performance. The direction for this change indicated that fewer departments were using rating sheets for clinical performance as part of their evaluation program. Changes in Hardware (Machine) Holdings Changes in departmental holdings of hardware (machines) are shown in Table 5. Although significance at the .05 level or beyond as measured by Chi-square statistics was not found in hardware holdings, several areas of growth and one area of decrease were noted. Departments increased ownership of slide projectors and learning carrels, and decreased use of learning carrels in the medical school library. Changes in Software (Teaching Aids) Holdings Changes in software holdings, e.g., motion pictures, tape and tape-slide programs, etc., were found. Data are shown in Table 6. A significant Chi-square was found for nontextbook or nonjournal software ownership. The direction indicated that more departments had these types of materials available than in the first assessment period. A second significant Chi-square was found for models owned by departments. The direction indicated that more departments owned teaching models during the second assessment period. It should be noted that software 19 sent in the Teaching Aids Package was not counted for the 1972-73 academic year and that these data indicate that departments have significantly acquired new and other materials. Summary Effective changes were brought about by the dissemination of the Teaching Aids Package modified, as indicated earlier, by other forces interacting on the discipline of obstetrics and gynecology. These changes were found to be significant at or beyond the .05 level as measured by the Chi-square statistic and are listed below: 1. There was an increase in student-faculty contact hours. 2. There was a decrease in the number of departments using written essay questions for evaluation purposes. 3. There was a decrease in the number of departments using rating sheets for clinical performance in departmental evaluation programs. 4. There was an increase in nontextbook or nonjournal software acquisitions in departments even after excluding materials sent to departments in the Teaching Aids Package. 20 TABLE 3 CHANGES IN CLERKSHIP CHARACTERISTICS OF UNDERGRADUATE TEACHING PROGRAMS IN OBSTETRICS AND GYNECOLOGY, 1970-71 (N = 97) AND 1972-73 (N = 95) Mean Percent Yes Percent No Category (70-71) (72-73) (70-71) (72-73) (70-71) (72-73) Number of Students in Clerkship 17 17 Number of Clerkships Held Per Year * i Clerkship Is Required (Yes) or Elective (No) * 92 * 8 Length of Clerkship (In Weeks) 6 1 'Scheduled Student-Faculty Contact, Hours Per Week 15 35 Operating Room Time * 7 Clinic Time * 11 Seminar Time * 4 Lecture Time * 3 Rounds Time * 4 Tutorial or Preceptor Time * 4 Time Spent Other Activities * i Departmental Objectives in Written Form 70 61 30 39 ♦Data not available for 1970-71 **Significant beyond the .01 level 21 TABLE 4 CHANGES IN EVALUATION PROCEDURES USED ON AN UNDERGRADUATE CLINICAL CLERKSHIP IN OBSTETRICS AND GYNECOLOGY, 1970-71 (N = 97) AND 1972-73 (N = 95) Percent Yes Percent No Category (70-71) (72-73) (70-71) (72-73) Departments Using Objectives Tests 39 52 61 48 *Departments Using Written Essays 56 25 44 75 Departments Using Oral Examinations or Faculty Evaluations 77 77 23 23 *Departments Using Rating Sheets for Clinical Performance 93 64 7 36 Departments Using Letter Grades 27 14 73 86 Departments Using Number Grades 21 16 79 84 Departments Using Pass/Fail/Honors Grading System 77 79 23 21 ♦Significant beyond the .01 level 22 TABLE 5 CHANGES IN HARDWARE (MACHINE) HOLDINGS USED IN UNDERGRADUATE TEACHING PROGRAMS IN OBSTETRICS AND GYNECOLOGY, 1970-71 (N = 97) AND 1972-73 (N = 95) Percent Yes Percent No Category (70-71) (72-73) (70-71) (72-73) Departments Owning Motion Picture Projector(s) 77 80 23 20 Departments Owning Slide Projector(s) 86 94 14 6 Departments Owning Audiotape Player(s) 72 83 28 17 Departments Owning Videotape Player(s) or Monitor(s) * 38 * 62 Departments Owning Computer Terminals * 10 * 90 Departments Owning Overhead Projector(s) * 33 * 67 Departments Owning Learning Carrel(s) 59 68 41 32 Departments Using Learning Carrel(s) in Library, Etc. 31 18 69 82 *Data not available for 1970-71 23 TABLE 6 CHANGES IN SOFTWARE (TEACHING AIDS) HOLDINGS USED IN UNDERGRADUATE TEACHING PROGRAMS IN OBSTETRICS AND GYNECOLOGY, 1970-71 (N = 97) AND 1972-73 (N = 95) (Software Sent in APGO Teaching Aids Package Not Included) Mean Percent Yes Percent No Category (70-71) (72-73) (70-71) (72-73) (70-71) (72-73) *Non-Textbook or Journal Software in Departments Motion Picture Films Owned by Departments Tape-Slide Programs Owned by Departments Programmed Texts Owned by Departments Audiotape Programs Owned by Departments Videotape Programs Owned by Departments *Models Owned by Department Handouts Given to Students by Departments Other Media Software Owned by Departments 3 26 1 2 15 17 1 1 1 3 1 1 1 2 ** 2 ** 3 68 98 57 56 64 70 30 35 33 38 23 24 40 76 ** 49 ** 100 32 2 43 44 36 30 70 65 67 62 77 76 60 24 ** 51 ** Q ♦Significant at .01 level **Data not available 24 CHAPTER 5 FACULTY REACTIONS TO THE APGO TEACHING AIDS PACKAGE The Steering Committee wished to determine the reactions of faculty members in departments of obstetrics and gynecology in medical schools in the United States and Canada to the instructional units in the Teaching Aids Package. Earlier, in Chapter 3, conclusions reached during the first year of the dissemination regarding faculty reactions were reported. (See also reference 25.) Purpose, Methodology, and Return Regardless of the viability of any unit of prepackaged instructional materials, the reactions of the faculty to such materials will depend upon its acceptance and use. The Steering Committee had designed the original Teaching Aids Package to provide for faculty examples of various modalities which could be used in teaching programs in order to indicate to them multiple options available for their use in enhancing teaching programs. Since the goal was to demonstrate to faculty a variety of modalities requiring Little hardware gear-up, no effort was made to integrate each Teaching Aids Package unit in terms of meeting an overall instructional goal. Rather, each unit selected for inclusion in the package was independent of other units within the package in terms of instructional objectives. Therefore, the Teaching Aids Package did not constitute a unified core of information which could be used as basic didactic information in obstetrics and gynecology. The purposes of the assessments conducted during academic years 1971-72 and 1972-73 were to obtain a sample of faculty reactions to the Teaching Aids Package, to determine if faculty showed specific modality preferences, and to see if they found materials included in the Teaching Aids Package useful in enhancing undergraduate instructional programs. In addition, the assessments sought other content areas for which materials needed to be developed, inquired whether faculty were willing to purchase new materials out of departmental funds if future packages were to be developed, and requested an overall evaluation of the Teaching Aids Package. During academic year 1971-72, Steering Committee members and consultants from a number of Research in Medical Education Centers worked to develop a survey form which would yield information concerning the questions noted above. After several drafts and revisions, the "Faculty Assessment of the Teaching Aids Package" form was accepted by the Steering Committee. These forms were administered to contact persons from departments of obstetrics and gynecology who had received the Teaching Aids Package. It was felt that the person who received the package would be in the best position to evaluate the units contained within it. In some cases, faculty were sent forms by mail and were asked to complete and return them to the Project Office. In other cases, faculty completed forms during a series of evaluation workshops held in their geographical 25 regions. Follow-up mailings were made in order to obtain as large a response pattern as possible. The first assessment period, 1971-72, yielded returns from 87 of the 116 departments which had received the Teaching Aids Package, for a return rate of 75 percent. As previously indicated, additional packages were mailed during the 1972-73 academic year. In all, 129 packages were sent to departments of obstetrics and gynecology in medical schools in the United States and Canada. Of the 129 schools, 124 schools were asked to respond to the Faculty Assessment of the Teaching Aids Package form. The five remaining schools were not asked to respond because the analysis of collected information was in progress when materials were sent. Of the 124 forms distributed, 103 were returned for a return rate of 89 percent for the 1972-73 academic year. Changes in Student-Faculty Contact Hours The significant increase in scheduled student-faculty contact hours per week was discussed earlier in the review of teaching program assessment studies. Amplification of this point is noted in Table 7. It should be noted that only one faculty member from each department was asked to complete the Faculty Assessment of the Teaching Aids Package form. Generally, persons who attended workshops completed these forms, and their responses did not reflect how each faculty member in his department could have responded. The information, then, is a sample of those faculty who seemed to be the most involved in the undergraduate teaching program at their institutions. TABLE 7 FACULTY TIME SPENT IN INSTRUCTION WITH UNDERGRADUATE MEDICAL STUDENTS IN OBSTETRICS AND GYNECOLOGY, 1971-72 AND 1972-73 Category Percent Faculty 1971-72 (N = 87) Percent Faculty (1972-73 (N = 103) Percent Difference Zero Percent 0 1 + 1 1 to 25 Percent 31 38 +7 26 to 50 Percent 35 28 -7 51 to 75 Percent 9 10 + 1 76 to 100 Percent 0 6 +6 No Response 25 17 -8 Totals 100 100 26 Chi-Square Comparison A Chi-square statistic program was used to determine if significant differences in the data reported in 1971-72 and 1972-73 existed. There were areas where significant statistical differences were noted. The Chi-square studies were limited to media categories in order to determine if modality preferences were indicated. However, changes in faculty reactions to each unit in the Teaching Aids Package are shown in Appendix B. Percentages were computed by multiplying the number of units in a media category by the number of usable forms returned, then dividing that value into the number of respondees who answered the item "yes," and then multiplying by 100 to obtain a percent value. Faculty Familiarity with Teaching Aids Changes in faculty familiarity with units in the Teaching Aids Package were found. The changes are shown in Table 8. A significant Chi-square was found in the area of written materials, a media category with which faculty indicated more familiarity. Faculty familiarity with materials in other media categories did not reach the .05 level of significance although an increase across most media categories was found. TABLE 8 COMPARISON BY MEDIA TYPE OF FACULTY RESPONSES ABOUT FAMILIARITY WITH UNITS IN TEACHING AIDS PACKAGE, 1971-72 and 1972-73 Media Category Percent Yes 1971-72 (N = 87) Percent Yes 1972-73 (N= 103) Percent Difference Written Material (8 Units) 66 67 + 1* Tape-Slide Programs (6 Units) 79 82 +3 Motion Pictures (5 Units) 76 85 +9 Audiotape Programs (3 Units) 76 75 -1 ♦Significant Beyond the .05 Level 27 Availability of Units to Students Changes were found in the willingness of faculty to allow students to have available to them units in the Teaching Aids Package. The changes are shown in Table 9. In terms of faculty's making materials in the various media categories available for student use, only one area of significance was found as measured by the Chi-square statistic. Not surprisingly, this significant Chi-square was in the area of motion pictures, a modality which faculty indicated they were making more available for student usage. No statistical significance was found for the other media category modalities. However, shifts noted were in the direction of making all media categories more available for student usage. TABLE 9 COMPARISON BY MEDIA TYPE OF FACULTY RESPONSES ABOUT AVAILABILITY FOR STUDENT USAGE OF UNITS IN TEACHING AIDS PACKAGE, 1971-72 AND 1972-73 Percent Yes Percent Yes Percent Media Category 1971-72 (N = 87) 1972-73 (N= 103) Difference Written Material (8 Units) 40 42 +2 Tape-Slide Programs (6 Units) 59 65 +6 Motion Pictures (5 Units) 68 76 +8* Audiotape Programs (3 Units) 46 51 +5 *Significant Beyond the .05 Level Required Use of Units Changes were found regarding faculty reactions in requiring students to use units in the Teaching Aids Package. Changes are shown in Table 10. Three areas in media categories concerning the faculty's requiring students to use materials disseminated in the Teaching Aids Package showed change over the two-year study. Significant Chi-squares were reached in written materials, tape-slide programs, and motion picture categories for which the shifts were in the direction of not requiring students to use materials in the Teaching Aids Package. Although significance was not reached in the media category of audiotape programs, the direction was the same as for the three previously mentioned categories. Units Helping Students to Develop General Knowledge in Obstetrics and Gynecology Changes in faculty reactions to units in the Teaching Aids Package in helping students to develop general knowledge in obstetrics and gynecology were found. Changes are shown in Table 11. 28 TABLE 10 COMPARISON BY MEDIA TYPE OF FACULTY RESPONSES ABOUT REQUIRING STUDENTS TO USE UNITS IN THE TEACHING AIDS PACKAGE, 1971-72 AND 1972-73 Media Category Percent Yes 1971-72 (N = 87) Percent Yes 1972-73 (N = 103) Percent Difference Written Material (8 Units) 13 6 -7* Tape-Slide Programs (6 Units) 25 16 _9* Motion Pictures (5 Units) 46 40 -6* Audiotape Programs (3 Units) 16 12 -4 ♦Significant Beyond the .01 Level TABLE 11 COMPARISON BY MEDIA TYPE OF FACULTY RESPONSES ABOUT UNITS IN TEACHING AIDS PACKAGE HELPING DEVELOP GENERAL KNOWLEDGE OF OBSTETRICS AND GYNECOLOGY, 1971-72 AND 1972-73 Media Category Percent Yes 1971-72 (N = 87) Percent Yes 1972-73 (N= 103) Percent Difference Written Material (8 Units) 49 40 _9* Tape-Slide Programs (6 Units) 68 63 -5 Motion Pictures (5 Units) 67 74 +7 Audiotape Programs (3 Units) 54 47 -7* ♦Significant Beyond .05 Level Chi-square studies were also completed to determine if shifts occurred during the two years regarding the materials distributed in the Teaching Aids Package as pertained to helping students to develop general knowledge. Significantly fewer faculty indicated that the media categories of written materials and audiotape programs helped students to develop general knowledge in obstetrics and gynecology. No significant Chi-squares were found in the areas of tape-slide and motion picture programs for the question posed. 29 Units Helping Students to Answer Questions of Residents and Staff Changes in faculty reactions to units in the Teaching Aids Package in helping students to answer questions of residents and staff were found. Changes are shown in Table 12. Chi-square studies were also completed to determine if shifts occurred during the two years regarding the materials distributed in the Teaching Aids Package as pertained to helping students to answer questions of residents and staff in the area of obstetrics and gynecology. Significantly fewer faculty indicated that the media categories of written materials, tape-slide programs, and audiotape programs helped students to answer questions of residents and staff. Although no significant Chi-squares were found in the areas of motion picture programs for the question posed, shifts were in the direction that faculty felt that motion picture units helped in this area. TABLE 12 COMPARISON BY MEDIA TYPE OF FACULTY RESPONSES ABOUT UNITS IN TEACHING AIDS PACKAGE HELPING ANSWER QUESTIONS OF RESIDENTS AND STAFF, 1971-72 AND 1972-73 Percent Yes Percent Yes Percent Media Category 1971-72 (N = 87) 1972-73 (N= 103) Difference Written Material (8 Units) 42 32 -10** Tape-Slide Programs (6 Units) 59 51 __8** Motion Pictures (5 Units) 56 63 +7 Audiotape Programs (3 Units) 43 39 _4* ♦Significant Beyond the .05 Level ♦♦Significant Beyond the .01 Level Units Helping Students to Answer Questions on Formal Examinations in Obstetrics and Gynecology Changes in faculty reactions to units in the Teaching Aids Package in helping students to answer questions on formal examinations were found. Changes are shown in Table 13. 30 TABLE 13 COMPARISON BY MEDIA TYPE OF FACULTY RESPONSES ABOUT UNITS IN TEACHING AIDS PACKAGE HELPING ANSWER QUESTIONS ON A FORMAL EXAMINATION, 1971-72 AND 1972-73 Media Category Percent Yes 1971-72 (N = 87) Percent Yes 1972-73 (N= 103) Percent Difference Written Material (8 Units) 38 30 -8* Tape-Slide Programs (6 Units) 56 48 -8* Motion Pictures (5 Units) 53 58 +5 Audiotape Programs (3 Units) 41 36 -5* ♦Significant Beyond the .01 Level Chi-square studies were also completed to determine if shifts occurred during the two years regarding the materials distributed in the Teaching Aids Package as pertained to helping students to answer questions on examinations in the area of obstetrics and gynecology. Significantly fewer faculty indicated that the media categories of written materials, tape-slide programs, and audiotape programs helped students to answer questions on examinations. Although no significant Chi-squares were found in the areas of motion picture programs for the question posed, shifts were in the direction that faculty felt that motion picture units helped in this area. Appropriateness of Unit Media Formats Changes in faculty reaction to the appropriateness of the media modality for the conveyance of content were found. Changes are shown in Table 14. In the area of the appropriateness of the media format for the content being conveyed, significant Chi-squares over the two-year period were found in the areas of written materials, tape-slide programs, and audiotape programs for which faculty indicated that the modalities were less appropriate. Again, faculty perceived motion pictures as more appropriate modalities for the conveyance of the content; however, statistical significance was not found. Overall Faculty Reactions Toward New Materials Over the two-year period, faculty were asked if they fell the need for a future Teaching Aids Package and if they would purchase a new package from departmental funds if the Steering Committee were to undertake the effort needed to construct one. Changes in faculty reaction are shown in Table 15. 31 TABLE 14 COMPARISON BY MEDIA TYPE OF FACULTY RESPONSES ABOUT APPROPRIATENESS OF THE FORMAT OF UNITS IN THE TEACHING AIDS PACKAGE FOR THE CON- VEYANCE OF THE CONTENT, 1971-72 AND 1972-73 Percent Yes Percent Yes Percent Media Category 1971-72 (N = 87) 1972-73 (N= 103) Difference Written Material (6 Units) 45 40 -5* Tape-Slide Programs (6 Units) 63 62 -1* Motion Pictures (5 Units) 64 72 +8 Audiotape Programs (3 Units) 43 41 -2* ♦Significant Beyond the .05 Level TABLE 15 COMPARISON OF FACULTY RESPONSES TO NEED FOR FUTURE TEACHING AIDS PACKAGES AND WILLINGNESS TO PURCHASE NEW MATERIALS, 1971-72 AND 1972-73 Percent Yes Percent Yes Percent Area 1971-72 (N = 87) 1972-73 (N= 103) Difference Need for Future Teaching Aids Package 85 87 +2 Future Purchase of Teaching Aids Package 74 84 + 10 No significant Chi-squares were found for the data presented above; however, the direction of change was favorable for constructing a new Teaching Aids Package. Most departments indicated they would purchase such materials from departmental funds. It should be noted, however, that many faculty who responded "yes" to purchasing new materials wrote comments that they would prefer to review units first and purchase only those items needed for then teaching programs. Overall Faculty Reaction to Teaching Aids Package Changes in faculty reactions to the Teaching Aids Package were found. Changes are shown in Table 16. 32 TABLE 16 COMPARISON OF FACULTY RESPONSES ABOUT OVERALL RATING OF THE TEACHING AIDS PACKAGE, 1971-72 AND 1972-73 Percent Percent Percent Responses 1971-72 (N = 87) 1972-73 (N= 103) Difference Excellent 36 26 -10 Good 47 53 +6 Fair 12 11 -1 Poor 0 1 + 1 No Response 6 9 +3 No significant Chi-squares were found for data shown above. It does appear that materials were still viable in the second year; however, faculty tended to be somewhat more critical. Summary Faculty reactions to material sent in the Teaching Aids Package were assessed over a two-year period. Changes found to be significant at or beyond the .05 level as measured by the Chi-square statistic are listed below: 1. Faculty indicated significant gain in familiarity with units in the written materials media category. 2. Faculty indicated significant gain in making units in the motion picture category available for student use. 3. Faculty indicated significant decrease in requiring students to use units in all media category areas except audiotape programs. 4. Faculty indicated significant decrease as to the value of units in students' developing general knowledge of content, answering questions of residents and staff, and answering questions on formal examinations and appropriateness of media format for units in the written materials, tape-slide, and audiotape media categories. 33 CHAPTER 6 STUDENT REACTIONS TO THE APGO TEACHING AIDS PACKAGE The Teaching Program Assessment form and the Faculty Assessment of the Teaching Aids Package form were sent to all medical school departments of obstetrics and gynecology participating in the project. Due to the large number of students in the clinical vears in obstetrics and gynecology (approximately 11,000), it was decided to gather information from only a small number of schools. It should be pointed out that the prime purpose for the dissemination of the Teaching Aids Package was not to provide students with a basic set of materials which could be implemented by all schools; rather, it was the intention of the Steering Committee to provide these materials in various modality formats in order to demonstrate to faculty alternative teaching/learning strategies. Even so, it was recognized that some faculty might make these materials available to students and that a representative group of students' reaction to these materials would be beneficial in determining possible future directions. It was pointed out earlier that, in nearly all teaching programs, the Steering Committee felt that faculty would first review materials and decide if the materials were appropriate for their teaching program. Accordingly, students probably would not have the opportunity of making a thorough review of all materials, as would faculty contact persons. Even with this limitation, it was determined to go ahead with this portion of the study. In order to maintain comparability of information, it was decided to attempt to make the Faculty Assessment of the Teaching Aids Package form and the Student Assessment of the Teaching Aids Package form as similar as possible, even though it was recognized that students would not have reviewed or had available to them many of the materials that faculty members had reviewed. Purpose, Methodology, and Sample The lack of information regarding student reactions to teaching/learning materials could seriously weaken the impact of any modality employed by a department. Medical students as a general rule, tend to come from the upper portions of their undergraduate classes and may be very discriminate as to their selection of learning strategies. Factors which may contribute to this idea include increased standards of living in the United States where children have grown up in an "electric" age of television, battery-operated radios, and tape recorders, and where public schools and colleges have utilized newer technologies in educational programs. With this in mind, the Steering Committee felt that student feedback about the Teaching Aids Package could have a beneficial effect as the Committee decided about possible further disseminations. As with the Faculty Assessment of Teaching Aids Package form, a sample of students for the academic years 1971-72 and 1972-73 was obtained. Again, the plan would allow assessment of the materials to occur over time under various contingent conditions. All departments are not 35 alike, and it was anticipated that a wide sampling would be desirable in order to determine the efficacy of the Teaching Aids Package in a variety of learning environments. An attempt was made to obtain reactions from students from at least one department within each district of the Cooperative Teaching Network. In some cases, district chairmen decided to gather data from more than one school. However, a plotting of responses from departments that supplied student responses across the country matched closely with the actual dispersion of medical schools throughout the United States and Canada for both years when the forms were administered. During the 1971-72 academic year, Student Assessment of Teaching Aids Package forms were obtained from 164 students who were from 23 departments of obstetrics and gynecology in medical schools in the United States and Canada. It should be noted that some of the departments forwarded only a few forms from students and that the total clerkship for some of the departments concerned was not surveyed. During the 1972-73 academic year, 12 schools surveyed students on their clinical clerkships and a total of 161 students responded. Returns for the second year of the study were considered more reliable as the 12 schools reporting appeared to include all students on the clerkship where the forms were administered. In light of the above, the reader should be aware that the forms obtained did not represent a scientifically designed random sample of students. Rather, it represented more of a regional sample design. Chi-Square Comparison A Chi-square statistic program was used to determine if significant differences in the data reported in 1971-72 and 1972-73 existed. There were areas where significant statistical differences were noted. The Chi-square studies were limited to media categories in order to determine if modality preferences were indicated. However, changes in student reactions to each unit in the Teaching Aids Package are shown in Appendix C. The same procedures as used with faculty data were used to analyze information. Percentages were computed by multiplying the number of units in a media category by the number of usable forms returned, then dividing that value into the number of respondees who answered the item "yes," and then multiplying by 100 to obtain a percent value. Student Use of Units Changes in student use of units in the Teaching Aids Package were found. Changes are shown in Table 17. In terms of students using materials in the Teaching Aids Package, two significant Chi-squares were found. These were in the areas of tape-slide and motion picture programs. In terms of the tape-slide programs, the direction was that fewer students were using those programs while the direction in terms of motion picture programs indicated that more students were using those units. Although significance was not obtained in the areas of written materials and audiotape programs, slight differences did occur. Essentially, student usage of these materials remained fairly constant over the two-year period. 36 TABLE 17 COMPARISON BY MEDIA TYPE OF STUDENT RESPONSES ABOUT USING UNITS IN TEACHING AIDS PACKAGE, 1971-72 AND 1972-73 Percent Yes Percent Yes Percent Media Category 1971-72 (N = 164) 1972-73 (N= 161) Difference Written Material (8 Units) 28 22 -6 Tape-Slide Programs (6 Units) 54 37 -17* Motion Pictures (5 Units) 56 54 _">* Audiotape Programs (3 Units) 47 34 -13 ♦Significant Beyond the .01 Level Required Student Use of Units Changes in student responses to faculty's requiring them to use units in the Teaching Aids Package were found. Changes are shown in Table 18. The media categories of written materials, tape-slide programs, motion picture programs, and audiotape programs were all found to be significantly different using Chi-square statistics. The direction was that fewer students were being required lo use media in all media categories. TABLE 18 COMPARISON BY MEDIA TYPE OF STUDENT RESPONSES ABOUT THEIR REQUIRED USAGE OF UNITS IN TEACHING AIDS PACKAGE, 1971-72 AND 1972-73 Media Category Percent Yes 1971-72 (N = 164) Percent Yes 1972-73 (N = 161) Percent Difference Written Material (8 Units) 8 1 -7* Tape-Slide Programs (6 Units) 10 3 -7* Motion Pictures (5 Units) 17 7 -10* Audiotape Programs (3 Units) 11 1 -10* ♦Significant Beyond the .01 Level 37 Units Helping Students to Develop General Knowledge Changes were found in student reaction toward units in the Teaching Aids Package in helping them to gain general knowledge in obstetrics and gynecology. Changes are shown in Table 19. Student reactions over the two-year period to the Teaching Aids Package as it helped them to develop general knowledge in obstetrics and gynecology revealed no significant Chi-square changes. The direction was that fewer students in the 1972-73 academic year answered positively to this question. TABLE 19 COMPARISON BY MEDIA TYPE OF STUDENT RESPONSES ABOUT USEFULNESS OF UNITS IN TEACHING AIDS PACKAGE IN DEVELOPING GENERAL KNOWLEDGE IN OBSTETRICS AND GYNECOLOGY, 1971-72 AND 1972-73 Percent Yes Percent Yes Percent Media Category 1971-72 (N= 164) 1972-73 (N= 161) Difference Written Material (8 Units) 23 19 -4 Tape-Slide Programs (6 Units) 47 32 -15 Motion Pictures (5 Units) 53 51 -2 Audiotape Programs (3 Units) 43 32 -11 Units Helping Students to Answer Questions of Residents and Staff Changes were found in student reaction toward units in the Teaching Aids Package in helping them to answer questions of residents and staff. Changes are shown in Table 20. Student reactions over the two-year period to the Teaching Aids Package as it helped them to answer questions of residents and staff in obstetrics and gynecology were analyzed using the Chi-square statistic. A significant Chi-square was found for the media category tape-slide programs. However, the direction was that fewer students in the 1972-73 academic year answered positively to this question for other media categories also. Units Helping Students to Answer Questions on Formal Examinations Changes were found in student reaction toward units in the Teaching Aids.Package in helping th^m to answer questions on formal examinations. Changes are shown in Table 21. Student reactions over the two-year period to the Teaching Aids Package as it helped them to answer questions on examinations in obstetrics and gynecology revealed little 38 TABLE 20 COMPARISON BY MEDIA TYPE OF STUDENT RESPONSES ABOUT UNITS IN TEACHING AIDS PACKAGE HELPING ANSWER QUESTIONS ON A FORMAL EXAMINATION, 1971-72 AND 1972-73 Media Category Percent Yes 1971-72 (N = 164) Percent Yes 1972-73 (N = 161) Percent Difference Written Material (8 Units) 17 14 -3 Tape-Slide Programs (6 Units) 37 26 -11* Motion Pictures (5 Units) 42 41 -1 Audiotape Programs (3 Units) 33 24 -9 ♦Significant Beyond the .01 Level TABLE 21 COMPARISON BY MEDIA TYPE OF STUDENT RESPONSES ABOUT UNITS IN TEACHING AIDS PACKAGE HELPING ANSWER QUESTIONS ON A FORMAL EXAMINATION, 1971-72 AND 1972-73 Media Category Percent Yes 1971-72 (N= 164) Percent Yes 1972-73 (N = 161) Percent Difference Written Material (8 Units) 15 13 -2 Tape-Slide Programs (6 Units) 32 22 -10 Motion Pictures (5 Units) 38 36 -2 Audiotape Programs (3 Units) 26 20 -6 changes, and no significant Chi-squares were found for the media categories. Yet, the direction was that fewer students in the 1972-73 academic year answered positively to this question. Student Reactions to the Appropriateness of the Media for the Conveyance of the Content Changes were found in student reaction to the appropriateness of the media for the conveyance of the content. Changes are shown in Table 22. 39 TABLE 22 COMPARISON BY MEDIA TYPE OF STUDENT RESPONSES ABOUT APPROPRIATENESS OF THE FORMAT OF UNITS IN THE TEACHING AIDS PACKAGE FOR THE CON- VEYANCE OF THE CONTENT, 1971-72 AND 1972-73 Percent Yes Percent Yes Percent Media Category 1971-72 (N= 164) 1972-73 (N= 161) Difference Written Material (8 Units) 22 18 -4 Tape-Slide Programs (6 Units) 43 29 -14 Motion Pictures (5 Units) 48 48 0 Audiotape Programs (3 Units) 36 28 -8 No significant Chi-squares were found for the two-year study period in terms of the appropriateness of the format for the conveyance of the content according to media categories. However, the direction was that fewer students in the 1972-73 academic year answered positively to this question for all media categories except that of motion pictures for which no change was found. Overall Student Reaction to Teaching Aids Package Changes in student reaction to the Teaching Aids Package and the need for a new Teaching Aids Package were found. Changes are shown in Tables 23 and 24. It was of interest to note that student reactions toward the quality of the Teaching Aids Package and to the need for a future Teaching Aids Package did not change significantly over the two-year period. Therefore, a large majority of the students felt that the Teaching Aids Package was either excellent or good and that a future Teaching Aids Package was needed. Summary Student reactions to materials sent in the Teaching Aids Package were assessed over a two-year period. Changes found to be significant at or beyond the .05 level as measured by the Chi-square statistic are listed below: 1. Students indicated significant decreases in the use of units in the tape-slide and audiotape media categories. 2. Students indicated significant decreases in their being required to use units in all media category areas. 3. Students indicated significant decrease in units in the tape-slide media category helping them to answer questions of residents and staff. 40 TABLE 23 COMPARISON OF STUDENT RESPONSES ABOUT OVERALL RATING OF THE TEACHING AIDS PACKAGE, 1971-72 AND 1972-73 Percent Percent Percent Responses 1971-72 (N= 164) 1972-73 (N= 161) Difference Excellent 29 31 +2 Good 48 53 +5 Fair 15 8 -7 Poor 2 1 -1 No Response 6 7 + 1 TABLE 24 COMPARISON OF STUDENT RESPONSES ABOUT NEED FOR FUTURE TEACHING AIDS PACKAGES Percent Percent Percent Responses 1971-72 (N= 164) 1972-73 (N= 161) Difference Yes 75 77 +2 No 16 17 + 1 No Response 9 6 -3 41 CHAPTER 7 INTRAORGANIZATIONAL COOPERATION AND THE THRUST IN NEW DIRECTIONS The original objectives of the Ad Hoc Committee, later to become known as the Steering Committee for Cooperative Teaching of the APGO, were expanded over the three-year period of its existence. The main objective was to include all departments of obstetrics and gynecology in medical colleges in the United States and Canada in their efforts. The activities of this group have also increased, especially in relation to other organizations serving undergraduate medical education in obstetrics and gynecology, groups representing other disciplines in medical education, and national groups representing all disciplines commonly taught in medical schools and health science centers. It seems appropriate to describe the strategies for obtaining intraorganizational cooperation and input and to indicate new directions being pursued by the Steering Committee. During its first year of operation, the period between July 1969 and July 1970, intensive efforts were made by the membership to develop and select materials. As the Ad Hoc Committee worked with the Association of Professors of Gynecology and Obstetrics and as increased interest was expressed by other medical school departments of obstetrics and gynecology in participating in the activities of the Steering Committee, attention was focused on dissemination of a "Sampler Package" and assessing its impact. These later activities were the focus of the second two years of the group's energy. However, once the Steering Committee had implemented its dissemination and assessment effort, new tasks and thrusts became more critical. Intraorganizational Cooperation In July of 1971, Stead and others published their study on "Educational Technology for Medicine: Roles for the Lister Hill Center."(34) This comprehensive study added direction to the efforts of the Steering Committee. The stimulus for attempting to create a United States and Canadian network involving all departments of obstetrics and gynecology in all medical schools was certainly influenced by that report. A second report entitled "Educational Technology for Medicine: Academic Institutions and Program Management,'Y72// addressed itself to many of the problems faced by users of newer technology for educational purposes. These documents, while addressing a much broader audience, raised relevant issues for the Steering Committee. It should be noted that a Steering Committee member was on the committee that worked on the first document referred to above and that advance copies of the second document were provided to the Steering Committee and certainly influenced directions being pursued. The staff of the Association of American Medical Colleges worked with the Steering Committee in a framework of mutual sharing of information and ideas. In such a manner, the efforts of the Steering Committee and participants on the network were integrated 43 into an interdisciplinary effort in medical education. While it is important to maintain communication lines with other disciplines and organizations which represent all of the disciplines involved in medical education efforts, the present structure of medical education requires even greater efforts to coordinate activities within a specified discipline. The Steering Committee has been most fortunate in avoiding serious internal problems within the discipline throughout its existence. Several factors contributed to its ability to avoid potential conflicts, and the strategy for accomplishing this aim probably needs to be stated. When the Ad Hoc Committee was founded, members of that committee belonged to a variety of national organizations - all of which had some interest in the education of students and practitioners within the discipline. As the Ad Hoc Committee worked at its original objectives, key persons and other organizations were kept informed as to its activities through the Newsletter communication technique. Eventual affiliation with the Association of Professors of Gynecology and Obstetrics was appropriate because persons from other disciplinary organizations held membership in that organization. For example, members of the Council on Residency Education in Obstetrics and Gynecology (CREOG) are associated with the Association of Professors of Gynecology and Obstetrics and are represented in the Steering Committee membership; a member of the Audiovisual Committee of the American College of Obstetrics and Gynecology (ACOG) is a member of the Association of Professors of Gynecology and Obstetrics and of the Steering Committee; several members of the Steering Committee conduct examinations for the American Board of Obstetrics and Gynecology, and two members consult with the National Board of Medical Examiners (NBME); one member has been very closely associated with the American Association of Obstetricians and Gynecologists Foundation, Inc. (AAOG), Kennedy Foundation Curriculum Committee efforts and has served to insure liaison between the efforts of that group and the Steering Committee; and finally, a leading producer in the Southern Medical Schools Consortium, "Self-Instructional Materials Project," recently was invited to join the Steering Committee in order to add the experience of that vigorous group. These important lines of communication to other groups markedly facilitated communication, provided important input, and catalyzed the development of ideas, programs, and a sense of accountability. Even within the Steering Committee, disagreement exists over specific policies. To mediate, as much as possible, potential damaging effects of such honest differences of opinions, one strategy developed by the Steering Committee membership has worked rather well. Basically, it is the concept of centralized decentralization. That is to say, each Steering Committee member, by virtue of his supervision of a number of schools within a geographical district, has been able to determine the evaluative and productive efforts of participants of the network within his district. This has led to a sense of responsibility on the part of each Steering Committee member as to what the proper functions of the central project staff ought to be. resulting in centralized functions being more "service oriented" rather than being "directed oriented" in nature. There is a value system expressed above which is firmly held by the Steering Committee, that decentralized power bases lead to more effective operation for centralized service activities. The Project Center, then, has been designed to serve in a "helping relationship" to the Steering Committee members and the districts they represent. It is also noted that each district chairman serves not only the Steering Committee for Cooperative Teaching of the APGO, but other organizations which have made seminal contributions to medical education within the discipline. Rather than compete with activities of other organizations, it has been a Steering Committee policy to encourage and integrate these activities to serve the network participants. While honest differences of opinion exist and, in all likelihood, will continue to exist, the Steering Committee shall work to bring reality to the concept of 44 cooperative efforts. In the final analysis, the benefits accrue to the medical student, the medical practitioner in the area of obstetrics and gynecology, and ultimately to the patient. Thrusts in New Directions By November 1972, it was apparent to the Steering Committee that its initial objectives were being met and that the assessment procedures and data obtained indicated that the network was, in fact, established and functioning. It was also recognized that the Steering Committee needed to change its emphasis from attention to dissemination and attitudinal assessment back toward its original aims of production, selection, communication, and evaluation in developing a new Teaching Aids Package which could be experimentally tested. By February 1973, efforts in these old, yet new directions were begun. The Steering Committee devised a media evaluation form for evaluating products which were becoming more and more available. These evaluation forms are being collected and copies of them are shared with the Audiovisual Committee of the American College of Obstetrics and Gynecology. By April of 1974, tabulations of these media evaluation forms will be provided to the Office of Audiovisual Educational Development, Bureau of Health Resources Development, and the National Medical Audiovisual Center of the National Library of Medicine to support them in their major clearinghouse efforts. The Steering Committee has selected twelve priority objectives from the Guide to a Basic Curriculum in Obstetrics and Gynecology(20) for which it hopes to identify and/or produce educational materials. In addition, the Committee hopes to develop formative tests related to each item selected and a final summative evaluation instrument to test the learning effectiveness of a new Teaching Aids Package as an integrative instructional system. The twelve terminal objectives selected from the forty terminal objectives published in the document referred to above are listed below. 1. Given an adult female patient, the student will obtain a complete history, including the following: (a) pelvic pain, bleeding, discharge; (b) menstrual history; (c) obstetrical history; (d) contraceptive history; and (e) sexual history. The student will have demonstrated competency in obtaining a complete history if he records information on all items as described in the enabling objectives. 2. Given a female patient, the student will perform a complete pelvic examination and will obtain a cytologic smear of the cervix. 3. Given a patient who has missed two menstrual periods, by means of history and physical findings, the student will be able to diagnose pregnancy. 4. When presented with an uncomplicated obstetrical patient, the student will be able to inform her about: (a) diet; (b) hygiene; (c) exercise; (d) preparation for labor; (e) family planning; (0 danger signs of pregnancy; and (g) use of drugs. 5. Given a patient in labor, the student will be able to monitor the patient until delivery and differentiate between normal and abnormal labor. 6. Given a patient in the postpartum clinic, the student will be able to reassess her at the end of the puerperium by: (a) performing a 45 physical examination; (b) performing a cytologic smear of the cervix: (d) ordering a hematocrit; and (d) counseling with regard to family planning. 7. Given a patient with vaginal bleeding in the first trimester of pregnancy, the student will diagnose and differentiate between abortion and ectopic pregnane}. He will then write out the appronn->te plan of management. 8. Given a patient in third trimester of pregnancy, the student should be able to recognize pre-eclampsia and outline a plan of management. 9. Given a case, the student will be able to make the diagnosis and describe the treatment of vulvovaginitis. 10. Given a patient with symptoms of menopause, the student will be able to: (a) elicit an appropriate medical history; (b) perform an appropriate physical examination; (c) order appropriate laboratory tests; (d) interpret these tests; and (e) make a differential diagnosis. 1 1. Given a patient with dysmenorrhea or the premenstrual tension syndrome, the student will be able to identify the symptoms, ascertain whether there is an organic cause, and plan appropriate management. 1 2. Given actual or illustrative cases, the student will be able to decide whether a contraceptive history should be taken and actually take one, decide whether to prescribe a contraceptive method, decide which contraceptive method is most appropriate for particular patients in illustrative cases, and discuss with patients the advantages, disadvantages, and potential contraindications of contraceptive methods. Individual Steering Committee members have selected objectives and are working independently and in conjunction with one another to identify and/or produce media for them. While it may not be possible for materials to be developed for all objectives, it is hoped that a future Teaching Aids Package will cover a minimum of six of the twelve priority objectives as far as didactic information is concerned. Several groups have been making progress in these areas and some materials are already under development. Materials in the original Teaching Aids Package have been reviewed in terms of their applicability to these new objectives, and the correlation was published in the Cooperative Teaching Bulletin, Volume 3, Number 2, May 1973 issue. In addition, previous review efforts by subcommittees of the Steering Committee identified some excellent materials that will undoubtedly become a part of the new Teaching Aids Package. A great deal of assistance in accomplishing these tasks has been obtained from the Office of Audiovisual Educational Development, Bureau of Health Resources Development, and the National Medical Audiovisual Center, National Library of Medicine. Professional educators from those offices are in the process of reviewing efforts of the Steering Committee in developing materials and are providing critiques as to plans and prototype versions of materials being produced. In addition, Steering Committee district chairmen often rotate the location of their workshops throughout their district, and experts from learning resource centers and divisions ol research in medical education have been, and shall be, very helpful in providing the technical and educational expertise needed to produce quality materials. It is planned that the new Teaching Aids Package will be completed and forwarded to 15 test institutions by April 1974. Resultant data, along with faculty and student 46 reaction instruments, should provide information as to the viability of the new materials under a controlled setting which did not exist prior to the distribution of the original Teaching Aids Package. A strategy for further dissemination of the new Teaching Aids Package will probably not be resolved until the efficacy of the new materials is determined. The Steering Committee is also engaged in two other supportive projects for medical education. It is developing for the American College of Obstetrics and Gynecology a package of materials for learning about human sexuality, contraception, and family planning, and it is anticipated that distribution of this package to all medical schools on the Cooperative Teaching Network will occur prior to the end of 1973. Included with the package will be a post-test to measure knowledge in this area. In addition, the American College of Obstetrics and Gynecology will use these materials in workshops especially designed for physicians other than Board-Certified obstetricians and gynecologists, thereby ensuring that materials in this vital area are readily available to practicing physicians who offer family practice or general medical care to patients. The Steering Committee is also working with Ortho Pharmaceuticals to revise materials prepared for nurse practitioners in the area of obstetrics and gynecology, in order that they may be used with medical students. An earlier review of some of these materials by the Steering Committee led to the conclusion that these materials, slightly modified, could be of value to medical students. A prime reason that the Steering Committee reached such a decision was the fact that the development of materials by Ortho Pharmaceuticals for use with nurse practitioners was based upon a task analysis of functions required to deliver health care in the area of obstetrics and gynecology to patients. While the Steering Committee is not prepared to endorse the idea that some of the functions presently performed by physicians should rest with a less trained individual. they are prepared to work with these materials in order that the medical student can profit from the obviously well-prepared and tested set of learning materials. It is apparent from what we have discussed above that the Cooperative Teaching Network in Obstetrics and Gynecology has much to do. The Steering Committee can contribute; however, the network's eventual course will be decided by the faculty who contribute their time to improving undergraduate education in obstetrics and gynecology, and a continuation of intergroup cooperation. In the final analysis, these two factors are both the great strengths and the possible hazards of a networking approach to cooperative sharing of educational materials. 47 CHAPTER 8 CONCLUSIONS It has been four years since the inception of the idea of cooperative teaching in obstetrics and gynecology by the Steering Committee in July 1969. A basic assumption of the Steering Committee is that medical student teaching is the primary mission of medical college departments. Correlated and integrated interests such as patient care, community service, and research are critical and necessary but must play a secondary role to the teaching mission. The Steering Committee for Cooperative Teaching, Association of Professors of Gynecology and Obstetrics, is working vigorously with its colleagues and other professional groups both in medicine and other behavioral disciplines. Conclusions reached from the four years of experience are: 1. Establishment and Implications of a Biomedical Communication Network. The most significant accomplishment of the Steering Committee is the establishment of the first comprehensive biomedical communication network for educational purposes. All departments of obstetrics and gynecology in all medical colleges in Canada and the United States are members of the network, receive a quarterly Bulletin for Cooperative Teaching, and have received a set of learning materials because of the efforts of the Steering Committee. The districting plan, in which each Committee member has responsibility for coordinating efforts for departments in his geographical area, adds structure and decentralization to a unified effort of the Association of Professors of Gynecology and Obstetrics. This accomplishment required cooperation of Steering Committee members; Council members of the Association of Professors of Gynecology and Obstetrics; professors from each department; the Fund for Medical Education, which provided additional nongovernmental resources to make the concept of networking a reality; and support (both financial and advisory) from the Office of Audiovisual Educational Development of the Bureau of Health Resources Development, and the Lister Hill National Center for Biomedical Communications and the National Medical Audiovisual Center of the National Library of Medicine, which identified the need for providing support to the networking effort of the Association of Professors of Gynecology and Obstetrics. Participating departments had new and imaginative pedagogical techniques. Our study showed that media sent to nearly all departments have been well received and that student data from a sample of 27 departments indicate media have been used by students, although availability of any item was a decision made by local faculty based on local needs. Evidently, faculty felt that student learning could be facilitated using educational media. Workshop sessions and early publications by Steering Committee members emphasized the critical role of faculty in not allowing educational media to substitute for student-faculty contact. Rather, they emphasized the use of educational media to enhance instruction and encourage students and faculty to reach for higher levels of 49 intellectual and skillful application of the arts and science of medicine. The newer technology in medical education should embellish, not replace, faculty-student contact. It seems reasonable to assume that the faculty have been relieved of repetitive tasks and have spent the increased time as shown in our data in meeting individual student needs, challenging the advanced, and helping the laggard. Participants at workshops freely communicated problems and possible solutions to improve teaching programs. They formed relationship which continued after workshops in the preparation of new and better educational materials. The benefits of these production efforts are potentially of great significance. Excellent materials produced by professors working cooperatively are readily available and quickly communicated to all the participants of the network. While the materials being produced and shared may not be of the highest quality, they are in all likelihood improvements over materials sent in the original Teaching Aids Package and materials sent in other distributions by other groups. The Steering Committee recognized that much work remains to be done in order to produce and disseminate truly excellent educational materials, but the future promises closer relationships between professors through communicating at future workshops, continuing dialogue using the Bulletin for Cooperative Teaching, joint projects between schools and consortia, and new, unified, tested Teaching Aids Packages for objectives found in the Guide to a Basic Curriculum in Obstetrics and Gynecology. (20) Recognizing these facts and hopes, the Steering Committee believes the student must have available to him alternative pathways to learning. Assessment procedures need only indicate that some, not all, students find a particular program very helpful in mastering an educational objective. It is the objective that should be the focus, not the method of learning. The medium is not as important as the content of the medium. Our experience with the Teaching Aids Package supports this view. Both faculty and students on the one hand expressed a need for more teaching aids and on the other hand commented that the format was inappropriate. This is not surprising, but refreshing! The packages were selected to demonstrate technology; they were first-generation attempts; they served the purpose of introducing to students and faculty alternate learning pathways. In the future, the emphasis will be on the objectives, the content, and the appropriateness of the format. Close attention must be paid to the transfer of training, i.e., to the application of knowledge in the real world. To ensure this, we will need valid evaluation instruments. Heretofore, resident and staff questions and formal examination questions were not directed to and were not measuring the same ability, knowledge, and skill used in actual practice. Educational programs and assessment programs of health care delivery systems need to be coordinated. 2. Problems in Need of Solution. Although the Cooperative Teaching Network has filled a need to enhance teaching-learning programs in obstetrics and gynecology, the filling of such a need has created new problems which must be faced and resolved. Four major areas appear to be most pressing at this time. They are (1) providing for future dissemination of teaching packages; (2) continuation of Steering Committee production activities; (3) meeting the needs of developing institutions — organizing teaching-learning programs; and (4) ultimate evaluation procedures. 3. Providing for Future Materials. The first Teaching Aids Package disseminated by the Steering Committee was fully paid for by the United States Government. At issue and yet to be resolved is the question, "Who will pay for future disseminations?" While it is possible for many departments to purchase materials, most, because of the extreme strains on medical school budgets, are faced with serious financial cutbacks. In addition, there are some departments which have made little use of the materials from the dissemination, and one may hypothesize that these departments are in the most dire need of such materials. Potential supporters for disseminations to departments on the network 50 are the United States Government, the Canadian Federal Government, private foundations, professional organizations, contributions from business and industry, a pooling of resources from all departments on the network, or combinations of resources. It is not within the wisdom of the Steering Committee to recommend "who should foot the bill," even though users are sharing and must continue' to share the cost. However, the problem needs to be resolved in order to continue and accelerate progress in improving teaching-learning programs in obstetrics and gynecology. In order to accomplish this, mechanisms must be developed by which talented, imaginative, and competent professionals (i.e., subject-matter and pedagogical experts) can combine to produce, test, and refine educational materials so that all students receive the benefits of the best that is available. 4. From Assessment to Evaluation. Early efforts of producers of educational media, working individually, toyed with new ideas and techniques in poorly controlled situations. While materials were shared between producers as early as 1969 on an organized basis, little attempt was made to develop an evaluation procedure utilizing experimental and control groups in obstetrics and gynecology. Because of the efforts of the last four years, such experimentation is now feasible. While attitudes and reactions of both faculty and students to materials will continue to be valid measurement parameters, the evidence of changes in student performance based on methodology of learning must become an integral part of the development of educational materials in obstetrics and gynecology. It is to this end that Steering Committee members will focus their efforts for the next few years. Another major objective for the Steering Committee evolved. The Steering Committee will continue to encourage cooperation between all organizations and consortia interested in the undergraduate curriculum in obstetrics and gynecology. The excellent sessions conducted by the Kennedy Foundation Curriculum Committee and the American Association of Obstetricians and Gynecologists, Inc., in teaching professors how to teach, and the efforts by the Southern Medical Schools Consortium "Self-Instructional Materials Project" in teaching professors to write objectives, write tests for objectives, and write materials which teach the objectives and are tested by the tests, have also contributed to the expertise now evident in departments of obstetrics and gynecology in the United States and Canada. 5. The Elusive, Ultimate Criteria. The final major problem concerns assessment and evaluation of instructional programs. Criteria for the "excellent physician" are not clearly delineated and may never be. However, means of relating teaching programs to improved patient care based on physician intervention are greatly needed. This is the greatest question. Its possible solution rests in the realm of longitudinal educational research. Only by continuing to have a viable group of content and educational experts working cooperatively over time can such solutions be found. It is the intention of the Steering Committee to try to resolve some of these problems and to make the long-term effort required of it. 51 BIBLIOGRAPHY 1. Association of American Medical Colleges, "Workshop on Medical School Curriculum." Atlanta, Georgia, September 18-22, 1968. (Mimeograph.) 2. Anderson, M. B., Owen, S. G., Hall, B. S., and Smart, G. A. "Comparison of a Teaching Machine Program and a Series of Lectures in Electro-Cardiography." Self Instruction Medical Education. Proceedings of the Second Rochester Conference. University of Rochester, 1967. 3. Azneer, J. E., Kesler, E. and Cacamo, L. "The Development and Testing of a Programmed Course in Diabetic Acidosis." Individualized Instruction in Medical Education. Proceedings of the Second Rochester Conference. University of Rochester, 1968. 4. Baas, J. H., as translated by Handerson, H. E. The History of Medicine, Volume II. Robert E. Krieger Publishing Co., Inc., Huntington, N.Y., 1971. 5. Broudy, H. S. "A Critique on Performance-Based Teacher Education." American Association of Colleges for Teacher Education, One Dupont Circle, N.W., Suite 610, Washington, D.C. 20036, May 1972. 6. Brown, T. C. "Centers for Research in Medical Education in the United States: A Preliminary Study." Unpublished Dissertation, University of Southern California, 1970. 7. Brown, T. C, McCleary, L. E., Stenchever, M. A., and Poulson, A.M., Jr. "A Competency Based Educational Approach to Reproductive Biology." Am. J. Obstet. Gynec. 116:1036-1043, 1973. 8. Brubacher, J. S., and Rudy, W. Higher Education in Transition. New York: Harper Brothers, 1958. 9. Castle, W. and Davidson, L. "An Evaluation of Programmed Instruction in a New Medical Faculty." British Journal of Medical Education, 3:359-361, 1969. 10. Chez, R. A. and Hutchinson, D. L. "Teaching Medical Students Obstetrics and Gynecology." Obstet. Gynec, 33:127'-130, 1969. 11. Christensen, H.N. "The Identification of Special Topics for Teaching by Short 'Plug-in' Learning Programs." Programmed Instruction in Medical Education. Proceedings of the First Rochester Conference. The University of Rochester, 1965. 12. "Educational Technology for Medicine: Academic Institutions and Program Management." Supplement to J. Med. Educ., 48:203-226, 1973. 13. Feldman, H. "Learning Transfer from Programmed Instruction to Clinical Performance." Nursing Research, 18:51-54, 1969. 14. Flexner, A. Medical Education in the United States and Canada. A Report to the Carnegie Foundation for the Advancement of Teaching, Bulletin No. 4, Boston, Massachusetts: The Marymount Press, 1910. 15. Geiger, L. G. Higher Education in a Maturing Democracy. Lincoln, Nebraska: University of Nebraska Press, 1963. 16. Githens, J. H. "The Current Scene in Medical Education." Clinical Pediatrics, 8:376, 1969. 17. Guyton, R. G. "A Multimedia Self-Teaching Package for Family Practice Education." Unpublished Dissertation, University of Utah, 1973. 53 18. Haskins, C. H. The Rise of Universities, Ithica, New York: Cornell University Press, 1957. 19. House, E. R. "Politics As Usual." School Evaluation: Thr Politics and the Process, McCluchen Publishing Corporation, Berkeley, California, 1973. 20. Hunter, C. A., Jr. (Ed.), Guide to a Basic Curriculum in Obstetrics and Gynecology. The American Association of Obstetricians and Gynecologists Foundation, Inc., 1972. 21. Hunter, C.A., Jr. "Methodology and Evaluation of Undergraduate Curriculum in Obstetrics and Gynecology" Am. J. Obstet. Gynec. 104:221-222, 1969. 22. Kearny, C. P. and Huyser, R. J. "The Politics of Reporting Results." House, E. R., (Ed.), School Evaluation: The Politics and the Process. McCluchen Publishing Corporation, Berkeley, California, 1973. 23. Lee, P. V. Medical Schools and the Changing Times. Evanston, Illinois: Association of American Medical Colleges, 1962. 24. Lloyd, J. "Student Reactions to Self-Instructional Materials." Self-Instruction in Medical Education. Proceedings of the Second Rochester Conference. University of Rochester, 1965. 25. Lyman, R. A. "Disaster in Pedagogy." New England J. Med., 257:504-507, 1957. 26. McCarthy, W. "Improving Classroom Instruction: A Programmed Teaching Method." J. Med. Educ, 46:605-609, 1971. 27. McGlothlin, The Professional Schools. New York Center for Applied Research in Education, Inc. 1964. 28. Miller, G. E. "Adventures in Pedagogy." J. Am. Med. Assoc, 162:1448-1450, 1956. 29. Miller, G. E. "Medical Education." Encyclopedia of Educational Research, 4th Ed. 30. Mulvihill, J. A student operated series of medical films. /. Med. Educ, 43:1202-1205, 1968. 31. Pusey, N. M. The Age of the Scholar. Cambridge, Massachusetts: Harvard University Press, 1964. 32. Rulin, M. and Chez, R. A. "Interdepartmental Sharing of Audiovisual Teaching Aids - A Valid Concept?" Obstet. Gynec., submitted 1973. 33. Russell, J. K. "Sound and Visual Teaching Aids in Obstetrics and Gynecology." Med. BioLIllus., 16:1966. 34. Stead, E. A., Jr., Smythe, CM., Gunn, C.G., and Littlemeyer, M. H., (Eds.), "Educational Technology for Medicine: Roles for the Lister Hill Center." /. Med. Educ, 46, Part II, 1971. 35. Steering Committee for Cooperative Teaching, APGO. "Faculty Reactions to an Instructional Media Network in North America." J. Med. Educ, 48:430^135, 1973. 36. Stenchever, M. A. "The Choice of a Teaching Modality." J. of Reprod. Med., 7:117-119, 1971. 37. Stenchever, M. A., and Brown, T. C. "A Network for the Dissemination of Teaching Materials."/. Med. Educ, 47:702-707', 1972. 38. Sweet, B. and Doyle, A. E. "Teaching Machines in Medical Education." Med. J. Australia. 1189-1191, 1971. 54 39. West, L. C. and Stickley, W. T., "Cinematic Self-Instruction in Laboratory Pharmacology." Programmed Instruction in Medical Education. Proceedings of the First Rochester Conference, University of Rochester, 1964. 40. Wilds, P. L. and Zachert, V. "Programmed Instruction as a Method of Teaching Clinical Problem Solving." Programmed Instruction in Medical Education. Proceedings of the First Rochester Conference, University of Rochester, 1964. 41. Wilson, J. A. R., Robeck, M. C, and Michael, W. B., Psychological Foundations of Learning and Teaching. New York: McGraw-Hill Book Company, 1969. 42. Wood, F. "Medical Education in America Before 1910." /. of Med. Educ. 38:631-637, 1963. 55 case study no. 1