!! spellx done
251  Peer Review System (NIH)
     February 19, 1976

     Having worked within the framework of the NIH Grants/Peer
Review system for most of my scientific career, I felt it was a 
particular privilege and obligation to comment on the efficiency and
fairness of this system at the occasion of the present hearings.  My
first encounter with this system began over 25 years ago: during the
12 years that I worked at the University of Wisconsin; and throughout
the course of my subsequent work at Stanford.  Especially during the
middle period I served on many NIH study sections and have also served
on the National Advisory Mental Health Council for the NIMH.  I
have also related to a variety of other government agencies and can
therefore make some comparative estimates of the way that different
variants of the system and other ways of administering research in
fact operate.

     At the outset I should summarize that although my own personal
experiences with Peer Review have sometimes been quite frustrating,
at others very smooth, it is difficult to imagine any alternative
system that could function nearly as well in practice in meeting the
national goal of strengthening the productivity of important health
research in this country and supporting the careers of dedicated 
research scientists.  This is certainly not to pretend that this is a
perfect arrangement; but to assert that it stands rather high in the
scale of practical human affairs in achieving its asserted goals.  If
the Congress and the people wish to put a higher priority on other
objectives - for example, by hypothesis, equal access to federal
support for research as a matter of equal personal right (rather than
in terms of scientific quality or its utility for the public good)
then the Peer Review System may not be the most effective instrument.
However, if there is to be a significant diversion of goals, then
this issue should be addressed directly and institutions established
whose avowed redirection of purpose is clearly stated.  There is 
nothing to prevent the allocation of specific components of the over-
all budget, within the framework of the existing grants system, in
such measure as may be politically determined to be most appropriate
for other purposes.  The most obvious of these would be the diversion
of resources to weaker institutions or less able scientists with the
hope that these funds might by themselves upgrade those capabilities.
Needless to say, one should not embark upon such new and expensive 
programs without a very careful consideration about the likelihood
that they will achieve the objectives of the intended investment.

     The existing system of Peer Review and its counterpart in the 
overall framework of the scientific enterprise is based essentially
on the concept of motivated mobility.  It is indeed true that the 
so-called prestigious institutions receive more than a demographic
proportion of support for scientific work.  Yet it is precisely the
same institutions that are the target for the careers of many
younger scientists, who aspire to work in the company of excellence,
and in a context that experience has shown to be so highly productive.
These institutions are deeply self-conscious that their most
treasured resource is the positions on the faculty; and they spend
an enormous effort in seeking out the most qualified candidates.
This is not to decry the attractions that other more distributed
institutions may have for many very competent people, and here we
have the network of open publication and the avid pursuit of evidence
of new discovery as one of the fundamental structures on which 
scientific progress is based.Given these competitive, open, pluralistic,
institutions it is really not likely, nor is there any significant
evidence, that true talent remains long hidden from view or that 
it persistently fails to find recognition from the existing review
system.

     If these arguments for maintaining the essential structure of
Peer Review are for the moment granted, we must then face the sincere
concerns of many who may feel that the review process should be
less secret, that it perhaps might involve wider circles of critics
than at present, that investigators should have greater opportunity
to learn the reasons why their proposals may not have been accepted 
and have a better opportunity to rebut criticisms that have been
made.  In particular, it has been suggested that the entire proceedings
of scientific review be made open and public, and that more 
elements of constitutional due process should be attached to a program
that involves a commitment of tax dollars to a public purpose.

     I am deeply sympathetic with these concerns; and in various
situations have attempted myself to bring about some of the reforms
that have in fact tended to alleviate some of them.  We do have a
dilemma with respect to the practical possibility of obtaining critical
and candid reviews if these are done in public.  The process
is very similar to what is involved in many personnel actions which
are the one area of public administration which has been widely
deemed to be the most qualified for some exception to complete public
ventilation of every claim and counter-claim.  It is not only the 
reviewers who stand at risk from a public proceeding - there are
also grave potential embarrassments to the applicants and there is
really considerable question as to whether justice would be better
served by a complete removal of the cloak of confidence.  However,
as an experimental pragmatist, I could hardly argue against attempting
a trial, within a well defined and limited context of such an
approach, and would certainly recommend this before there were any 
sudden upheavals in the overall system.

     These concerns have, of course, evolved in a setting of more
and more rapidly growing confrontation and with the assertion of the
right to due process as a basic guideline in every human interaction.
Before yielding to any more than one would advocate that trial
juries deliberate in public the temptation to avow such an approach
as the essence of liberty, we should keep in mind the costs to the  
operation of any social system if each transaction could be valid
only if the parties appeared in court and had the right of counsel.
We would find that it would take six years rather than six months to
complete a review of any given proposal if we had to give credit to
all the arguments that litigious contestants might wish to offer.
This is hardly a speculation since we know that already the gravest 
impediment to the allocation of criminal justice is the inordinate
delay and the clogging to our judicial pipelines that apply even
under existing legislation.

     We should then be seeking effective reforms that attempt to
meet these sometimes conflicting objectives, in a way results in a
practical degree of fairness and of efficiency and of concern for
the variety of interests involved.  I believe that the key to this 
objective was already foreseen by the Congress in the form of its
legislation for the establishment of the various institutes of the
NIH.  With each such institute the Congress has almost always had to
prevail against a reluctant executive in insisting that the legal 
power to award grants be vested in an Advisory Health Council rather
than in the appointee of the President directly.  These Councils
have occasionally, but only occasionally, been allowed to operate in
the fashion implied by the enabling legislation.  They were intended
to represent public interest and a wide variety of interests of the
participants in the research enterprise to help be sure that the
overall purposes of the research budget of the NIH were met by the
overall program of supported grants.  The study sections, by law,
are in fact agents of these Advisory Councils, report to them and it
is the Councils' legal approval that is binding as a necessary con-
condition in the grants process. Throughout several administrations the
Executive has been reluctant to provide the highest and most 
representative quality of appointments to many of these councils - with 
some notable exceptions - and the directors of some institutes but
not others have made the most appropriate use of them.  Obviously,
such a council with a heavy representation of lay-membership can
hardly be expected to redo the work of the professional peer groups,
the study sections, in the detailed examination of the substance of
thousands of applications.  These councils do however have the
potential to be sure that the review process is conducted fairly, to
provide avenues of appeal and redress, and to be sure in other ways
that the public interest is best served by the entire operation.  It
is at this level, in my opinion, that concerns about the equity of
distribution of grants by geography, or by any other criterion that
has been suggested, could most appropriately be validated: by 
encouraging the appointment of an appropriately representative group
of members to these councils.  As council membership must, by law,
rotate fairly quickly, there is little opportunity for the
development of the kinds of cliques and cabals that are claimed to 
impede the fair operation of the peer review process.  I do not 
think this is entirely a hypothesis since my reflections on these 
procedures stem from my term with the National Advisory Mental
Health Council, at the time when it was perhaps at the peak of its
competence and influence to fulfill its legislative and social mandate.
It was subsequently subjected to considerable interference, 
expressed mainly in the form of inordinate delays in the appointment
of new members and in the use of the crassest political criteria in
the attempted choice of new nominations.  Since these matters have
been said about the role of the advisory councils as the tempering 
influence to make the Peer Review System work well and fairly, and
in a way that appears to be fair.

     As to the rights of applicants to understand more clearly the
reasons for the fate of their applications, it first of all has to
be said that the vast majority of disappointed applicants these days
have had their work approved on a scientific basis; but had failed
to find funding because of inadequate budgets, and administrative
decisions about the cutoff points for this funding.  This disparity 
between the reservoir of frustrated talent and the resources that
are being made available for the pursuit of research should not be
laid to the door of the Peer Review System!  Nevertheless, there are
a number of reforms that I have strongly advocated in the past, and
have been gratified to see adopted in considerable measure in recent
years.  These have to do mainly with taking more time and trouble to
indicate to applicants more detail about the difficulties that may
have occasioned low priority scores, or rejection, or deferral of
their applications.   I believe that many institutes have in practice
worked out of fair accommodation between the needs for practical
operation of the review system and the right of information on
the part of the applicants.  The structure of the system should not
be molded by a very small minority of the most litigious and often 
scientifically unproductive workers.  There are a number of 
administrative details that should be worked on but which doubtless will
require the investment of more not less money to make the system
work well.  It is especially irksome that it takes so long - often
the best part of a year - between the time that an application must
be submitted and the time that one gets any significant information
about its outcome.  There is usually not enough opportunity for some
dialogue, for the working out of compromises, for the removal of
particular difficulties; and a process that in principle could be
done within a few months can sometimes be stretched out for several
years or beyond the life-time of the idea and the zest to pursue it
on the part of the investigator.  This process has not been helped
by the rapidly increasing accumulation of regulatory encumbrances
that have sought to enforce highly desirable social objectives - for
example the protection of rights of human subjects - in ways whose
administration seems calculated simply to add to delays in the 
processing of applications, even before they are submitted rather than
during the many months during which they are under consideration;
and if these could be conditions of the award of a grant rather 
than the acceptance of an application, many months would already be
saved.  These encumbrances aggravate the problems of communication
that investigators face, and which they may attribute to the 
essential core of peer review quite inappropriately.

     I have just one other concern about the detailed operation of
the Peer Review System and that is for reasons of the administrative
simplicity its judgments end up being transformed into a single number,
the priority rating, that is supposed to represent the consensus
of the study section.  I believe that this is far too crude a
measure of a sometimes diverse set of views that reflect different
dimensions of approval, or of concern; and that this number does not
give enough latitude in the further negotiation of the grant between
the NIH and the investigator, to correspond to the true range of
opinions of the reviewers about the project.  For this reason alone,
I believe that an intense, empirically oriented study of the peer
review process would be most desirable, not with the aim of wrecking
it, but with the aim of making some new technical innovations in the
way in which reviews are conducted, and the results of reviewers' 
deliberations are scaled.

     I also believe that there has been a growing tendency, under
budgetary stringencies, to increase the cost and judicial burdens of
the review process by demanding re-examination of research projects
on an increasingly shorter term.  It used to be unusual but imaginable
that an ongoing research project could look forward to as much
as 7 years of support.  Now a growing number of projects are only
funded for 3 years and sometimes less; which is hardly time to 
recuperate from the rigors of applying for one grant before having to
begin to renew the process.  Of course, there are many circumstances
where it is vitally important to track the progress of a particular
piece of work in some detail.  But these exceptional circumstances
really should not be confused with the bread and butter type of 
support - for the fundamental operation of our institutions of
higher learning and research - which are not likely to change very
much in the fundamental quality of their work in so short a time.  
Some of the faults of the Peer Review System can certainly be laid
to the sheer volume of material that has to be processed by a group
of dedicated but overworked and essentially unpaid volunteers who
serve on the review committees.  More meticulous procedures for
dealing with the process could become more practically available if
it were not involved so frequently.

     A number of other administrative reforms might be recommended
to alle-some of the difficulties of dealing with the inevitable 
disappointments of some applicants, and their understandable 
tendency to question the fairness of the system -- although I believe
that the central ingredient is better and brisker interchange of
information between the investigators and the NIH itself.  (site
visits, which have been restricted for budgetary reasons, are among
such devices.)  Since the competitive rivalry among scientists,
which is a central part of the system for innovative discovery, can
of course breed occasional personal antagonisms, applicants may feel
that they have been abused by the presence of a specific individual
on the review committee.  We might experiment with a procedure that
allowed an applicant to request that some single individual member
of a review panel be excused from deliberations on his own case,
leaving the burden to the other members of that group.  Likewise
applicants should be encouraged to designate the names of referees
who are familiar with their work, and should be consulted at least
by mail as potential advocates to submit further information into
the review process.

     These and many other procedural variations should be within the
latitude of an innovative and concerned Advisory Council to try,
towards the continued perfection of the peer review system.

     The integrity of that system is important first of all, of
course, as a matter of responsible government administration.  The
public should also note that the scientific community has developed
its own devices for the rapid and effective advancement of knowledge,
which operate somewhat differently from the adversary system  with
due process that describes more formal government procedures.
Scientists are expected to tell the truth, not just to present the 
best case for their own side, and to look for technicalities and
dilatory tactics with which to win a case.  For a community of
mortal humans, that system has worked very well: the best evidence
perhaps is that isolated infractions -- like the rare scientists
who have lied in print  -- make headlines.  For it to continue
working it must of course get social reinforcement: recognition of
the basic values of approaching scientific truth through the open 
publication of results, and their availability to reexamination of
every detail by others. And there must of course be serious sanctions
-- as there are perhaps more in science than in any other field
of human activity -- against lying.  The social losses from a
disturbance of that system, and its replacement by a quasi-judicial
-legalistic-adversary one could be so large as to undercut both the
material and the philosophical advances on which our modern culture
is founded.
---------------------------------------------------------------------