THE INTEGRITY OF THE PEER REVIEW SYSTEM
AS USED AT THE NATIONAL INSTITUTES OF HEALTH

I. Background

From a level of approximately $35 million provided for the
National Institutes of Health (NIH) in fiscal year 1950, the
Federal investment for biomedical research in that agency

has grown to over $2.5 billion in fiscal year 1977. With re
only minor perturbations overall growth has been steady, oe
reflecting confidence in biomedical research for improvements f
in the health of our citizens and reaffirming the primary role 5

of the Federal government in the support of advancing knowledge.
From the time that the Federal role was established, support
for research training has also been an explicit recognition
that the long term vigor of the enterprise is dependent on

a constant infusion of well-trained investigators. The
consequence has been the creation of the world's foremost
biomedical research endeavor as a unique partnership between
the public sector and the biomedical science community.

To carry out the responsibility for the large-scale investment
of tax-derived funds, the NIH developed a system for the
allocation of its research funds. A key factor in the concept
adopted was that prominent scientists active at the cutting

edge of science in the laboratory or clinical setting were

in the best position to assess the scientific merit of proposals
submitted for Federal funding and to recommend those potentially
most contributory to the objective of improving health through
increasing new knowledge.

As the size of the research program expanded, the scientific
members of the legislatively mandated National Advisory
Councils soon were unable to cope with the research components
of the grant applications, given the diversity and number of
proposals submitted. To meet the new demands the NIH created
by administrative action a two tier review system that, with
refinements, is still in use. This arrangement consists of:

1) discipline or specialty oriented initial review groups
(IRG's), comprised of recognized experts in those areas and
charged with assessing the scientific quality of each grant
application, and 2) legislatively mandated National Advisory
Councils, including both scientific and lay personnel, which
had been assigned the task of recommending those proposals
deemed most worthy of the investment of Federal funds. This
system has been repeatedly examined by a variety of prestigious
groups and has invariably been acclaimed as serving the public
interest extraordinarily well in allocating Federal funds for
research. In fact other research supporting organizations in
both the public and private sectors have emulated the NIH concept
and system. Most recently the Congress has seen fit to
explicitly mandate review by a peer system for various types
of research programs.

It should be noted that the overall grant application review
system at NIH also represents one of the best bargains
currently available to the Federal government. Present
estimates place the cost of that system at about one dollar
for every $100 requested. Were the NIH forced to reimburse
the individuals constituting these panels for the total time
expended in the preparation for application review and at

the rate they are able to command as consultants or to expand
its own staff to encompass the review responsibility, the
cost would be enormously greater.

Despite minor criticisms -- which for the most part have been
successfully refuted -- the system has withstood the test of
time. It is not surprising, therefore, that threats to it,
arising either externally or internally, are viewed within

the scientific community and elsewhere with the gravest concern.

II. The Problem

Several recent developments together with longstanding

governing principles have converged to cause a strikingly

serious overloading of the NIH grants review system that
constitutes in the opinion of long-time, well-informed observers,
the most ominous threat to its integrity that the system has
faced since its inception. The elements, cited below, contri-
buting to this situation are several:

® <A significant increase in the capacity of the biomedical
research enterprise in this country over the past ten
years. This has been a planned and purposeful expansion
on the part of the Federal government, involving not only
increases in funds appropriated for research but the
establishment and support of a large-scale research train-
ing enterprise also financed with Federal funds.

e An almost doubling in five years of the number of research

grant applications reviewed by the NIH (Figure 1). The
evidence available suggests a strong relationship between
this phenomenon and the expansion of the research. training
programs which was initiated in the mid-60'ts. Therefore,
the first and most important influence is the increase

in the number of scientists who are now able to compete

for existing funds. The second is the uneasiness within
the scientific community concerning the inconstancy of 7
Federal programs and funding during the past few years on
and the increased competition within the system. Discus-
Sions with numerous institution officials and individual
scientists support the belief that these uncertainties

in funding have prompted more numerous applications to
NIH for the same project or parts of it in order to
increase the chances of success in competition for the
available funds.

7 Reductions in position ceilings within the Executive
Branch which, in the case of the Division of Research
Grants (DRG), NIH, have resulted in fewer employees to
staff the Study Section operation than was true a decade
ago (Figure 1).

°

The zealous implementation of the Federal Advisory Committee
Act and Presidential directives to reduce the number of
advisory committees in the Executive Branch (Figure 2).

e Interpretations of the Privacy Act by the General Counsel,
Department of Health, Education and Welfare (DHEW) which
permit individual principal investigators to request iW ge
access to their application review files at any time we
during the review process and to submit amendments and (
rebuttals to the evaluations the application has received.

e Continuing intra-departmental scrutiny of the legitimacy
of the statutory grounds (in the Freedom of Information
Act and the Federal Advisory Committee Act) for declining
to release applications prior to funding and for conducting
the Study Section (IRG) review of applications in closed
session appear on the verge of destroying a level of
confidentiality viewed as essential to the effective
operation of the system.

e The tradition of the American scientific community and
the Federal government that every qualified scientist has
free and open access to compete for public funds for
research.

III. The Consequences

 

While any of several of these influences would have created
serious problems, the aggregation of all of them has resulted
in a devastating impact on the system, a severe erosion of the
capability for effective and timely scientific review, and

a situation that will certainly worsen significantly unless
immediate and major corrective steps are undertaken. Some

of the specific consequences can be identified as follows:
The workload on individual Study Section members in
preparation for the three meetings per year has

vastly increased. It should be noted that these
individuals are practicing scientists whose career
advancement depends on their personal productivity in
research and that they are the same individuals who
must spend more time in paperwork -- such as more
detailed and time-consuming preparation of applications
and progress reports -- associated with securing their
own research support.

Although the program managers of the peer review system
for project grants at NIH conclude that, on the average,
Study Section workloads should not exceed approximately
225 to 240 applications per year, recent years have seen
that average rise steadily to the level in 1977 of 355
applications per Study Section. In rapidly developing
scientific areas, the number is significantly greater.

Resignations by Study Section members have increased
sharply over the past three years, from four in 1974 to

thirty one in 1976. There have also been larger numbers jel
of formal declinations by qualified individuals to serve oe
a

%¢

on initial review panels. Additionally, we are told that

in some institutions and professional organizations,

younger scientists are being discouraged from accepting “Se
appointments to Study Sections as being too demanding

of their time. This circumstance is the obverse of that

in previous years when such service was a coveted honor

and important in the advancement of a biomedical scientist's

career.

The access provision of the recent Privacy Act, as inter-
preted by the DHEW Office of General Counsel, permitting
unrestricted access to the application files by the
principal investigator, is now used extensively hy
individual scientists as an informal appeals mechanism,
During the 1977 fall review cycle, almost 1,300 grant
Summary statements ("pink sheets") were requested, fre-
quently before they were sent to the members of the
Advisory Councils. This represented about one third of
the applications under review at that time; the comparable
number in 1976 was but a small fraction of that total. The
result is a serious disruption of the usually orderly
process of grant application review, especially in the
period between Study Section consideration and Council
review. Members of both initial review groups and
Advisory Councils are being badgered by principal investi-
gators concerning comments by site visitors or by Study
Section members about the review of their applications
prior to final recommendation by the Advisory Council,
Because of the work overload, the Executive Secretaries
of the Study Sections are unable to spend sufficient

time in the preparation of the all-important pink sheets
and other pertinent staff activities related to the review
system. The latter include, for example, the identifica-
tion of applications involving DNA research or human
subjects and review for assurance of conformance with
current agency policies. Furthermore, the fact that they
are subject to disciplinary action for improperly with-
holding information (under the Freedom of Information
Act) and to criminal penalties for disclosing material
(under the Privacy Act) to any person not entitled to
receive it has created a working environment that is not
conducive to effective and productive administration.

This burden on the NIH staff, together with the similar
overload on the consultants, especially at the initial
review stage, must inevitably lower the quality of the
review process.

IV. Possible Solutions

 

Generally speaking, solutions to the problem would involve a
decrease in the number of applications submitted for review
and/or an increase in the capacity of the system to handle

the larger workloads. Numerous possibilities have been explored
as potential solutions. Close examination, however, leads to
the conclusion that no single action will suffice. Among

ideas which are under consideration are the following:

Relief from the ceilings on staff positions and numbers
of Advisory Committees.

 

-- The White House, the Office of Management and Budget,
and the several levels of DHEW must be persuaded that
the prevailing Executive and Legislative branch stric-
tures on increases in both staff positions and the
number of advisory committees are counterproductive
in this instance and must be substantially modified.
One way of doing it without appearing to backdown on
highly trumpeted past achievements in reducing the
number of Advisory Committees would be to invoke the
concept of a flexible Study Section (in essence, a
large panel of members constituting a given Study
Section which could then be broken into several
smaller groups); this seems to be gaining favorable
attention within the Department. However, even if
approved, there would still remain the very serious
problem of staffing the sub-groups.
(3)

Wider use of the Intergovernmental Personnel Act

and expert consultants under the authorities of the
National Cancer Institute and the National Heart,
Lung and Blood Institute to augment the number of
Executive Secretaries staffing Study Sections. This
approach is already being utilized in an effort to
relieve the pressure on IRG staff. But individuals
whose services are available under either of these
approaches are necessarily limited in the duration

of their assignments. A longer continuity of
involvement is required for an individual to effec-
tively fill an Executive Secretary role. Furthermore,
there emerges a serious question as to the appearance
or even actuality of a conflict of interest.

Reallocation within NIH of staff positions under

existing ceilings. It is uncertain how feasible

this approach might be. Total personnel ceilings

at the NIH have fallen steadily over the last decade

while program responsibilities have increased substan-
tially. As the latter involve enlarged funding,

complexity of programs and ancillary issues related to such
problems as ethical considerations in the use of human sub-
jects in research, Equal Employment Opportunity, refinement
of accountability requirements, development of long-

range planning documents, etc., there would not appear

to be any significant existing source of positions

which could be tapped without very deleterious conse-
quences. To reduce the intramural research program

staff for this objective would cripple and demoralize
extremely important and productive scientific activities.

e Modifications in application processing by NIH

Deferral of the final decision as to funding on those we
applications whose review has been interrupted by ro
challenges before it has been completed (e.g. between mt
Study Section and Council meetings). This approach

apparently has been interdicted by DHEW legal staff

on the basis that it would constitute a punitive

measure against the principal investigators who

challenge tentative recommendations.

Withholding documents requested under the Privacy Act
(e.g. pink sheets) until review has been ccmpleted
and a decision has been made to fund a particular
proposal. It is uncertain whether this is feasible,
in view of previous interpretations of the Act by

at least some government counsels.

as

We
-- Limitation on the length of applications. A proposal
to hold applications to an arbitrary length (e.g.
20 pages} might accelerate review and reduce the
workload but could weaken the primary method of
communication from the investigator to the reviewers.

-- Limitations on the number of applications submitted
per principal investigator or per institution. The
NIH should be justifiably reluctant to impose such
limitations by directive, given the history of the
Federal government's attitude toward the accessibility
of appropriated funds to eligible grantee institutions
and scientific investigators.

® Restraints within the grantee institution

-- Establishment by grantee institutions of formalized
local screening systems to reduce the number of poorly
executed or incomplete applications submitted to the
NIH. Considerable doubt exists as to whether most
institutions could constitute an effective screening
process of this nature, especially without a directive
from the NIH requiring such a prescreening phase.

-- Voluntary restraint by individual investigators in
submitting applications and in requesting pink sheets
prior to the completion of the application review.

On the basis of discussions with individuals in the
academic community, it seems highly unlikely that there
would be sufficient response to a call for voluntary
restraint on application submission by individual
investigators and their institutional officials. The
importance of grant support to investigators and the
compétition in the grant system at this time force

the conclusion that unless voluntarism occurred
simultaneously and to a significant degree in most
grantee institutions there would be little effect.
However, voluntarism might prove successful in reducing
the number of premature requests for pink sheets.

V. Possible Actions

At the moment, the scientific community and other interested
sectors of our society are largely unaware of the complexity
and portentousness of this situation. It also seems certain
that with that awareness there will be the same immediate and

widespread concern which is held by the relatively few individuals

who are cognizant of the overall situation at this time. It is
important that there be developed a strategy to handle this
multi-faceted problem so as to maintain the integrity of the
system as a matter of public good. Concommitantly, the
strategy should avoid the possibility of inappropriate
efforts by well meaning but poorly informed individuals or
of arousing in critics the use of this set of circumstances
to substitute a less effective approach to the allocation
of Federal funds than the mechanism now used by the NIH.

It is necessary, therefore, to examine efforts which the
academic and scientific communities as well as the govern-
ment might undertake. The following are recommended for
consideration:

® Reaffirm support for the peer review system

Leaders of the academic and scientific communities should
publicly recommit themselves to the value to the public
and to science of the peer review process and the need
and value for qualified individuals, particularly among
young investigators, to serve on initial review panels.
The consequences of alternatives, such as total block
grants or distribution on a geographic basis, should be
explained.

® Inform the communities

Basic to any successful effort is the need to broadly
inform the scientific and academic communities as well

as the public as to the nature and seriousness of this
threat to the system. Undoubtedly, several different
approaches will have to be made, including presentations
and discussions at scientific meetings as well as articles
in appropriate scientific and possibly other publications.

® Enhance the analysis of the research grants program
While considerable effort has been expended by the NIH
in maintaining statistics about the grants program, the
current set of problems reveals the inadequacy of those
efforts. It is suggested that NIH should expand its
analytical activities so as to provide the basis for a
better understanding of the dynamics of the program.

Ve

® Address the personnel requirements for the system
This aspect involves staffing of the Study Section
apparatus with more full-time positions as well as
increasing the number of Study Sections and members.
These changes will have to be approached in both the
Legislative and Executive Branches. The Congress can
speak to the need for expanding the capability of Study
Sections to handle the workload but it is the Executive
Branch through White House policy and action at several
HEW levels that must implement the required changes.
Eliminate the impact of the current interpretation of the
Privacy Act

The requests for pink sheets and other documents by principal
investigators prior to the completion of the review process
should be interdicted by some method. Two possibilities
exist:

-- Convince the Office of the General Counsel, DHEW, that o-
either requests for pink sheets can be refused prior v
to final decision on the application or permit the
application to be deferred until the next review cycle
if a premature request for information is made, on the
basis that the process has been disrupted and the
application is incomplete, or

-- Seek legislative modification of the access provision
of the Privacy Act.

Establish a balance between competing public policies

Just as the approach to less secrecy in government is a
desirable public objective, so is the assurance that the
nation's scientific enterprises will flourish by being
provided with an environment attractive to the best scientists
and therefore conducive to the most productive research.

At the present time, there appears to be an imbalancing of
these oft times competing values in that under prevailing
procedures, responses to demands for disclosure of research
proposals submitted to the NIH and other Public Health
Service agencies do not permit adequate protection for the
ideas of individual scientists. Those ideas are the
scientist's intellectual capital and his professional
advancement is primarily dependent on then.

It is recommended that both the Congress and the Executive
Branch review the present laws and regulations and their
administration to determine in explicit terms the manner in
which each of these policies should relate to one another.
The solution to reconciling the conflict between these

values to best serve the overall public interest would be

a legislative shield under which research proposals would be
protected from premature disclosure permitted by 5 U.S. Code,
Section 552(b) (3).
- 10 -

VI. SUMMARY

The effectiveness of the system of peer review used to allocate
research funds appropriated for programs at the National Institutes
of Health is severely threatened by a series of policies, events
and circumstances which require immediate attention by the
Executive Branch, the Congress and the scientific community.
Failure to take prompt corrective measures for these problems

could result in the destruction of a system widely acclaimed as
both efficient and equitable, without the availability of an
appropriate alternative mechanisn.

Association of American Medical Colleges
May 15, 1978
APPLICATIONS REVIEWED AND NUMBER OF EMPLOYEES

 

 

 

1970 - 1977
APPLICATIONS
20 ,000
17,741
4”
¢
$
$
a)
§
s
9
pete
15,000 sf "
‘
$
s
4
‘
$
¢
¢
¢
¢
¢
¢
10,614 +
jhe ’
jf “mf
10,000 yt +4
‘
aassad 9,541
8,774
EMPLOYEES
500
5 ,000
400
300
0 Q

70 71 72 73 74 75 76 77

FISCAL YEAR
(Figure 1)
(Z eainsty)

APPLICATIONS, STUDY SECTION MEMBERS AND STUDY SECTIONS

ut

1969-1977
INDEX
225
200
175
~ NUMBER OF GRANT APPLICATIONS
REVIEWED
150
125 s NUMBER STUDY SECTION MEMBERS

 

aaa

10 = _ ee

NUMBER STUDY SECTIONS

 

 

 

 

 

 

75
YEAR NO. APPLICATIONS NO. MEMBERS NO. SECTIONS APPLICATIONS/MEMBER
1969 8,227 690 48 12
1973 9,541 685 47 14
1977 17,741 789 50 23
1969 70 7] 72 73 74 75 76 7

YEAR
Specific remarks keyed to text: JUL 14 1978
1. Has there been steady growth in real dollars?

2. Combination of truncated terms of awards and increasing red tape and delay/
lead time: many applicants are in a state of perpetual application anxiety.
Particular impact on complex, interdisciplinary projects: have become almost
impossible. Result is splintering of research into nothing but projects,
tempered by traditional modes of scientific communication among experts in a
discipline, but erosion of glue that holds an institution together.

3. I take serious and principled exception to the point made here and
throughout. Investigators should have, if anything more, not less opportunity
for feedback. Without that corrective, any system will become corrupt.

4. Obviously others are ready to fill the vacuum, who may have limited opportunity
to exhibit their talents in constructive research. Worst of all, rebuffed
and ressentient youngsters.

5. Individual badgering is of course a serious nuisance. The reviewers could
be armed with a form letter from the Director NIH that such communications
are an interference with the review process and that information not sent
through channels must legally be disregarded, to the point where the
reviewer should disqualify himself if he feels that his autonomy has been
impaired by such communications.

6. If the councils met more often, and had better technical support,

or if other means to sustain interim support were feasible, P.I.s

could well REQUEST deferral for more info. In present circumstances, this is
a disaster.

7. Sections could at least be admonished not to discredit short applications
if there are effective references to previous literature and they come from
established investigators. I have repeatedly been met with insistence that

I include detailed accounts of experimental procedures that were in the
literature!! Conversely, in another case, the meat of an application was a
preprint of a paper in press, which staff refused to incorporate in the
application because it had not been retyped as part of*a conventional proposal.

8. Page 8. Yes I have to agree about these cautions and needs.

9. As before, vehemently NO. I agree about need for confidentiality of
individual reviewers' names. But pink sheets are rife with unsupported
assumptions about what an applicant meant by a given phrase, with factual
errors (particularly about budgeting and institutional situations), and
about the existing state of the literature. I do not have good solutions to
these problems, but will at least advert to an appellate role of a COUNCIL-
(based) mechanism, which must be made managerially brisker.

@