Reforming (funding) peer review

NIH has announced an initiative to enhance its peer review practices. In so doing, the world’s largest science funding agency acknowledges the validity of two criticisms often levied by grantees and science policy-types:

  • that the NIH discriminates against young researchers by putting a premium on preliminary results when evaluating proposals. This practice has created a situation in which funding is better viewed as an intermediate research output – a prize for work well done, in the words of Ed Lazear – rather than an input into the production of future results.
  • that peer review as practiced by the CSR (Center for Scientific Review) results in the funding of incremental, “normal science”, at the expense of truly creative proposals.

Both criticisms have some merit, and we’ll discuss them (separately) at some later date. For now, I’d like to make a very simple point. To paraphrase Churchill, peer review is a terrible system to allocate funding across projects…except that all the others are worse still. The alternatives, such as an institute model (as practiced in France, or in many of the US National Labs) can easily produce fiefdoms insulated from competition and increasingly disconnected from the outside world. There are also prizes, but they seem much more appropriate to reward innovation towards goals well-specified in advance. In short, there are many reasons to suspect that the competitive, peer-reviewed grant is an important meta-institution whose adoption greatly increased the world’s epistemic base.

So I am hoping that the NIH task force will remember a few things as it develops its reform proposals further. First, do no harm! Remember that those who have the most gripes against the current system might have agendas of their own. Second, experiment! Although scientists often have strong priors regarding what is likely to work, subjecting reform proposals to careful, randomized evaluations is the way to go…


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: