Predicting Productivity Returns on Investment: Thirty Years of Peer Review, Grant Funding, and Publication of Highly Cited Papers at the National Heart, Lung, and Blood Institute
There are conflicting data regarding the ability of peer review percentile rankings to predict grant productivity, as measured through publications and citations. To understand the nature of these apparent conflicting findings, we analyzed bibliometric outcomes of 6873 de novo cardiovascular R01 grants funded by the National Heart, Lung, and Blood Institute between 1980 and 2011. Our outcomes focus on Top-10% papers, meaning papers that were cited more often than 90% of other papers on the same topic, of the same type (e.g. article, editorial), and published in the same year. The 6873 grants yielded 62,468 papers, of which 13,507 (or 22%) were Top-10% papers. There was a modest association between better grant percentile ranking and number of top-10% papers. However, discrimination was poor (area under ROC 0.52, 95% CI 0.51-0.53). Furthermore, better percentile ranking was also associated with higher annual and total inflation-adjusted grant budgets. There was no association between grant percentile ranking and grant outcome as assessed by number of top-10% papers per million spent. Hence, the seemingly conflicting findings regarding peer review percentile ranking of grants and subsequent productivity largely reflect differing questions and outcomes. Taken together, these findings raise questions about how best NIH should use peer review assessments to make complex funding decisions.
- Received May 8, 2015.
- Revision received June 11, 2015.
- Accepted June 18, 2015.