NSF funding: myth, hyperbole and luck

May 04 2017 Published by under [Biology&Environment]

Preproposal news is starting to leak out of the BIO directorate, and with it a renewed sense of anguish among those who were not invited to submit a full proposal. In particular, the invite rates were painfully low this year, coming in around 22%. Considering that only 25-30% of THOSE will ever see funding, it appears that success rates in the NSF core programs continues to decline.

This is bad. Worse than we've seen in this generation.

Naturally people see these numbers and turn on the process, calling it random and a lottery, then focusing on the expertise of reviewers as a problem.

I get it, I really do. But before we decide to tear down the machine, let me make two points:

1) There is a certain amount of randomness to this process, but it is not as great a negative force as people seem to like to believe. Everyone who has been doing this long enough has a story about a proposal that did well once or twice and then never crossed the line. Everyone. Of course, if you have survived long enough to tell that story, you probably have a story about a proposal that got denied a couple of times, but through your will and brilliance managed to break down the doors to funding!

If you have served on enough panels, you may or may not recognize these as two sides to the same coin. The only difference is on one hand we were screwed by variance in the system and on the other we won by grit and determination. Both these narratives fit nicely in our comfort zone, while ignoring the reality of the enterprise. For every proposal you managed to "get through despite the odds", some other PI "got screwed by the system". And thus the mythology of funding goes....

2) Funding is no lottery. That is not to say there is no luck involved (there is), but a lottery treats every entry the same. I can assure you, having reviewed well over 100 proposals to two different panels, all proposals are NOT the same. At least half are not even in the running for funding. At the preproposal stage IOS has taken to triaging probably about 1/4 right off the bat, and another 1/4 might as well be. In that upper half though, things get more interesting.

To make the jump from upper 50% to invited for full proposal, you really need to hit all the right marks and have people on panel who you get excited over your ideas. Easier said than done, I know. Once you get into the full proposal stage, well, now is when separating signal from noise gets pretty tough at 6% funding rates. I would argue that probably 70-80% of the science that makes it this far is fundable and would produce quality publications. We even have data on funded projects that demonstrates that reviewers can not predict the success of the project and it's hardly a stretch to say that our ability to draw a line between what should and should not get money at that level is suspect.

And here's where we identify the real problem.

All of the issues people identify in the system are symptoms of there just not being near enough money to fund all the good science. Of course there is luck involved when so many proposals that make the initial cut are worthy of support! The variance we see is mainly because of the constraints we are operating under and not a fault of the system, per se. If the 22% invite rate was the funding rate, none of this becomes a problem. But when you need to fit 100 hippos in a Chevy Nova, shit breaks down.

We need to stop complaining about the process and continue to apply pressure in the only place that will fix any of this. Without additional funds, this continual flatlining of the NSF budget will continue to make consistent funding a pipe dream.

4 responses so far

  • eeke says:

    Also keep in mind that reviewers review these proposals until we're blue in the face. Carefully discuss all of them, write out the reviews, everything. We try to craft reviews of proposals we know will not be funded in such a way as to not discourage the applicant. Great idea! Great project, but.... That sort of thing. But at the end of the day, the decision lies with the Program Directors. The reviewers make recommendations, and the PD's make the decision based on those recommendations, and also on location, demographics, topic of interest, etc. The point is that the science itself is only part of the package. I've reviewed a lot of grants that I wish could have been funded - it's heart-breaking to see them go down at a review panel, but even more so when you know only a small number of those excellent projects can be funded. It sucks.

  • AcademicLurker says:

    @eeke: Being on an NIH study section is even more depressing than being a supplicant...er, I mean, applicant. You get to see all of the amazingly strong proposals that don't get funded.

    There's an argument about getting more junior people on study sections, but in a way I'm glad I never served on a (NIH, I reviewed for other organizations like AHA) study section as a junior faculty member. It would have been so discouraging I might have been tempted to give up.

  • Well, I still think that the education from serving on panel or study section is too valuable. It really changed how I wrote grants.

  • A Salty Scientist says:

    I'm a junior faculty member, and just served on my first NSF panel. Having successful grants from colleagues to use as a guide is useful. Seeing why those grants were successful during panel discussion is invaluable. There are certain things that seasoned reviewers look for that a noob won't know.

    Discouragement and the temptation to give up is something I worry about. I had an NIH grant triaged shortly before the pre-proposal deadline, and was somewhat tempted to not even bother flipping the proposal to NSF. I did though, and that grant ultimately got invited and funded. I think about my decision to submit the grant in the context of no hard grant deadlines, which BIO is apparently considering. I might have been tempted to acquire even more preliminary data and postpone submission. I think that rolling deadlines might be a good thing, but I do wonder if certain demographics may be adversely affected.

Leave a Reply