What your NSF grant ranking really means

We're coming up on the time when people are going to start to hear from the NSF POs about whether or not their proposals got funded. With that in mind, I thought it might be a good time to talk about how proposals are ranked and what it means.

When I get a rejection back, and I have a good amount of practice here, I tend to start with the rating and panel summary and then dive into the reviews. I've talked about why a panel summary may or may not be a good summary of the discussion, but when you get a good one there is a tremendous amount of information in there. Sometimes you have to read between the lines a bit, but it is more important than I once thought to address the issues that arise in the panel summary because those are specifically discussed in the subsequent panel.

One other important thing to take note of is the context statement. Unlike NIH, there is no numerical rating of the proposals and corresponding cut-off value. Instead, there are four categories: high, medium and low priority, and not competitive. The context statement will tell you how many proposals came into that program and what percentage got ranked in each category. This tells you a lot about both the panel and your competition. In looking back through my previous declines, those numbers vary widely and my proposals have been in programs where 5% of proposals were rated as not competitive (VERY unusual) and 55% were in that same category. Based on my experience, the latter is far more normal.

"Not competitive" is roughly equivalent to being triaged at NIH. Although every proposal is discussed at the panel, if you fall into that category the panel is telling you that you have a proposal flaw big enough that, even if there were infinite resources available, they would not trust your proposal to produce results worthy of the financial support. Yes, that sucks, but the reasons your proposal falls into that category should be clearly spelled out in the panel summary, but read it carefully, because many summaries are not as blunt as they should be.

If you managed to steer your proposal into one of the other three categories, congratulations, the panel that you had good ideas that would result in interesting data and publications. But here's where it gets a little tricky. Obviously, the proposals in the high priority category have a better chance of being funded than those in the low, but there are several factors that keep the POs from just funding the top rated proposals. Once the panel has done its job, the POs get down to brass tacks. They have a portfolio to fill and simply funding all of the panel's favorite proposals may not tick all of the boxes the PO needs to check.

Part of NSF's mandate is to do things like making sure that PUIs get a share of the pot. Proposals from EPSCoR states are another consideration. Career stage is important as well, not just for beginning investigators, but mid and late-stage PIs are considered as well (e.g. projects like OPUS proposals). It's not just about how your proposal was ranked, but about making all the pieces fit together.

For this reason, anything in the top three categories is technically fundable, making the differences between the top three categories less important than making it into those top three in the first place. Obviously, preference is going to be given to the higher rated proposals, but the POs have a lot of leeway here when it comes to who gets the money. Particularly because there is no ranking within each category, proposals in the medium priority have a decent shot at getting picked up if there is money to do so. The flip side of that, of course, is that proposals in the high priority category may also go unfunded.

This is what can make NSF so maddening or forgiving, depending on where your proposal falls. Watching a proposal that was previously ranked in the high priority category fall to a lower one was tough on the panel I went to, because I know that as a proposal writer I would be screaming my ass off when I got that summary back. OTOH, as someone on a panel A) we can only judge each proposal in the context of what else is on the table, and B) having a proposal just tossed back into the next round without addressing the previous concerns just because it was ranked highly in the last round is not a great way to make an impression. It is a delicate balance of changing just enough to improve the proposal without opening up new issues, but that is true for any proposal that gets out of the pit of "not competitive".

12 responses so far

  • drugmonkey says:

    if you fall into that category the panel is telling you that you have a proposal flaw big enough that, even if there were infinite resources available, they would not trust your proposal to produce results worthy of the financial support.

    hmm. I get the sentiment but this is not universally true anymore in the NIH review. Yes, there will be some applications for which this is true. However, with the number of excellent apps the NIH is getting, there are going to be some that get triaged that would be coolio with infinite (or even merely trebled) funding.

  • Odyssey says:

    Exactly. My field and panel don't have 55% in the non-competitive category, more like 20-25%, but otherwise my experience mirrors this.

  • avocado says:

    How soon after the NSF panel meets, should you anticipate hearing back from the PO??

    I know that Fastlane gives you 6 months, but I would anticipate that successful proposals would hear from the PO shortly after the panel meeting.

  • proflikesubstance says:

    DM, I'm not saying that the proposal is shit or that it wouldn't produce something of value, but that the funding to useful data ratio is skewed to the point that the panel didn't think the investment was a good one.

    Avocado, the million dollar question... Unfortunately, there is no right answer and I have had anything from 4 to 6 month turn around. It depends on so many factors that you just have to wait. The lack of a congressional budget only adds to the problems, but NSF has been approved to spend to their previous budget. What that means for people on the bubble, however, is that the PO may hold on to these proposals for longer hoping for some $$ to filter down once the budget is officially decided.

  • drugmonkey says:

    Since I'm not familiar, are the reviewers more engaged in deciding "this should/not be funded" in NSF review? In some senses the mere use of "not competitive", which suggests "not competitive for funding", may be answering my question ?

  • proflikesubstance says:

    Yes. The panel ranks the proposals, and although the POs have leeway across the "priority" categories, my understanding is that they do not reach below the not competitive line, no matter how much money they have to play with.

  • yellowfish says:

    My application from January, which was my first one, came back reviewed as one of 3 tiers- "Competitive- persuasive compelling" (~7% of them, based on their numbers), "Competitive- but lacking in some respects (~38%), and "Not competitive" (~55%). (I feel like for some reason the wording "competitive but lacking", which I got, sounds extra insulting when you say it out loud, although thats probably just because they didn't fund me)

    Do different panels use different rating systems, or is the High/Medium/Low mapped onto this in some way I'm not clear on? I don't see that rating system anywhere in my review, but I'd also never seen an NSF review before this one.

  • proflikesubstance says:

    Yes, there is some discrepancy here and I know that different programs have, at times, tried different things. In fact, the terminology used on the panel I was on does not match the panel summary I have received from said panel in previous rounds.

    That said, the basic premise remains the same. Stay out of the 'not competitive' bin and you have a shot.

  • drugmonkey says:

    Officially the NIH system stamps down hard on any mention of whether or not a grant should be funded at the level of peer review. Review is supposed to evaluate the relative *merit* of an application but not discuss whether it should be funded or not funded.

  • proflikesubstance says:

    In a coninuous numerical rating scheme, you can do this. NSF has bins, however, making this inherently more difficult.

  • laserboy says:

    More globally, things are quite varied. For instance, I have been in funding rounds where the committee decided that they had too many applicants and decided to throw out half the proposals before review. If you end up in unlucky half, you don't get any feedback whatsoever. Makes it hard to improve the proposal.

    In other cases, all proposals are reviewed, no matter how many are received. But the field is as broad as possible. Essentially anything that might attract co-contributions from companies can be funded, so biology is mixed with physics is mixed with civil engineering etc etc. The reviewer comments are very useful, but the committee never offers any comments about why particular proposals didn't succeed. I suspect that because a wide range of topics are reviewed together, the committee finds it very difficult to decide. That same wide variety probably makes it difficult to give a coherent justification for deciding between proposals as well.

  • [...] At least in the 4 panels I have heard about in BIO, funding lines are coming in around 6-9%. The continuing resolution and 1% cut in the new Fed budget mean that things are tight and this is certainly no surprise. Knowing this was going to be a slim year, POs have been urging panels to wield I mighty hammer when it comes to putting proposals in the black in the above graph and 80% of proposals ending up "non-competitive" is not unusual this year, in stark contrast to previous years. [...]

Leave a Reply