We're coming up on the time when people are going to start to hear from the NSF POs about whether or not their proposals got funded. With that in mind, I thought it might be a good time to talk about how proposals are ranked and what it means.
When I get a rejection back, and I have a good amount of practice here, I tend to start with the rating and panel summary and then dive into the reviews. I've talked about why a panel summary may or may not be a good summary of the discussion, but when you get a good one there is a tremendous amount of information in there. Sometimes you have to read between the lines a bit, but it is more important than I once thought to address the issues that arise in the panel summary because those are specifically discussed in the subsequent panel.
One other important thing to take note of is the context statement. Unlike NIH, there is no numerical rating of the proposals and corresponding cut-off value. Instead, there are four categories: high, medium and low priority, and not competitive. The context statement will tell you how many proposals came into that program and what percentage got ranked in each category. This tells you a lot about both the panel and your competition. In looking back through my previous declines, those numbers vary widely and my proposals have been in programs where 5% of proposals were rated as not competitive (VERY unusual) and 55% were in that same category. Based on my experience, the latter is far more normal.
"Not competitive" is roughly equivalent to being triaged at NIH. Although every proposal is discussed at the panel, if you fall into that category the panel is telling you that you have a proposal flaw big enough that, even if there were infinite resources available, they would not trust your proposal to produce results worthy of the financial support. Yes, that sucks, but the reasons your proposal falls into that category should be clearly spelled out in the panel summary, but read it carefully, because many summaries are not as blunt as they should be.
If you managed to steer your proposal into one of the other three categories, congratulations, the panel that you had good ideas that would result in interesting data and publications. But here's where it gets a little tricky. Obviously, the proposals in the high priority category have a better chance of being funded than those in the low, but there are several factors that keep the POs from just funding the top rated proposals. Once the panel has done its job, the POs get down to brass tacks. They have a portfolio to fill and simply funding all of the panel's favorite proposals may not tick all of the boxes the PO needs to check.
Part of NSF's mandate is to do things like making sure that PUIs get a share of the pot. Proposals from EPSCoR states are another consideration. Career stage is important as well, not just for beginning investigators, but mid and late-stage PIs are considered as well (e.g. projects like OPUS proposals). It's not just about how your proposal was ranked, but about making all the pieces fit together.
For this reason, anything in the top three categories is technically fundable, making the differences between the top three categories less important than making it into those top three in the first place. Obviously, preference is going to be given to the higher rated proposals, but the POs have a lot of leeway here when it comes to who gets the money. Particularly because there is no ranking within each category, proposals in the medium priority have a decent shot at getting picked up if there is money to do so. The flip side of that, of course, is that proposals in the high priority category may also go unfunded.
This is what can make NSF so maddening or forgiving, depending on where your proposal falls. Watching a proposal that was previously ranked in the high priority category fall to a lower one was tough on the panel I went to, because I know that as a proposal writer I would be screaming my ass off when I got that summary back. OTOH, as someone on a panel A) we can only judge each proposal in the context of what else is on the table, and B) having a proposal just tossed back into the next round without addressing the previous concerns just because it was ranked highly in the last round is not a great way to make an impression. It is a delicate balance of changing just enough to improve the proposal without opening up new issues, but that is true for any proposal that gets out of the pit of "not competitive".