Dealing with proposal rejection

Apr 18 2011 Published by under [Education&Careers]

NSF BIO panels are meeting right now, or have met in the last couple of weeks, depending on your panel of choice. That means funding decisions are working their way through the system and notifications will be forthcoming in the next few weeks. Unfortunately, ~90% of are going to be disappointed in the results. Also unfortunately, I have a LOT of experience being in that 90%.

While I have no problem providing advice for proposal writers based on my experience from both being on an NSF panel and through making (and theoretically, learning from) many mistakes along my path of grant writing, one of the biggest things I have learned is how to deal with proposal reviews for an unsuccessful proposal.

The first thing I do with my reviews is print them out and read them over fully. I've talked before about the NSF rankings and what they mean. I get the whole "these reviewers don't know their ass from their elbow" thing out of my system and put the reviews away for a couple of days. Then I'll go back to them and read them again, while making a list of things I need to address and things I got right. What did the panel focus on? What were common flaws perceived by more than one reviewers? How can I fix those? Will it require data or a change in the focus of the question being asked? What proposal flaws did they not come out and say, but are between the lines? What are the strengths to build on? What did reviewers not understand?

Once I have a good list, I use that instead of the reviews as a guide. For one, it separates any emotional reaction to the reviews from the revision process, but it also gives me a handy check list and reminder of what I need to address and clarify. After that, I'll set up a phone appointment with the PO. This is critical for getting a better feel for how the panel conversation went and why certain points became focal issues. It is also essential for advocating for the changes you plan to make and getting a read on whether the PO thinks those will adequately deal with the concerns. Take notes.

I can't stress enough how important it is to bounce ideas off the PO. More than once I thought I had a new strategy figured out only to have the PO say "but that's not going to address XXX". Whereas I was initially frustrated by some discussions I had with POs because it felt very one-sided, I was ignoring the subtly of the language they have to use. Talk about your revision ideas but even though you'll be doing most of the talking, you have to listen carefully to the reaction. Remember that they know exactly what people did and did not like about your proposal and that information may not appear in your panel summary, depending on how it was written.

Funding rates are not good right now, so it's important to make every application count. Remember that if there were parts of your proposal that the reviewers didn't "get" then you need to clarify the language there and make sure it is obvious what you want the reviewers to take away. It is YOUR JOB to make sure even the most distracted reviewer walks away from your proposal convinced it can work, because you have everything to lose if they don't. Your PO can help you figure out the direction to take your research questions, but you need to package it up all purty like.

24 responses so far

  • Namnezia says:

    I like your strategy of hiding the reviews once you take notes on them. That way you don't get pissed off every time you see them, but rather have a nice list of "things to do to improve my grant" which will also allow you to start working in a more directed manner.

  • drugmonkey says:

    I very often go through the summary statements for NIH grant reviews with two highlighters- one for positive comments and one for negative. Then when I'm preparing the revision I have a nice outline.

    oh and that last bit, spot on, my friend, spot on.

  • gerty-z says:

    This sounds like a great strategy. One of the hard things to deal with in this business is the rejections-you have to be able to extract useful information without getting emotional and then move on.

  • brooksPhD says:

    "I can't stress enough how important it is to bounce ideas off the PO. More than once I thought I had a new strategy figured out only to have the PO say "but that's not going to address XXX". Whereas I was initially frustrated by some discussions I had with POs because it felt very one-sided, I was ignoring the subtly of the language they have to use."

    this is so damned important. "have you considered adding a biostatistician?" isn't a question, it's a statement saying, "Get a statistician on board STAT."

    I just (co-)hosted a 2-day grant writing workshop and one of the sessions was about strategies for success in grant writing/getting funding. A large part of it was devoted to when and how to get in touch with your PO.

  • ScienceGeek says:

    I have been meaning to set up a time to talk with my PO and just did b/c of your great post.

  • Neuropop says:

    Sorry to burst a bubble but funding decisions on proposal reviewed in this round will not be forthcoming for the next couple of months at least. Many directorates at NSF are still trying to clear the backlog of decisions from submissions in the past summer due to the lack of a FY11 budget. However, the rest of the post is spot on. My PO didn't even bother to use the zombie PO code. He just said "exciting but no panel is going to fund this as is. Show that it works first. If you need money to do so, come back for a small exploratory grant." Feedback like that is invaluable.

  • Josh says:

    PLS,
    I've glanced at your blog a few times in the last several months. Given this limited sampling, I wanted to say THANK YOU for providing me with an excellent resource and educational tool. You have single-handedly provided me with a resource for my graduate students that exemplifies both the myriad work-related concerns that a young academic will face and the delicate, whiny nature of the majority of their future peers, should they decide to commit to a career as an academic. Just one question, what will you complain about if you do get a grant before tenure?

  • proflikesubstance says:

    Probably trolling internet douchbags, if I had to guess.

  • fcs says:

    Definitely sounds like a good strategy. (Though, wow, those 10% funding rates scare me, and I'm sure with the current budget they're just going to get worse).

    With papers, I always have a backup plan. Like, submit to top-tier conference with <20% acceptance rate, get rejected, take reviewer comments into account, then resubmit to a lower-tier conference. Would you advice a person to do something like that with grants? (For example, if you had a NSF grant rejected, you revise, tweak, and turn it around and submit to NIH or DOD ?)

  • For example, if you had a NSF grant rejected, you revise, tweak, and turn it around and submit to NIH[?]

    AHAHAHAHAHAHAH!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    In relation to getting feedback from POs, yes it can be very valuable post-review, if they were at the review panel meeting. One thing I have learned to be very wary of, however, is the utility of getting PO feedback on a grant pre-submission. POs are never gonna tell you, "This thing stinks", even if it stinks.

  • proflikesubstance says:

    If you have a proposal that you CAN tweak and send to another agency, but aren't doing it, all I would ask is why? There isn't really a tier system there. Everyone is sending their proposals to the agency that is most likely to fund them and anyone else who will look at them.

  • fcs says:

    Ah, so the reason I ask is I once attended a workshop with POs from most of the major funding agencies in my field. One PO said, "You need to focus - just submit one grant to one agency your first year." Another PO (from another agency) said, "That's crazy - submit as many grants as you can to as many different places as you can."

    Because I work in an interdisciplinary area, I can sell my research to several different agencies. But I'm still trying to get a feel for the landscape to understand what good practice is. For example, from what I heard at the workshop Agency A would be displeased to hear you have a very similar proposal in simultaneously at Agency B. (Possibly to the point of disqualifying you). But they of course wouldn't care if you submitted to Agency B after they've rejected your proposal. Whereas Agency B could care less if you also have something similar in at Agency A.

    As a newbie to this I don't yet understand all of these inter-agency political nuances.

  • CoR says:

    Nice, thanks. I'm very proud to have inspired a post on rejection. 😉

  • GMP says:

    FCS, PlS has hinted that there is substantial difference in funding focus among agencies, so you really want to submit to the best fit agency from the get go. It's not that the NSF is somehow better than NIH or DoD; they simply all fund different things, may have more or less overlap in basic science funding but typically have very different priorities. So you want to submit to the agency that holds your work in the highest regard/relevance. FWIW, most CS folks I know are largely funded by the NSF, although I have seen one with NIH funding and a couple with DoD and DOE funding (these are typically multi-investigator grants where the CS person does highly applied work tailored for biomedical or defense or energy applications).

  • rknop says:

    Last time I got a proposal back from NSF -- my last rejection, for it sent me into a spiral of despair that eventually led to me leaving academia (temporarily, it turned out) -- there was pretty much no hope. Talking to the PO, it sounded like I had very little chance. (The best he could offer me was to spend a year trying to figure out what the hot topic next year was going to be and write a proposal on it.) The comments from the reviewers were *directly contradictory*. It was just ridiculous.

    Despair is about the only reasonable reaction I've ever managed to find for funding proposal rejection.

    I've done much better with telescope proposal rejections. (I say that partly because I've had telescope proposals receive time as well as proposals be rejected.)

  • proflikesubstance says:

    More often than not, conflicting reviews indicate that you need to clarify your proposal so that reviewers are on the same page when it comes to the science. It's frustrating as hell to get contradictory reviews, but the panel summary will come down on one side or the other (or the PO will clarify) and it is up to you to ensure that reviewers don't misunderstand you the next round.

  • proflikesubstance says:

    FCS said: Ah, so the reason I ask is I once attended a workshop with POs from most of the major funding agencies in my field. One PO said, "You need to focus - just submit one grant to one agency your first year." Another PO (from another agency) said, "That's crazy - submit as many grants as you can to as many different places as you can."

    FCS, I think this is where one PO is pushing for the solution that works best for them. Every PO would rather have one tight proposal rather than three that got 1/3 the attention. It makes their job easier and keeps the funding stats up (fewer proposals in the pot means higher % funded). But from the individual PI perspective, having one iron in the fire significantly reduces your chances of getting funded. Of courses, spamming panels or study sections isn't ever going to get you funded, but have multiple quality projects that you are generating good data for is pretty much key. Also, using those data for multiple proposals is also critical.

    FCS said: Because I work in an interdisciplinary area, I can sell my research to several different agencies. But I'm still trying to get a feel for the landscape to understand what good practice is. For example, from what I heard at the workshop Agency A would be displeased to hear you have a very similar proposal in simultaneously at Agency B. (Possibly to the point of disqualifying you). But they of course wouldn't care if you submitted to Agency B after they've rejected your proposal. Whereas Agency B could care less if you also have something similar in at Agency A.

    I'm not sure this is all that big of a problem. In fact, I have heard more stories of co-funding between agencies than people being snubbed by one because of another. In my conversations with POs, the tone has always been that seeing where else a proposal is being submitted creates opportunities, not reduces them. Someone else may disagree.

  • proflikesubstance says:

    Rejection muse.... has a nice ring.

  • Namnezia says:

    FCS - You can submit to as many places as you can, but if you happen to get it funded at two places, then you have to forfeit one, usually the one with less $$$ (and it's the other way around, for life-sciences NIH budgets are far bigger than NSF ones).

  • fcs says:

    Thanks, all, that helps clarify things.

  • CoR says:

    Sounds like the name of one of those heavy metal bands that you consider 'music'.

  • Josh says:

    Well now that I've been accused of being an internet douchbag, which, by the way confirms the "delicate" part of my previous post, I'll try to contribute something meaningful. Have you ever considered that we are all just waiting in the queue? Unless your grant is fatally flawed, all this talk of trying new agencies, etc. is just a form of bet hedging with no proven links to increasing likelihood of funding. While it is always wise to have more than one proposal going at once, there is another strategy that I don't see mentioned very often - patience and resubmission. At NSF there is significant turnover in reviewers, panel members and even PO's. As such, there is a good chance that a proposal will be reviewed by different people, at every level, for any given resubmission. If your grant gets good reviews, there is not much value in "revamping it" entirely, as you may get a whole new set of reviewers, with new complaints. Do the bare minimum on that grant to address previous reviews, resubmit, and focus most of your creative time and energy on developing completely different proposals, not "completely overhauling" a grant that is already written. As a pre-tenure PI I have adopted this approach with my NSF grants with success.

  • proflikesubstance says:

    Well now that I've been accused of being an internet douchbag

    I merely remarked that I would complain about such people when I run out of faculty concerns to discuss. Don't be so delicate.

    I haven't seen anyone anywhere suggest that overhauling a proposal that reviews well is an advisable strategy, nor do I do that here. But no matter where a proposal falls in the review spectrum, talking to the PO is a good idea. Experience indicates that what ends up in the panel summary was not necessarily the focus of the discussion and the PO can illuminate that for you. I have also watched proposals that were well reviewed in a previous round go backwards at the subsequent panel after doing the "bare minimum" (at least in the eyes of the panel), so making some effort to modify according to suggestions is not necessarily a waste of time. If it only needs a tweak here or there, the PO can also help out there.

    Overall, I agree that you just have to revise to the best of your ability and resubmit.

  • Josh says:

    While I agree that it is important to speak with the PO, I think you are putting too much emphasis on interpreting the nuances of the PO's response. I have found conversations with PO's to be highly variable and often influenced heavily by the panel. For example, for my last grant proposal, I resubmitted 3 times - it was funded on the 3rd. The PO was the same for the first two submissions and different for the third. Prior to the third submission, things had gone downhill badly - to the point that the PO suggested, essentially, that we either "completely revamp" or not resubmit. I addressed the reviews reasonably (against the PO's suggestions) and resubmitted. Turns out the panel was similar for the first two submissions but different for the 3rd. End result - very little revision of a very good proposal that was eventually funded. I have 2 other, older colleagues who have had this exact same experience. I guess the take home message is: if your proposal is good, it will eventually be funded, if you are patient and have good data to support your proposal.

Leave a Reply