How your proposal gets reviewed

PiT has an excellent post up about the manuscript review process, which I thought might be worth following up with a post on grants. See, when I first sent proposals in I pictured a process during which my 15 pages of text was carefully read, considered over some time and probably looked over a second time for clarification and summation of key points.

Hahahahahahhaahaha.

I have done my share of ad hoc reviewing, usually of between 1 and 4 grants a round for the last couple of years. When you have a month to comment on one or two grants, it's not hard to read it through a couple of times (at least some parts), look up some references that you are curious about and think about the proposal before writing your review. Often, I'll come back to the review a week or so later and make sure I got my points across in a way that reflects what I think of the project.

When you have a stack of reviews (can you believe that I even got 2 more ad hoc requests today, despite being on a fucking panel!) it changes the equation. Why is this important? Because I'll be the dude at the table either advocating for your proposal or not. With 15 proposals on my desk, there are two ways they get reviewed:

1) At work I read through the proposals with a minimum of 20 interruptions per grant. A good day is one where I can actually finish an entire proposal in a day, but most get spread over two days. I can't close my door and not answer phone or email for these because I would be dead to the world for a week if I did that for 15 proposals. Let's not forget that the semester is starting and I have a couple of things to attend to around that time.

2) At home I open a proposal after dinner, after the Wee One has been put to bed and I've had a long day. Maybe I'm drinking a beer while reading your proposal, maybe something stronger. I'm probably tired and maybe thinking of (or, *ahem*, writing) my next blog post.

This is actually kinda key to think about when you are writing a proposal. Undivided attention to your proposal is probably unlikely. What have you got that is going to capture my attention and keep me from answering that phone or email? Is your proposal going to make me tell one of my grad students to come back later with their problem?

Maybe more importantly, have you written the proposal in a way that is conducive to reading with non-continuous attention? Did you stack your hypotheses all at the beginning, only to never have them appear again? Maybe not a good idea. Is your summary catchy and poignant or an afterthought you whipped together with scraps off the cutting room floor? Yes, it is odd that the people who have more time to look over your proposal are not in the room when it is being discussed, but that's the deal.

This is not to say that I'm going to skim these grants and do a half-assed job, by any means. I take it as seriously as I would want someone else to do when reading my proposal. What it does mean, however, is that how the grant is written is going to factor into my impression of it more than you might expect. I don't have time to sift carefully through your three paragraph explanation of how wonderful a technique is when all I want to know is how you are going to apply it. Make it easy for your reviewers to take home your main points, otherwise you're going to get drowned out in the static.

p.s. And don't make lots of stupid grammar and spelling mistakes. Everyone makes some, but if I am seeing one or two per page I start to wonder if I am putting in more effort than you did.

13 responses so far

  • Namnezia says:

    Well put! But unfortunately in may cases you are forced to spend three paragraphs writing every minutiae of a technique because some hard-assed reviewer in the previous round questioned its validity. I recently started relegating this stuff to a methods section, so if a reviewer just wants to read the narrative they can do so without being interrupted with technical asides, which they can read later if they want.

  • proflikesubstance says:

    Agreed, this is a huge challenge in the NSF system because of the constant change in the panels. I have tried to deal with this by making it clear up front what parts reviewers asked for more detail on, but by the time a reviewer gets to page 8, they probably don't remember. I think a methods section (and delineating sections clearly, in general) is useful to orient the reader and keep them focused on what you want them to get out of the section.

  • physioprof says:

    For NIH grant review, even for assigned reviewers, the Overall Impact score is probably about 90% determined after having read nothing more than the Specific Aims page and the Biosketch of the PI. This is why I strenuously urge people I am mentoring in grantsmanship to spend a huge amount of time crafting and fine-tuning the Specific Aims page. If by the end of the Specific Aims page, the reviewer is excited by the proposed studies, then you are gonna do great, and if they are not, then you are fucked. This is regardless of how amazing or poor the rest of the grant is.

    And BTW, no one reads the "Summary" paragraph when reviewing grants, which is what is intended for public consumption.

  • proflikesubstance says:

    The 'summary' in NSF speak is not the same thing as the NIH one. It's a one pager somewhat similar to the specific aims page.

  • Thank you for explaining why my PI kept asking me to repeat myself when I was writing my mock grant. It was driving me nuts, because I kept thinking "fucking PI"s are reading this, not idiots. How many times do I need to tell them why I"m proposing what i'm doing" .

    Reading this post, the bell went off. Oh yeah it like me trying to get through paper while doing experiments. I find it so challenging bc I forget what I was reading during the last incubation..same thing for PI's and grants.

  • GMP says:

    PLS, great post. There is (unfortunately common) option 3) for when an NSF proposal gets read for the first time: 2 am in a hotel room in Arlington, VA, the night before the panel. Or if you are lucky, on the plane ride before.

  • Our most recent grant has a separate methods section, after we ran into some criticism of the flow of similar grants in the last year. It makes it soooo much easier to read.

  • antipodean says:

    Vertical readability is your friend

  • Steve Gough says:

    Nice advice, thanks. I'm very sad to read this, though.

    My little semi-profit R&D firm has unsuccessfully submitted an education project three times to NSF. We've expended perhaps $30K on employee time, and hundreds of uncompensated hours for me, the owner. Our last try included 17 collaborators on 12 campuses. I organized all that (imagine the phone calls, emails, and paperwork), some of the best in the field joined us. I figure at least another $30K in faculty/staff time was spent by others; so about $60K in resources went into this. We resubmitted because it was so clear what we were proposing was extraordinarily good stuff. The last round of reviews was a joke; at points I thought perhaps I was reading reviews for the wrong proposal. The program officer clearly steered the panel who (except for a couple who raved about it) went along and skimmed it. Things we mentioned multiple times in response to past reviews (going along with your busy/nonlinear reader theory) were completely ignored by some, criticized for "overemphasis" by others. All this for ~$500K we were seeking (over 10% spent writing/researching.)

    This system is broken, and your description of how reviewers work affirms my suspicions. There needs to be a preproposal process so people like me don't waste hundreds of hours writing good proposals only to have biased officers and overworked/easily steered reviewers reject them. And in our case NSF clearly violated stated policy by rejecting the proposal because a "for profit" was involved.

    A preproposal process would let people like me see these problems (especially the bias and lack of policy enforcement by the program officer, which is unknowable beforehand) without such tragic investment and we could go elsewhere for money.

  • proflikesubstance says:

    Steve, while I am sure that you put in a lot of effort I'm not sure how that is different from the time and effort that any PI puts into the grant writing process. I submitted a proposal last year that had 5 co-PIs in 3 different countries and we put in a huge amount of time on it. This was on top of me teaching a new course and adding other responsibilities as well. It didn't get funded, so we're going to fix it and resubmit.

    Multi-author grants often suffer from not being as cohesive as single-author proposals because so many hands are in the pot. Might it be possible that the reason it was reviewed the way it was is simply because you didn't spell out your plan as carefully or clearly as you think? If you get back reviews that don't look like they are addressing your proposal, usually that is a sign of an unclear proposal, not 5 reviewers and a PO out to get you - though that is a easier claim for the proposal writers.

    I'm not sure how a pre-prosal would help that anyway. If you want policy enforced, then refer to it in your proposal. If you had a "for profit" involved, they should have written a letter of support and this would have been the place for such a policy to be highlighted.

    Rather than storming off in a huff, it might be more useful to look at what the reviews tell you about your proposal more than what you think your proposal said.

  • Steve Gough says:

    proflikesubstance,

    First, huge props for putting this topic on the web, it's been so far unspeakable, for obvious reasons. And I appreciate your thoughts on what we did.

    We submitted three times, and carefully refined, corrected between each; this was a massive, carefully focused effort. Thousands of person hours.

    And the for profit was me. I was integral. Carefully explained. Panel members violated policy, and I believe after being coached by the PO, by assuming any for profit was a ripoff and finding reasons not to fund, while NSF's official policy is to encourage public-private partnerships. Believe me, it was a flagrant violation of stated policy.

    We would have protested, and talked about it, but of course all my collaborators had pending proposals (they all got funded, no fair! :-).

    And still not in a huff, just saying to NSF: Define what you want, and don't move the goalposts after the proposal's submitted. The panel members/program officers are not following stated policy, the field is very far from level, good teaching/science is trampled by personal bias and policy violations, and that's not right.

    My point again: A pre-proposal system could fix that. With much less effort I can see the PO hates my idea or who I am and move on. I'm cool with that!

    Many thanks for this thread!

    Steve

  • proflikesubstance says:

    Steve,
    Thanks for coming back to clarify your situation. My assumption that you were one-and-done with NSF and complaining that the system was broken because you didn't get funded was based on numerous comments, and an indeed a website dedicated to those same complaints. I see now that is not the case.

    The program officers do indeed have a lot of latitude when it comes to their decisions and your experience seems to be a case where this was clearly not a good thing. Despite this, I don't agree that a preproposal would be a good way to go.

    My reason for that is twofold. First, I doubt that there would be enough information in the preproposal for the PO (or whoever was reading it) to be able to clearly say no in your case or in cases that face the most common problem: not enough 'preliminary' data. Second, adding another written step to the process would slow things down and kill those of us submitting multiple proposals. I you are writing 5-8 NSF proposals a year the preproposal becomes a major burden. Same goes for the POs, who seem to me to be pretty damn busy already. If anything, it would mean that even less attention gets paid to them. It may weed out the proposals that have no business getting into the system, but certainly not well thought-out proposals that may have some flaws once they are dug into.

  • [...] first post was by Prof-like Substance, who explained exactly how your grant proposal gets reviewed. (Kinda reminded me of “I’m Only a [...]

Leave a Reply