What you learn from reviewing a batch of proposals

Oct 06 2015 Published by under [Education&Careers]

It is full proposals review season for NSF's IOS and DEB panels. Theoretically, what we are reviewing is the cream skimmed from the top of the preproposal pile (+ CAREER proposals). The reality, of course, is that we have the resulting proposals from those who were convincing at the preproposal stage. Subtle, but very different.

One thing that has really jumped out at me this year is the number of proposals that are clearly stretching to make the 15 pages. There's lots of tricks to do this, some more obvious than others, but it's still clear if you add a superfluous table that spans two pages or use five pages for your Broader Impacts that could be summarized in two. I don't remember seeing that in previous full proposal panels, but perhaps I was less attuned to it before. Indeed, some people this round aren't even bothering to fill the full 15 pages, ending their proposals a page or more early. My sample size is too small to know whether this is A Thing, or if I'm just getting an unusual number of these proposals.

I honestly also don't know whether this is a consequence of the preproposals stage or not. I mentioned this on twitter and got different reactions, but the general feeling was there was an influence of how we select preproposals.

As we've discussed before, pre- and full proposals are different documents with different goals. It is entirely possible to sell an exciting preproposal that doesn't hold water as a full proposal. I don't think that's in inherent flaw, just a new feature of the system.

But I digress.

Another obvious issue with some proposals is their lack of balance between writing for the ad hocs and the panelists. What's the difference? The ad hoc reviewers will be people in your field who know the ins and outs of the system. They will be the ones to spot a flaw that doesn't take into account some recent literature in your field. They will be the ones questioning part of your specific methodology. The panelists will likely be evaluating the goals of the project at a different level. Beyond the minutiae of your system, panelists will be determining whether or not your over-arching questions appeal broadly. Is there more than just answering a subfield question here?

Unlike the preproposals that do not get reviewed outside of the panel, every full proposal walks the tightrope between these two audiences, attempting to please them both. The key to doing this is structuring the proposal so that the broadest questions and approaches are front and center, with the gritty details towards the end and well labeled. Mixing and matching the two works for some, but often makes a mess in less experienced hands. When reading a number of proposals at once, this is very clear.

As a panelist I often struggle with how much grant writing advice I should include in my reviews. I mean, that's not technically the job, but that feedback might be just as important as the scientific feedback. Not everyone out there has mentors willing or able to provide critical feedback. Unlike NIH, where the review is written specifically to the study section, NSF reviews are written more with the applicant in mind. I admit that I've found more of this type of critique slipping into my reviews than it once did. I'll be curious to see how that is handled at panel.

Of course, the biggest trick of all is ensuring one takes one's own advice when it comes time to write your own proposal.

8 responses so far

  • Zen Faulkes says:

    I am surprised by the perception that a proposal less than 15 pages is seen as... inadequate. Given how often I hear reviewers complaining "I AM READING SO MANY PROPOSALS DON'T MAKE READING YOUR PROPOSAL HARD FOR ME", I would have thought that a concise proposal would be welcomed, not dinged.

  • It is possible you can make your proposal convincing in less than 15 pages. Very possible. But in doing so you leave your back door WIDE open to claims of "you never convinced me of X and you left plenty of space to do it!"

  • Ola says:

    There's one thing worse than leaving blank space - filling it with guff. Worst example I ever saw was a BSD proposal in which they spent the entire last page mansplaining the lack of experimental detail. You know the score - even of the mice don't exist yet we should trust them because of their amazing track record.
    If only they'd spent that space actually filling in the details!

  • jimbo says:

    I'm one of those who submitted a <15 page proposal (likely not in your area.) First time I've been invited to do a full proposal. My preproposal had a clear flaw that at least one panelist pointed out, but which somehow I didn't clue into until a few days before the full proposal was due; my scrambling at the end (I've always struggled with procrastination) resulted in a weak third aim and room to spare. I'm already assuming it won't be funded and thinking about the next cycle (probably a good strategy even when one is more confident), but it's somehow reassuring that I'm not the only one to have left space.

  • There's countless reasons people might not fill the whole thing. Again, it's not a fatal flaw and POs will tell you that if you can say it all in less space, that's fine. IME, though, it almost always gets commented on by the reviewers. Even when you do fill the space, at least one will tell you where you wasted some space when you could have been explaining what they wanted to hear more about.

  • kjfafiu says:

    I think that the shorter proposals is a clear and obvious outcome of having to write preproposals. When you write the proproposal it take a lot of effort to condense your ideas down to just 4 pages. Then if you are invited for a full proposal you are asked to expand back out to 15 pages without adding in any new ideas. At that point you often find that the super tight concise version works really well and that it is nearly impossible to add 11 more pages of text without putting in some obvious fluff or introducing new ideas. Of course the worst is when the full proposal is rejected and then you are forced to condense back down to 4 pages...

    And what's wrong with a long broader impacts statement? Broader impacts is one of the primary components on which proposals are to be judged – why should they be restricted to 2 pages?

  • I never said they should be restricted to 2 pages, simply that use 5 when 2 would suffice is an issue.

  • Juan Lopez says:

    Applications are much less than 15 pages at most places I submit or review for. Typically, 2-12 pages. Could it be that some people are used to shorter applications and then find it hard to expand without looking diluted? It's the same argument of kjfafiu above, but based on other applications not just the pre-proposal.

    Ola, you never get tired of being sexist? It would have been enough to say that they explained something you didn't buy.

Leave a Reply