It is full proposals review season for NSF's IOS and DEB panels. Theoretically, what we are reviewing is the cream skimmed from the top of the preproposal pile (+ CAREER proposals). The reality, of course, is that we have the resulting proposals from those who were convincing at the preproposal stage. Subtle, but very different.
One thing that has really jumped out at me this year is the number of proposals that are clearly stretching to make the 15 pages. There's lots of tricks to do this, some more obvious than others, but it's still clear if you add a superfluous table that spans two pages or use five pages for your Broader Impacts that could be summarized in two. I don't remember seeing that in previous full proposal panels, but perhaps I was less attuned to it before. Indeed, some people this round aren't even bothering to fill the full 15 pages, ending their proposals a page or more early. My sample size is too small to know whether this is A Thing, or if I'm just getting an unusual number of these proposals.
I honestly also don't know whether this is a consequence of the preproposals stage or not. I mentioned this on twitter and got different reactions, but the general feeling was there was an influence of how we select preproposals.
@ProfLikeSubst Both odd and not good IMO. Suggests PIs are punting with >1 pre-props, despite not having enough 'meat' to fill a full prop.
— Leonie Moyle (@SpeciationLab) October 4, 2015
As we've discussed before, pre- and full proposals are different documents with different goals. It is entirely possible to sell an exciting preproposal that doesn't hold water as a full proposal. I don't think that's in inherent flaw, just a new feature of the system.
But I digress.
Another obvious issue with some proposals is their lack of balance between writing for the ad hocs and the panelists. What's the difference? The ad hoc reviewers will be people in your field who know the ins and outs of the system. They will be the ones to spot a flaw that doesn't take into account some recent literature in your field. They will be the ones questioning part of your specific methodology. The panelists will likely be evaluating the goals of the project at a different level. Beyond the minutiae of your system, panelists will be determining whether or not your over-arching questions appeal broadly. Is there more than just answering a subfield question here?
Unlike the preproposals that do not get reviewed outside of the panel, every full proposal walks the tightrope between these two audiences, attempting to please them both. The key to doing this is structuring the proposal so that the broadest questions and approaches are front and center, with the gritty details towards the end and well labeled. Mixing and matching the two works for some, but often makes a mess in less experienced hands. When reading a number of proposals at once, this is very clear.
As a panelist I often struggle with how much grant writing advice I should include in my reviews. I mean, that's not technically the job, but that feedback might be just as important as the scientific feedback. Not everyone out there has mentors willing or able to provide critical feedback. Unlike NIH, where the review is written specifically to the study section, NSF reviews are written more with the applicant in mind. I admit that I've found more of this type of critique slipping into my reviews than it once did. I'll be curious to see how that is handled at panel.
Of course, the biggest trick of all is ensuring one takes one's own advice when it comes time to write your own proposal.