How I prepare for an NSF review panel

Apr 02 2012 Published by under [Education&Careers]

The first time I went to NSF for a panel I wasn't quiet sure what I needed to do to be prepared. I read like crazy and showed up fairly apprehensive about whether or not I would be prepared to do a good job. Facing my second panel in a few weeks, I have a much better feel for preparing this time around. Below are some suggestions if you are making a trip to NSF in the future.

- Get you reviews in be the requested week before... or close. This isn't just for the benefit of your PO, it is for you. NSF has a couple of different choices behind the panelist sign-in:

You reviews all get filed under the Panel Review System tab, but once you have submitted a review for a proposal you can go check out the other reviews under the Interactive Panel System tab. This is important. Not only because you'll want to know if your assessment is radically different from the other panelists (and ad hocs if you are on a full proposal panel) so that you can reread it, but for the reasons below.

-Decide which proposals you are going to fight for and find out who you will be fighting. You will be going into the discussion with a short list of proposals you really liked. By looking up the other reviews ahead of time you can figure out how much of a fight you have on your hands. What did the other panelists think of your Good List? Go look those panelists up and find out what they do. Is their expertise strong in the proposal topic? People may write opinionated reviews, in either direction, before the panel discussion and change their minds during the discussion. This seems especially true if they were out of their element a little when writing and didn't "get" the proposal. Don't be scared off of fighting for something that got hammered by another panelist, but figure out where they are coming from.

-Identify your panel enemy. Alright, maybe "enemy" is a little strong but in both of my experiences I have found that I share the majority of the proposals I am responsible for with a small group (3-5) of other people. Among that group, there seems to be one person who thinks every idea put forth is doomed for failure. Their reviews often focus on poking holes in any perceived methodological flaws, as if the PI(s) have no recourse but to follow exactly what is written without modification. This is the person you will need to convince that the proposals you are fighting for will work. Read their review, know what they work on and be prepared to spar with them.

-Get a sense of the panel's mood. After reading the reviews you have access to, are they generally positive or negative? Is this going to be a panel that spends most of its time finding reasons to whack a proposal or finding reasons to support them. There is always a mix of both, but there can be a very different feel to panels that go one way or the other.

-Figure out if anyone else shares expertise in your area. Scan the list of names on the panel. Know anyone? Read anyone's work? I wouldn't go e-stalking anyone or reading people's stuff just for kicks, but I find it useful to have a little idea what other people do and what they may weigh in on.

That's what I do in the lead up to a panel. I'm sure some of the more experienced Peeps out there may have other suggestions as well. Above all, remember that you are there to talk science and participate in the process of getting people funding. As a group you'll have to make some hard decisions and it's easy to walk out of one of these feeling a little depressed by the amount of good science that doesn't make the cut. But do what you can, learn what you can and get to know the other panelists - there's a decent chance they will be reviewing one of your grants some day.

13 responses so far

  • anon says:

    Really interesting post, thank you. If you find that the panel seems negative overall, do most proposals end up shitcanned? Is there some quota that the panel must fill in ranking the proposals as high, medium, or low priority?

  • proflikesubstance says:

    For the full proposals, the POs try to limit what gets into the top two categories, IME. For these preproposals we have been told that the target for getting invited to the full proposal stage is between 15-20%. So yes, there are a lot of proposals that get tossed. Some deserve it ad others just don't quite measure up for any number of reasons.

  • Whoosh... says:

    I love the thought that someone on a panel might fight for my proposal and spare with the "enemy" to get it through! (=
    This year I got my first duty as a proposal pre-reviewer, before they are going to the actual panel. Any advice on what makes a good and helpful review for the panel members?

  • proflikesubstance says:

    IME, ad hoc reviews are not given the same weight as the panelist reviews. Panelists do not get the ad hoc reviews until getting into the room (or maybe a couple of days before, depending on you PO), whereas they have had the other panel reviews for at least a week or so. Therefore, there is not a huge amount of time to evaluate them. Often they are scanned to look for things the panelists missed because they are submitted by people more closely associated with the work. They are consulted if there is a major discrepancy between the ad hocs and the panel reviews, but otherwise they just get a once over.

    That said, they are provided as feedback to the proposal writer, making them an important tool for the PI. Whereas the panel reviews are usually short because the panelists know that the discussion is what will ultimately matter, the ad hocs often delve into more detail. Therefore, they often tend to be of more value to the PI than the panel.

  • Whoosh... says:

    so reviewing for the proposal writer not for the panel? ... then there definitely must be room for grantsmithing feedback!

  • arrzey says:

    This doesn't address the new strategy (in many panels) of a shorter first application (4 pages) followed by an invitation to submit a full proposal to something more than 10% and something less than 50% of those submissions. The first panel has NO ad hoc reviewers and a huge amount of work to do.

  • proflikesubstance says:

    I'm aware, dude, I'm on one shortly. The ad hoc thing is not an issue for the preproposals, but certainly will continue to be a factor in both the full proposal stage and many other panels that haven't gone to a preproposal.

  • [...] What Jon Bardin misses: the costs of graduate school (must-read) Shouldn’t We Be More Skeptical of the DeepChallenger Dive? Reforming Science: Methodological and Cultural Reforms How I prepare for an NSF review panel [...]

  • Just to follow up, since I'm working on proposal reviews, I thought I would try the interactive panel trick and it did not work. I get a message saying the the panel rules do not permit reading the reviews. I'm not sure who sets the rules, might be that the program manager for this panel doesn't want people adjusting their views based on the comments of others.

  • Eli Rabett says:

    Well, in NASA panels, you can see the other reviews AFTER you submit yours.

    My experience with the ad hoc reviews is that they are the most important if no one on the panel is expert in the area of the proposal, and I have been on panels where the panel asks the program managers to go out and see if they can get a quick read from a particular person overnight. Sometimes this works and it really helps.

  • Eli Rabett says:

    Oh yes, a better word is opponent.

  • [...] MUCH. And I want to share my new-found knowledge with YOU, of course. PLS had a great post from when he was on an NSF review panel, so hopefully my experience on this NIH study section will also be useful. Therefore, consider the [...]

Leave a Reply