DEB and IOS: differences in the review process

Apr 22 2013 Published by under [Education&Careers]

Before I had the opportunity to get involved with the review process at NSF, my impression was that it was standardized across the board. At least across the BIO directorate it had to be, right? Well, yes and no.

Obviously there are core aspects of review that remain the same, but there are some subtle differences as well.

Triage: DEB panels are somewhat aggressive about triaging proposals that panelists universally disliked. If you can't get three people to give a proposal more than a "Good" each, then there is no discussion about that proposal. The PI will get the three reviews, but no panel summary. IOS maintains that every proposal should be discussed, so you will always get a panel summary back from your IOS submission. I can see the advantages to both and don't think one is better than the other approach.

One unintentional consequence of this, however, is the deadline for reviews. In DEB they try very hard to get all reviews ahead of the panel in order to make the triage list and determine how many proposals are going to be discussed. For an IOS panel, they want the reviews ahead of panel, but it doesn't always work out that way. Whereas they are required to be filed ahead of the summary, they are not required prior to panel. That is not to say that these reviews were done any differently, just later.

{UPDATE: This policy may have been revised in DEB this year. See comments}

Caucusing: DEB allows it, IOS does not. The idea is simply that the reviewers discuss the proposal ahead of table discussion to see if they can work out differences. It makes the group process more efficient, but also has the potential to obscure some details from the rest of the group who might otherwise weigh in.

Previous panel summaries: On a DEB panel the reviewers are given access to previous panel summaries if a proposal is a resubmit. With funding rates as low as they are, the majority of proposals fit that category. This access allows reviewers to determine whether the PIs took previous panel advice (good or bad). IME, if there is relevant information along these lines, the POs in IOS will convey the information to the panel in discussion.

Rating: Based on panels I have been on, DEB and IOS use their categories differently, which should be considered when you read the opening review statement. DEB tends to toss a much higher percentage of proposals in "Not Competitive". In their terminology, this equates to the particular proposal not having a shot against others received. IOS, OTOH, uses the NC designation sparingly to mean that you should basically pursue other ideas. This may vary a bit between panels, but the opening review statement (where percentages in each category are reported) should be a good indication of how the panel used each designation.

As one might expect, however, the process is generally pretty similar between IOS and DEB. The science being proposed in each division is excellent and the bar is high for even the preproposals. The target invite rate seems to be in the 20-30% range at the preproposal stage, significantly winnowing the field.

9 responses so far

  • anon says:

    DEB did not triage any proposals this time around, at least for the panel I was involved

  • proflikesubstance says:

    That's really interesting. I wonder why?

  • Emilio Bruna says:

    Apparently there are differences even within DEB - the panels I've been on in DEB (full and preproposal) discussed and provided a panel summary for all submissions, even the ones with poor scores. We may not have discussed some of them for very long, but we at least discussed to agree on the flaws and then put this on paper for the PI to get feedback.

  • anon says:

    My impression was that DEB thought that triaging was not a good idea after all; I think there was no triaging in any DEB panels.

    Also previous resubmission data were not provided and were discussed only if the PI mentioned that the proposal was a resubmission.

  • proflikesubstance says:

    That's a significant departure from last year. Interesting....

  • ecologist says:

    DEB does not provide any previous panel summaries for pre-proposals (we asked, in the process of preparing to re-submit one that had been invited in last year's pre-proposal competition, but failed at the full proposal stage). In the past they have made use of that material at the full proposal stage. I don't know what they will do now when a new pre-proposal in inserted between rejection of one full proposal and submission of a revision.

  • Terry says:

    I was wondering how much these practices vary from panel to panel and depending on the program officer's overall preferences and practices. I've gotten some uniformly crappy reviews back from DEB in the past, and it still came with a panel summary.

  • physioprof says:

    Caucusing: DEB allows it, IOS does not. The idea is simply that the reviewers discuss the proposal ahead of table discussion to see if they can work out differences. It makes the group process more efficient, but also has the potential to obscure some details from the rest of the group who might otherwise weigh in.

    Any discussions of proposals outside of the formal panel discussion are an ethical breach under NIH rules.

  • proflikesubstance says:

    I've been on DEB panels that have allowed small discussions and ones that have not, but it is clearly allowable at NSF.

Leave a Reply