Open Thread: How does an NSF panel work?

Mar 21 2013 Published by under [Education&Careers]

A lot of the conversation I had on twitter following my post on panel prep earlier in the week made me realize how many people are curious what actually happens in a panel. I was going to write a description, but that seems kinda boring, so I figured I would leave it open for questions and for others with experiences in different NSF divisions to weigh in. They all have different cultures and their own way of making the process work. So if you're curious what the review panel experience is like, fire away.

34 responses so far

  • Colin says:

    How much of a bonus do new PIs really get? How is this factored in in practice? Same question for diversity and gender weighting

  • proflikesubstance says:

    It doesn't factor in at the panel level, IME. Status is something considered by the POs as they are balancing portfolios. I think panelists are aware of it, but it is rarely specifically addressed.

  • KateClancy says:

    My experience is same as PLS - status wasn't discussed. Diversity might have been mentioned once or twice if it related to broader impacts, but it never seemed to positively or negatively influence where the proposal got placed.

  • odyssey says:

    IME status is only discussed if you have a senior PI applicant who has not been very productive. And that's not a discussion you want to be the subject of.

  • evolcoop says:

    I've read that the pre-proposal stage evaluation emphasizes big and innovative ideas/questions. I have also heard off-hand comments from established PIs that in terms of preliminary data, it's almost like you have already done the experiments you are proposing. How do panels view innovative and high risk ideas/approaches in terms of the preliminary data they would think is necessary (more because it's high risk?), particularly at the preproposal stage, but also throughout review. I would love opinions on this matter in DEB vs MCB, if some of you have participated in either. Thanks!

  • proflikesubstance says:

    I can only provide a DEB perspective, but Od might be able to weigh on from the MCB side. The "preliminary" data argument goes round and round all the time. What it comes down to is fairly simple: Can you convince a panel that you are on to something interesting and that you can get the results you say you can? Fair or not, early career people are going to have to provide a bit more preliminary data at all stages of review to demonstrate feasibility. I don't think people necessarily think of it this way, but established PIs will have more published and unpublished data to rest their proposal on than someone starting out their lab.

    WRT how much, that's going to vary. But for new people, erring on the side of too much is probably a good plan. First, for the reasons above, and second, because you're probably underestimating how much data you need.

    For some perspective, I submitted a proposal my first year that was going to cover some of the core work we do. I had very little data and it got denied. That proposal is in the "likely to fund" pot right now (four years later) and the preliminary data in support of it encompasses the first and second aims of the original proposal. The current proposal takes those data to demonstrate feasibility and then goes well beyond that in the new research plan.

  • New panelist says:

    Question: how often does someone who is not one of the assigned reviewers weigh in in the discussion of a proposal. And how often does that (seem to) change the outcome?

  • proflikesubstance says:

    IME, it is relatively rare and unless they bring up a big concern related to their specific expertise, the impact is usually not huge. The POs generally like to keep the discussion centered on the reviewing panelists. Particularly because of the need to stay on time.

  • ScienceGuy says:

    It all depends on the PO in charge. Sadly, this is how it works in my experience:

    1) Primary reviewer blabs on and on about the proposal, often ignoring all salient facts, and most definitely any external ad hoc reviews. Only three people are listening anyway, because most people are typing their summaries or prepping to blab on and on as soon as they get their turn to play primary reviewer. The three people listening are the PO, the secondary reviewer, and the summary writer. The secondary reviewer is mostly just wondering whether they'll have anything new to add (usually not), the summary writer is trying to figure out how to condense the blather spewing from the primary reviewer's mouth, and the PO has already made up hi mind anyway but wants to hear whether anyone detects any flaws.

    2) Everyone votes on the priority, based on whether the primary reviewer sounded positive or negative or competent or incompetent.

    That's it -- Basically the problem with NSF is that your proposal's success depends on one person. If the primary reviewer likes it and is charismatic, you're funded. If the primary reviewer doesn't like it or is a dork, nothing except a forceful PO will save it.

  • proflikesubstance says:

    IME, it takes one of the reviewers to champion it. Doesn't matter which one. The preproposals are a little loser as well, and it doesn't take someone pushing hard for it to clear the bar.

  • ScienceGuy says:

    Extra tip: The primary reviewer/panel presenter is one of the reviews you'll receive. Look for the individual review that most closely matches the 'panel summary'. That's the person that determined whether your proposal was funded. If they're crazy (and they often are; sadly those are the lonely people that most like to go to three day meetings and eat in mall courts), then you can chalk the lack of funding up the bad luck and resolve to try again. Even though they don't have an effect on whether you get funded because they are often ignored at panel meeting (although good POs will call out reviewers who ignore them), DO pay attention to the external reviews when revising your proposal or moving the work forward. Those comments are usually the best and most useful because the people writing them had only your proposal to review and not a stack of them. Thus they usually spent more time on your proposal. They are also experts in the field and you are reading their comments because of that expertise (instead of just their willingness to stay in a chain hotel in the most boring part of DC imaginable).

  • odyssey says:

    evolcoop:
    MCB doesn't have pre-proposals. They're moving to a once a year submission (this November), but have said they're not considering pre-proposals at this stage. Not clear yet how that's going to pan out.

    As far as prelim data goes, what PlS said. Except I would caution young people to be very careful not to get caught up in the never-ending "just one more piece of prelim data" cycle. Realize that your proposal will never be "perfect". Get it to "good" and submit. There is only one guarantee in all this: no submissions, no money.

    Also, I recommend you read DrugMonkey's related post. His advice applies to the NSF as well.

  • odyssey says:

    New panelist:
    I agree with PlS - it's rare. Not unheard of, but rare.

  • ScienceGuy says:

    ExtraExtratip: READ the NSF descriptions of what 'Broader Impacts' is, and address them all. A lot of people think it's just training stuff. It's not. Publishing in Nature and talking to the press is actually an official 'Broader Impact'.

    That said, many NSF panelists are people from small undergraduate institutions, and they think that an important part of NSF's mission is training and support of smaller primarily undergraduate institutions. So even though it's not official NSF policy, people tend to value the involvement of undergrads or high school kids more than other potential broader impacts.

    But overall, broader impacts are like icing. They can work in your favor, but unless you really screw this section up it won't hurt the success of a great project. Except for some proposal types (CAREER, etc), which explicitly require a training component.

  • odyssey says:

    ScienceGuy:
    That's not my experience on panels. Yes, it takes at least one of the reviewers championing your proposal. But that's true on all panels/study sections. Funding rates suck and it is what it is at the moment.

    The panels I've been on have largely been filled with successful PI's who take the job very, very seriously. Yes, one or two might be typing up the summaries of the last couple of proposal reviews, but the rest do pay attention. If for no other reason than to know how best to position the proposals they want to champion.

    External reviews are not ignored IME. Unless it's one that's devoid of content, which makes them useless.

  • odyssey says:

    ScienceGuy:
    Broader Impacts are not icing - they are an integral part of an NSF proposal. And Nature papers are not viewed as BI's IME.

  • ScienceGuy says:

    odyssey: I agree that broader impacts are an integral part of an NSF proposal, but great broader impacts will never make up for bad science. In contrast, great science can get an enthusiastic response even with bland broader impacts ("...this project will provide training opportunities for undergraduate and graduate students...") It's icing. Lots of times I have heard a PO have to ask a panelist reviewer: "And what about the broader impacts?" And when BI are mentioned, it's usually in this context: [After a breathless description of the great project] "...AND the broader impacts are good!"

    I agree that panelists tend to weigh more heavily broader impacts having to do with education. But that is because many of them are from undergraduate institutions and they value that, as I said above. But look up the NSF BI definition & evaluation criteria -- Societal impact and wide dissemination of the results are important. NSF has an educational mission, yes. But it also wants to fund great science. Its motto is 'Where Discoveries begin'. They are not the Dept of Education.

  • ScienceGuy says:

    odyssey: As for who serves on panels... I have served on panels, and I like to think highly of myself. I serve on panels because I know POs have a very hard time finding reviewers and panelists. And I feel a duty to serve. But overall, I must admit, the people who serve on panels are the same sorts of people who like committee meetings back at their home institution.

    You wrote: "Yes, one or two might be typing up the summaries of the last couple of proposal reviews, but the rest do pay attention. If for no other reason than to know how best to position the proposals they want to champion."

    You admit right there that most people are not paying attention. "...to know how best to position the proposals they want..." is not paying attention. It's prepping for their little moment in the limelight, just like I said above. Paying attention is listening to the proposal description, thinking about the review criteria, actively asking questions, and making an *informed* recommendation for how a lot of heard-earned taxpayer money should be spent.

  • DrugMonkey says:

    I hate committee meetings and avoid them at all costs. I like serving as a NIH grant reviewer because it feels like a community good and payback for all the grants I submit for consideration. Plus, I like to make a difference and this is one way to do that. One of the best feelings in this business is saving someone else's grant. Doesn't happen often that you single handedly make a difference but it does occur.

  • ScienceGuy says:

    "That's not my experience on panels. Yes, it takes at least one of the reviewers championing your proposal."

    Actually, odyssey, we agree on this. That's what I said. I think you simply disagree with my cynical description of the process.

    For the rest of the readers: "Championing" = A panel reviewer must argue for your proposal, because everything else (ad hoc external reviews, etc) tend unfortunately to be ignored.

    Yes -- A half dozen Nobel laureate external reviewers can love your application (or hate it), but some dork with nothing better to do except go to DC to bloviate on panel will be the one who actually makes the decision.

    Sorry, that's the way it is. If you hate that and you are competent, then REVIEW WHEN ASKED AND GET YOUR DAMN ASS ON TO PANEL. This is your friends and neighbors' tax money! Make sure it's spent wisely.

  • odyssey says:

    ScienceGuy:
    Firstly, I've couched everything in terms of my own experience. You seem to feel what you have experience applies to all NSF panels. It doesn't, in the same way that what I have experienced doesn't apply to all panels. Blanket statements are not helpful.

    Secondly, you are reading into what I wrote. No where did I say most people are not paying attention. Maybe I could have been clearer. Listening to the other reviewers in order to better position the proposals you wish to champion IS paying attention. What I meant by that is the following: people pay attention in order to know whether the science/BI's in the proposals they want to champion is as good as, worse than, better than the proposal being discussed. You can only do that by paying attention to the proposal description, the reviews and thinking about the review criteria.

    I'm sorry to hear your panel experiences have been so negative.

  • While my experience of the general process is similar to ScienceGuy's (there are panelists, the primary panelist for a proposal presents a summary review), many of my details and perceptions are very different.

    On both panels I have been on there was often vigorous discussion about proposals, but it was occurring before the 'official' panel discussion. The POs on the panels I have attended encourage the three panelists on a given proposal to discuss the proposal and come to consensus before the panel discussion. So, in my experience (limited to DEB), just because one panelist is giving the summary of that discussion doesn't mean they are the one who 'made the decision' or that the decision was made by one person. If that is not how this works on other panels, I'm very sorry to hear that.

    As for the comment that most panelists are from small undergraduate institutions, in my experience that is only true if you consider everything but Harvard, Yale, and Princeton 'small undergraduate institutions'. On my last panel, of the 20 odd people, I can only think of 3-4 that were from the undergrad focused institution category. Over half were heavy weights in my field (from the junior rising stars to the senior silver backs). But again, my experience is limited to a very small section of NSF. Clearly mileage can vary.

  • Jim Woodgett says:

    Review panels have changed over time. In the past few years, the dynamic has moved from focusing on science to jockeying for positioning of assigned grants within an ever decreasing funding rate. Too many criteria are listed for evaluation with the interpretation being largely left to the reviewer (despite guidelines, none are enforced). Chairs seem to concerned with keeping the meeting short. There are more top-down competitions and I am certainly concerned that truly excellent grants are being trashed by people who are more than ever concerned about their own research support. The panels are too big and there needs to be much greater peer-pressure - on the reviewers themselves.

  • odyssey says:

    One person does make the decision about whether or not to recommend funding. But it's not any of the panelists/reviewers.

  • I stand corrected. Yes, the POs make the final decision. The panel only recommends. I should have said ' In my experience it is not just one bloviating panelist that is making the recommendation for a given proposal'. 🙂

  • Angie says:

    My experience after 6 panels (albeit USDA) is that the majority of people take the responsibility seriously and do the work that needs to be done. The panels I've been on have functioned almost exactly as described for NSF (but USDA does not require BI or ad hoc reviews), and I feel like the cream effectively rises to the top for funding.

    I also concur that good panelists choose the proposals out of their bunch (1-3 out of maybe 20) that they want to "fight" for, and if they are charismatic, knowledgeable, and aggressive it usually means that proposal will be funded (if the other reviewers don't have significantly different opinions). A good panelist will identify and prepare a response to conflicting reviews ahead of time, if they feel like the proposal is worth fighting for. But nobody takes any panelist seriously if they make a case for every proposal they got.

    The ratio of good panelists who understand this system and are personally capable of taking up the fight to the more generally clueless/less confident is maybe about 2:1.

    Some USDA panels are moving to virtual meetings (to save on expenses), where everyone tunes in online from their home institutions. I haven't done that yet. I am really curious about how that is working out and whether it effects the dynamics significantly.

    I know the thread is specifically on how NSF panels work, but maybe someone finds my perspective useful.

    "If they're crazy (and they often are; sadly those are the lonely people that most like to go to three day meetings and eat in mall courts)"

    ScienceGuy, WTF?

  • ScienceGuy says:

    DrugMonkey: My experiences at NIH are different. Being on NIH panel is a status thing, often (sadly) reduced to dick-waving. It's the equivalent of the old 'smoke-filled room' where decisions get made.

    And yea, everyone else, of COURSE my perspective of NSF panels is only my perspective. I assumed we all understood that as a given. And no, not all my experiences have been negative. In fact, the first time I went to NSF panel, I did nothing for weeks but tell people how cool it was and how impressed I was with the colleagues I met.

    But I am not so naive as to tell you that NSF is perfect. As a taxpayer and scientist, I wish it were so. But it's not. Our job as scientists is to be cynical, and I am as cynical of NSF and and the review process as I am my science and the science of others. Others should be too. That's the first step to making it better.

    Familial and professional obligations prevent me from working at NSF right now, but they know when that will change, and I have been welcomed. When/if I get the power to do so, I will make it clear that everyone who gets funded is EXPECTED to serve on panel ASAP and as often as possible. I don't like a lot of things that NIH does, but I like that they draw their reviewers from their pool of fundees.

  • Bashir says:

    Don't you mean skeptical, not cynical.

  • A/C says:

    "Our job as scientists is to be cynical..."

    I call BS. Our job as scientists is to be critical thinkers, but if you are only cynical, then you're more likely to be dismissive. That's not the same as making a discerning judgement based on the facts at hand and the input of other perspectives.

    Also, your "I have been welcomed" at NSF business? Seems to be of the same behavior category in which you put NIH panelists.

  • drugmonkey says:

    It's the equivalent of the old 'smoke-filled room' where decisions get made.

    that implies that it is some sort of power-dependent collusion at work instead of a defined process. This is false by design and, IME, false by practice. Sure, there are cultural forces at work that tend to shape the behavior of a given section and a single highly charismatic and persuasive reviewer is always going to have outsized influence. but it is hardly a smoke filled room where the power-mongers conspire to pick all the winners and losers within their business for the greater aggrandizement of ....what exactly? a subfield?

  • ScienceGuy says:

    I meant cynical, not skeptical. We're supposed to be skeptical, but our job requires cynicism. tragic, I know. But that's the way it is.

    DM: "it is hardly a smoke filled room where the power-mongers conspire to pick all the winners and losers within their business for the greater aggrandizement of ....what exactly? a subfield?"

    Yea... that's exactly what I think it is. Many would like to think otherwise, but c'mon... On your own blog you have categorized study section personalities. The illusion of objectivity is just that.

    These are human endeavors, folks. What's with all the apologism?

  • Odyssey says:

    I meant cynical, not skeptical. We're supposed to be skeptical, but our job requires cynicism. tragic, I know. But that's the way it is.

    No, our job does not require that at all. Or at least mine doesn't.

    And there's no apologism. Others, myself included, just don't view things the way you do.

  • drugmonkey says:

    On your own blog you have categorized study section personalities. The illusion of objectivity is just that.

    The failure to achieve invariant "objectivity" (whatever that means) does not in any way imply the complete absence of this property.

  • BirdNerd says:

    Sorry to be late to the discussion, but I just wanted to add my experience serving on multiple panels for DEB, both under the old system and on a panel to discuss pre-proposals under the new system. Again, this is all from my experience, I obviously can't speak to the experience of others.

    First, for the panels I've served on the panelists are very accomplished, experienced scientists from a wide range of schools (major, major research institutions, mid-level state schools, RUIs, everything in between). The panelists also span a wide range of ages, from assistant professors to fully tenured. They all appear to take their responsibility quite seriously and most of them do it because they believe it's important and are willing to take time out of their lives to spend 4 days in DC. I don't think anyone does it just because they have the time.

    To echo other comments, it takes someone in the room to champion a pre-proposal to get it placed in the 'Invite' category (of course, POs have the final authority). At the pre-proposal stage there are no external reviews, only the three assigned panelists clustered in the room. That being said, one of the two secondary reviewers can easily standup for a proposal and change its fate for the better (or worse). Given the absolutely dreadful funding rates (5% or less seems to be the word this year), in order to make it onto the 'Invite' list, a proposal basically had to be in two of the three panelists 'top three' pre-proposals (out of ~24 pre-proposals assigned to each panelist). It's not much of a consolation to the PIs, but plenty of really outstanding pre-proposals don't make it because rates are so low.

    On the DEB Blog they have some numbers about the fate of pre-proposals submitted by Young Investigators versus more established researchers and at least in the first round of pre-proposal to full proposal to funding, the young guns didn't seems to fare any worse than the others. In the pre-proposal panel I served on, it was discussed quite a bit and panelists were often very concerned with providing Young Investigators with copious constructive comments in the panel summaries, especially when the pre-proposals fell in the 'Do Not Invite' category.

    BIs are always a topic of discussion at panels, and I think even more so with the pre-proposal system. As others have noted, I've never seen a bad batch of BIs sink and otherwise outstanding proposal, but it does seem to help when a pre-proposal's Intellectual Merit put it right on the cusp of making it into the 'Invite' pile.

    Lastly, IME, given the incredible financial constraints under which NSF operates, especially since there are so many proposals, they actually do a remarkable job of funding outstanding science. The problem is that there are many more outstanding proposals submitted than can possibly be funded.

Leave a Reply