Thoughts on the transparency of panel service

Dec 03 2012 Published by under [Education&Careers]

As I mentioned on Saturday, I strongly encourage junior people to try and get on an NSF panel. A couple of people commented that they have tried to get on a panel and not been successful. Although this has not been my experience, I can see how this could easily be the case. As I already stated, POs need to balance a number of things when putting together a panel. If your research topic is smack in the wheelhouse for a particular panel, there may be many willing participants who fill your "panel niche" and it may be tough for you to get into the rotation.

What surprised me a little was Zen's comments:

As solid as this advice is, it bugs me. It makes it look like getting an NSF grant is determined by unwritten standards maintained by an established network of well-connected people, with fairly high barriers to entry.

The NSF shouldn't a be a country club....

....That there is a substantial number of replies indicating people want to serve, but haven't been invited, and that there's no clear understanding of why that is, does not help promote the appearance that panel selection is a transparent process.

This line of thinking doesn't make sense to me. How, exactly, would you make panel invites a transparent process? Considering that NSF heavily guards it's panel membership, I don't see how this goal is attainable. I'm not sure it should be.

The reality is that some people make better panelists than others, based on their experiences. Maybe the PO has seen your proposals and likes what they see. Maybe you tick the right boxes in terms of the knowledge that a panel needs to fill a hole. If it were just first come first serve then there would likely be large gaps in the field coverage of the panel. When you account for the need to include gender, ethnicity and employment stage/place balance, yes, there will be some people who have a hard time getting on a panel.

There's nothing Country Club or In Crowd about it. I did not know any of the POs on my first panel and my training was in another country, further reducing my connectivity. I contacted my PO after an unsuccessful application and asked to serve on the next panel while revising my proposal. For me, it worked. For others it may not be that simple, depending on their expertise and the needs of the panel.

I don't see how the selection process could or should be a "transparent" process, but maybe there are some arguments out there.

14 responses so far

  • Bashir says:

    NSF grant is determined by unwritten standards maintained by an established network of well-connected people, with fairly high barriers to entry.

    I am more familiar with NIH but are you saying this isn't true. I don't mean to say that there's a conspiracy or anything nefarious. Just factually, that sentence isn't accurate?

  • proflikesubstance says:

    I think the standards are pretty clear. There's a whole grant manual that NSF puts out annually on this very topic. I would also argue that the "established network of well-connected people" is much more fluid in the NSF world than the NIH one. There really isn't the same renewal mechanism in DEB and IOS that exists at NIH, and to a lesser extent, some NSF directorates.

  • Zen Faulkes says:

    I was hoping my aluminum hat crack would indicate I was indulging more in idle conversation than substantive criticism.

    I understand that in complex organizations, there is a lot of tacit information, and individuals often serve as institutional memory rather than documents. I am currently in the throes of program evaluation, where we are asked for all kinds of information that is not necessarily written down in any one place. You find out by asking people.

    Let's see some comments in the previous thread, not written by me.

    "Are international postdocs eligible to sit on a panel?"

    "Is there some club or something? Do you have to know one of the PO's personally? How much do I need to be a pest about it?"

    "Do you know if faculty at SLACs/PUIs have served on panels, or if NSF prefers only those at R1s?"

    All of those questions are probably better described as questions arising from tacit information, rather than lack of transparency.

    Sorry.

  • proflikesubstance says:

    All of those questions are probably better described as questions arising from tacit information, rather than lack of transparency.

    Or, 1) from people either not looking the information up, or 2) not just asking a PO. These aren't secrets and any PO will give you the answer if you just ask. I'm willing to bet it's even written somewhere if we bothered to look.

    People ask for directions all the time, even though we have maps and GPS devices on practically everything. Does that mean that navigation needs to be more transparent?

  • Zen Faulkes says:

    I wasn't trying to say any of that stuff is secret. I've heard "tacit information" described as the sort of water cooler stuff that people know but may not be explicitly written anywhere.

  • Confounding says:

    "I think the standards are pretty clear. There's a whole grant manual that NSF puts out annually on this very topic."

    I think this statement better captures the conflict. Either the NSF grant process is a well-documented, clearly standardized process, or one where your chances of getting a grant funded are greatly improved once you are "in the know" by sitting on a panel.

    I think it's kind of hard to argue it can be both at the same time.

  • proflikesubstance says:

    The standards are clear, what you gain insight on is the experience of the reviewer. Until you're up at 12:30 am working through you 12th proposal, you may not fully appreciate the need for clarity, subheading and figures to break up text. This is not some unwritten rule, it's just experience.

  • femalephysioprof says:

    Ask a PO, but my understanding is that NSF maintains a database of reviewers and uses this database together with scientific expertise needs to help decide on candidate panel members. So, build your portfolio (or credit score, if you will) by agreeing to review the next NSF grant application that arrives in your in-box. This way, the PO will have some information about you as a reviewer to rely on when considering your request to join a panel.

  • tree guy says:

    It does say somewhere on NSF's website that if you are interested in reviewing or serving, you should get in touch with a program officer. I did. Wasn't asked to review anything, but asked to sit on a panel about a year later. Don't know what the secret is, but if you are bombarding them with proposals, sooner or later, they are going to notice and ask you to return the favor. I have a mental picture of what NSF panels are like and I try to use that in crafting proposals. However, this is pure fantasy on my part, I want to see how the sausage gets made.

  • chicaScientific says:

    Does anyone know whether or not you can find information about the rank of all proposals at a certain panel? I would very much like to put my rank in context but can't find this information anywhere.

  • proflikesubstance says:

    Does anyone know whether or not you can find information about the rank of all proposals at a certain panel? I would very much like to put my rank in context but can't find this information anywhere.

    In fastlane there is a cover page for your reviews that should give you all the details. It's the page where you click on each review. There should be two or three paragraphs of text above the links that give you the number of proposals evaluated and the percentage that fell into each rating category.

  • Maybe that is division specific or at the POs discresion. Some POs will not update the date of the panel and others will. Though, I have never heard of anyone in engineering finding out the number of proposal and percentage that fell into each category for a certain panel. I'm trying to check again on Fastlane, but of course the site keeps crashing.

    I would think that information would not be available since the PO can fund almost any proposal that they want. If a panel only ranks one or two proposals highly recommended and you are one of them and you didn't get funded, then you probably wouldn't let your PO hear the end of it, but if you don't know you can't say with certainty that you were skipped over.

  • proflikesubstance says:

    Every Bio review I have ever gotten back has had a cover page with the stats. It doesn't tell you who got funded, just something like "There were 129 proposals, of which 15% were rated high priority, 22% medium priority, 25% low priority and 38% not competitive". This goes back to 2008 in my fastlane dashboard.

  • Bethany says:

    I think the real strategic advantage of serving on a grant panel is that it lays bare the true responsibility you have as a grant writer to write well. I am a second year Assistant Professor and I was able to serve on a panel recently. I think it made my next grant infinitely clearer. By being on the panel, I was able to pick up on the writing cues that make what is obvious to an expert reader equivalently obvious to experts in fields that are quite far removed from my own. Thinking about it strategically, I got to see a larger variety of grants as a panelist than I am likely to write myself over the next 5-6 years. Why not learn from other peoples' mistakes before making more of my own? As I read and re-read the proposals I was responsible for and skimmed through many of the others, it became easy to see the shape of well argued science and organized research teams. It reminded me that what I think is self-evident probably isn't evident at all outside of my micro-discipline.

    The experiences interacting with other panelists was both inspiring and enraging in equal measures. Some, I felt were too quick to dismiss particular fields and other prone to giving reviews that were not helpful to the scientists who has prepared the proposal. Overall, it was a good reminder that there is a clear difference between a good and bad proposals, but that there is an element of idiosyncrasy that can't be accounted for.

    More nobly, I found the experience to be an overwhelming responsibility and honor as I was able to play a small role in the scientific agenda of a major funding agency and the trajectory of scientific discovery. While science budgets for a single call for proposals is trifling in comparison to other national expenditures, it was a weighty experience and well worth the chaos it created in my teaching schedule, the hours and hours of extra work, and subsequent illness.

    It was also a reality. I was one of only a handful of women on the panel and the only representative of social sciences in an interdisciplinary call where many proposals confused societal impacts with social science (to the biophysical scientists and engineers: I am equally capable of doing arcane, jargon rich, impenetrable research and, unfortunately, sometimes do). It was hard, it was intimidating, but I would readily do it again (especially if the grant I wrote three months later gets funded).

Leave a Reply