Oct 13 2010 Published by proflikesubstance under [Biology&Environment], [Education&Careers]
1) should unfunded PIs be included on panels or study sections?
2) Should postdocs (if funded by Federal funds) be included on panels or study sections?
Discuss with justification of your position.
24 responses so far
1. No, from a qualitative perspective. Receiving funding 'qualifies' you to comment in the eyes of your peers. If you can get funded you shouldn't be passing judgment on your fellows also trying to get funded.
2. No, from a quantitative perspective. While there are many competent postdocs who know their science and their techniques, I think there are few who are qualified to sit in judgment on something they have no *fundamental* understanding of. And secondly, for the same reason as #1, qualitatively I want to see my peer review panel being made of experts, not wannabes.
Let the flaming begin.
1. I agree with Tidey. Even though it's a bit of a lottery around the funding line having an understanding of what it takes must surely be a pre-requisite.
2. I agree again because of the reason above. But also that being on a panel or study section might well be the kiss of death for a post-doc's career. You need to be getting papers out the door. The time wasted in reviewing will cost you both the time and intellectual energy required to first author papers (and your own grants) -I reckon you may need a fair few of those in order to progress to the point that you do need to get on some panels?
Hehe... Let me guess: PlS probably askes because he has specimens of both kinds on his panel. (PlS, have you been funded by the NSF so far?)
I know I have had both unfunded PI's and funded postdocs on the panels where I served. An unfunded insanely grumpy assistant prof, cutting every proposal at the knees (judgmental much? How great are you at writing proposals if you haven't gotten any funded yet?) and a funded postdoc on another occasion. It's a different funding directorate than PlS's, but but this looks like including these types of panelists is cross-cutting.
I know the rationale for inviting unfunded PI's was to help them learn about panel review so they'd write better proposals and get funded. Many of these people are technically solid but just new to proposal writing, so they don't necessarily make bad panelists.
The same with funded postdocs. They are usually technically perfectly comptetent to serve on panels. However, they lack the experience of being independent PI's to assess some of the finer points (e.g. what can be pulled off with a certain amount of money, a bigger picture of where the field in the broader sense is going etc).
But, I would take a technically strong postdoc in my field to review my proposal any day over a crusty PI outside my area.
How's that for a wishy-washy response to your questions... 🙂
I take it you are enjoying Arlingon?
I would also add that panelists must have NSF funds, not other federal agency funds (EPA, NOAA, D0D, NIH). One of my former PIs *never* had an NSF grant, yet he's a regular on NSF panels.
Also, postdocs who were successful with their own NSF postdoc fellowships can certainly review postdoc proposals and maybe graduate fellowships. I wish NSF would reach out more to postdocs for ad hoc reviews of proposals. Many postdocs don't realize that all they need to do is email a PO to let hir know they would like to review.
Are your knuckles and elbows bloody yet?
I'm on record in favor of PIs who are not yet funded by the NIH being represented on review panels. So Yes on #1. I throw out a "maybe even some senior postdocs as well" but I always figured that was an extreme Overton shifting position. Are you telling me that NSF lets postdocs review research grants? Interesting.
I'm in favor of this because it seems like basic fairness, one, and the only way to combat biases, two.
Look at it this way- The NIH has explicit rules that study sections must have diversity. Check this link
There must be diversity with respect to the geographic distribution, gender, race and ethnicity of the membership.
In my experience this seems to be taken quite seriously. Ethnic minorities would appear to be well represented on panels on which I've served. Women run about 35-40% I think at one point I check this for my most frequent section against the CSR overall numbers. When suggesting reviewers to an SRO / discussing why so-and-so had been ad hoc'ing for two years and not appointed, it became clear to me that the geographic distribution is a pretty hard line.
Notice anything missing in this "diversity" statement? No? Well how about this comment...
There is a need for balance in the level of seniority represented among members of a study section. Too many senior-level reviewers are just as problematic as too few.
Right on. And too few junior reviewers are as problematic as too many....what? Where is that statment? Not to be seen...
So why do we have diversity requirements if not to make things *fairer* for all applicants? What would be the point of requiring a diversity of reviewer backgrounds, perspectives, seniority, geography, etc, if not to ensure fairness through the competition of biases? hmm?
So why would one suspect class of applicant be overtly and intentionally excluded? The NIH made a lot of noise recently about purging assistant professors off the panels. Their justification for doing so was almost entirely unstated and for damn sure free of any backing data.
1. Yes. Some of the more insightful reviews I've seen on NSF panels have come from unfunded PI's.
2. No. It would be an exceptional postdoc indeed who understood, really understood, what being a PI and running an independent research program is all about.
You have postdocs one your panel?!?!?!?!?!? Never heard of that before...
I'm not "saying" anything, simply asking the questions and curious how people feel. I would agree that most postdocs, in isolation, is probably not going to have the breadth of knowledge required to make a funding call on a proposal. However, bear in mind that there are three people discussing every proposal. How about these questions, then?
3) If grouped with more senior people, could a postdoc's perspective be valuable on a panel?
4) Do people really think that, given the 10% paylines, one needs to have hit the right combination of proposal and panelists to vault over that line in order to be able to judge whether some proposals are more meritorious than others?
Please allow me to be Devil's Advocate for a minute:
Really? Most of you think that a post-doc can't review grants as part of a panel?
I would agree that post-docs don't necessarily have the experience to know what can be done with a given amount of money, but we certainly have a lot of experience with TIME, since often what we are hearing is "this experiment will only take X amount of time" and we are left to translate that to mean approximately (3X +28 days) by the time the experiment is done. But that's really just talking about Stock Critiques, and there are presumably other reviewers to make sure those get covered.
Senior post-docs should be pretty good at reading and evaluating science. And if they are not, why are they still in science? Why would a post-doc be less good at evaluating the science than an associate professor that is only tangentially involved in the specific area of research discussed in a grant?
If post-docs are so bad at evaluating science, why allow them to review manuscripts for journals?
I understand, up to a point, having a standard for panelists, but the minimum standard of "funded by us" does seem a little circular.
NatC, I guess this is what I was getting at with question #3.
If you have a funded post-doc on the panel, Eli suspects they were winners of the American Competitiveness in Chemistry Fellowships which is a rather special program that funds a post doc to do independent research with academic and commercial partners. The idea is to fast track the winners.
Fellows will pursue research with industrial, and/or national laboratory, and/or Chemistry Division-funded center partners that will enrich their in-house research program. In addition, fellows will develop and implement their own plans for broadening participation in the chemical sciences.
Also, government scientists can sit on panels but cannot get NSF funding.
1. Yes, I think unfunded PIs should be able to sit on panels. Otherwise, government and industrial scientists are excluded, and I think they often have a valuable "outside academia" perspective. Also, PIs who aren't funding can still have good insight into the science, and have the requisite on the job experience to evaluate the practicalities of the proposal.
2. No. I think postdocs would be a great resource if the only criteria was how cool the science is, but I don't think a postdoc understands enough about how to run a lab/research project to be able to realistically evaluate the resource allocation/feasibility/budget appropriateness/milestone reasonableness of a proposal due to lack of experience. This was certainly the case for me. I do think it is a good idea to use postdocs as ad hoc reviewers to get their perspective and scientific expertise.
Let me correct my comment above: the non-PI academics I have met on panels may have been research scientists rather than postdocs (they were very young though!) I believe they were PI's on their funded NSF grants and I don't think a university will grant either permanent or limited PI status to a person with a traditional postdoctoral appointment (I know mine doesn't ); the appointment must be of a less transient nature. So I believe these people were actually young research scientitsts.
1) yes, certainly if by 'unfunded' you mean not holding a current NSF grant. Funding could be through another agency, which means they know how to write grants, and calling them a PI implies they run a research group, however funded (intramural, private foundation, between grants). Also, I know of some junior, yet-to-be-funded PIs who sit on panels because they want some inside experience - nothing helps you write a grant more than reading lots of them and hearing critiques.
2) Yes, on a case by case basis. I don't see why a senior postdoc with their own funding is any less capable of scientific critique than the junior faculty of yore. Give me a break. In some fields, I think the younger point of view is invaluable to shake up the same-ol', same-ol' you can sometimes get from the establishment scientists. Especially considering 3. You don't need experience running a whole lab to know if an experiment is going to yield answers, or whether the question is worth asking. Sometimes the younger guard actually understand the newer techniques better. I'll grant the senior guard more broad perspective, and big picture thinking. It seems to me a mix would be great for discussion, if only the juniors and seniors could respect one another's opinions.
The ironic thing for me is that when putting together a grant review panel, it usually ends up MOSTLY being composed of unfunded people and postdocs. Very few funded or tenure-track people will agree to spend a week of their lives at a panel review and time before that to actually read the proposals. On the panels on which I have served where there ARE these more established people, they tend to be some of the worst panelmembers who show up without having done their homework. If you want a diligent panel, pick young, unfunded people.
My experience is very at odds with this statement.
Just to be clear it's not that I think a good postdoc isn't up to the task. I just think they ought to be shielded from this sort of necessary but unproductive work.
I find it hard to think of panel work as unproductive. Maybe in the sense that the postdoc looses two weeks of potential lab work, but the benefits of seeing the process from the inside surely outweigh the very minor loss in lab time. Fuck, I've had a cold lay me out for longer than that.
It was too long a reply to post as a comment. So it's here as a blogpost.
From a lowly postdoc's POV, I don't see an inherent problem with having unfunded PIs on a review panel. As others have hinted at, this might help keep panels from skewing toward the old hunting club composite.
Regarding postdocs on review panels, I would say 'no' to unmentored review. As some have pointed out, postdocs lack the experience of having run a lab to understand what's feasible and what's not and often err on the side of being overly critical of others' work. However, I do think it would be a great idea for postdocs to engage in a mentored review process--as in pairing up with an experienced PI, a la PlS Q#3 (As an aside, it might be preferable if it was someone other than the postdoc's research adviser). I disagree that it would be 'unproductive' time for postdocs who are heading down the research track. One of the things that is perhaps most mysterious about the PI gig is grants and review processes. Having postdocs engaged in mentored review of grants would serve 2 purposes: teaching them how to review and showing them what panels are looking for in review, which could be useful when preparing applications.
My paneling is mostly for NASA, not NSF -- perhaps that's the distinction. Many people in my field are funded entirely on soft money, as there are not very many professorships in this subfield. This definitely affects who is willing to serve on panels and who isn't.
I also find that it is quite easy even for a neophyte to pick out the good proposals from the bad. It's getting the great from the good that's trickier.
Do you mean that it took you 2 weeks the first time or that it takes you two weeks now?
I should clarify that wasn't talking about mentored review, but reviewing as it is currently set up where there are three people who read all proposals. It was not uncommon in my panel for one reviewer to score a proposal very differently than the others. It was the discussion that allowed everyone to evaluate the opinions of everyone at the table and the ad hoc reviews as well. In some cases, two people changed their opinions to match the outlier because they had seen something the others had missed. In others, the outlier changed their opinion.
In that setting, I don't see how a postdoc couldn't do just fine in the review process. If a postdoc can't recognize good science form bad, there is a problem. And the proposals that end up in the top category do so by way of discussion, not on any one person's opinion.
In addition, it was my understanding that postdocs on panels get a reduced load of proposals - 9, with 3 as primary, instead of 15, with 5 as primary.
It took me two weeks of time (spread over a month, probably) to review all the grants and write up my comments. That time also includes the panel itself. This was the first time I've done this, so I guess the answer is yes for both.
I would also add that panelists must have NSF funds, not other federal agency funds
Why? Does this really give you insight into what is good science? Maybe when I finally have some NSF funding I will suddenly feel all enlightened and know all sorts of new and interesting shit just because I get money from this fountain of knowledge, but somehow I doubt it.
The typical scientist who would be interested in sitting on an NSF panel has reviewed MULTIPLE NSF proposals and written a good pile of them as well. Familiarity with the process or expectation really isn't something that is an issue here. Therefore, I can't really see how having been funded by NSF helps to judge proposals to that agency.
Notify me of follow-up comments by email.
Notify me of new posts by email.
Site Admin | Theme by Niyaz
The Spandrel Shop Copyright © 2017 All Rights Reserved