Thoughts on the ESA letter to NSF Bio

From multiple fronts I have recently been made aware of an effort that started at the Ecology Society of America to formally protest the new NSF Bio regulations on grant submissions. For those not aware, the short version of the changes includes a move from two annual cycles to one, the institution of a preproposal stage and a limit of two grants per PI in any capacity.

These changes were met with resistance and the general feeling that early career people were going to take it in the teeth. Only the DEB and IOS programs in Bio initially made these changes, but MCB recently let it be known that they were going to do the same after a year utilizing a different approach.

The ESA letter focuses on the concern that the new policy is going to significantly slow the pace of science in the Bio Directorate, an opinion I have written in the past. I'll deal with each of the points of the letter separately:

The process creates an exceedingly long lag between the time when ideas are first proposed and when funding becomes available to investigators. Even if the two preproposals allowed per Investigator per year are successful, it takes over one year from submission to funding (as opposed to 6-9 months in the former system). This lag time increases to over two years or longer if a preproposal is unsuccessful. The increased lag period comes at a time when the rapid pace of environmental change requires science-based solutions to address societal needs. It also hinders the development and deployment of new tools and technologies (e.g., molecular and informatic) that inform solutions that address such rapid environmental change. The long lag between idea generation and funding is particularly hard on junior scientists who are establishing their research programs, but also hinders progress of more senior scientists seeking to sustain active research programs and to educate the next generation of scientists.

It's hard to argue with this one, which basically summarizes several points that have become familiar refrain. When I served on the preproposal panel last spring, these exact points came up. The NSF party line* is basically that the extra time now built into the process is critical for PIs to incorporate the feedback they received from the previous round. The claim is that internal studies demonstrated that proposals that were turned around from one round to the next did not fair as well as those that took a round off. Unfortunately, there is no NSF equivalent of Rock Talk to show these data, so we can't pick it apart. We're left wondering whether these data are a game of averages or represent real trends. Nevertheless, from the perspective of a PI, the loss of one round per year is perceived as a significant development, no matter how NSF wants to spin it.

The process limits the scope of science by (1) selecting against complex, interdisciplinary science that cannot be convincingly described in four pages and (2) hindering collaboration among scientists (by limiting the number of submissions per investigator per year) at a time when research programs and teams need to be increasingly multifaceted, innovative, and interdisciplinary to address complex issues.

This is where I'm a little less enthusiastic about the approach. The first argument about complexity that can't be described in four pages is, to me, a little weak. Is it hard to encapsulate a complex project in four pages? Yes. But claiming my science is SO complex you just don't get it because I can't tell you all about it is not exactly winning me over. Make it work.

The second point is stronger. For a decade plus we have been told that interdisciplinary is the way to get things done. Multi-PI projects have been encouraged at every level, from federal to institutional, and suddenly it is a liability. Now we have to pick and choose what we can contribute to in order to stay within an arbitrary limit of proposals. Okaaaaaay.

The process limits feedback to scientists, slowing the pace at which creative ideas advance during the iterative submission-resubmission process, because of the lack of ad hoc reviews for proposals. Although investigators faced low rates of proposal success with the former process, it at least offered comprehensive feedback and allowed for relatively quick resubmission, increasing the chances for success with future submissions.

This is another point that I don't see eye to eye with. If you have ever served on a panel, you know that the opinions that really matter are in the room. Whereas the ad hoc reviews can be very informative to the panelists, they can just as easily be almost ignored. The POs will ask the lead reviewer to comment on the ad hocs, especially if they are at odds with the panelist opinions, but they do not carry the weight of the panelist reviews. Additionally, my experience in the preproposal panel suggested that NSF can get pretty good coverage of the vast majority of the proposals with the panelists, not unlike NIH's study sections. Besides, the PI still gets at least three reviews back and if it didn't get trashed in those reviews, a panel summary as well. Are the additional couple of ad hoc reviews really that important?

The delays, the reduced opportunities for collaborative proposals, and the more limited feedback are likely to have a disproportionate effect on young scientists and members of groups who are not yet well represented in our science. We fear that this new process will result in the loss of some very promising people from the pipeline who are already discouraged by bleak prospects for funding research.

This is a concern shared by NSF. When the Director of Bio came to speak to our panel in the spring, this was something that he cited as being at the top of the list of things they are monitoring. I have to say that most panelists were not necessarily comforted that the situation was being monitored, but it was made clear that the POs were being directed to keep their portfolios balanced as before. How the preproposal process is going to affect that is not entirely clear.

One rumor I have heard repeatedly, however, is that only established labs made it through the preproposal process. I can say from my experience as both a panelist and a preproposal PI, this is not true. I can understand the perception that those with more proposal writing experience were able to navigate the new process better, but we pushed many early career preproposals through in my panel and I had one accepted from my own lab as the sole PI. I don't buy the fear mongering.

The letter concludes with the following:

We are optimistic that thoughtful modifications of the new preproposal process, made in consultation with the ecological and environmental sciences community, will ensure that science progresses as rapidly as possible given the level of funds available, thereby providing maximum benefit to society. In any such modifications, we believe it is essential (1) to ease current restrictions that limit collaboration and the pursuit of high-risk, high- reward ideas and (2) to provide two deadlines per year, even if that requires taking other measures, such as reducing the number of ad hoc reviews or reducing proposal length, to ensure reasonable workloads for NSF staff and the reviewer community.

Here's my biggest problem with this letter. Where is the solution? It is very easy to say that the current system is not working and we need to make changes, but don't place that burden right back in the lap of NSF if you want to make changes! Make a PROPOSAL. I've made some very clear suggestions geared towards solving these exact issues, but I have seen almost no other concrete proposals made. I think this is where we, as a community, are doing ourselves the most harm. We need to hammer out what we want and how to implement it if we want to gain any traction, otherwise NSF can throw up their collective hands and say "this is the best we can do."

So will I be signing this petition? No. As much as it summarizes some of the concerns I share, I don't think it has much value without suggestions for real change. I think NSF was fully aware they would get push back from the community based on these changes and this type of letter is going to fall right into the "expected whining" camp. Until the community can present a meaningful document with realistic changes, there is no impetuous for NSF to do anything.

I see this as a labor negotiation. NSF changed the rules of "payment" and has left their "payees" upset with the new environment. Unless we can make a convincing case (including some concessions**) for change, we're going to have to live by the new rules.

*Make no mistake, there are wildly different opinions on the changes among the Program Officers.

**Sorry folks, but NSF believed strongly that the system was nonsustainable. If you want to convince them to change you need to make concessions that address the issues.

16 responses so far

  • Dr24hours says:

    One cycle per year is madness. Especially for soft money people. It means that NSF is not a realistic source of funding for me. I won't waste my time.

  • Dave Baltrus says:

    Where are you pulling the MCB info from?

  • proflikesubstance says:

    Link at the bottom. Sorry, I'm writing from the road so I didn't get all the links in. http://scientopia.org/blogs/proflikesubstance/2012/05/25/is-the-big-pitch-testing-what-nsf-thinks-it-is/

  • Genomic Repairman says:

    This measure whether intentional or unintentional appears to be killing off the investigator pool through a slow choking off of funding, especially to the newbies. I doubt that NSF's official stance of monitoring this grave concern does little to make the junior TT profs sleep well at night.

  • Confounding says:

    This is going to hurt mathematical modelers and other theoretical types who rely on lots of bits from a number of grants instead of one or two "sustain my lab" grants.

    Which is a pity.

  • physioprof says:

    I can't speak to the details, but it sounds like a lot of this complaining has the same delusional basis as that of the NIH extramural community: an inability to accept the laws of mathematics and the fact that alterations in peer review cannot possibly make it more or less likely for *everyone* to get funded. The only important of peer review policies is the extent to which they alter *who* gets funded.

  • EcoNerd says:

    I am a junior tt faculty and a signatory to the letter, but basically agree with your assessment of it. As you say, this is a labor negotiation of sorts. For me the highest priority by far is to get the number of pre-proposals increased to 3. I am even ok with capping the number of invited proposals at 2. I was fortunate to have two full proposals invited this year, but if neither gets funded, what do I do? I know those ideas can get me through the pre-proposal stage ( albeit certainly not guaranteed next year), so this really limits how many new collaborations I can pursue. Obviously there are other funding sources inside and outside NSF, but they have their own constraints and aren't always appropriate for a given idea. So 3 pre-proposals, even if they can't all be invited, would really provide a lot more flexibility in my collaborations.

  • proflikesubstance says:

    CPP, I agree that the formula doesn't change - there is only so much money to go around. However, the ESA letter appears to be aimed at the pace of the proposal process more than the numbers.

    EcoNerd, three preproposals per round may be helpful, but I would be more supportive of two rounds per year with a two proposal limit. Even if the time frame is 8 months instead of 6, I think turn around time is a big concern.

  • Nice post, PLS. I know we haven't seen eye to eye on the details of this, but I agree completely with your take home message - turning back the clock is not an option. Our best route forward is serious discussion about what the real problems with the system are and what things can and cannot be fixed through the proposal process.

    I have to admit while I understand the psychology of wanting more tries/year, I don't understand the math on the benefits of increasing the rate of proposal turn around. If there are 2 or more funding decisions per year, the probability of your proposal getting funded decreases in each of those in comparison to only 1 funding decision per year. The crux of the problem is that there are only so many winning tickets (so to speak) per fiscal year. Either that fixed number of winning tickets can be awarded in a single cycle or can be split across more cycles. Unless I'm missing something there is nothing about more cycles per fiscal year that makes things better, it just provides the illusion of things being better. What is really screwing the young scientists isn't necessarily* the number of times they can submit a proposal per year, it's the number of awards that can be made per year relative to the number of proposals being submitted.

    Having said that, I have seen the 2-proposal limit start to negatively impact collaborations. I would love to see NSF start a serious discussion on the tradeoffs of either lifting the limit on co-PIs or increasing the preproposal limit to 3.

    *However, I recognize the 'young scientists might need more practice writing a grant before they write a successful one' as a potentially valid argument, but wonder if there might be other mechanisms that can help with this that we should be thinking about.

  • proflikesubstance says:

    The argument behind the 2 cycles per year is simply more feedback and potentially shorter time to funding. Whereas this has the biggest impact on junior people, if someone is hoping to write a follow-up proposal to a current grant there is now one shot at getting the funding without a gap. In the previous system one could apply two cycles (one year) before a grant ended and if it wasn't funded, have the feedback and a second shot. Even if it took three tries, the gap in funding for that line of research would only be 6ish months. In the current system, three tries would potentially shut that line of research down.

  • [...] Thoughts on the ESA letter to NSF Bio Why Nick Kristof’s Scientific Illiteracy Threatens Us All Owning a Sequencer is NOT a Prerequisite for Sequencing! Autism, immunity, inflammation, and the New York Times Medscape and theheart.org get harsh sanctions for European Society of Cardiology embargo break (I hate the whole idea of embargoes anyway) G-nome Surfer (not sure how valuable an analytical tool this will be but it does seem kinda fun) [...]

  • European Academic says:

    Sorry, posted the below comment on an old post (from 15 Sept 2010) - which however appeared in my RSS feed this morning as a new post (and I was redirected to somewhere else by clicking on it). So something seems to be wrong with the Scientopia feeds. Anyone else having this problem?

    *******
    Off topic for this post, but perhaps of interest to whoever is running Scientopia: since about last week your RSS feeds throw my Google Reader to another place, to scientopia.info rather than scientopia.org. So for example, when I saw this post of yours and clicked on it, I ended up here:

    http://scientopia.info/blogs/proflikesubstance/2010/09/15/co-post-a-conversation-on-the-dangers-of-field-science/

    Same with other scientopia blogs.

  • [...] to anyone who checks up on this space on an even semi-regular basis. I generally like to post about my experiences with funding agencies because I think that can be helpful to people dealing with similar issues. Just browsing the last [...]

  • [...] year with a preproposal round. We're now 9 months into that and different groups, including ESA, are making noise about this [...]

  • [...] There are definitely good reasons for having a pseudonym, especially if you want to write/tweet about things that might be more controversial. For example, ProfLikeSubstance is an evolutionary biologist who tweets and blogs under a pseudonym, which has allowed him to post more easily about topics such as, say, the new NSF BIO preproposal system. [...]

  • [...] interact with the agency. Researchers who depend on those other agencies don't necessarily have to send letters or make online surveys to try and get your attention, then wonder if anyone received [...]

Leave a Reply