How did NSF beginning investigators fare in the new system?

Dec 10 2012 Published by under Uncategorized

There's a webinar today for the IOS Division of NSF to discuss the current state of NSF affairs. Based on the slides sent out, there is a lot of general info about the process that will likely be serious rehash for anyone who has been paying attention to all the changes. However, there is some interesting data on how reviewers felt about the process and how Beginning Investigators (those with no previous federal funding) faired.

General:

1) Ad hoc reviews were cut from 14,000 in the old system to 2,500 in the new.

2) The 30% full proposal target was hit the year.

3) IOS reports that 80% of PIs used to only submit once a year anyway (who are these people?) so the new system doesn't change behavior for the majority of PIs.

Reviewer specific

1) 65% of preproposal panelists thought the new system improved their experience, with only 20% feeling it detracted.

2) Only 20% of preproposal reviewers across IOS and DEB spent more time reviewing, compared to full proposals. This is relevant because there were many more preproposals to read than panelists would get in a full proposal round.

3) In the full proposal round, 74% of IOS panelists thought their experience was better this year than previous, with only 5% indicating a worse experience.

4) Whereas nearly 70% of panelists did not feel there was a change in the Intellectual Merit at the full proposal stage, nearly a third saw improvement.

5) Over half felt the full proposal Broader Impacts were improved, compared with previous years.

Investigator specific

1) The percentage of Beginning Investigators who were ranked in the High Priority category was on par with previous years.

This is notable, as some feared the Beginning and New Investigators were likely to take it in the teeth. These data suggest otherwise.

2) Both the percentage of submissions and high priority rankings from those working at RUIs (mostly undergrad institutions) was up in the new system.

Unfortunately, we do not have funding rate data yet because the budget isn't settled, but these indicators appear to be validating NSF's position here. Panelist satisfaction is up, ad hoc requests are down and the fear that some groups of investigators would be unfairly affected is not borne out by the data. I'll be listening in for other tid bits later today.

18 responses so far

  • Zen Faulkes says:

    As always, fascinating stuff on this. Keep it comin'!

  • I wasn't planning on attending the webinar because I thought it would be complete rehash and I don't submit to IOS anyway, but those stats are very, very interesting. Are they just IOS or is DEB in there too? I saw DEB mentioned above for preproposal reviewing time, but nothing else. I'm particularly glad to see the new investigator stats. My impression from being on a preproposal panel was that new investigators were faring well, but you never know if your n=1 generalizes. In general, I agree that the data seems to validate NSF's changes - there's at least nothing in what you've presented that causes me any heartburn except the low funding rates overall and that's not NSF's fault. It might also explain why they were less than receptive to the organized protest from the ecologists.

  • proflikesubstance says:

    It wasn't totally clear from the slides which numbers are ISO specific and which included DEB, but I imagine we'll find out later today.

    WRT the ESA letter, I don't think NSF had the numbers yet. I think they are/were inclined to see what the data look like before changing course. I'm of the opinion that NSF is not who we should be complaining too anymore, but rather Congress.

  • DrugMonkey says:

    IOS reports that 80% of PIs used to only submit once a year anyway (who are these people?)

    Amateurs?

  • DrugMonkey says:

    What was the "organized protest"?

  • proflikesubstance says:

    What was the "organized protest"?

    A letter, signed by several hundred NSF scientists.

  • proflikesubstance says:

    And DM, regarding your first comment, this number is going to depend on how the numbers were counted up. Many who can apply to IOS can also submit to DEB. If the reported numbers are only for IOS, I would be curious to see how it looks when the DEB submissions are included.

    It is possible that some PIs were hitting IOS once a year while also applying to DEB once or more per year.

  • Joshua King says:

    Nothing conclusive here in any way. This is only one year and we have no idea if it confounds DEB with IOS, etc. It is going to be a little while (2 years?) before you can conclude much about the impact on juniors.

  • Sorry I haven't emailed you back about the Congress petition. I'm in end of the semester email avoidance mode 🙂

    I completely agree that the crux of the issue is the funding and that's not NSF, that's congress. My memory is that DrugMonkey blogs once a year about the NIHers doing a 'contact your Congressperson' drive. The NSFers should get more organized too, though the pessimist in me suspects we'll have limited impact. Might be worth asking the revered blogosphere sage about strategies the NIHers use to rally the troops.

  • proflikesubstance says:

    JK - These data turn out to be just IOS, I have been informed, so no confounding.

    ME - No problem, the responses have been.... interesting. I think we need some organization on that front, so hopefully we can make some collective progress.

  • rs says:

    This is regarding your second table (RUI). I have heard from my friend who is a faculty in teaching focused university that the bar for tenure has been raised in recent years. All tenure eligible faculty are expected to submit research grants and publish one or two research articles to get the tenure. Earlier, if you had a good teaching record, tenure was guaranteed. This could explain your second table. Welcome to the rat race.

  • proflikesubstance says:

    Here's the link to the recorded webinar: https://mmancusa.webex.com/mmancusa/ldr.php?AT=pb&SP=MC&rID=49439132&rKey=445f080953dc37fb

    FYI, It's picky about the browser you use.

  • proflikesubstance says:

    WRT the RUI institutions, I don't think one would see a big jump in a single year based on something that has been building at different rates at different institutions of a number of years. In the webinar it was suggested that the decreased effort to put together a preproposal may better fit the RUI timeline in the middle of the year.

  • Michelle Elekonich says:

    "Ad hoc reviews were cut from 14,000 in the old system to 2,500 in the new."

    This was actually review *requests* rather than reviews.

  • proflikesubstance says:

    And what was the return rate on those requests?

  • Michelle Elekonich says:

    It was about 50% but that is not a solid number since in IOS there is variance in how the Program Directors put the requests in the system. Some do all their requests through the official E Correspondence system in Fastlane and others contact reviewers by email first and then only put the ones who say yes in the system. So we working on other ways to get more solid numbers. Working with the NSF Fastlane collected data is unlike doing your own research -you can't just decide on a measure and use it; you are limited by what is taken and the variance in practices. Much of the demographic data are self-report which has its own issues.

  • [...] off the data release by IOS, DEB has sent out a similar (though slightly harder to decipher) dataset. Let's break down the [...]

  • [...] changes on the scientific process.  Well, over the winter break, DEB released its data to date (IOS did this a little earlier and comparisons between IOS and DEB are discussed here). So let’s see what [...]

Leave a Reply