There's a webinar today for the IOS Division of NSF to discuss the current state of NSF affairs. Based on the slides sent out, there is a lot of general info about the process that will likely be serious rehash for anyone who has been paying attention to all the changes. However, there is some interesting data on how reviewers felt about the process and how Beginning Investigators (those with no previous federal funding) faired.
1) Ad hoc reviews were cut from 14,000 in the old system to 2,500 in the new.
2) The 30% full proposal target was hit the year.
3) IOS reports that 80% of PIs used to only submit once a year anyway (who are these people?) so the new system doesn't change behavior for the majority of PIs.
1) 65% of preproposal panelists thought the new system improved their experience, with only 20% feeling it detracted.
2) Only 20% of preproposal reviewers across IOS and DEB spent more time reviewing, compared to full proposals. This is relevant because there were many more preproposals to read than panelists would get in a full proposal round.
3) In the full proposal round, 74% of IOS panelists thought their experience was better this year than previous, with only 5% indicating a worse experience.
4) Whereas nearly 70% of panelists did not feel there was a change in the Intellectual Merit at the full proposal stage, nearly a third saw improvement.
5) Over half felt the full proposal Broader Impacts were improved, compared with previous years.
1) The percentage of Beginning Investigators who were ranked in the High Priority category was on par with previous years.
This is notable, as some feared the Beginning and New Investigators were likely to take it in the teeth. These data suggest otherwise.
2) Both the percentage of submissions and high priority rankings from those working at RUIs (mostly undergrad institutions) was up in the new system.
Unfortunately, we do not have funding rate data yet because the budget isn't settled, but these indicators appear to be validating NSF's position here. Panelist satisfaction is up, ad hoc requests are down and the fear that some groups of investigators would be unfairly affected is not borne out by the data. I'll be listening in for other tid bits later today.