What should the PUI share of NSF awards be?

Mar 15 2013 Published by under [Education&Careers]

The NSF DEB blog is running the numbers from the first round of preproposals and resulting full proposals in a series of posts right now (part 1, Part 2). Other than the simple fact that I'm giddy about the information and analysis coming out of this blog, there are some really interesting insights into the funding decision making. One thing that caught my eye from part 2 of the numbers series, was the following (emphasis added):

At this point in time, the whole process appears to be avoiding disproportionate impacts on Beginning Investigators and Primarily Undergraduate Institutions. However, both of these groups have historically been minority groups in submissions to DEB and furthermore underrepresented in award portfolios compared to submissions. While we can report changes over time and compare between groups, these numbers do not provide answers to the questions that underpin evaluations of progress: What is the right mix of Beginning Investigators and Primarily Undergraduate Institutions in an award portfolio? And, how can we reach the right mix in awards? Would such a mix require a change in submission patterns and/or changes in peer review practices and/or changes in Program Officer handling of the submissions we already receive?

We all know that the NSF system is for amateurs, but even in this context, the researchers a Primarily Undergraduate Institutions (PUIs) are at the biggest disadvantage when it comes to research productivity. Typically using only undergraduate help and while teaching 3-4 classes per semester, faculty at PUIs have to fit research into the thin cracks of open time and try to make the most out of summers. This is toughest on early career PIs, creating multiple courses. Many do it successfully, but the PUI category exists because of the likely limit in productivity.

DEB's numbers for the last 7 years indicate that proposal success rate originating from PUIs hovers between 12-15%, with submissions to the total pool between 14-18%. And this is WITH portfolio balancing* by POs. Presumably NSF is happy with this success rate, reflected by the consistency of the numbers over time. The question is, should tax payers be happy with that rate?

When budgets get tight institutions need to look at value for their money. In NIH land, there have been two responses to the too-many-mouths-at-the-trough problem; 1) Kill the rich, 2) Eliminate the small town grocer. NSF has toed the more "Canadian" line by mixing portfolio balancing and basically telling PIs with major roles in two or more proposals that they shouldn't bother submitting another lead proposal.

So I'm curious whether people believe this is the best route for science? For the tax payer? For innovation? What say ye?

*POs are required to balance their grant portfolios based on several factors, including PI gender, ethnicity, institution type, state, etc.

33 responses so far

  • Terry says:

    You know, PLS, you don't have to talk about it as if we aren't in the room. Us at PUIs are reading this, too. You don't have to talk about it as if we're not involved in the conversation, even though we don't get invited often enough to panels.

    You shouldn't be starting with the presumption that, on a dollar-to-dollar comparison, that the quality or productivity of the work coming out of RUI grants is less than that of regular awards.

    Once you show that these RUI awards actually do result in less science per dollar, then maybe you could discuss whether you think this is a waste of money. I'm not even sure that's true. The people I know who have RUI awards seriously rock and have a good quantity and quality of pubs. Since these awards are cheaper than others, often (no grad student or postdoc costs, and limits to PI salary are the same for big awards, and we earn less than R1 professors on average), we do a lot with these awards.

    It ain't a frickin' token gift. They're funding real science and real student training.

    It's a separate category because there is a lower bar for productivity. But the awards are smaller too. (I think - I don't have the numbers on that.)

    Also, NSF takes broader impacts seriously, including how they allocate dollars. One meaningful broader impact is student training. All of these regular NSF awardees wouldn't have any decent grad students to train unless they came out of labs that trained undergraduates. It's a long-term disservice to just give money to big labs when the small labs are often training the best future grad students. Sure, some good grad students come from R1s, but not as many.

  • proflikesubstance says:

    I'm not in any way assuming people from PUIs aren't in the room, I figured that this post would garner the most attention from that segment of the population.

    I know the PUI world well and collaborate with PIs who work in these institutions. IME, the awards are only smaller when it comes to salaries. If the proposal is going to be undertaken solely at a PUI, then grad stipend and postdoc salaries are often not requested. However, even proposals with a single PUI PI among >1 lead PIs fall into this category.

    I think it is interesting that you read this post as taking a shot at PUI science, since I'm not sure where I wrote anything about tokenism here. I'm simply reporting the numbers, echoing the question posed by DEB and curious what the response will be.

  • anonymous says:

    I think that one real strength of the NSF system is that it explicitly recognizes that not all institutions are the same, and that there is more than one route to success. Grants to PUIs are not some sort of charity.

  • Terry says:

    Even asking the question about whether funding RUI proposals might not be the best route for innovation isn't healthful.

    You need to demonstrate that RUI awards are less valuable in their products before moving on in the conversation.

    Saying that you have PUI experience and that you work with them doesn't help. "I'm not racist, I have black friends." "I'm not homophobic, I have plenty of gay friends."

    If you genuinely wondering whether RUI awards are a good investment, and whether NSF is wise in maintaining consistent support for RUIs as budgets get tighter, then the actual differences between RUI and regular awards should be delineated rather than implied.

    I do love the "small town grocer" analogy. I'm glad I'm not Wal-Mart.

  • anonymous says:

    The phrase "small town grocer" and the reference to lower productivity did come off as more condescending than intended, I suspect. While on average R1 folks are more productive than people at PUIs, the variance is high.

  • proflikesubstance says:

    Even asking the question about whether funding RUI proposals might not be the best route for innovation isn't healthful.

    Healthful to who? And how did we get to PUIism? Science and how it gets done is a spectrum that is not directly correlated with place of employment.

    However, if NSF is asking what the "right" ratio of PUI grants is, I think it's something that should be talked about. Unless you want to leave that decision to them....

  • proflikesubstance says:

    The phrase "small town grocer" and the reference to lower productivity did come off as more condescending than intended, I suspect. While on average R1 folks are more productive than people at PUIs, the variance is high.

    Anon, Are you suggesting that, on average, we would find equal or greater research productivity by those who teach a 3/3 and use undergrads, when compared to PIs with full labs and teaching one or two classes a year? What if we compared grant load across these populations? For the most part, PUI faculty are EXPECTED to be small town grocers. It's not condescending, it's the job.

    I'm not making a judgment, I'm stating the facts. There are different academic jobs out there, with different research expectations.

  • proflikesubstance says:

    Lest we lose focus here, NSF IS ASKING THIS VERY QUESTION. Do y'all want to be part of the conversation or spend your time getting upset over what you perceive as a pigeon-holing of your institution?

  • anonymous says:

    Hi PLS, Anon here again. Long-time lurker, love the blog.

    I get what you mean about the difference between PUI and R1 institutions. I've worked at a PUI, although I'm not at one now, and the job expectations are different.

    I think the problem is that when someone at an R1 refers to those who work at PUIs as "small town grocers", it does come off as a bit of a put-down. There is a difference between "small-town grocer" and "person who spends less time on research".

  • Zen Faulkes says:

    "What should the PUI share of NSF awards be?"

    Somewhere above 0% and below 100%. Probably roughly proportional to the PUI share of applications, I think.

    I'm not convinced there is any sort of optimum. Different institutions fill different sorts of niches.

  • Hermitage says:

    There's a problem/issue the NSF is trying to solve, but we shouldn't talk about because....FEELINGS? What fuckery is this? Are we scientists are cranky toddlers just up from naptime?

    Is NSF PO 'portfolio balancing' as systematic as, say, NIH's curve bump for ESI/NI status submissions? And what is the overall purpose, be more on the NIH-land side of things and push for maximum productivity per dollar, or be more egalitarian? It seems they've been the latter so far, but is that on purpose and do they want to continue it as sequester puts the squeeze on?

  • proflikesubstance says:

    I think the problem is that when someone at an R1 refers to those who work at PUIs as "small town grocers", it does come off as a bit of a put-down. There is a difference between "small-town grocer" and "person who spends less time on research".

    It's a spectrum. By the standards of some labs in my college, mine barely reaches small town grocer status. But that's what I do and I am meeting the expectations of my position, the Dean and my department . It's just reality.

    And not the point here.

  • Terry says:

    Great points. I appreciate the reflection and the acknowledgment that expectations are different.

    I'm not upset about getting pigeon-holed, but I did want to call out an unsubstantiated implication - that RUI awards might result in less innovation. I think I've been pretty civil about it. I appreciate the friendly and civil response. (I am a regular and appreciate what's in here, after all.)

    Keep in mind that it is expectations really are different at the institutional level, not just at the federal level. R1s expect faculty to have grants, while most PUIs realistically don't have this expectation of their faculty. The means that PIs at PUIs are outliers on campus. They might have teaching assignment accommodations that others lack, enabling more time for research. While NSF does expect less, I'm genuinely curious if the result is truly less on a dollar-for-dollar basis. I agree with anon that there is extraordinarily high variance, perhaps so much that differences can't be found (using a frequentist approach, at least).

    To answer the questions, I think the fraction of RUI awards should stay the same or maybe even increase, I guess. Most everyone says, "do whatever to fund me!" when asked. I think a strong support of PUIs is important because this really is foundation of future researchers. It's no accident that liberal arts colleges produce so many more grad students than research institutions. Supporting research in teaching environments is necessary to sustain competitiveness. (For the same reason, lower down in the so-called-pipeline, we need more science in elementary schools.)

    With grants harder to get, research institutions will continue to do research, though at a lower level. At PUIs, without funding, research can fizzle to nothingness. Once a PUI professor stops research, it won't happen again. A total shutdown would be more traumatic at that level than at R1s, I think. That trauma would ultimately harm R1s because there will be fewer competitive grad students looking for slots in grad school.

    I don't like how RUIs are handled at the level of each directorate. In Biology, it's just one subjective factor that's used by the program director. In some cases it has more weight than others. The general perception about PIs of RUI proposals is that the designation doesn't influence the review process enough - that the reviewers don't have enough of an understanding of institutional context to evaluate the proposal fairly. I don't know if this is true, but I've seen people get majorly dinged for not having enough research happening, in a panel summary, when their labs are on the high end of productivity for their fields at a PUI. Every program director is different, and I wish there was more consistency in how RUIs are handled. A separate competition isn't the answer, because you need the specialized panels.

    [A self-promoting plug, by the way - my blog is about research at PUIs.]

  • anonymous says:

    PLS, I just was trying to point out that perhaps there were some statements in the initial post that could be taken in a different way than intended. But I think this is a valuable discussion to have, so thanks for initiating it.

  • Terry says:

    Here's another way to put it, with a specific example. About how student training matters.

    If I get the 250K proposal funded for my PUI, which has been pending for 7 months, there are going to be twelve talented students with mountains of potential, who will get incredible research experiences for nine months each. These will be life-altering experiences for every single one of 'em, I swear. Most of them will go on to grad school, and be a lot better because of it. Without the program, they won't even be applying to grad school. And nearly all of these are URM students, which matters because we really do need to diversify the field to keep up with the world.

    Would a 250K award do that at an R1? That's a pretty good damn investment.

    [Let's table the notion that we don't need to train new grad students that they won't have jobs at the other end. That's a whole other discussion.]

    And that's not even looking at the pubs - there'll be 6-10 of those, I guess, some of which will get cited for a good while to come.

  • proflikesubstance says:

    And this is why a dollar to dollar comparison doesn't work. As the research expectations are different, so are the training expectations.

    But, as much as I am a supporter (and product) of this notion it's valid to ask whether supplying the grad school pipe is a major need right now (excluding cases of URMs), with many calling for a significant reduction in PhD slots to try and stem the vaunted overproduction of PhDs. I don't know whether I buy the overproduction story, but it is part of the discussion.

  • Terry says:

    I am really curious about the $-to-$ comparison, even if it's not entirely valid because there are different measuring sticks.

    It's easy enough for NSF with their reporting system to count up the number of pubs with grant numbers on them from PUI and non-PUI awards. I haven't seen these numbers. I'm just curious. I think most people would be surprised at the productivity of the PUIs that get NSF money. Or maybe some would be shocked at the "waste."

    On the time horizon of final reports, NSF can't learn about the outcomes of undergrads funded on their grants. How many REUs, for example, end up in grad school? REU awards are long enough, and often renewed, that these data are available. But for regular awards? It's harder to show training outcomes.

    Since nearly all of my students are URMs, I don't have any ethical concern about sending them off to grad school, it's good for both them and for the entire profession. But if I was teaching at a nearly-all-white PUI, I'd have some mixed feelings, or at least would communicate the challenges of getting a faculty position very clearly.

  • LD says:

    NSF has always been a bit of a National Spread the Money around Foundation. Is this bad? No, clearly not entirely, but productivity counts a bit more to me than training. After all, it is grad "school" and training happens there.

  • Alex says:

    I'm at a PUI, I'm sometimes sensitive to how PUIs are discussed, and I didn't see anything wrong with PLS's post.

    And it is a fact that my research productivity is lower than that of my R1 colleagues. And that's pretty typical. Yes, we have one superstar whose lab is as productive as anybody at the most elite R1, but the rest of us are trying to get out high-quality work at a slower pace. I will defend the quality of my work as being comparable to my R1 colleagues, but the quantity is lower. And while some of my work is on hot stuff, I do have to be careful to avoid the most competitive niches, because I know I can't keep up. So I pick hot and important topics and then pick the aspect of the topic that nobody else is looking at. That way I'm in the Big Game but not at risk of getting run over by people with armies of postdocs to throw at the problem.

    My other solution is to spin off side projects that aren't necessarily big or hot but involve genuinely unanswered questions, and put students on those. Those projects aren't fundable (so resources get scraped together in bits and pieces) but students learn a lot. And, frankly, having a side project on top of my bigger, hotter main project is good for the intellectual atmosphere of the group. There's always more than one thing going on in the lab, more than one question being discussed, more than one technique being learned. It's a healthy balance between things that keep me in the game and things that keep it mixed up and broad for the students.

  • Terry says:

    My lens was (is?) askew. I'm sorry for derailing the conversation.

  • proflikesubstance says:

    Meh, isn't that 50% of blogging?

  • qaz says:

    Terry - I don't think you've derailed the conversation. I think you've hit the conversation on the head. There are people who think that their way is the only way and do not appreciate the fact that there is good science happening at PUIs, by university faculty with hard money positions and teaching loads, and by institute faculty living on soft-money. This discussion is loaded with disdain and a lack of appreciation for other ways of accomplishing the business side of science.

    I don't know where the term "small-town grocer" comes from, but I first saw it on DrugMonkey's blog, and I'm pretty sure it was meant to be insulting. The key is to remember that there is a big difference between providing a community with local needs (as a small-town grocer does) and contributing to the grand progress of SCIENCE (which is what grants are for). Remember that DrugMonkey did not use "small town grocer" to refer to PUI professors, but to all those scientists who want to run a small lab, living on one grant, trying to make small progress on a small topic, and not chasing that Nobel prize. The implication is that these people are not contributing as much as the cut-throat people surviving in the Thunderdome of soft-money jobs.

    When discussing how many PUI grants should be included in NSF's portfolio, we have to determine what NSF's goals are and what those grants are doing. For one thing, NSF has the goal of actually educating the populace and not just making more scientists, so giving research opportunities to undergraduates (the real small-town grocer part) IS part of their portfolio. But, also, important work has been and will continue to be done at colleges and universities (and non-university laboratories) around the world, including PUIs, big state research Universities, and snooty private schools.

    Diverse portfolio balancing is the key to maintaining growth. This has been known for decades in the financial world. Chasing the latest fad leads to bubbles and crashes.

    Most importantly, portfolio balancing is the job of program, not of the scientific review committees. The scientific review committees simply do not have the vantage point to see the shape of the field.

  • qaz says:

    And can we get over the pipeline issue? It's bad enough that people think we're overtraining grad students because the only path they see is for those grad students to try to get faculty jobs. I still say grad school is great fun and (in the fields we're talking about) you get paid for it. Now people are going to say we shouldn't give undergraduates research experiences because the grad school pipeline is too full? Are we going to stop high school college programs? Can't we just say that getting research experience is a good thing? I wish more people in the world had some experience with the process of science. Imagine if members of congress or journalists or the rest of the public actually had an appreciation for how science worked?! Most of the people making decisions and influencing the world are college graduates - let's make sure they get research opportunities in college. Maybe then they'll realize how important science is and maybe they'll start to appreciate the scientific process.

  • Alex says:

    Also, whether the grad school pipeline is too full or not (I'm much more pessimistic than qaz) is completely separate from whether undergrads should get research experiences. There are things in science that undergrads can ONLY learn from doing research, and many of those things will be valuable in their futures no matter what road they go down. Also, even if you think fewer people should go to grad school, _somebody_ should go to grad school, and research experience is a good way to help people figure out whether it's the path they want to follow.

    Now, somebody might say that undergrads should get their research experiences via REU programs at large R1 schools, but an REU is 8-10 weeks. Research at their home institution can be year-round.

  • proflikesubstance says:

    Anyone want to take a stab at the original question? Should PUI awards roughly reflect the submission rate? Be less than that? More?

  • RogerDodger says:

    OK, I'm game. Here's my answer:

    PUI award rates should be whatever NSF thinks they should be given NSF's 'mission'. If they want their mission to include general scientific literacy and STEM education, then they should dump a lot of money on PUIs. If they don't think that's their mission, then they should not put money there.

    One of the great things about NSF is the strong sense of mission. The POs and administrators have a lot of latitude when it comes to funding, and use it. NIH panels are often circle-jerks between like-minded egos all hell-bent on funding the same minutia that got them all on panel in the first place.

  • RogerDodger says:

    Qaz's comment above is right on target. Science is not the load of crap that fills textbooks. Science is a way of thinking, a method for discovery, and the only way to learn science is by doing science. THE ONLY WAY TO LEARN SCIENCE IS BY DOING SCIENCE.

    If we taught math the way most science is taught, people would learn about famous historical problems and memorize mathematical proofs. But they'd never know how to actually do math. If we taught art the way most science is taught, people would learn lots about art history and be able to name every famous painting, but they'd never have tried to draw anything. If we taught languages the way most science is taught, people would have dictionaries memorized and know all the parts of grammar and how the language changed over thousands of years, but they'd still never be able to communicate.

    THE ONLY WAY TO LEARN SCIENCE IS BY DOING SCIENCE.

    And that's worth learning even if not everyone will end up a 'professional' scientist, for the same reason it's worth learning math and art and foreign languages.

  • Terry says:

    Meanwhile, it's only crickets over on the DEB blog.

    When I first read it, I started to write, "well, it depends on your mission, which in my opinion should address all kinds of institutions in the US, which looks like status quo is okay, aside from a general lack of funds" but I didn't want to commend unless I had something really novel or useful to say. I don't think the above fits into either of those categories.

  • proflikesubstance says:

    That mindset of only commenting when you have true insight drives most bloggers crazy (so I hear). Comments have a way of snowballing, but not until someone starts the first ball rolling. A blank comment section somehow looks like the deep dark woods to most potential commenters.

    The DEB blog is going to take some time to build an audience of people willing to comment. It's only about a month old. I wouldn't be surprised in traffic doubled every couple of months. The posts are fantastic for people interested in the inner machinations.

  • Terry says:

    Well, yeah. But this is the D. E. B. blog. I think NSF is imagining that people would be reluctant to post on an officially sanctioned NSF venue. I don't want to occupy too much real estate over there, even if I am really enjoying the flurry of data that they're offering. Sometimes the best profile is a low profile. I don't want to be an NSF blog gadfly.

    (Even if the program directors are reading this, which some of them undoubtedly are.)

  • proflikesubstance says:

    Haha, yeah but if you google "NSF DEB blog" my blog is still the first hit. Once they get going a little it will pick up.

  • Zen Faulkes says:

    "Comments have a way of snowballing, but not until someone starts the first ball rolling."

    I have ROLLED THE BALL. Like a dung beetle.

    (Comment is in moderation right now.)

  • [...] a position of “running a lab” instead of “doing research.” I wanted to be a small-town grocer instead of [...]

Leave a Reply