The elephant in the room

May 21 2013 Published by under [Education&Careers]

A comment from Emilio Bruna last weeks's post encapsulates a mentality that is common in review recipients, manuscript or proposal.

I actually think a much bigger reason a (pre)proposal might not make it through in one year or panel when it would in a different one is the luck of the draw wrt the three people reviewing it. No one who is an expert in your subdiscipline? Trouble.

Anyone who has ever served in the role of presiding over the review process will recognize this sentiment well. Having had to bite my tongue as authors complained about non-specialist reviewers when I had sent their manuscript out to one or more of their suggested reviewers, I can only imagine the calls POs receive.

But here's the question: Why are you writing a proposal for people in your subdiscipline? This is doubly pertinent at the preproposal stage, where only panel members read the proposals. As a panelist I have often read proposals outside my wheelhouse, but some of those have been among mine and the panel's highest rated proposals. Why? Because if you're writing for you subdiscipline, you're doing it wrong.

24 responses so far

  • Emilio Bruna says:

    PLS, You're not being fair by using a comment on a blog post to extrapolate about what you think my approach to grant writing is (I assume that's what you mean by my "mentality") is or the reasons for which I or others think our proposals or manuscripts are rejected. Maybe, and this is just a suggestion, you could have asked if this is truly my 'mentality' or even for clarification. I would have gladly fleshed out my response a little. I don't have time to do that here, but in the interest of stimulating conversation a few thoughts below.

    I don't think you should write to your subdiscipline, or I would never have gotten a grant or published a paper. I've done both. WRT NSF, full proposals are read by panelists and ad hocs, and adhocs are typically specialists, or at least more specialized on your topic than an average panelist. They play a unique role - they might know the literature, techniques, or system really well - and that allows them to both provide unique criticisms and also emphasize how it might be unique in ways that aren't immediately clear to the panelists that aren't specialists. In preprosals, that specialist insight might be the difference for a preproposal that is on the fence. I was not suggesting you should write a preproposal for specialists - one must obviosuly demonstrate relevance to a broad audience. But I am suggesting that if you have a preety good preproposal that is on the border of competetive/not competetive, the insights provided by a specialist could help push you into the competetive column. FWIW, the specialist insight cuts both ways, as I saw on a recent panel. If that specialist on the panel points out a key flaw the generalists overlooked becuase they aren't *ahem* specialists, it could take a highly ranked proposal down. So I guess a more comprehensive perspective is that specialists reviewing proposals can give good science that needs some work on the pitch a chance to make the full proposal round, and also keep science with a fundamental flaw that only a specialist could recognize out when it would have otherwise snuck through.

    No you shouldn't write to specialists. Yes, whether or not a specialist reads your preproposal can make a big differrence. No, you should not assume too much about a scientist's mentality or approach to the profession based on blog comments. Yes, anonymity allows people to be extra snarky on the internet. Can we agree on most or all of these statements?

  • odyssey says:

    WRT NSF, full proposals are read by panelists and ad hocs, and adhocs are typically specialists, or at least more specialized on your topic than an average panelist.

    While this is can be true (not all PO's make use of ad hocs), what is also true is that you need at least one very strong advocate for your proposal on the panel. And they often are not a specialist in your field. As you note, an ad hoc specialist can point out fatal flaws someone else may not pick up on, but it is very, very difficult for an ad hoc to convince a panelist to advocate for a proposal if that person is not already convinced the proposal is worth funding.

  • Emilio Bruna says:

    Odyssey - totally agree. I'm trying to recall if on the panels on which I've served if the the advocates for proposals 'on the edge' have been specialists or not, and I confess I can't recall. I've definitely advocated for both proposals in my area and those that I knew less about but were just very cool.

  • Emilio Bruna says:

    PS sorry about the typos earlier, was trying to write on a phone and I apparently have fat thumbs

  • martini says:

    The problem is not that a grant is written for a sub-discipline, it is that non-expert panelists sometimes think they know know more about your discipline than they actually do.

    I have had a grant panelist criticize my my introduction and approaches citing a view of my field that is more than 20 years out of date. In fact, had he taken my introduction at face value, or had read a recent review, he would have found his criticisms of my proposal were simply wrong. In an era where you require 3 Excellents to move forward in DEB, having one ignorant reviewer is more than enough to kill your chances. Same goes for reviewers with attitudes.

  • Joshua King says:

    The reviewing process at NSF, under the incredibly competitive conditions we are currently in, will remain capricious because we are all "specialists" to one degree or another and on any given day, a given panelist can run with or shred our proposals to bits. PLS spends a lot of time splainin' this away, but I'm not sure why.

  • proflikesubstance says:

    Some of us observe reality and some make it fit our own convenient circumstances.

  • Who knew? says:

    wow, a philosopher...and adherent of correspondence theory, it would appear. Who would have thought?

  • Emilio Bruna says:

    PLS, I'm not sure what you mean about making reality fit our circumstances, but any thoughts on people's comments? Genuinely interested...

  • proflikesubstance says:

    You tell me, Emilio. When you're on a panel, do you ONLY champion proposals in you subdiscipline? Genuinely interested.

  • Emilio Bruna says:

    Answered thet...third comment, response to Odyssey.

  • Terry says:

    When you're writing a proposal, you have to make a jazillion tiny decisions about what details to include and what to exclude, many of which are based on presumptions about the composition of the panel and ad-hocs.

    For example, if you draw ad-hocs that don't bother to put into context some area- or taxon-specific approaches or methods, then you could get dinged in panel only because you decided to cut the paragraph explaining how and why you've chosen a specific method and why it works. But if you include that paragraph, then you don't have the time to cover all of the bases that the panel is expecting otherwise.

    Without specialists in your own subfield, certain things need to explained more. That takes real estate, and also draws away from the concept-driven narrative. So I think most of us count on having at least one person who knows what they're doing to vouch with "okay, these methods are good." If I don't draw a person who is very familiar with tropical ant natural history, then I'm screwed, because it'd take 2-3 pages for me to explain how certain methods are effective, and those with only experience in the temperate zone would think that what I do is totally whack. So, I'm really counting on an ad-hoc in my speciality. And it's worked. But when I don't get an ad-hoc who is familiar with what are standardized methods in my subfield, it's a long hill to climb.

  • proflikesubstance says:

    Emilio, it's not my job to express your feelings, I can only work with the comments you write. If you think I've taken something out of context, that's fair, but I lifted the whole paragraph. If that's not how you feel then maybe you should be writing for your audience.

    I think people place WAY too much weight on the composition of the panel, because it's easier than looking at their proposal and trying to determine whether or not what they are writing is truly a top 10% proposal. It's all well and good to churn a middling proposal hoping that someone will see the brilliance, but it's not enough in the current environment. That's why I push the introspective approach, even if it's not everyone's cup of tea.

    Personally, I wouldn't bet on Terry's approach. That might be because I'm in a field less prolific than the tropical ant disciples, but I would rather not roll the dice like that. My preference is to write the proposal to excite a wide range of reviewers and assume nothing outside of what the panel fields, about the people deciding my fate. There's a pretty clear correlation for me between writing a broadly convincing proposal and getting funded. Yes, ad hocs are important, but in 15 pages there is enough real estate to please everyone, IMO. Either you have something exciting or not, and you should be able to figure that out after a couple rounds (now delayed) of feedback.

  • Joshua King says:

    PLS, something to consider. In many ways panels are like editors at high impact journals. Lots of submissions, often high quality, of stuff that is out of their area of expertise. The goal is to identify the novel and transformative to publish while leaving the incremental, perhaps less glamorous work behind. You seem to be stressing the point that the burden here is almost entirely on the authors and their writing to move the needle for the panel members or editors who are, if not blameless, at least should not be burdened with worry that they are missing things if a proposal is written, at least in part, with the "specialist" in mind. The problem here is that this whole mindset and approach (only the "novel" and "transformative" and broadly targeted proposal will stand the best chance of success) pretty much ignores how the vast majority of scientific advance actually occurs. I would urge you to read this editorial for some perspective on this kind of problem and why this approach is actually potentially a problem for science.

    http://www.cell.com/trends/ecology-evolution/abstract/S0169-5347%2813%2900126-2

    The funding process in the US, and elsewhere is capricious because it is a human endeavor. The authors need to write good proposals, no question. But the panels, and their composition, also have an impact and sometimes even very exciting science gets passed over because panel members blow it.

  • proflikesubstance says:

    Josh, While I'm still reeling from an actual comment form you, I'll make it clear that I don't think panels are blameless. BUT, as a proposal contributor, I think one is unwise in automatically blaming the panel. This is often the default.

    Do panels sometimes get it wrong? Of course. But not nearly to the extent they are blamed by those not selected for funding. My argument is simply that it is to the proposal submitters advantage to be more introspective than less.

  • Emilio Bruna says:

    But PLS, you DID take it out of context. It was from the tail end of a much longer discussion on a previous post on proposal neighborhoods and rankings, and I used the word *might* for a reason. It might. It might not. It may, but only for preproposals where space is limited. Or not. Hey, I wonder what the other readers of your blog think? But to smugly suggest that this "encapsulates a mentality" is just bogus. I'm also not sure it's true. I think "they just don't get it!" Is a common and natural first response to rejection, but that most people eventually get past it and see the flaws in their paper/proposal for what they are. Too narrow in scope, a boring question, bad design, poorly articulated, whatever.

    Still, it is your blog so interact with your commenters as you wish, and lesson learned. Had you asked, though, I would have told you as a submitter and author that in 99.9% of cases where a paper or proposal is rejected, its my fault for not explaining things well. But maybe specialists matter if your proposal is on the edge. but maybe not, and don't count on it. In other words, what you wrote in your response to Josh. Couldn't agree with you more.

  • Joshua King says:

    PLS, well, then we are in general agreement. Introspection all around. I also agree that there is usually room for improvement in most proposals, as Emilio points out. The crux of the problem, ultimately, is not panels or proposals, though. It is limited funding for basic biological, systematic, physiological, and ecological research that seems to grow tighter every year. This is what really gets me down rather than the vagaries of the process.

  • proflikesubstance says:

    But PLS, you DID take it out of context. It was from the tail end of a much longer discussion on a previous post on proposal neighborhoods and rankings

    And included a link to the specific comment, nested within all its glorious context. It's generally not good blogging practice to recreate an entire comment thread as a new post. The comment highlighted in the post is simply an example of the type of thing I hear often and I disagree with the sentiment. Hence the post.

    Had you asked, though, I would have told you as a submitter and author that in 99.9% of cases where a paper or proposal is rejected, its my fault for not explaining things well.

    Fair enough, and I think people who are successful get to this point, which is why they manage to rethink their proposals and put things out there that eventually do excite enough panelists to get through.

  • proflikesubstance says:

    Josh, yes, limited funding squeezes everything. Given that, I think it's all the more critical to try to identify where reviewers are getting hung up on your proposal rather than use the "They just didn't get it" or "moving goal posts" crutch than one can see constantly reiterated on the toobs or by the water cooler. Other than writing to your congressperson there is little we can do to fix current funding rates. So, the only other options are to give up or figure out what maximizes your chances to survive.

  • I have said this many times: the *last* people I want reviewing my grants are "experts in my subdiscipline. That's because these are exactly the motherfuckers who can see all the weaknesses and everything that will "never work". I want people reviewing my grants who know just enough to see how fucken cool my shitte is, but not enough to obsessively drill down into the details.

  • BirdNerd says:

    Couple of observations/comments from having submitted multiple pre-proposals to DEB and IOS (some invited, some not) and from having served on a DEB pre-proposal panel.

    1. Given the current funding rates, I don't think the panels 'get it wrong' in the sense of recommending invitation or funding of proposals that aren't worthy. I do think they potentially miss inviting or funding really cool stuff, but that's mostly because so few proposals actually get funded.

    2. I'd bet that we all have different ideas on what constitutes a 'sub-discipline.' That being said, if you're unable to convince a broad range of ecologists or evolutionary biologists (or whatever you consider your primary discipline) that your stuff is interesting and worth funding, you're not going to get funded, it's just that simple (in my opinion). Sure, everyone either knows someone or knows someone who knows someone that seemed to have lost out do to some stochastic panel process, but across the board that seems to be rare. Truly Excellent (in the actual NSF definition of the word) pre-proposals get invited and funded. Unfortunately, the Very Good pre-proposals typically do not, but that's because of the funding rates, not because there is anything wrong with the panels or the panelists reviewing the proposals. Again, in my opinion, 'ignorant reviewers' aren't too blame. If a reviewer isn't up on the latest research in your sub-discipline, they typically defer to those panelists that are more familiar with that area.

    It's easy to complain about the review process and how if a pre-proposal goes un-invited or a full proposal goes unfunded the panelists are to blame. I just don't think that's the case. Every panelist I've interacted with has wanted to invite/fund more proposals, the funding rates make it impossible.

  • Hey, I wonder what the other readers of your blog think?

    I think you're a fucken whiny asse titty baby and ought to toughen the fucke uppe and stop polluting interesting substantive discussions with logorrheic sniveling about whether your precious opinion has been "misrepresented".

  • Joshua King says:

    BirdNerd sums it up, yet again. No matter how good the writing in the proposal or how thorough the panel, very good to outstanding scientific work will go unfunded year in and year out. = capricious process.

    But by definition the capriciousness of the process should encourage the greater scientific community to think more about addressing the underlying funding issues (for example, state-level funding for universities) where we might be able to affect some change. Likewise, it also forces most of us to look for other ways to get things funded (SBIR, alternate agencies, private money, etc.). These are not necessarily bad things.

    Comradde, by the way, you are clearly one tough motherfukker. How'd you get so calloused and leathery?

  • [...] its offer to neuroscientists a no-brainer Federal Cuts to Force Closure of Fusion Project at MIT The elephant in the room The Outrageous Cost of a Gene Test Officially Entering “And Then They Fight” Phase of Open [...]

Leave a Reply