On top of several other things I have going on these days, I'm also reviewing a stack of grants for an NSF panel. I have done a good amount of ad hoc reviews for NSF and other funding agencies before, so I didn't think this would be all that much different. Oh, but it is.
To start with, some of these proposals are pretty far outside of my expertise. When you get an ad hoc review it often (though not always) falls within your close field, which is why they sent it to you. When you have been assigned 12 reviews out of ~100, the odds are against even half of those being related to your work.
Another issue is the sheer volume. Proposals are dense and it takes a bit to get through on, summarize your thoughts on their system and approach and comment intelligently on the whole thing. Doing that 4 or 5 times takes a lot of time. 12 times for grants that are mostly outside of work? That's a whole new ball game. I would be lying if I didn't tell you that I am about 3/4 through and feeling like my brain has turned to oatmeal. Remember that I am also teaching, dealing with everything going on in my lab and trying to write a new grant for the Oct NIH deadline.
Finally, the feedback is giving me a bit of stage fright. When you do ad hoc reviews you submit them and go about your daily business. When you are a panelist, you instantly have access to all of the other reviews on the proposals you are reviewing, as soon as yours is submitted. As soon as the submit button is hit, you can see how your impression measures up against all the other reviewers, and I admit that has made me a little more vigilant in making sure I can back everything I say up with data. It's not that I did crappy reviews in the past, but the instant comparison between my opinion and that of Big Name X is slightly intimidating as I hit the submit button. In some cases I am right in line with others and in some cases I have a very different opinion of the proposal. In the latter cases, it means that I need to do more reading to make sure I can defend my position when it comes to the panel, which BTW, is yet another unknown for me.
I am glad I am doing this and looking forward to the experience, but the process has been more draining than I thought it would be and finding time to get to all of them has really cut into the time I spend with my family. I'll be happy when it is over.