This semester I have been teaching a sort of "How Science Works" class for grad students. We've covered a lot of things, but one part of the class I really wanted to see work was doing a grant panel. I asked the class to write an NSF preproposal according to the new standards of Bio's DEB and IOS programs. Once they were handed in, I split the class into two parts and gave each student a "primary" and "secondary" proposal from the opposite class section.
I anonymized the proposals as best I could and distributed them. The panels were held during two separate class periods and every proposal was discussed with the class, led by the primary reviewer. All the proposals were rated and the ratings went up on the board. At the end of class I told them they could only put 1/3 of the proposals through and that they had to make a call on which preproposals get requested for a full proposal (we're not doing the full proposals).
I really didn't know what to expect from the class or how they would react to the exercise. I have to say I was very happy with how it all went. My primary concern was that they would all be afraid to criticize their peers, but that did not turn out to be the case. If anything, they were harsher than I expected - in a good way. Both panels were forced to make a tough call for the final spot and a lot of discussion ensued about the proposals in question.
The only things about the exercise that did not work as well as I had hoped mainly had to do with time. I did not ask the secondaries to do a panel summary, which I would have if there was a bit more time. I also would have liked to give each person three proposals, but we barely had time for the discussion we had with two each. I may consider scheduling the panels for a longer session next year, maybe in the evening rather than our typical class time.
All in all, however, I think it was a very valuable experience. With permission I have posted the proposals that did make the cut to the class internal webpage, do give all the students access to examples of good proposals. I have also sent the anonymized reviews of their proposal to all of the students for feedback. We'll be discussing the process later in the week, but several students have already expressed how much they got out of the experience. If you can fit this kind of thing into a graduate class, I highly recommend it.