As part of a graduate class assignment I gave students a pre-review draft of a manuscript* and asked them to review it. We had discussed review, but none had experience with it. To complicate the matter, the class draws a wide range of students with varying backgrounds. Some were obviously more familiar with the background and topic of the manuscript than others. I later gave them the actual reviews of the paper and the final published version. Overall, I think this was a useful exercise and they appear to have as well.
But now I am grading their reviews and realizing that they are incredibly difficult to evaluate. In retrospect I could have given them clearer expectations for this assignment, but their work on this assignment is far more scatter shot than any of the previous assignments. I have no idea what to do with these.
*The manuscript was a review, intended for a broad audience.