How to Review Papers

• A review should always be polite, respectful to the author(s), and helpful for improving the paper, regardless of whether you recommend acceptance or rejection.

• Start the review with one or two sentences summarizing the paper.

• Communicate clearly the strengths and weaknesses of the paper.

• Make sure that your comments give the reasons for your recommendation.

• Be especially clear in justifying a recommendation of rejection and provide feedback the author(s) can use to improve the paper/work.

• If you feel the author(s) should be aware of related work, try to provide specific references.

• Try to give, in terms of quality, the kind of review you would like to receive for your own work.

• Keep the original paper you reviewed for a few months (3 in case of conferences, up to 6 in case of journal papers) so that you can discuss your review in case any controversy arises.

• Do not distribute submitted papers, they are meant to be confidential.

• Here are some (transcribed) tips for paper reviewing from Allan Newell:

  When we get a paper to review, at the beginning we should always have as the default that we accept the paper. While reading the paper, we may start raising specific objections along the issues in the review form, namely the goals are not stated, the system is not well described, the approach is not novel or not validated, etc. etc.

  Each objection weighs a little against our initial default acceptance. Rejecting a paper is to see if these objections weigh more than our threshold, based on our experience with other other conferences, papers, and advice from the specific conference. (Of course, the review can also raise the initial default acceptance, and then it's even a clearer accept.)

  One word of care that I recall: It may happen that we raise either unjustified objections or support to a paper. In particular in AI, it may be rather common that we completely "disagree" or "agree" with the paper's approach/results. We have to be very careful, as much as possible, not to include objections or support that corresponds to subjective, or sometimes dogmatic, opinions about the work.