Reviewers give a score of between 1 and 10 for each paper. The program committee will interpret the numerical score in the following way:
10: Top 5% of accepted NIPS papers, a seminal paper for the ages.
I will consider not reviewing for NIPS again if this is rejected.
9: Top 15% of accepted NIPS papers, an excellent paper, a strong accept.
I will fight for acceptance
8: Top 50% of accepted NIPS papers, a very good paper, a clear accept.
I vote and argue for acceptance
7: Good paper, accept.
I vote for acceptance, although would not be upset if it were rejected.
6: Marginally above the acceptance threshold.
I tend to vote for accepting it, but leaving it out of the program would be no great loss.
5: Marginally below the acceptance threshold.
I tend to vote for rejecting it, but having it in the program would not be that bad.
4: An OK paper, but not good enough. A rejection.
I vote for rejecting it, although would not be upset if it were accepted.
3: A clear rejection.
I vote and argue for rejection.
2: A strong rejection. I'm surprised it was submitted to this conference.
I will fight for rejection
1: Trivial or wrong or known. I'm surprised anybody wrote such a paper.
I will consider not reviewing for NIPS again if this is accepted
Reviewers should NOT assume that they have received an unbiased sample of papers, nor should they adjust their scores to achieve an artificial balance of high and low scores. Scores should reflect absolute judgments of the contributions made by each paper.
Confidence Scores
Reviewers also give a confidence score between 1 and 5 for each paper. The program committee will interpret the numerical score in the following way:
5:
The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature.
4:
The reviewer is confident but not absolutely certain that the evaluation is correct. It is unlikely but conceivable that the reviewer did not understand certain parts of the paper, or that the reviewer was unfamiliar with a piece of relevant literature.
3:
The reviewer is fairly confident that the evaluation is correct. It is possible that the reviewer did not understand certain parts of the paper, or that the reviewer was unfamiliar with a piece of relevant literature. Mathematics and other details were not carefully checked.
2:
The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper.
1:
The reviewer's evaluation is an educated guess. Either the paper is not in the reviewer's area, or it was extremely difficult to understand.
Qualitative Evaluation
All NIPS papers should be good scientific papers, regardless of their specific area. We judge whether a paper is good using 4 criteria; a reviewer should comment on all of these, if possible:
Quality
Is the paper technically sound? Are claims well-supported by theoretical analysis or experimental results? Is this a complete piece of work, or merely a position paper? Are the authors careful (and honest) about evaluating both the strengths and weaknesses of the work?
Clarity
Is the paper clearly written? Is it well-organized? (If not, feel free to make suggestions to improve the manuscript.) Does it adequately inform the reader? (A superbly written paper provides enough information for the expert reader to reproduce its results.)
Originality
Are the problems or approaches new? Is this a novel combination of familiar techniques? Is it clear how this work differs from previous contributions? Is related work adequately referenced? We recommend that you check the proceedings of recent NIPS conferences to make sure that each paper is significantly different from papers in previous proceedings. Abstracts and links to many of the previous NIPS papers are available from http://books.nips.cc
Significance
Are the results important? Are other people (practitioners or researchers) likely to use these ideas or build on them? Does the paper address a difficult problem in a better way than previous research? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions on existing data, or a unique theoretical or pragmatic approach?
zz from http://nips.cc/PaperInformation/ReviewerInstructions
