Skip to yearly menu bar Skip to main content


2024 Ethics Reviewers Guidelines

The role of ethics review is to assess NeurIPS submissions for risks in at least one of the following areas:

  • Research involving human subjects
  • Data privacy, copyright, and consent
  • Data quality and representativeness
  • Safety and security
  • Discrimination, bias, and fairness
  • Deception and harassment
  • Environmental Impact
  • Human rights (including surveillance)

NeurIPS has asked authors to consider ethics and broader impact when submitting their papers since 2021, and adopted a Code of Ethics in April 2023. Ethics reviews are a second round of review that take place should the program committee flag any potential concerns during the technical review phase. Ethics reviewers provide feedback to the program committee regarding risks and harms of the work, and recommend potential mitigations prior to acceptance. To minimize exposure risks, ethics reviews are only published at the author’s discretion; unlike the main reviews, which are all public. Please refer below and to the Ethics Review Guidelines for further details.

We ask that ethics reviewers:

  • review up to 5 papers each,
  • provide ethics reviews during at least one of:
    • The period of July 15 - 26, 2024 for submissions to the main conference track, and/or
    • The period of July 26 - August 6, 2024, for submissions to the Datasets & Benchmarks track
    • The period of August 7-30, 2024, for emergency reviews to cover late review requests
  • participate asynchronously in any relevant follow-up discussions with the authors and program committee during the author response period of August 2024. 
 

When Do You Get Involved?

You become involved when a paper is flagged for ethics review. This primarily occurs during the initial review process, though some cases may arise later during the discussion period or during reviews by AC and PC chairs.

What is an Ethics Review?

The ethics review differs from the primary review in several ways. Ethics review is limited in scope to evaluating the possibility of ethical risks, specifically those flagged by the primary reviewers. When the primary review flags a paper, it indicates a claim that an ethical concern exists. Your role as an ethics reviewer is to validate or disprove this claim, focusing on the evaluation of the claim and providing a critical discussion supporting your evaluation. For example, if a paper is flagged for potential ethical concerns around safety and security (e.g., misuse of autonomous weapons, misinformation, deepfakes, etc.), you will comment on the validity of these risks and whether the authors have done enough to mitigate them. Note that it is perfectly acceptable to conclude that no ethical concern exists, and we expect this to be true for some of the papers assigned to you. You are also encouraged to recommend reasonable risk mitigation strategies wherever appropriate.

To summarize, the following key aspects are typically addressed in an ethics review:

  • Are the flagged ethical concerns valid?
  • Are there any additional ethical concerns that were not flagged?
  • Do the authors suggest or implement mitigation strategies that sufficiently address the ethical risks? Note that a paper does not need to completely resolve the ethical risk.
  • As appropriate, reviewers are encouraged to suggest additional mitigation strategies that the authors could consider to alleviate the risks.

Reviewer Guidelines & Best Practices

1. Availability

It's okay to be unavailable for part of the review process (e.g., on vacation for a few days). However, if you will be unavailable for a longer period, especially during crucial times (e.g., the discussion phase), please inform us and your ACs.

2. Responsibility

With great power comes great responsibility! Take your role seriously and be fair.

3. Constructive Reviews

Write thoughtful and constructive reviews. Ensure your reviews align with the NeurIPS code of conduct. Although the double-blind review process helps reduce discrimination, subtle discrimination can still occur and should be actively avoided.

Example: Instead of saying, "Please have your submission proof-read by a native English speaker,” use a neutral formulation such as "Please have your submission proof-read for English style and grammar issues.”

4. Confidentiality and Communication

Do not discuss submissions assigned to you with other reviewers, ACs, or SACs without prior approval from your AC. Other reviewers, ACs, and SACs may have conflicts with these submissions. Your primary point of contact for any discussions should be the corresponding AC for that submission.

Do not discuss your own submissions (i.e., those you are an author on) or submissions with which you have a conflict with other reviewers, ACs, or SACs.

Keep papers assigned to you absolutely confidential.

5. Professionalism

Be professional, polite, and listen to other reviewers. Do not succumb to undue influence.

Actively engage in the discussion phase for each of your assigned submissions, even if not specifically prompted by the AC.

6. Thoroughness

It is unfair to dismiss any submission without thoroughly reading it. Reflect on times when you received an unfair, unjustified, short, or dismissive review and strive not to be that reviewer. Always be constructive and help authors understand your viewpoint without being dismissive or using inappropriate language. If you need to cite existing work to justify one of your comments, be precise and provide a complete citation.

If you would like the authors to clarify something during the author response phase, articulate this clearly in your review (e.g., “I would like to see a discussion on risk X”).

7. Ethics Review Sensitivity

Bear in mind the positive and objective aspects, especially in ethics reviews, where the content itself could provoke emotionally charged responses.

8. Cultural Inclusiveness

Ethical concerns are sometimes deeply intertwined with personal lived experiences and cultural norms. We ask ethics reviewers to consider that paper submissions at NeurIPS come from all over the world, and encourage you to evaluate if the ethics concerns under review are sensitive to cultural context and may not be a worldwide issue.