2023 Datasets and Benchmarks Track Reviewer Guidelines

 

The Datasets and Benchmarks track serves as a venue for high-quality publications, talks, and posters on highly valuable machine learning datasets and benchmarks, as well as a forum for discussions on how to improve dataset development. 

This page provides an overview of reviewer responsibilities and key dates. Please note, some dates differ from the main track.

Contact Information

The Area Chair (AC) assigned to a paper should be your first point of contact for that paper. You can contact the AC by leaving a comment in OpenReview with the AC as a reader. 

If you encounter a situation that you are unable to resolve with your AC, please contact the Datasets and Benchmarks chairs at datasetsbenchmarks@neurips.cc. 

 

Important Dates 

Dates may be subject to change.

  • Paper submission - Wed, June 7, 2023
  • Reviewers bid on papers: Thurs, June 8 - Wed, June 14, 2023
  • Check paper assignment: Wed, June 19, 2023
  • Reviewing period: Wed, June 19 - Fri, July 21, 2023
    • Extended to July 27, 2023
  • Ethics review period: Mon, July 24 - Tues, Aug 1, 2023
  • Reviews released to authors: Wed, Aug 2, 2023
  • Reviewer-Author Discussions: Wed, Aug 2 – Wed, Aug 30, 2023
  • Reviewer-AC Discussions: Wed, Aug 30  - Fri, Sept 8, 2023
  • Metareviews due:  Fri, Sept 15, 2023
  • Author notifications: Fri, Sept 22, 2023

In terms of workload, we plan to assign no more than 3 submissions per reviewer and no more than 5 submissions per AC. Please note that we may need to deviate from this rule in certain cases.

 

Main Tasks

Fulfilling your responsibilities as a reviewer in a high quality and timely manner is critical to the success of the review process. Here is a list of key dates and tasks for reviewers:

  1. Preparation:
    • Read and agree to abide by the NeurIPS code of conduct.
    • NeurIPS 2023 is using OpenReview. Please make sure that your OpenReview profile is up to date. If you have changed or plan to change your email address, please update the address set as “preferred” in your OpenReview profile and confirm it. It is crucial that we are able to reach you quickly. We will send most emails from OpenReview (noreply@openreview.net). Such emails are sometimes accidentally marked as spam (or classified as Updates in Gmail). Please check these folders regularly. If you find such an email in there, please whitelist noreply@openreview.net so that you do not miss future emails related to NeurIPS 2023 Datasets and Benchmarks.
    • Note that your assignments and tasks will appear at the reviewer console in OpenReview
    • Read what constitutes a conflict of interest for NeurIPS 2023 and how to declare them in your profile.  
  2. Bid on papers: Thurs, June 8 - Wed, June 14 2023.
    • Your bids are an important input to the paper matching process. 
    • Unfortunately, in past years there have been a small number of reviewers who engage in deceptive bidding practices. If we have a reason to suspect that a reviewer is engaged in deceitful bidding to influence reviewing outcomes, we will request an ethics investigation, and malicious actors may be removed from future involvement in the program committee. 
  3. Check paper assignments: Wed, June 14 2023.
    • As soon as you are notified of papers to review, you are expected to log in to OpenReview to check for conflicts and to check that papers fall within your area of expertise.
    • If you don’t feel qualified to review a paper that was assigned to you, please communicate this to your AC right away.
    • These assignments may change during the first week, as some reviewers and ACs request re-assignments. Please watch for notification email from Openreview.
  4. Reviewing: Wed, June 14 -  Fri, July 21 2023.
    • We know that serving as a reviewer for NeurIPS Datasets and Benchmarks track is time consuming, but the community depends on your high quality reviews to uphold the scientific quality of NeurIPS. 
    • Please make your review as informative and substantiated as possible; superficial, uninformed reviews are worse than no review as they may mislead the review process.
    • You can see the review form questions and guidance on how to answer each question in the "Review Form" section below.
    • Make sure to flag any questionable papers for ethics review. These papers will be assigned ethics reviewers, who will effectively join the paper's assigned program committee. See the NeurIPS ethics guidelines.
    • Please  use the NeurIPS paper checklist included in each paper as a tool when preparing your review (some submissions may have the checklist as part of the supplementary materials, this is allowed). Remember that answering “no” to some questions is typically not grounds for rejection. In general, authors should be rewarded rather than punished for being up front about the limitations of their work and any potential negative societal impact. You are encouraged to think through whether any critical points are missing and provide these as feedback for the authors.
    • Do not worry about minor violations of the required format (e.g., papers that exceed the page limit by a few lines), but immediately report any major violations that you notice to your AC.
    • When writing your review, please keep in mind that after decisions have been made, reviews and meta-reviews of accepted papers as well as your discussion with the authors will be made public (but reviewer and AC identities will remain anonymous). This year, authors of rejected papers will have the option to make this information public for their rejected papers as well.
  5. Discussion period: Wed, Aug 2 – Fri, Sept 8 2023.
    • Reviews will be released to authors on Wed, Aug 2 2023. 
    • Author-Reviewer Discussions (Wed, Aug 2 – Wed, Aug 30 2023): During this first phase, please carefully read all other reviews and the author responses to all reviews for the papers assigned to you. Then please engage in an open exchange with the authors.
    • Reviewer-AC Discussions (Wed, Aug 30  - Fri, Sept 8 2023): During this second phase, please discuss the paper, the reviews, and the author responses among the reviewers and with the area chair.
    • As you read each author response, please keep an open mind.  Even if the author response didn’t change your opinion about the paper, please acknowledge that you have read and considered it.  
    • Participating in discussions is a critical part of your role as a reviewer.  The discussion period is especially important for borderline papers and papers for which the reviewers’ assessments differ, and we hope that you take discussions seriously.  If your evaluation of the paper has changed, please revise your review and explain the change.
    • When discussing a paper, remember that different people have different backgrounds and different points of view. Reviewer consensus is valuable—only rarely are unanimous assessments overruled—but it is not mandatory.
  6. AC’s metareviews due: Fri, Sept 15 2023
    • Reviewer workload between the end of the discussion period (Fri, Sept 8 2023) and the metareview due date (Fri, Sept 15 2023) should be light, but if ACs come back to you with additional questions, please respond promptly.
  7. Author notification: Fri, Sept 22 2023.

 

 

Review Form

Below is a description of the questions you will be asked on the review form for each paper and some guidelines on what to consider when answering these questions. Please use the NeurIPS paper checklist included in each paper as a tool when preparing your review (some submissions may have the checklist as part of the supplementary materials). Remember that answering “no” to some questions is typically not grounds for rejection. When writing your review, please keep in mind that after decisions have been made, reviews and meta-reviews of accepted papers and opted-in rejected papers will be made public.

 

  1. Summary and contributions: Briefly summarize the paper and its contributions. This is not the place to critique the paper; the authors should generally agree with a well-written summary.
  2. Strengths: Describe the strengths of the submission, considering significance of the contribution, relevance to the broader research community, quality of the research, clarity of paper, and ethical and social implications. If the submission includes a dataset and/or benchmark, consider accessibility, accountability, and transparency or the artifact.
  3. Opportunities for improvement: Explain the limitations of this work along the same axes as above.
  4. Limitations: Have the authors adequately addressed the limitations and potential negative societal impact of their work? If not, please include constructive suggestions for improvement.
    In general, authors should be rewarded rather than punished for being up front about the limitations of their work and any potential negative societal impact. You are encouraged to think through whether any critical points are missing and provide these as feedback for the authors.
  5. Correctness: Are the claims made in the submission correct? If the submission is a dataset, it is constructed in a sound way? If it is a benchmark, are the evaluation methods and experiment design appropriate and performed correctly?
  6. Clarity: Is the paper well written?
  7. Relation to prior work: Is it clearly discussed how this work differs from previous contributions?
  8. Documentation: For datasets, is there sufficient detail on data collection and organization, availability and maintenance, and ethical and responsible use? Note that dataset submissions should include documentation and intended uses; a URL for reviewer access to the dataset; and a hosting, licensing and maintenance plan. For benchmarks, is there sufficient detail to support reproducibility?
  9. Ethical concerns: If there are ethical issues with this paper, please flag the paper for an ethics review. For guidance on when this is appropriate, please review the NeurIPS ethics guidelines.
  10. Overall: Please provide an "overall score" for this submission. 
  11. Confidence:  Please provide a "confidence score" for your assessment of this submission to indicate how confident you are in your evaluation.  
  12. Code of conduct acknowledgement. While performing my duties as a reviewer (including writing reviews and participating in discussions), I have and will continue to abide by the NeurIPS code of conduct (https://neurips.cc/public/CodeOfConduct).

 

 

Other Roles

During the review process you will be working with:

  • Area Chairs (ACs). ACs are the principal contact for reviewers during the whole reviewing process. ACs are responsible for checking the initial reviewer assignment and optionally recommending new reviewers for submissions, ensuring that all submissions receive quality reviews, facilitating discussions among reviewers, writing meta-reviews, evaluating the quality of reviews, and making decision recommendations.
  • Ethics Reviewers. You may flag submissions for additional review by ethics reviewers. The comments from the ethics reviewers will be visible to all reviewers, the AC, and the authors. You may use their comments to inform your deliberations.

 

Policies

Please make sure to review the policies in the NeurIPS 2022 Datasets and Benchmarks Call for Papers. 

Executing Code & Clicking on Links

Please remember that just like any other untrusted code, any submitted code may contain security vulnerabilities. If you are planning to run any submitted code, please make sure you are doing this in a secure environment because this code is not vetted by our submission system. We recommend running source code (1) inside a Docker container, or (2) a Virtual Machine image (using VirtualBox or VMWare), or (3) on a network-isolated cloud instance.

You may wish to also be cautious about accessing other web links provided from the paper, as these may contain vulnerabilities or may log visitor IP addresses.

Confidentiality

You must keep everything relating to the review process confidential. Do not use ideas, code, or results from submissions in your own work until they become publicly available. Do not talk about or share submissions with anyone without prior approval from the program chairs. Code submitted for reviewing cannot be distributed or used for any other purpose.

Please note that in the Datasets and Benchmarks track, authors can choose to submit single-blind since it may be impossible to share hosted and well-documented datasets or benchmarks entirely anonymously. In some cases, the submission may already be publicly available.

 

 

Formatting instructions

As a reminder, submissions are limited to nine content pages, including all figures and tables, in the NeurIPS “submission” style; additional pages containing only references and the NeurIPS 2023 paper checklist are allowed. Any content beyond that can be reviewed at your discretion. This includes any additional content that was submitted as part of the main PDF as well as any supplementary material uploaded separately.  In general, we were lenient with minor formatting violations (e.g., a spillover to page 10), as long as these violations can be easily rectified in the final version.  If you find violations that are not easily rectified without causing other presentation issues, please flag them to your AC.

Some submissions may have included the NeurIPS 2023 checklist into their supplementary material by mistake, so you may find the checklist there (to be viewed at your discretion).

Dual submissions

NeurIPS does not allow submissions that are identical or substantially similar to papers that are in submission to, have been accepted to, or have been published in other archival venues. Submissions that are identical or substantially similar to other NeurIPS submissions fall under this policy as well; all NeurIPS submissions should be distinct and sufficiently substantial. Slicing contributions too thinly is discouraged, and may fall under the dual submission policy.  If you suspect that a submission that has been assigned to you is a dual submission or if you require further clarification, please contact the corresponding AC. For more information about dual submissions, please see the Call for Papers and FAQ.

Useful Links