CHES 2021 aims to support open and reproducible research within the field of cryptography. As such, authors of papers accepted to CHES 2021 are invited to submit artifacts associated with their papers, such as software or datasets, for review, in a collaborative process between authors and the artifact review committee. The goal of the process is not just to evaluate artifacts, but also to improve them. Artifacts that pass successfully through the artifact review process will be archived alongside the paper on the website of the Transactions on Cryptographic Hardware and Embedded Systems (TCHES) journal.
Note that this is the first occurrence of an artifact review process within the IACR, and the process should be viewed as experimental at this stage. After the CHES 2021 cycle, the artifact review process will be revisited and may be changed or discontinued.
The two main goals of the artifact review process are to improve functionality and reusability of artifacts to enable reproducibility and extension by the scientific community.
Reproducibility, in the context of computational experiments, means that the scientific results claimed can be obtained by a different team using the original authors’ artifacts. The artifact review process for CHES 2021 does not include attempting to reproduce the experiment and verify the scientific claims in the accepted paper. Rather, the artifact review process for CHES 2021 aims at ensuring sufficient functionality of the artifact to enable a research team to attempt to reproduce the results.
Examples of this in the field of cryptography include:
Where possible, such as in software-based artifacts relying solely on open-source components, the artifact review process will aim to run the artifact and test harness, and see that it produces outputs that would be required to assess the artifact against results in the paper. For artifacts that depend on commercial tools or specialized physical hardware, the goal of the artifact review process will be to confirm that the artifacts are functional, and could plausibly be used by someone with access to the appropriate tools to reproduce the results.
Reusability means that the artifacts are not just functional, but of sufficient quality that they could be extended and reused by others. Reusable artifacts have clear user and developer documentation, and are well-structured in ways that make them easy to modify or extend.
The artifact review committee may recognize zero or more artifacts at the CHES 2021 conference as exemplars in terms of functionality, amenability to enabling reproducibility, or reusability.
The artifact review process begins after the paper has been accepted for publication in TCHES. Only papers accepted to CHES 2021 will be considered under the artifact review process.
Following notification of acceptance (or acceptance with minor revisions) to CHES 2021, authors should register their intention to submit an artifact for review. The artifact may be submitted for review any time up to the artifact submission deadline, but authors are encouraged to submit sooner. The review process will begin as soon as the artifact is submitted.
For papers accepted to TCHES Volume 2021 Issue 1
For papers accepted to TCHES Volume 2021 Issue 2
For papers accepted to TCHES Volume 2021 Issue 3
For papers accepted to TCHES Volume 2021 Issue 4
Once the artifact is submitted, two or more members of the artifact review committee will be assigned to review the artifact. The artifact review process will be a continuous process, and may involve requests from the reviewers for additional help on how to run the artifact, interpret its results, etc. It is acceptable (and expected) that the interaction between the reviewers and the authors leads to the artifact being updated during the review process. Updates that affect scientific characteristics reported in the paper (such as changes to performance) should be clearly documented.
We aim for the artifact review process to be completed within 6 weeks of the artifact being submitted, but this will vary depending on the scale of the artifact and the timeliness of interaction between the authors and reviewers. Authors of artifacts that are accepted for archiving will be provided instructions on how to submit the archival version of their artifact.
This is the first time an artifact review process has been run for an IACR event, and we may need to adapt the process as we go along. We ask for authors to be understanding and to join us in viewing this as a collaborative process trying to produce better artifacts for the scientific community.
The artifact review process will be single-blinded: the authors of the paper and artifact are not anonymous, but the reviewers will be anonymous. Communication between the authors and the reviewers will be facilitated via the HotCRP review site. Authors should not attempt to learn the identities of the reviewers, for example by not embedding analytics or tracking elements in the artifact or a website; if you cannot comply with this for some reason out of your control, please notify the chairs immediately to discuss.
The CHES 2021 artifact review process follows the same conflict of interest policy as TCHES, which is the IACR policy with respect to conflicts of interest. A conflict of interest is considered to occur automatically whenever an author of a submitted paper and a reviewer:
Conflicts may also arise for reasons other than those just listed. Examples include closely related technical work, cooperation in the form of joint projects or grant applications, business relationships, close personal friendships, instances of personal enmity. For more information please see the IACR Policy on Conflicts of Interest. Authors will be asked to identify conflicts of interest with the committee members at time of artifact registration.
If your artifact is accepted, you will be required to grant the IACR a non-exclusive, irrevocable license to distribute the artifact, via an open source license such as the Creative Commons Attribution (CC-BY), Attribution-NonCommercial (CC-BY-NC), Attribution-NonCommercial-NoDerivatives (CC-BY-NC-ND), or another open source license of your choice. If your artifact also contains third-party material that you did not create, you must ensure that you have permission to redistribute that material, for example because it is also open source or because you have obtained the appropriate permissions.
It is not a requirement that any patent rights be granted.
Artifacts shall be registered and submitted via the IACR HotCRP server at https://submit.iacr.org/ches2021artifacts.
A submission shall include:
The artifact itself shall include at least the following files:
Files such as LICENSE and README can be plain text files or Markdown files.
Source code files within the artifact are encouraged to be organized, formatted, and documented using best practices and conventions appropriate to the programming language in question. For example, formatted using a consistent style such as PEP8 for Python; documentation of APIs using JavaDoc for Java or Doxygen for C; unit tests using an appropriate framework.
The primary form of the artifact should be as source code, with suitable build scripts and instructions on how to install the appropriate dependencies.
For artifacts with complex dependencies or build requirements, the authors are encouraged to also package the artifact in the manner that makes it most amenable to successful execution. Potential formats include:
When in doubt, imagine a first-year grad student in 2029 who is told by their supervisor "See if you can change this artifact from CHES 2021 to do X." We want to give them the best chance of success with the least amount of pain.
ches2021artifacts@iacr.org