SysTEX '24

Important dates

All deadlines are Anywhere on Earth (AoE = UTC-12h).

Artifact Evaluation

SysTEX this year for the first time includes an optional artifact evaluation phase for accepted papers, inspired by similar efforts in the security and systems communities.

The artifact evaluation is carried out by a dedicated volunteer artifact evaluation committee (AEC). The explicit focus is on providing constructive feedback and cooperatively improving the quality and reproducibility of any code or data research artifacts. This initiative aims to add an optional "artifacts evaluated" badge on the website, linking to the corresponding open-source repository. The presence of this badge will not only acknowledge the commitment to open science by the authors but also enhance the visibility of these efforts.

Submission Instructions

Authors of accepted paper who wish to take part in the optional artifact evaluation should register their artifact by the deadline June 7, 2024 (23:59 AoE) via https://systex24-artifacts.hotcrp.com/.

Submissions should include:

1. the (non-anonymous) camera-ready PDF of the accepted paper;
2. a clear, stable URL pointing to the public (non-anonymous) artifact;
3. which badges you want your artifact to be considered for (cf. section below: Available, Functional, Reusable). Assignment of final badges is at the discretion of the AEC after reviewing.

Any documentation and building/running instructions should be self-contained in the artifact (e.g., via a README file). In case custom hardware (e.g., Intel SGX, ARM TrustZone, etc.) is required, this should be clearly documented in the artifact itself. Authors may choose to optionally provide remote access to an evaluation machine with the required hardware for the artifact reviewers. In this case, login information may be privately shared via hotcrp comments.

The evaluation will be single-blind, i.e., only the identity of the reviewers is hidden but the paper and artifact should not be anonymous.

Badges

We will aim to have badges of increasing levels, similar to ACM and USENIX:

1. Artifacts Available: When the AEC could verify that the artifacts were made available permanently and publicly via a stable URL. Any public hosting website is allowed, but not a personal webpage. When using a git repository (e.g., GitHub), a commit hash or tag is required for a stable URL. To earn this badge, no further requirements on documentation, completeness, or functionality are needed.
   
2. Artifacts Functional: When the AEC deems the artifact to be (i) complete (i.e., all key components described in the paper are included); (ii) reasonably well-documented; and (iii) exercisable (e.g., building of the software could be verified).
   
3. Artifacts Reusable: When the quality of the artifact significantly exceeds minimal functionality, e.g., the code can be ran and at least some results could be reproduced.
   

Note: As this is a new initiative, and to keep the reviewing load manageable, verifying full reproducibility as per ACM's strict definition is put explicitly out of scope this year. The "Artifacts Reusable" badge is intended to recognize those artifacts that go beyond minimal expectations.