ICML 2019 has an option for supplementary code submission that the authors can use to provide additional evidence to bolster their experimental results. Since we have been getting a lot of questions about it, here is a Frequently Asked Questions for authors.
1. Is code submission mandatory?
No. Code submission is completely optional, and we anticipate that high quality papers whose results are judged by our reviewers to be credible will be accepted to ICML, even if code is not submitted.
2. Does submitted code need to be anonymized?
ICML is a double blind conference, and we expect authors to put in reasonable effort to anonymize the submitted code and institution. This means that author names and licenses that reveal the organization of the authors should be removed.
Please note that submitted code will not be made public — eg, only the reviewers, Area Chair and Senior Area Chair in charge will have access to it during the review period. If the paper gets accepted, we expect the authors to replace the submitted code by a non-anonymized version or link to a public github repository.
3. Are anonymous github links allowed?
Yes. However, they have to be on a branch that will not be modified after the submission deadline. Please enter the github link in a standalone text file in a submitted zip file.
4. How will the submitted code be used for decision-making?
The submitted code will be used as additional evidence provided by the authors to add more credibility to their results. We anticipate that high quality papers whose results are judged by our reviewers to be credible will be accepted to ICML, even if code is not submitted. However, if something is unclear in the paper, then code, if submitted, will provide an extra chance to the authors to clarify the details. To encourage code submission, we will also provide increased visibility to papers that submit code.
5. If code is submitted, do you expect it to be published with the rest of the supplementary? Or, could it be withdrawn later?
We expect submitted code to be published with the rest of the supplementary. However, if the paper gets accepted, then the authors will get a chance to update the code before it is published by adding author names, licenses, etc.
6. Do you expect the code to be standalone? For example, what if it is part of a much bigger codebase?
We expect your code to be readable and helpful to reviewers in verifying the credibility of your results. It is possible to do this through code that is not standalone — for example, with proper documentation.
7. What about pseudocode instead of code? Does that count as code submission?
Yes, we will count detailed pseudocode as code submission as it is helpful to reviewers in validating your results.
8. Do you expect authors to submit data?
We understand that many of our authors work with highly sensitive datasets, and are not asking for private data submission. If the dataset used is publicly available, there is no need to provide it. If the dataset is private, then the authors can submit a toy or simulated dataset to illustrate how the code works.
9. Who has access to my code?
Only the reviewers, Area Chair and Senior Area Chair assigned to your paper will have access to your code. We will instruct reviewers, Area Chair and Senior Area Chair to keep the code submissions confidential (just like the paper submissions), and delete all code submissions from their machine at the end of the review cycle. Please note that code submission is also completely optional.
10. I would like to revise my code/add code during author feedback. Is this permitted?
Unfortunately, no. But please remember that code submission is entirely optional.
The detailed FAQ as well other Author and Style instructions are available here.
Kamalika Chaudhuri and Ruslan SalakhutdinovICML 2019 Program Chairs