- Data access under agreement
- Evaluation sets released with their evidence rather than as answer keys: the retained documents, the verbatim excerpts, the locators and the dates that produced each value, so that a disagreement with our reading is checkable rather than a matter of opinion. The agreement fixes what may be redistributed, because part of the material is licensed and part of it is private.
- Co-authorship on evaluations
- If your group designs the measurement, you are an author on it. We are not looking for a validation exercise with our name on the result: we would rather the counting rule was written by somebody with no stake in which way it comes out.
- The protocol before the result
- What is being measured, on what data, with what counting rule and what would count as a failure, dated and published before the run. A protocol we publish after the numbers are in is a description of the numbers, and we treat it as one.
- How to cite
- The position paper carries the argument the product rests on, with its citations; the research page carries a BibTeX entry for it. If you cite a measurement rather than the argument, cite the dated protocol and the dated result separately, because they are two documents.
- No embargo and no approval right
- We do not read your write-up before submission, we do not hold a veto over a conclusion drawn from material we provided, and there is no clause that lets us withdraw data after seeing a result. The only constraint in the agreement concerns redistributing material we are not free to release, and it is written down rather than left to a conversation.