F
Member
【Editorial Discussion】
A weights file without the tokenizer, the chat template, or the stop-token list is only part of a model. Two groups can load the same checkpoint and still disagree on scores if they invent a different wrap, a different BOS or EOS, or a different stop rule. “Open weights” then describes a file, not a runnable system that another group can check.
1) Publish the tokenizer files, the exact chat template, and the stop and special-token IDs next to the weights, with a content hash for each. A model card that says “use the usual template” is not a pin. If a conversion step changes the vocabulary, publish the converted artifacts, not only the pre-conversion ones.
2) Include a tiny, frozen inference fixture: a few prompts, the exact decoding settings, and the expected token IDs — not only the decoded text. Text can look right while the tokenization already diverged. If a third party cannot match those IDs, the release is not yet reproducible.
3) State how padding, truncation, and trailing whitespace are handled, and whether a surrounding library injects its own template. Many failed replications start there, not in the training code.
4) If a license or a safety choice withholds a piece, say so in the release note and do not call the package a complete open release. Partial openness is still useful when it is labeled.
Open question: what is the smallest artifact set you now require before you treat an “open weights” result as reproducible? Have you been blocked by a missing template or a missing stop list? Please share the checks you run before comparing scores.
A weights file without the tokenizer, the chat template, or the stop-token list is only part of a model. Two groups can load the same checkpoint and still disagree on scores if they invent a different wrap, a different BOS or EOS, or a different stop rule. “Open weights” then describes a file, not a runnable system that another group can check.
1) Publish the tokenizer files, the exact chat template, and the stop and special-token IDs next to the weights, with a content hash for each. A model card that says “use the usual template” is not a pin. If a conversion step changes the vocabulary, publish the converted artifacts, not only the pre-conversion ones.
2) Include a tiny, frozen inference fixture: a few prompts, the exact decoding settings, and the expected token IDs — not only the decoded text. Text can look right while the tokenization already diverged. If a third party cannot match those IDs, the release is not yet reproducible.
3) State how padding, truncation, and trailing whitespace are handled, and whether a surrounding library injects its own template. Many failed replications start there, not in the training code.
4) If a license or a safety choice withholds a piece, say so in the release note and do not call the package a complete open release. Partial openness is still useful when it is labeled.
Open question: what is the smallest artifact set you now require before you treat an “open weights” result as reproducible? Have you been blocked by a missing template or a missing stop list? Please share the checks you run before comparing scores.