F
Member
【Editorial Discussion】
Token-level perplexity is the exponential of average negative log probability per scored token. Change the tokenizer and the unit in that average changes. A lower number across two differently segmented models is therefore not, on its own, evidence of better prediction on the same text. Even a perfectly implemented likelihood evaluator can produce an invalid comparison if the headline treats unlike tokens as a common unit.
The problem is not an absent tokenizer or an unreproducible release. Assume both tokenizers and both models are fully available. The question is which quantity permits a meaningful comparison, and what additional differences remain after the denominator is repaired.
**Expose the denominator first**
For each identical source document, retain the total negative log likelihood of its canonical tokenization, the number of scored tokens, and the number of source bytes. Report the totals as well as the averages. This makes it possible to see whether a changed token count is driving the apparent advantage. Aggregate corpus totals before dividing if the target metric is a corpus-wide per-byte average; averaging document averages instead gives short documents more relative weight.
A useful common-unit diagnostic is code length in bits per source byte: convert the negative log likelihood from natural-log units to bits and divide by the byte count of the same text. This normalizes the denominator; it does not make two model pipelines otherwise identical. In particular, a canonical token-path probability is not automatically the marginal probability of a string summed over every token sequence that could decode to it.
**Keep the scored material comparable**
1) Verify lossless recovery of the source text, including whitespace and Unicode distinctions relevant to the corpus. A tokenizer pipeline that silently normalizes or drops content is scoring a different representation. Do not call the removed bytes successful compression by the model.
2) State how document starts, document ends, and special tokens contribute to the numerator. Reset context consistently between independent documents. For an initial diagnostic, use documents that fit both models' context limits so different truncation and sliding-window policies do not dominate the comparison.
3) Examine results by text type and language, with identical document membership. A tokenizer may provide a different effective text span within a fixed token budget. A separate fixed-token-window experiment studies that deployment trade-off, not the same full-document likelihood comparison.
Per-byte code length is not a universal quality score. It cannot establish instruction following, factual correctness, or useful reasoning, and comparisons still depend on preprocessing and context conventions. It also cannot isolate a tokenizer's causal effect when model weights and training data changed simultaneously.
How do you present likelihood comparisons across tokenizers without implying that tokens are interchangeable units? Which diagnostic has exposed an apparent perplexity improvement that disappeared once the scored text and denominator were made comparable?
Token-level perplexity is the exponential of average negative log probability per scored token. Change the tokenizer and the unit in that average changes. A lower number across two differently segmented models is therefore not, on its own, evidence of better prediction on the same text. Even a perfectly implemented likelihood evaluator can produce an invalid comparison if the headline treats unlike tokens as a common unit.
The problem is not an absent tokenizer or an unreproducible release. Assume both tokenizers and both models are fully available. The question is which quantity permits a meaningful comparison, and what additional differences remain after the denominator is repaired.
**Expose the denominator first**
For each identical source document, retain the total negative log likelihood of its canonical tokenization, the number of scored tokens, and the number of source bytes. Report the totals as well as the averages. This makes it possible to see whether a changed token count is driving the apparent advantage. Aggregate corpus totals before dividing if the target metric is a corpus-wide per-byte average; averaging document averages instead gives short documents more relative weight.
A useful common-unit diagnostic is code length in bits per source byte: convert the negative log likelihood from natural-log units to bits and divide by the byte count of the same text. This normalizes the denominator; it does not make two model pipelines otherwise identical. In particular, a canonical token-path probability is not automatically the marginal probability of a string summed over every token sequence that could decode to it.
**Keep the scored material comparable**
1) Verify lossless recovery of the source text, including whitespace and Unicode distinctions relevant to the corpus. A tokenizer pipeline that silently normalizes or drops content is scoring a different representation. Do not call the removed bytes successful compression by the model.
2) State how document starts, document ends, and special tokens contribute to the numerator. Reset context consistently between independent documents. For an initial diagnostic, use documents that fit both models' context limits so different truncation and sliding-window policies do not dominate the comparison.
3) Examine results by text type and language, with identical document membership. A tokenizer may provide a different effective text span within a fixed token budget. A separate fixed-token-window experiment studies that deployment trade-off, not the same full-document likelihood comparison.
Per-byte code length is not a universal quality score. It cannot establish instruction following, factual correctness, or useful reasoning, and comparisons still depend on preprocessing and context conventions. It also cannot isolate a tokenizer's causal effect when model weights and training data changed simultaneously.
How do you present likelihood comparisons across tokenizers without implying that tokens are interchangeable units? Which diagnostic has exposed an apparent perplexity improvement that disappeared once the scored text and denominator were made comparable?