A public test set that lives on GitHub will leak into crawls. Private or rotating held-out sets are how labs and arenas stay honest. If your 'eval' was in the pretrain mix, the number is a contamination artifact, not a capability measurement.
Data kept out of training — and often unpublished — so scores reflect generalization rather than memorization.
A public test set that lives on GitHub will leak into crawls. Private or rotating held-out sets are how labs and arenas stay honest. If your 'eval' was in the pretrain mix, the number is a contamination artifact, not a capability measurement.