Probabilities outside the selected set are removed and the remaining values are renormalized before sampling. A fixed k gives a constant candidate count even when the distribution is unusually sharp or flat.
Top-k sampling restricts each generation step to the k tokens with the highest model probability.
Probabilities outside the selected set are removed and the remaining values are renormalized before sampling. A fixed k gives a constant candidate count even when the distribution is unusually sharp or flat.