Bidirectional attention usually allows each input position to use context from both sides. A task-specific head can then turn the representations into classifications, token labels, embeddings, or other predictions.
An encoder-only model produces contextual representations of an entire input rather than generating it token by token.
Bidirectional attention usually allows each input position to use context from both sides. A task-specific head can then turn the representations into classifications, token labels, embeddings, or other predictions.