The encoder converts the source into contextual states, while the decoder predicts output tokens using its prior outputs and information from the encoder. Cross-attention commonly connects the two parts for tasks such as translation and summarization.