In a transformer block, the same position-wise network is applied to every token after attention has mixed information across positions. Its hidden dimension and activation determine much of the block's parameter count and nonlinear capacity.
A feed-forward network transforms each input independently through learned linear layers and nonlinear activations.
In a transformer block, the same position-wise network is applied to every token after attention has mixed information across positions. Its hidden dimension and activation determine much of the block's parameter count and nonlinear capacity.