Pairs of narrow matrices are inserted into selected weight transformations, and their product represents the update. This reduces trainable parameter count and makes separate task adapters easier to store and swap.
Low-rank adaptation fine-tunes a model by learning small low-rank updates while keeping the original weights frozen.
Pairs of narrow matrices are inserted into selected weight transformations, and their product represents the update. This reduces trainable parameter count and makes separate task adapters easier to store and swap.