Memory dies are placed close to the processor and connected through wide interfaces. Capacity and bandwidth affect whether large models fit and how quickly parameters and activations can reach compute units.
High-bandwidth memory is a stacked memory technology that provides accelerators with high data-transfer bandwidth.
Memory dies are placed close to the processor and connected through wide interfaces. Capacity and bandwidth affect whether large models fit and how quickly parameters and activations can reach compute units.