Workers gather or exchange the pieces needed for each computation and may shard gradients and optimizer state as well. This lowers per-device memory use at the cost of additional communication and coordination.
Parameter sharding divides model parameters and related states across devices instead of replicating them everywhere.
Workers gather or exchange the pieces needed for each computation and may shard gradients and optimizer state as well. This lowers per-device memory use at the cost of additional communication and coordination.