Each worker computes gradients from its local examples, then a collective operation synchronizes gradients before parameters advance. Effective scaling depends on batch size, network speed, and keeping workers supplied with data.
Data parallelism runs copies of a model on different data batches and combines their training updates.
Each worker computes gradients from its local examples, then a collective operation synchronizes gradients before parameters advance. Effective scaling depends on batch size, network speed, and keeping workers supplied with data.