Open Internet by MindsNet

Optimizing Local SGD Synchronization

The current ad-hoc heuristic for determining when to AllReduce in heterogeneous Local SGD is limited. It relies on a hand-tuned threshold and an opaque '3 consecutive rises' rule, which may not be optimal or generalizable across different architectures and setups.

Computing & Technology, Computer Science, Machine Learning