We design and implement AXLearn, a manufacturing deep studying system that facilitates scalable and high-performance coaching of enormous deep studying fashions. In comparison with different state-of-art deep studying techniques, AXLearn has a novel deal with modularity and assist for heterogeneous {hardware} infrastructure. AXLearn’s inside interfaces between software program elements observe strict encapsulation, permitting completely different elements to be assembled to facilitate fast mannequin improvement and experimentation on heterogeneous compute infrastructure. We introduce a novel technique of quantifying modularity by way of Strains-of-Code (LoC)-complexity, which demonstrates how our system maintains fixed complexity as we scale the elements within the system, in comparison with linear or quadratic complexity in different techniques. This permits integrating options resembling Rotary Place Embeddings (RoPE) into AXLearn throughout hundred of modules with simply 10 traces of code, in comparison with a whole bunch as required in different techniques. On the identical time, AXLearn maintains equal efficiency in comparison with state-of-the-art coaching techniques. Lastly, we share our expertise within the improvement and operation of AXLearn.
- ยง Duke College
- ** Work carried out whereas at Apple
