Bookstore

This book presents the state of the art in distributed machine learning algorithms that are based on gradient optimization methods. In the big data era, large-scale datasets pose enormous challenges for the existing machine learning systems. As such, implementing machine learning algorithms in a distributed environment has become a key technology, and recent research has shown gradient-based iterative optimization to be an effective solution. Focusing on methods that can speed up large-scale gradient optimization through both algorithm optimizations and careful system implementations, the book introduces three essential techniques in designing a gradient optimization algorithm to train a distributed machine learning model: parallel strategy, data compression and synchronization protocol.

Written in a tutorial style, it covers a range of topics, from fundamental knowledge to a number of carefully designed algorithms and systems of distributed machine learning. It will appeal to a broad audience in the field of machine learning, artificial intelligence, big data and database management.

Distributed Machine Learning and Gradient Optimization

1.1.  Background

1.2.  Distributed machine learning

1.3.  Gradient optimization

1.4.  Challenges

Chapter 2: The preliminaries

        2.1. Overview

        2.2. Parallel strategy

        2.3. Gradient compression

        2.4. Synchronization protocol

Chapter 3: Parallel strategy

1.1.  Background and problem

1.2.  Data parallelism

1.3.  Model parallelism

1.4.  Hybrid parallelism

        3.5. Benchmark

        3.6. Summary

        4.1. Background and problem

        4.2. Lossless gradient compression

        4.3. Lossy gradient compression

        4.4. Sparse gradient compression

        4.5. Benchmark

        4.6. Summary

Chapter 5: Synchronization protocol

        5.1. Background and problem

        5.2. Bulk synchronous protocol

        5.4. Stale synchronous protocol

        5.5. Benchmark

Chapter 6: Conclusion

        6.1. Summary of the book

        6.2. Future work