The End Of Knowledge - Vault 6/27 - CVPR - 2017 - Distributed Deep Learning with MxNet Gluon

graph LR classDef main fill:#f9d9c9, font-weight:bold, font-size:14px classDef intro fill:#d4f9d4, font-weight:bold, font-size:14px classDef features fill:#d4d4f9, font-weight:bold, font-size:14px classDef examples fill:#f9f9d4, font-weight:bold, font-size:14px classDef advanced fill:#f9d4f9, font-weight:bold, font-size:14px classDef resources fill:#d4f9f9, font-weight:bold, font-size:14px Main[Distributed Deep Learning
with MxNet Gluon] Main --> A[Introduction] Main --> B[MXNet Features] Main --> C[Examples and Usage] Main --> D[Advanced Topics] Main --> E[Resources and Performance] A --> A1[Tutorial on distributed learning
with MXNet 1] A --> A2[Installation instructions slide reference 2] A --> A3[Assistance available for installation
issues 3] A --> A4[Speakers: Alex and Arakan
from Amazon 4] A --> A5[Get started on luong.mxnet.io 5] A --> A6[MXNet combines symbolic and
imperative benefits 6] B --> B1[Performance and flexibility for
deep learning 7] B --> B2[NDArray handles multi-device execution 8] B --> B3[Device context enables easy
data movement 9] B --> B4[No incrementing individual NDArray
elements 10] B --> B5[Autograd computes gradients of
dynamic graphs 11] B --> B6[Automatic gradient computation for
optimization 13] C --> C1[Linear regression model example 12] C --> C2[Multi-layer perceptron neural network
example 14] C --> C3[Network architecture using sequential
layer API 15] C --> C4[Flexible network definition with
symbolic performance 16] C --> C5[Activation functions, loss, gradient-based
optimization 17] C --> C6[Synthetic data generation and
preparation 18] D --> D1[Model setup, loss function,
optimizer, fitting 19] D --> D2[Evaluating model predictions and
test loss 20] D --> D3[Concise model definition and
automatic training 21] D --> D4[Complex architectures like CNNs
possible 22] D --> D5[Pre-defined architectures in model
zoo 23] D --> D6[Multiple GPUs machines for
distributed training 24] E --> E1[MXNet speed comparisons to
other frameworks 25] E --> E2[Distributed key-value store for
parameter synchronization 26] E --> E3[Multi-GPU training strategies 27] E --> E4[New optimizations for distributed
cluster training 28] E --> E5[Support for sparse data,
HDF5, math ops 29] E --> E6[Tutorials, examples, and learning
resources 30] class Main main class A,A1,A2,A3,A4,A5,A6 intro class B,B1,B2,B3,B4,B5,B6 features class C,C1,C2,C3,C4,C5,C6 examples class D,D1,D2,D3,D4,D5,D6 advanced class E,E1,E2,E3,E4,E5,E6 resources

Resume:

1.- The tutorial is on distributed learning with MXNet by Alex Smolak and Arakan from Amazon.

2.- Take a picture of the slide with installation instructions as a reference.

3.- Raise your hand if you have issues during the installation process and someone will assist you.

4.- Introduction to the speakers Alex and Arakan who work for Amazon, but not in package delivery.

5.- Get started with MXNet on luong.mxnet.io.

6.- MXNet combines benefits of symbolic frameworks (Caffe, Keras) and imperative frameworks (PyTorch, TensorFlow).

7.- MXNet offers performance of symbolic frameworks and flexibility of imperative frameworks for deep learning.

8.- MXNet's NDArray handles multi-device execution on CPU, GPU, and more using non-locking, lazy evaluation.

9.- Device context in MXNet allows easy data movement between devices.

10.- MXNet doesn't allow incrementing individual elements in NDArrays to enforce parallelism and performance.

11.- MXNet's autograd allows computing gradients of dynamic graphs and running optimization.

12.- Example of defining and training a simple linear regression model in MXNet.

13.- Using MXNet's autograd to automatically compute gradients for optimization, even for dynamic graphs with control flow.

14.- Moving to multi-layer perceptrons, a type of neural network, as a more sophisticated example.

15.- Example code defining a multi-layer neural network architecture using MXNet's sequential layer API.

16.- The code allows flexibly defining networks in a procedural manner while getting symbolic framework performance.

17.- Discussion of activation functions, loss functions, and gradient-based optimization for training neural networks.

18.- Generating synthetic data and preparing it for training a neural network.

19.- Setting up the model, loss function, optimizer and fitting the model to the generated training data.

20.- Evaluating the trained model's predictions and loss on test data to assess performance.

21.- The MXNet code allows concisely defining models that can be automatically trained via gradient descent.

22.- More complex architectures like convolutional neural networks can be defined using a similar API.

23.- Pre-defined neural network architectures are also available in MXNet's model zoo for common tasks.

24.- MXNet can utilize multiple GPUs and machines for distributed training of large models.

25.- Performance comparisons showing MXNet's speed relative to other popular deep learning frameworks.

26.- Distributed key-value store enables easy parameter synchronization for distributed training.

27.- Discussion of strategies for efficient multi-GPU training, like data and model parallelism.

28.- Latest MXNet release includes new optimizations for fast distributed training on large clusters.

29.- Additional features in MXNet include support for sparse data, HDF5 format, and optimized math operations.

30.- Pointers to MXNet tutorials, examples, and resources to learn more and get started quickly.

Knowledge Vault built byDavid Vivancos 2024