The End Of Knowledge - Vault 6/16 - CVPR - 2016

graph LR classDef main fill:#f9d9c9, font-weight:bold, font-size:14px classDef rl fill:#d4f9d4, font-weight:bold, font-size:14px classDef dl fill:#d4d4f9, font-weight:bold, font-size:14px classDef methods fill:#f9f9d4, font-weight:bold, font-size:14px classDef applications fill:#f9d4f9, font-weight:bold, font-size:14px Main[Deep Reinforcement Learning] Main --> A[Reinforcement Learning] Main --> B[Deep Learning] Main --> C[Methods and Algorithms] Main --> D[Applications] A --> A1[RL optimizes decisions,
maximizes future rewards 1] A --> A2[RL formalizes agent-environment
interaction for rewards 5] A --> A3[RL includes policy,
value function, model 6] A --> A4[RL: general framework
for decision-making 7] A --> A5[Value-based RL estimates
optimal value function 8] A --> A6[Optimal value obeys
recursive Bellman equation 9] B --> B1[Deep learning composes
parameterized functions 2] B --> B2[Neural networks combine
transformations, optimized parameters 3] B --> B3[Weight sharing enhances
neural architectures 4] B --> B4[DQN uses neural
networks for Q-function 11] B --> B5[Neural networks represent
Go positions, probabilities 24] B --> B6[Value networks trained
on self-play games 26] C --> C1[Q-learning estimates action-value
function Q s,a 10] C --> C2[DQN improvements: Double
DQN, Prioritized Replay 12] C --> C3[Distributed DQN enables
faster parallel training 13] C --> C4[A3C uses parallel
actor-learners for stability 14] C --> C5[Policy gradients optimize
policy using gradients 15] C --> C6[Actor-critic learns policy
and value function 17] D --> D1[Continuous control possible
with actor-critic variants 19] D --> D2[Complex variants solve
challenging control problems 20] D --> D3[Strategic games combine
RL, counterfactual regret 21] D --> D4[Model-based RL learns
environment for planning 22] D --> D5[AlphaGo combines deep
RL, search, self-play 28] D --> D6[Future: healthcare, assistants,
conversational AI 30] class Main main class A,A1,A2,A3,A4,A5,A6 rl class B,B1,B2,B3,B4,B5,B6 dl class C,C1,C2,C3,C4,C5,C6 methods class D,D1,D2,D3,D4,D5,D6 applications

Resume:

1.- RL optimizes decisions to maximize future rewards. Deep learning enables learning representations from raw inputs. Combining them allows solving complex tasks.

2.- Deep learning composes parameterized functions into a deep representation. Gradients can be computed via the chain rule to optimize the loss.

3.- Deep neural networks combine linear transformations, nonlinear activations, and loss functions. Parameters are optimized using stochastic gradient descent.

4.- Weight sharing over time (RNNs) and space (ConvNets) leads to powerful neural network architectures.

5.- RL formalizes the interaction between an agent and environment, with the goal of the agent learning to maximize rewards.

6.- RL may include a policy (agent's behavior), value function (estimate of future rewards), and model (understanding of the environment).

7.- Why RL? It's a general framework for decision-making, relevant wherever optimal actions need to be selected to achieve goals.

8.- Value-based RL estimates the optimal value function (max achievable rewards). Once known, an optimal policy follows by selecting value-maximizing actions.

9.- The optimal value function obeys a recursive Bellman equation due to the iterative nature of the reward maximization process.

10.- In Q-learning, an action-value function Q(s,a) is estimated, representing the value of each action a in each state s.

11.- Deep Q-Networks (DQN) use deep neural networks to represent the Q-function, trained using Q-learning with experience replay for stability.

12.- Improvements to DQN include Double DQN (reducing overestimation bias), Prioritized Experience Replay, and Dueling Networks (separating value/advantage streams).

13.- Distributed DQN variants like Gorila enable faster training by parallelizing across machines. Similar speedups achievable using multiple threads on a CPU.

14.- The Asynchronous Advantage Actor-Critic (A3C) algorithm uses parallel actor-learners, each with its own network, to decorrelate and stabilize learning.

15.- Policy gradient methods directly optimize the policy as a neural network using an objective function and gradient ascent.

16.- The policy gradient theorem expresses the gradient of the RL objective in terms of reward-weighted log policy gradients.

17.- Actor-critic methods learn both a policy (actor) and value function (critic). The critic guides policy updates.

18.- Deterministic policy gradients provide an efficient policy gradient formulation by exploiting action-value function gradients, avoiding integration over actions.

19.- Continuous control with deep RL is possible using actor-critic variants like DDPG, which interleaves learning a Q-function and deterministic policy.

20.- Complex variants using parallelism and RNNs can solve challenging problems like continuous control from pixels (e.g. DPPO).

21.- Strategic games like poker are approachable by combining RL with counterfactual regret minimization, using deep learning for function approximation.

22.- Model-based RL aims to learn an environment model and use it for planning. Key challenges are model inaccuracies and compounding errors.

23.- Go is challenging for AI due to its massive search space and the difficulty of evaluating board positions.

24.- Deep neural networks can be used to represent Go board positions and move probabilities (policy) or position values (value).

25.- Supervised learning on expert games can yield strong initial policy networks. RL via self-play can further improve the policy.

26.- Value networks can be trained on self-play games to provide position value estimates. Data diversity is critical to avoid overfitting.

27.- Combining neural network policies and values with Monte Carlo Tree Search enables highly selective search in Go.

28.- AlphaGo defeated the strongest human Go players by combining deep RL, search, and self-play training.

29.- Deep RL has seen progress and applications beyond just DeepMind. Key focuses are innovation, generality, and real-world impact.

30.- Promising future areas for deep RL include continued algorithmic improvements, healthcare, smartphone assistants, and conversational AI.

Knowledge Vault built byDavid Vivancos 2024