Deep Reinforcement Learning: Neural Networks for Learning Control Laws

welcome back I'm Steve Brenton and today I'm  gonna talk to you a bit more about reinforcement learning so in the last video I introduced the  reinforcement learning architecture how you can learn to interact with a complex environment from  experience and today we're gonna talk about deep reinforcement learning or some of the really  amazing advances in this field that have been enabled by deep neural networks and these advanced  computational architectures so again I am I can Steve on Twitter please do subscribe and do like  do share this if you find it useful and tell me other things that you would like me to talk about  ok so again in the last video we introduced this this agent the environment the agent measures the  environment through this state s it takes some action to interact with the environment given by  a policy so it has a control policy based on how it acts based on what state it's in and it is  optimizing this policy to maximize its future rewards that it gets from the environment and so  mathematically this policy is probabilistic so the agent has some kind of some probabilistic  strategy for interacting with the environment because the environment might be stochastic or  have some randomness to it and there is a value function that tells the agent how valuable being  in a given state is given the policy PI that it is enacting and so today what we're going to do is  we're going to augment this picture by introducing deep neural networks for example to represent the  policy and so here we have now we've replaced our policy with a deep neural network so this PI is  parametrized by theta where theta describes this this neural network and again it Maps the current  state to the best probabilistic action to take in that environment and so the whole name of the  game is to update this policy to maximize future rewards and again we have this discount rate gamma  here that says that rewards in the near future are worth more than rewards in the distant future okay  because again remember these rewards are gonna be relatively sparse and infrequent most of the  time because we're in a semi supervised learning framework where these rewards are only occasional  and so it's difficult to figure out what actions actually gave rise to those rewards this is  gonna be a pretty hard optimization problem to learn this best policy of what actions to take  but you know and actually the whole reinforcement learning paradigm is biologically inspired it's  it's essentially inspired by this observation so there's this this notion called hebbian learning  you may have heard this before and the little rhyme goes neurons that fire together wire  together and basically what that means is that when you have neural activity kind of when things  fire together they will essentially strengthen the wiring and the connections between those neurons  and biological systems and so in these kind of deep reinforcement learning architectures  the idea is that the reward signal that you get occasionally should somehow strengthen  connections that led to a good policy when when the right policy is firing when these neurons  are connected in a way that causes the policy the correct policy and you get a reward you want to  somehow reinforce that architecture and there's lots of ways of doing this you know essentially  through back propagation and so on and so forth so another area where a lot of research is going into  deep learning for reinforcement learning is called cue learning I talked about this in the last video  where this cue or quality function essentially kind of combines the policy and the value and  it tells you jointly how good is a current state given a current action a-okay and so assuming that  I do the best possible thing for all future states and actions so right now if I find myself in state  s an action a I can assign a quality based on the future value that I expect given that state and  given the best possible policy I can cook up and again there are deep queue networks where you  learn this quality function and once you learn this quality function then when you find yourself  in a state s you just look up the best possible a that gives you the highest quality for that state  and this makes a lot of sense this is a lot like how a person would learn how to play chess is they  would kind of simultaneously be building a policy of okay here's how I move in these situations  these are the trades I'm willing to make and you're also building a value function of how you  value different board positions and kind of how you gauge your strength and the strength of your  position based on on the state so it kind of makes sense that this would be an area for really  expanding with deep neural networks because these functions might be very very complex  functions of s and a and that's exactly what neural networks are good at is giving you very  very complex representing very complex functions if you have enough training data so that's what  we're talking about here these still suffer from all the same challenges of regular reinforcement  learning like the credit assignment problem so the fact that I might only get our reward at the very  end of my a chess game makes it very hard to tell which actions actually gave rise to that award  reward and so you're gonna do some of the same things that you would normally do like you might  use reward shaping to give intermediate rewards  based on some expert intuition or guidance and there's lots of other strategies like hindsight  and replay and things like that The basic idea is we're gonna take the same reinforcement  learning architecture and we're going to either replace the policy or the the Q function with  a policy network or a Q network And all of this kind of exploded on the scene because  of this 2015 nature paper "Human level control through deep reinforcement learning" where these authors from DeepMind essentially showed that they could build a reinforcement learner that  could beat human level performance in lots of classic Atari video games. So I'm gonna hit play. I love this one.

. . this is one of the  first ones that got me really excited about this.

So this reinforcement learner is essentially trying to maximize this score by breaking all of these these blocks and after a few hours of training it has an epiphany that only really excellent human players ever reach so it essentially finds an exploit in the game where it realizes that if it tunnels in one side. . .

so It's going to tunnel through here. If it tunnels through one side it can essentially use the physics of the game to break all of these blocks for it. And that's pretty amazing.

So it in a short amount of time learns a really advanced strategy that only a few humans only a small percentage of humans would actually learn eventually so really impressive. . .

this is a beautiful paper by the way,   you should should go read this they talk about how they actually build their networks so they use the  pixels of the screen itself as the input and they use convolutional layers and fully connected  layers eventually deciding what the joystick should do what actions to take. And a lot like other examples of neural networks this was the paper that really brought reinforcement learning  and deep reinforcement learning back to everyone's to the forefront of everyone's mind because this  showed performance that hadn't been attainable before so this is a lot like the image net of  reinforcement learning okay this brought it back into the forefront.