{"componentChunkName":"component---src-templates-blog-post-js","path":"/does-dqn-coverge/","result":{"data":{"site":{"siteMetadata":{"title":"Hi, Blog","author":"Yoni Schirris"}},"markdownRemark":{"id":"fc788dbc-e290-5085-beab-9c271c3a6835","excerpt":"Introduction Reinforcement Learning (RL), the idea of having an agent learn by\ndirectly perceiving its environment and acting to manipulate it, is one\nof the…","html":"

Introduction

\n

Reinforcement Learning (RL), the idea of having an agent learn by\ndirectly perceiving its environment and acting to manipulate it, is one\nof the great branches of Artificial Intelligence. In this blog post you\nwill read about a specific breakthrough by DeepMind: its success in\ncreating a single deep RL architecture that was able to achieve gameplay\nin Atari games comparable to that of humans across almost all the 494949\ngames [1]. They called it DQN, which stands for “Deep\nQ-Network”. When its predecessor was first introduced in 2013 [2],\nit was, according to the authors, the first system that was able to\nlearn control policies “directly from high-dimensional sensory input\nusing reinforcement learning”.
\n
\nIn the coming sections, we will first study Q-learning, a specific RL\nalgorithm, in its tabular form. Then we will learn how Deep Q-learning\nmakes it in principle possible to apply RL to more complex problems and\nsee the so-called deadly triad, which are three properties that are\noften present in these problems and lead to divergence. In the section\nthat follows, we will shortly introduce the two main two tricks that DQN\nutilizes for stability, namely experience replay and a fixed target\nnetwork. Finally, we answer, through experimentation, how these tricks\ncan overcome the divergence problems that partly come from the deadly\ntriad when evaluating the performance in the OpenAI CartPole gym\nenvironment. Concretely, our research questions are:

\n\n

We will answer these questions by investigating the return, i.e.\ncumulative reward, over time achieved by the agent. We expect that\nadding both tricks allows the method to converge on the chosen\nenvironment, while only using either of the tricks won’t. Additionally,\nwe will look at the phenomenon of “soft divergence”, i.e.\nunrealistically large Q-values. We hope to shed some light on the\neffectiveness of these tricks in less complex environments than Atari\ngames, allowing beginners in the field of RL to get an intuition of why\nDQN works so well, and to think of ways how to improve them even\nfurther.
\n
\nThis post assumes some familiarity with RL. If you feel a lack of prior\nknowledge while reading the post, we can recommend reading parts of the\nclassical introduction into Reinforcement Learning\n3 or watching the great (available online)\nlecture by David\nSilver,\none of the creators of the famous\nAlphaZero.

\n

Q-Functions and Q-Learning

\n

In the explanations that follow, there is always the implicit notion of\na Markov-Decision process (MDP): it consists of a finite state space\nS\\mathcal{S}S, a finite action space A\\mathcal{A}A, a reward space\nR⊆RR \\subseteq \\mathbb{R}R⊆R, a discount factor γ\\gammaγ and transition\ndynamics p(s′,r∣s,a)p(s', r \\mid s, a)p(s′,r∣s,a) that defines the probabilities to arrive\nin the next states with a certain reward, given the current state and\naction.
\n
\nThe agent’s goal is to maximize the expected return GGG, which is the\ncumulative discounted reward. This can be modeled by QQQ-functions,\nwhich represents the expected return for each state action pair\nQ(s,a)=E[G∣S=s,A=a]=E[R1+γR2+γ2R3+⋯∣S=s,A=a],Q(s,a) = E\\left[G \\mid S = s, A = a \\right] = E[R_1 + \\gamma R_2 + \\gamma^2R_3 + \\dots \\mid S = s, A = a],Q(s,a)=E[G∣S=s,A=a]=E[R1​+γR2​+γ2R3​+⋯∣S=s,A=a],\nwhere the rewards are stochastically determined by the actions according\nto the agent’s policy (i.e. a distribution over actions\nπ(a∣s)\\pi(a \\mid s)π(a∣s) for each state s∈Ss \\in \\mathcal{S}s∈S) and the environment\ndynamics p(s′,r∣s,a)p(s', r \\mid s, a)p(s′,r∣s,a). In particular, the QQQ-function provides\nthe information on which action leads to the highest value, given a\nstate, when following the given behaviour afterwards.
\n
\nQ-learning is the process of finding the QQQ-function of the optimal\npolicy while following a behaviour policy, e.g. ϵ\\epsilonϵ-greedy. This\ncan be done either in a tabular fashion or using function approximation:

\n

Tabular Q-Learning

\n

In tabular Q-learning, we use the following update rule:

\n

Q(s,a)←Q(s,a)+α⋅(r+γmax⁡a′Q(s′,a′)−Q(s,a)).Q(s, a) \\leftarrow Q(s, a) + \\alpha \\cdot \\left(r + \\gamma \\max_{a'} Q(s', a') - Q(s,a) \\right).Q(s,a)←Q(s,a)+α⋅(r+γmaxa′​Q(s′,a′)−Q(s,a)).

\n

Here, s′s's′ is the next state and α\\alphaα is a hyperparameter, namely\nthe learning rate. Under the assumption that each state-action pair\n(s,a)(s,a)(s,a) is visited infinitely often, this algorithm always converges to\nthe QQQ-function of the optimal policy [2].

\n

Note that with growing dimension of the state space the number of\npossible states grows exponentially. This is also called the curse of\ndimensionality since it makes tabular QQQ-learning and similar tabular\nmethods intractable in high-dimensional spaces. Therefore, we need ways\nto approximate this process.

\n

Q-Learning with Function Approximation

\n

In order to model these more complex scenarios, we model the Q-function\nas a parametrised differentiable function Q(s,a;θ)Q(s, a; \\theta)Q(s,a;θ) that takes\nin the state and action and, depending on the parameter values, outputs\na specific value. In practice, this is often modelled as a function\nQ(s;θ)Q(s; \\theta)Q(s;θ) that only takes in the state and outputs a whole array of\nQQQ-values, one for each action. For this blog post, you can assume that\nQ(s;θ)Q(s; \\theta)Q(s;θ) is a neural network with θ\\thetaθ as its weights. Then,\nwhat is learned are the parameters θ\\thetaθ, of which there are fewer\nthan there are states in the state space. The update rule of\nsemi-gradient Q-learning is then the following:

\nΔθ∝(r+γmax⁡a′Q(s′,a′;θ)−Q(s,a;θ))∇θQ(s,a;θ)(1)\\Delta\\theta \\propto \\left(r + \\gamma \\max_{a'} Q(s', a'; \\theta) - Q(s, a; \\theta)\\right) \\nabla_{\\theta} Q(s, a; \\theta) \\tag{1}Δθ∝(r+γa′max​Q(s′,a′;θ)−Q(s,a;θ))∇θ​Q(s,a;θ)(1)\n

where sss is your last state, aaa is the action you chose in that state,\nrrr is the reward that followed it and s′s's′ is the new state you found\nyourself in. It is a “semi-gradient method” because Equation (1) is not\nthe full gradient of the loss\nL=(r+γmax⁡a′Q(s′,a′;θ)−Q(s,a;θ))2.L = \\left(r + \\gamma \\max_{a'} Q(s', a'; \\theta) - Q(s, a; \\theta)\\right)^2.L=(r+γmaxa′​Q(s′,a′;θ)−Q(s,a;θ))2.\nInstead, the target r+γmax⁡a′Q(s′,a′;θ)r + \\gamma \\max_{a'} Q(s', a'; \\theta)r+γmaxa′​Q(s′,a′;θ) is treated\nas a constant.

\n

The Deadly Triad

\n

The resulting method now has the following properties:

\n
    \n
  1. The method is off-policy. That is, no matter what the behaviour\npolicy is, due to the maximization process in the update-rule, we\nactually directly learn the state-action values of the optimal\npolicy. This allows the agent to explore all possible actions, while\nat the same time learning what the optimal actions are.
    2. \n
  2. It is a bootstrapping method. That means, instead of updating its\nQ-value with an actual outcome of the behaviour, the agent uses an\nestimate Q(s′,a′;θ)Q(s', a'; \\theta)Q(s′,a′;θ) for the update which is itself not\nperfect and might be biased.
    3. \n
  3. The method uses function approximation as explained before.
    4. \n
\n

The first two properties were already present in tabular QQQ-learning,\nbut together with the added function approximation it is called the\n“deadly triad” since it can lead to divergence of the algorithm1.
\nIn the following section, we explain the tricks that [1]\nintroduced to tackle the deadly triad and other convergence problems\nthat may arise in Deep RL. It is largely an open research question how\nlarge the influence of the deadly triad is on divergence problems in\nDeep RL, but specifically the fixed target network seems targeted for\nprecisely that, as described in [4]. Other problems come from the\nnature of RL itself, with its highly non-i.i.d. data.

\n

DQN and its Tricks

\n

In principle, the method is just usual semi-gradient QQQ-learning with\ndeep neural networks as function approximators. But the authors employ\ntwo important tricks:

\n

1. Experience Replay

\n

This describes the process of storing transitions\n(st,at,rt,st+1)(s_t, a_t, r_t, s_{t+1})(st​,at​,rt​,st+1​) in a memory DDD from which minibatches are\nlater sampled during training.

\n

That process has several advantages:

\n\n

2. Fixed Target Network

\n

Recall that the update in semi-gradient Q-learning is given by Equation (1), where\n(s,a,r,s′)(s, a, r, s')(s,a,r,s′) is a saved transition and θ\\thetaθ is the current\nparameter vector.

\n

There can be the following problem with this framework. First, some\nnotation: Denote the target by\nT(θ)=r+γmax⁡a′Q(s′,a′;θ)T(\\theta) = r + \\gamma \\max_{a'} Q(s', a'; \\theta)T(θ)=r+γmaxa′​Q(s′,a′;θ) and imagine it is\nlarger by 111 compared to Q(s,a;θ)Q(s, a; \\theta)Q(s,a;θ). Then we obtain\nΔθ∝∇θQ(s,a;θ)\\Delta \\theta \\propto \\nabla_{\\theta} Q(s, a; \\theta)Δθ∝∇θ​Q(s,a;θ) and move in the\ndirection of increasing Q(s,a;θ)Q(s, a; \\theta)Q(s,a;θ). But QQQ is a function\napproximator, so this is not the only value that changes and similar\nstate-action pairs will change their value as well. Now, s′s's′ and a′a'a′\noccur only one time-step after sss and aaa, so they may be perceived as\nvery similar when processed by the QQQ-network. Consequently,\nQ(s′,a′;θ)Q(s', a'; \\theta)Q(s′,a′;θ) may increase as much as Q(s,a;θ)Q(s, a; \\theta)Q(s,a;θ). The\nresult: T(θnew)T(\\theta^{\\text{new}})T(θnew) may be one larger than\nQ(s,a;θnew)Q(s, a; \\theta^{\\text{new}})Q(s,a;θnew) again!

\n

I guess you already see the problem: assuming we would run this\nindefinitely, the Q-value could explode. This happens due to the\noff-policy nature of the algorithm which does not ensure that\nQ(s′,a′;θ)Q(s', a'; \\theta)Q(s′,a′;θ) gets the relevant feedback later on which would make\nit smaller again.

\n

For this problem, it may help to fix the target network, which was\nproposed in [1] for the first time. In that framework, you\nhave parameters θ\\thetaθ which you update constantly, and parameters\nθ−\\theta^{-}θ− which you hold constant for CCC timesteps, until you update\nthem to θ\\thetaθ and hold them constant again. This is claimed to\nprevent the spiraling that we describe here.

\n

The Experiment

\n

Aim of the Experiments

\n

The experiments we run aim to visualize the impact that the tricks of\nDQN have. Note that in [1], the authors find that each of\nthe two tricks, experience replay and a fixed target network, have their\nown contribution to the success of the agent and that their contribution\namplifies when they are combined. We are interested in whether we can\nconfirm these results in a simpler environment and highlighting the\nindividual contributions of the effects.

\n

The self-written source code of the experiments can be found on\nGitHub2, where we reference to others if we were inspired by their\nwork.

\n

The Environment

\n

The desiderata for our environment were the following:

\n\n

With these criteria in mind, we will see if the tricks proposed in DQN\nmake it solvable.

\n

We qualitatively reviewed the performance of our model trained on all\nclassic control environments of OpenAI Gym3. Performance is measured\nas the (negative) episode duration (sign dependant on the whether the\nagent is tasked to have long or short episodes), and plotted over the\ntraining period. We found that the Acrobot-v1 and MountainCar-v0\nenvironments would require further modifications to DQN to be solvable,\ni.e. even the tricks were not able to lead to convergence. However, the\nCartpole-v1 environment\ndisplayed exactly the required properties: it diverged when using\nstandard semi-gradient Q-learning with function approximation, but\nconverged when adding experience replay and a target network. In this\nenvironment the return is the total number of steps that the pole stays\nupright, with the upper limit of the number of episodes being 500.

\n

Methods

\n

We have trained the network in the following four versions:

\n
    \n
  1. vanilla semi-gradient Q-learning with function approximation.
    2. \n
  2. the same as (1) and including experience replay.
    3. \n
  3. the same as (1) and including a fixed target network which is\nupdated every c∈Cc \\in Cc∈C steps. We experiment with different values\nof CCC in the set {1,50,100,200,500}\\{1, 50, 100, 200, 500\\}{1,50,100,200,500}.
    4. \n
  4. a combination of (2) and (3). This is the full training process\nproposed in [1].
    5. \n
\n

Each variant uses the same simple neural network implemented in PyTorch:

\n
class QNetwork(nn.Module):\n    \n    def __init__(self, num_s, num_a, num_hidden=128):\n        nn.Module.__init__(self)\n        self.l1 = nn.Linear(num_s, num_hidden)\n        self.l2 = nn.Linear(num_hidden, num_a)\n\n    def forward(self, x):\n        x = F.relu(self.l1(x))\n        x = self.l2(x)\n        return x
\n

All variants use the same static set of hyperparameters listed in the table below, where we also explain the\nreasoning behind their choice.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
HyperparameterValueRationale
Number of episodes100100100Methods that proved convergence properties converge after around 50 episodes.
Batch size646464Standard ”exponent of 2” value with no particular meaning.
Discount Factor0.80.80.8Hand-picked during experimentation and implementation process.
Learning Rate10−310^{-3}10−3Hand-picked during experimentation and implementation process.
Memory Capacity10.00010.00010.000Hand-picked during experimentation and implementation process
Replay Start SizeBatch SizeAfter 646464 state transitions, we begin training, since then the memory contains enough transitions to fill a batch.
Seed0−100-100−10Set of different seeds used to ensure reproducibility and to get uncorrelated results.
ϵstart\\epsilon_{\\text{start}}ϵstart​1At the beginning we play a maximal exploration strategy since the policy did not find good behaviour yet.
ϵend\\epsilon_{\\text{end}}ϵend​0.050.050.05We don’t act greedily even during validation, so that the agent cannot overfit on the training experience
ϵsteps\\epsilon_{\\text{steps}}ϵsteps​100010001000Within 100010001000 updates, ϵ\\epsilonϵ is linearly annealed to 0.050.050.05. Hand picked during implementation as it gave fast convergence.
OptimizerADAM (default parameters)State of the art optimizer suitable for most applications.
NN architectureInput dim: number of states, i.e. 4; Hidden dim: 128 followed by ReLU; Output dim: number of actions, i.e. 2Hand-picked based on problem and input complexity.
\n

The Implementation

\n

We have utilized the code obtained during Lab sessions of the course\nReinforcement Learning at University of Amsterdam as the skeleton of our\ncode, and added a flag to enable/disable the experience replay trick,\nand added an integer input to define for how many steps the target\nnetwork would be fixed (1 meaning it is updated each time i.e. it is not\nfixed). All the environments used are from the OpenAI Gym\nlibrary. For the implementation we use several\nopen-source standard machine learning python packages (for details see\nthe source code). The\nexperiments are run on a regular laptop with Intel i5-7200U CPU and 8GB\nof RAM on which a single run of 100 episodes takes about 30 seconds.\nThis means that the results we obtained here can easily be checked on\nyour computer in a few minutes.

\n

Results

\n

For assessing the performance of our agent, we look both at the return\nand at so-called soft divergence. Note that with return, we always mean\nthe undiscounted return, even if the agent discounts, as in our case,\nwith γ=0.8\\gamma = 0.8γ=0.8. That is, we treat γ\\gammaγ not as part of the\nobjective, but as a hyperparameter. We will explain soft divergence in\nthe corresponding subsection.

\n

Return

\n

We will present the results of the experiments with plots in which the\nx-axis represents the number of the episodes and the y-axis represents\nthe return obtained during validation. This means that every 5 episodes\nwe take the trained model, fix ϵ\\epsilonϵ to 0.05 and evaluate the\nperformance of this model, without training it during this process.

\n

Each curve belongs to one variant of the algorithm. For each variant, we\ntrain the agent from scratch 10 times with different seeds and create\nthe line by averaging the returns. The shaded area around each curve is\nthe 95%95\\%95% confidence interval (i.e. it represents μ±2σ\\mu \\pm 2\\sigmaμ±2σ).

\n
\n\n \n \n \"yoni\n \n \n
Fig 1: Comparison between DQN with and without experience\nreplay
\n
\n

Figure 1 presents the comparison between\nthe models with and without experience replay. In both cases, we did not\nuse the fixed target network. The picture clearly displays the\ndifference in quality of the two different methods showing that\nexperience replay is a very important factor of convergence in this\nenvironment. We also ran the method without experience replay for 500\nepisodes, but still did not see any clearly improving values as seen in\nFigure 6 in the appendix.

\n
\n\n \n \n \"no\n \n \n
Fig 2: Comparison between DQN without experience replay for different rates of\nupdate of the target\nnetwork.
\n
\n

In Figure 2 we display the effect of\nthe target network with different update rates on the obtained returns\nwhile turning experience replay off. We observe that the version with an\nupdate rate of 505050 is often in the lead, however, we do not see an\nobvious statistical effect of the target network alone. To keep the\ngraphs uncluttered, we do not show the values of 100 and 500 which\nperform roughly equal to the 50 value.

\n
\n\n \n \n \"experience\n \n \n
Fig 3: Comparison between DQN with experience replay for different rates of\nupdate of the target\nnetwork.\nreplay
\n
\n

In Figure 3 we illustrate the effect of the\ncombination of both tricks. In the case with experience replay, over\nlarge parts of the training process the version with an update rate of\n200200200 steps is leading, although there can not be any statistical\nsignificance inferred in comparison to the update rate of 505050 and 111\n(no target network).

\n

Finally, in Figures 1 and [3], we observe that the average episode\nduration decreases after 75 episodes. To investigate if this is a\nsystematic or purely random effect, we trained the model for 300\nepisodes which can be seen in Figure 7 in the appendix. In both\nsettings the performance seems to decrease slightly after 100 episodes.\nTo be on the safe side, we therefore suggest to periodically take\nsnapshots of the weights and in the end use the model with the highest\naverage episode duration.

\n

Soft Divergence

\n

Now we look at assessing the so-called soft divergence. Soft divergence\nwas for the first time studied in 4 as follows:

\n

Let γ\\gammaγ be our discount rate, in our case 0.80.80.8. Since our rewards\nare all in the interval [−1,1][-1, 1][−1,1] (actually, it is always 111 until the\nagent loses or the episode is over), there is a bound on the maximally\nachievable true Q-values. We can see this by bounding the discounted\nreturn:

\n

G=∑n=0∞γnRn≤∑n=0∞γn=11−γ.G = \\sum_{n = 0}^{\\infty} \\gamma^n R_n \\leq \\sum_{n = 0}^{\\infty} \\gamma^n = \\frac{1}{1 - \\gamma}.G=∑n=0∞​γnRn​≤∑n=0∞​γn=1−γ1​.

\n

In our case, this bound is 11−0.8=5\\frac{1}{1 - 0.8} = 51−0.81​=5. Now, soft\ndivergence is the phenomenon of learned Q-values that exceed this\nbound. This is connected to the deadly triad: off-policy learning with\nbootstrapping and function approximation can lead to a situation in\nwhich the Q-values chase the targets, which themselves can get bigger\nand bigger until they exceed realistic bounds.

\n
\n\n \n \n \"q\n \n \n
Fig 4: Max Q-Values within evaluation episodes with experience\nreplay\nnetwork.\nreplay
\n
\n
\n\n \n \n \"q\n \n \n
Fig 5: Max Q-Values within evaluation episodes with experience\nreplay.\nreplay
\n
\n

In order to assess the presence of soft divergence, we measure the\nmaximal Q-values that appear within evaluation episodes. In Figure 4, you can find this for the case\nwithout experience replay, and in Figure 5 for the case with experience replay.

\n

First of all, we can see that if we use experience replay, the maximal\nQ-value converges precisely to 555. This is expected, since we already\nknow that an agent using experience replay achieves high return, and so\nif the Q-value measures the return correctly, it should output something\nclose to the optimal value, which is 555.

\n

However, the version without experience replay actually shows soft\ndivergence, with Q-values being consistently too large in the end of\ntraining with values ranging between 555 and 888. Additionally, since we\nalready know that the agent performs very bad, we would actually expect\nQ-values lower than 555. Now we get an explanation for why the agent is\nnot able to perform properly: the measurements of the Q-values are\npartly too high during the entire training process, and therefore not\naccurate, and cannot guide the agent to the actions with the best\nbehaviour.

\n

Another interesting phenomenon can be observed by looking at the\ndifferences between the different target network update rates. In both\nplots we see that, in the beginning of the training process, the\nQ-values get considerably larger than 555, ranging up to 252525. That is,\nall versions of the algorithm show soft divergence in the first 50\nepisodes, with a peak after 202020 episodes. One explanation might be the\nmaximization bias which is present in Q-learning. Maximization bias is\na phenomenon which happens when the algorithm overestimates the Q-values\nas a consequence of the max⁡\\maxmax arguments in the update rule. In\npractice, this bias has been studied and can be solved for example with\ndouble Q-learning explained in [5]. However, since we did not\ninclude experiments using double Q-learning, we cannot assess the\nvalidity of this explanation.

\n

A second and complementary explanation for this might be the following:\nin the beginning of training, the learned representations of the\nQ-network are very bad and might not be successful in distinguishing\nbetween different state-action pairs. If that happens, then the increase\nof one Q-value might just increase the Q-values of all other states in\nthe same way, which creates the already explained phenomenon that\nQ-values can become unbounded. This would also explain why the problem\nis less severe when using the fixed target network, and even less severe\nif the updates of the parameters happen less often.

\n

Conclusion

\n

In this blog post we have analyzed what effect the tricks experience\nreplay and a fixed target network have on the convergence of the DQN\nalgorithm. Our experiments have shown that, by itself, the DQN algorithm\nwithout experience replay and fixed target network fails to converge.

\n

Furthermore, we observe that experience replay has a significant impact\non the objective, i.e. return, whereas the fixed target network showed\nalmost no effect on that. This is different from what we expected, as we\ninitially expected that only the addition of both tricks would allow\nthe agent to perform optimally.

\n

For the large effect of the experience replay on the return, we believe\nthis is because in a physical system like the cart pole environment\nsubsequent state observations are deterministically dependent. This is\nthe strongest form of correlation and might lead to too large steps in\none direction. This can be specifically crucial, because a fast move in\none direction of the cart can lead to a change of orientation of the\npole in the other direction and to the end of the episode.

\n

The small effect of the fixed target network on the return is\ncontradicted, however, by analyzing its effect on soft divergence. We\nsaw that the fixed target network actually plays an essential role in\nmitigating the initially very high Q-values seen in all variants of the\nalgorithm. This does not itself change the return of the agent after\nconvergence, since in all cases, the algorithm is able to correct for\nthe mistake of vastly too large initial Q-values. Still, these results\nhint that in more complex environments, we would also see an influence\nof the fixed target network on the return achieved by agents as updating\nthe target network more rarely can help fight the soft divergence.\\

\n

We conclude that generally DQN contains strong and theoretically sound\nideas, and we saw that both tricks can, in different ways, help avoiding\ndivergence. However, to train it successfully and to use the utility of\nthe tricks fruitfully, still requires practice and effortful tuning.

\n

Appendix

\n

DQN without Experience Replay after 500 Episodes

\n
\n\n \n \n \"Results\"\n \n \n
This result shows that even after $500$ episodes, the version without\nexperience replay fails to converge in the cartpole environment. There\nis some slight upward trend that is nevertheless not very\nstable.
\n
\n

DQN with Experience Replay after 300 Episodes

\n
\n\n \n \n \"Results\"\n \n \n
This result shows that DQN with experience replay and with a fixed\ntarget network for 50 steps tends to reduce performance when training\nlonger. Note that the shaded area here is the 95% confidence interval\nover 5 runs from scratch, less than the 10 runs as in the other\nexperiments.
\n
\n

Referenes

\n

[1] Human-level control through deep reinforcement learning
\n[2] Playing Atari with Deep Reinforcement Learning]
\n[3] Sutton & Barto: Reinforcement Learning Book
\n[4] Deep Reinforcement Learning and the Deadly Triad
\n[5] Double Q-learning

\n
\n
\n
    \n
  1. \n

    It is out of the scope of this blog post to go into the details as to why the deadly triad can lead to divergence. We recommend the interested reader to read chapter 111111 of [@sutton2018reinforcement].

    \n↩\n
    2. \n
  2. \n

    https://github.com/YoniSchirris/rl-lab3

    \n↩\n
    3. \n
  3. \n

    https://gym.openai.com/

    \n↩\n
    4. \n
\n
","frontmatter":{"title":"Diving into the atari-game playing algorithm - Deep Q-Networks","date":"December 01, 2019","description":"A small collaborative research about the convergence properties of the Deep Q Network. Written by Leon Lang, Igor Pejic, and Simon Passenheim, and Yoni Schirris."}}},"pageContext":{"isCreatedByStatefulCreatePages":false,"slug":"/does-dqn-coverge/","previous":{"fields":{"slug":"/ideas-for-future-posts/"},"frontmatter":{"title":"Ideas for future posts"}},"next":null}}}