The Game is On! The DQN Trilogy -3: Duelling Double Deep Q-Network (D3QN)

Following up from last article. In the last blog I discussed about Double Deep Q Network. In this blog, I am discussing the more advanced version! Duelling Double Deep Q Network!

Index

• Background
• Advantage
• Architecture
• Q Network Class
• Agent

Background

Duelling Double Deep Q Network is the last network in the series of DQNs . Also developed by Researchers of DeepMind. The paper is here.

<a id=”advantage></a>

Advantage of an action, $A(s,a)$

From previous articles, we have come to know the term Q value of a state $s_t$ and action $a$ , $Q(s_t,a)$ . But there is a problem with this notion. Q-value means , the Value of an action in a certain state. We cannot distinguish which one causes higher/lower value. The state or the action or both? What if the action has no value at all ?! For that reason, we have two other concepts. Value of a state $V(s)$ and Advantage of an action $A(s,a)$

V(s_t)

In simple language , Value of a state is the Expected return of total rewards when the agent is in a certain state

\begin{equation} V(s_t) = \gamma R_{t+1}+\gamma^2 R_{t+2} + \gamma^3 R_{t+3} + …………….\gamma^{t+k-1} R_{t+k} \end{equation}

or, \begin{equation} V(s_t) = \sum_{i=t+1}^{K} \gamma**{k-1} R_k \end{equation}

Advantage A(s_t,a)

Advantage of an action means, the value of an action only . i.e. we get the value of taking certain action in a certain state without the Value of the state.

\begin{equation} A(s_t,a) = Q(s_t,a)-V(s_t) \end{equation}

or we get this final equation \begin{equation} Q(s_t,a) = A(s_t,a)+V(s_t) \end{equation}

Why is it necessary?

In many environments, in certain states, no action has any value. The paper mentions the example of the Atari Game Enduro

PS: I am not using Duelling Deep Q Network for Enduro in the blog.

Architecture

The sample architecture from the paper

First one is a simple DQN and the second one is D3QN

So basically our network predicts advantage and value from the given state and adds them right? Well , Not exactly!

Twist!

According to the paper,If you use this equation, the learning will be poor! Why ? The problem is , our output is $Q(s_t,a)$. When we backpropagate the loss, we will not be sure should the loss focus more on the Advantage $A(s_t,a)$ or value $V(s_t)$ ! That is a problem! To solve this problem , the paper suggested a trick!

Solution

In optimal policy $Q(s_t,a) = V(s_t)$ . so in that case Advantage $A(s_t,a)$ must be zero! . SO , as our policy is to choose the action with maximum Q value - also we can say action with maximum advantage- if we subtract maximum Advantage value from Q values in every step of action, we will definitely get a state where Advantage is zero. So, the loss will backpropagate through correct activation!

Correct equation should be,

\begin{equation} Q(s_t,a) = V(s_t)+A(s_t,a)-max(A(s_t,a)) \end{equation}

SO, lets play the game!

1. Import the Necessary Packages

import gym
import random
import torch
import numpy as np
from collections import namedtuple,deque
import matplotlib.pyplot as plt
%matplotlib inline
from torch import nn
import torch
import torch.nn.functional as F
import torch.optim as optim

2. Instantiate the Environment and Agent

Initialize the environment in the code cell below.

env = gym.make('LunarLander-v2')
env.seed(0)
print('State shape: ', env.observation_space.shape)
print('Number of actions: ', env.action_space.n)

State shape:  (8,)
Number of actions:  4


/floyd/home/gym/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))

state_shape= env.observation_space.shape[0]
action_size= env.action_space.n

import torch
import torch.nn as nn
import torch.nn.functional as F

<a id=”qnetwork></a>

QNetwork Class

class DuellingQNetwork(nn.Module):
    """Actor (Policy) Model."""

    def __init__(self, state_size, action_size, seed, fc1_units=64, fc2_units=64,fc3_units=64):
        """Initialize parameters and build model.
        Params
        ======
            state_size (int): Dimension of each state
            action_size (int): Dimension of each action
            seed (int): Random seed
            fc1_units (int): Number of nodes in first hidden layer
            fc2_units (int): Number of nodes in second hidden layer
        """
        super(DuellingQNetwork, self).__init__()
        self.seed = torch.manual_seed(seed)
        self.fc1 = nn.Linear(state_size, fc1_units)
        self.fc2 = nn.Linear(fc1_units, fc2_units)
        self.fc3 = nn.Linear(fc2_units,fc3_units)
        self.fc4 = nn.Linear(fc3_units,64)
        self.valuefc=nn.Linear(64,64)
        self.valuefc2=nn.Linear(64,32)
        self.advfc=nn.Linear(64,64)
        self.advfc2=nn.Linear(64,32)
        self.value=nn.Linear(32,1)
        self.advantage = nn.Linear(32, action_size)
        

    def forward(self, state):
        """Build a network that maps state -> action values."""
        x = F.relu(self.fc1(state))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        value_x=F.relu(self.valuefc(x))
        value_x=F.relu(self.valuefc2(value_x))
        adv_x=F.relu(self.advfc(x))
        adv_x=F.relu(self.advfc2(adv_x))
        value=self.value(value_x)
        advantage=self.advantage(adv_x)
        Q =value+advantage-torch.max(advantage)
        
        return value,advantage,Q

Agent

BUFFER_SIZE = int(1e5)  # replay buffer size
BATCH_SIZE = 64         # minibatch size
GAMMA = 0.99            # discount factor
TAU = 0.01              # for soft update of target parameters
LR = 0.001               # learning rate 
UPDATE_EVERY = 4        # how often to update the network

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

class Agent():
    """Interacts with and learns from the environment."""

    def __init__(self, state_size, action_size, seed):
        """Initialize an Agent object.
        
        Params
        ======
            state_size (int): dimension of each state
            action_size (int): dimension of each action
            seed (int): random seed
        """
        self.state_size = state_size
        self.action_size = action_size
        self.seed = random.seed(seed)

        # Q-Network
        self.qnetwork_local = DuellingQNetwork(state_size, action_size, seed).to(device)
        self.qnetwork_target = DuellingQNetwork(state_size, action_size, seed).to(device)
        self.optimizer = optim.Adam(self.qnetwork_local.parameters(), lr=LR)

        # Replay memory
        self.memory = ReplayBuffer(action_size, BUFFER_SIZE, BATCH_SIZE, seed)
        # Initialize time step (for updating every UPDATE_EVERY steps)
        self.t_step = 0
        self.criterion=nn.MSELoss()
    
    def step(self, state, action, reward, next_state, done):
        # Save experience in replay memory
        self.memory.add(state, action, reward, next_state, done)
        
        # Learn every UPDATE_EVERY time steps.
        self.t_step = (self.t_step + 1) % UPDATE_EVERY
        if self.t_step == 0:
            # If enough samples are available in memory, get random subset and learn
            if len(self.memory) > BATCH_SIZE:
                experiences = self.memory.sample()
                self.learn(experiences, GAMMA)

    def act(self, state, eps=0.):
        """Returns actions for given state as per current policy.
        
        Params
        ======
            state (array_like): current state
            eps (float): epsilon, for epsilon-greedy action selection
        """
        state = torch.from_numpy(state).float().unsqueeze(0).to(device)
        self.qnetwork_local.eval()
        with torch.no_grad():
            value,advantage,Q = self.qnetwork_local(state)
        self.qnetwork_local.train()

        # Epsilon-greedy action selection
        if random.random() > eps:
            return np.argmax(Q.cpu().data.numpy())
        else:
            return random.choice(np.arange(self.action_size))

    def learn(self, experiences, gamma):
        """Update value parameters using given batch of experience tuples.

        Params
        ======
            experiences (Tuple[torch.Tensor]): tuple of (s, a, r, s', done) tuples 
            gamma (float): discount factor
        """
        states, actions, rewards, next_states, dones = experiences
        target_values,target_advantages,target_Qs=self.qnetwork_target(next_states)
        _,_,Qs=self.qnetwork_local(states)
        _,_,next_state_Qs=self.qnetwork_local(next_states)
        predicted_actions=torch.argmax(next_state_Qs,1)
        predicted_actions=predicted_actions.reshape(-1,1)
        y=rewards+gamma*target_Qs.gather(1,predicted_actions)*(1-dones)
        #print(gamma*torch.max(self.qnetwork_target(next_states),1)[0]*(1-dones))
        loss=F.mse_loss(y,Qs.gather(1,actions))
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        
        

        # ------------------- update target network ------------------- #
        self.soft_update(self.qnetwork_local, self.qnetwork_target, TAU)                     

    def soft_update(self, local_model, target_model, tau):
        """Soft update model parameters.
        θ_target = τ*θ_local + (1 - τ)*θ_target

        Params
        ======
            local_model (PyTorch model): weights will be copied from
            target_model (PyTorch model): weights will be copied to
            tau (float): interpolation parameter 
        """
        for target_param, local_param in zip(target_model.parameters(), local_model.parameters()):
            target_param.data.copy_(tau*local_param.data + (1.0-tau)*target_param.data)

Replay Buffer

class ReplayBuffer:
    """Fixed-size buffer to store experience tuples."""

    def __init__(self, action_size, buffer_size, batch_size, seed):
        """Initialize a ReplayBuffer object.

        Params
        ======
            action_size (int): dimension of each action
            buffer_size (int): maximum size of buffer
            batch_size (int): size of each training batch
            seed (int): random seed
        """
        self.action_size = action_size
        self.memory = deque(maxlen=buffer_size)  
        self.batch_size = batch_size
        self.experience = namedtuple("Experience", field_names=["state", "action", "reward", "next_state", "done"])
        self.seed = random.seed(seed)
    
    def add(self, state, action, reward, next_state, done):
        """Add a new experience to memory."""
        e = self.experience(state, action, reward, next_state, done)
        self.memory.append(e)
    
    def sample(self):
        """Randomly sample a batch of experiences from memory."""
        experiences = random.sample(self.memory, k=self.batch_size)

        states = torch.from_numpy(np.vstack([e.state for e in experiences if e is not None])).float().to(device)
        actions = torch.from_numpy(np.vstack([e.action for e in experiences if e is not None])).long().to(device)
        rewards = torch.from_numpy(np.vstack([e.reward for e in experiences if e is not None])).float().to(device)
        next_states = torch.from_numpy(np.vstack([e.next_state for e in experiences if e is not None])).float().to(device)
        dones = torch.from_numpy(np.vstack([e.done for e in experiences if e is not None]).astype(np.uint8)).float().to(device)
  
        return (states, actions, rewards, next_states, dones)

    def __len__(self):
        """Return the current size of internal memory."""
        return len(self.memory)

agent = Agent(state_size=state_shape, action_size=action_size, seed=0)

3. Train the Agent with DQN

Run the code cell below to train the agent from scratch. You are welcome to amend the supplied values of the parameters in the function, to try to see if you can get better performance!

n_episodes=1000
max_t=1000
eps_start=1.0
eps_end=0.1
eps_decay=0.995

def train_agent(agent,env,target_folder,eps_start=eps_start,eps_decay=eps_decay,eps_end=eps_end,gamma=GAMMA,n_episodes=n_episodes,max_t=max_t):
    """Deep Q-Learning.

    Params
    ======
        n_episodes (int): maximum number of training episodes
        max_t (int): maximum number of timesteps per episode
        eps_start (float): starting value of epsilon, for epsilon-greedy action selection
        eps_end (float): minimum value of epsilon
        eps_decay (float): multiplicative factor (per episode) for decreasing epsilon
    """
    scores = []                        # list containing scores from each episode
    scores_window = deque(maxlen=100)  # last 100 scores
    eps = eps_start                    # initialize epsilon
    for i_episode in range(1, n_episodes+1):

        state = env.reset()
        score = 0
        for t in range(max_t):
            action = agent.act(state, eps)
            next_state, reward, done, _ = env.step(action)

            agent.step(state, action, reward, next_state,done)
            state = next_state
            score += reward
            if done:
                break 
        scores_window.append(score)       # save most recent score
        scores.append(score)              # save most recent score
        eps = max(eps_end, eps_decay*eps) # decrease epsilon
        print('\rEpisode {}\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end="")
        if i_episode%10==0:
            torch.save(agent.qnetwork_local.state_dict(),"{}/dqn_agent{}.pkl".format(target_folder,i_episode))
        if i_episode % 100 == 0:
            print('\rEpisode {}\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))
        if np.mean(scores_window)>=200.0:
            print('\nEnvironment solved in {:d} episodes!\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))
    return scores

scores_dddqn=train_agent(agent,env,target_folder="duelling_double_dqn/")

Episode 100	Average Score: -172.07
Episode 200	Average Score: -64.69
Episode 300	Average Score: 46.16
Episode 400	Average Score: 115.03
Episode 500	Average Score: 175.13
Episode 520	Average Score: 200.28
Environment solved in 420 episodes!	Average Score: 200.28
Episode 521	Average Score: 200.58
Environment solved in 421 episodes!	Average Score: 200.58
Episode 522	Average Score: 200.49
Environment solved in 422 episodes!	Average Score: 200.49
Episode 567	Average Score: 200.08
Environment solved in 467 episodes!	Average Score: 200.08
Episode 568	Average Score: 200.45
Environment solved in 468 episodes!	Average Score: 200.45
Episode 569	Average Score: 203.26
Environment solved in 469 episodes!	Average Score: 203.26
Episode 570	Average Score: 203.18
Environment solved in 470 episodes!	Average Score: 203.18
Episode 571	Average Score: 201.89
Environment solved in 471 episodes!	Average Score: 201.89
Episode 572	Average Score: 201.77
Environment solved in 472 episodes!	Average Score: 201.77
Episode 573	Average Score: 206.00
Environment solved in 473 episodes!	Average Score: 206.00
Episode 574	Average Score: 206.68
Environment solved in 474 episodes!	Average Score: 206.68
Episode 575	Average Score: 206.41
Environment solved in 475 episodes!	Average Score: 206.41
Episode 576	Average Score: 206.56
Environment solved in 476 episodes!	Average Score: 206.56
Episode 577	Average Score: 207.28
Environment solved in 477 episodes!	Average Score: 207.28
Episode 578	Average Score: 207.80
Environment solved in 478 episodes!	Average Score: 207.80
Episode 579	Average Score: 209.33
Environment solved in 479 episodes!	Average Score: 209.33
Episode 580	Average Score: 211.68
Environment solved in 480 episodes!	Average Score: 211.68
Episode 581	Average Score: 211.87
Environment solved in 481 episodes!	Average Score: 211.87
Episode 582	Average Score: 212.28
Environment solved in 482 episodes!	Average Score: 212.28
Episode 583	Average Score: 209.17
Environment solved in 483 episodes!	Average Score: 209.17
Episode 584	Average Score: 209.53
Environment solved in 484 episodes!	Average Score: 209.53
Episode 585	Average Score: 213.97
Environment solved in 485 episodes!	Average Score: 213.97
Episode 586	Average Score: 216.57
Environment solved in 486 episodes!	Average Score: 216.57
Episode 587	Average Score: 216.80
Environment solved in 487 episodes!	Average Score: 216.80
Episode 588	Average Score: 216.95
Environment solved in 488 episodes!	Average Score: 216.95
Episode 589	Average Score: 217.12
Environment solved in 489 episodes!	Average Score: 217.12
Episode 590	Average Score: 218.13
Environment solved in 490 episodes!	Average Score: 218.13
Episode 591	Average Score: 217.34
Environment solved in 491 episodes!	Average Score: 217.34
Episode 592	Average Score: 217.30
Environment solved in 492 episodes!	Average Score: 217.30
Episode 593	Average Score: 221.42
Environment solved in 493 episodes!	Average Score: 221.42
Episode 594	Average Score: 222.49
Environment solved in 494 episodes!	Average Score: 222.49
Episode 595	Average Score: 223.84
Environment solved in 495 episodes!	Average Score: 223.84
Episode 596	Average Score: 224.06
Environment solved in 496 episodes!	Average Score: 224.06
Episode 597	Average Score: 223.74
Environment solved in 497 episodes!	Average Score: 223.74
Episode 598	Average Score: 224.08
Environment solved in 498 episodes!	Average Score: 224.08
Episode 599	Average Score: 224.04
Environment solved in 499 episodes!	Average Score: 224.04
Episode 600	Average Score: 224.17

Environment solved in 500 episodes!	Average Score: 224.17
Episode 601	Average Score: 223.78
Environment solved in 501 episodes!	Average Score: 223.78
Episode 602	Average Score: 223.95
Environment solved in 502 episodes!	Average Score: 223.95
Episode 603	Average Score: 226.31
Environment solved in 503 episodes!	Average Score: 226.31
Episode 604	Average Score: 226.20
Environment solved in 504 episodes!	Average Score: 226.20
Episode 605	Average Score: 225.95
Environment solved in 505 episodes!	Average Score: 225.95
Episode 606	Average Score: 226.22
Environment solved in 506 episodes!	Average Score: 226.22
Episode 607	Average Score: 228.01
Environment solved in 507 episodes!	Average Score: 228.01
Episode 608	Average Score: 228.27
Environment solved in 508 episodes!	Average Score: 228.27
Episode 609	Average Score: 231.08
Environment solved in 509 episodes!	Average Score: 231.08
Episode 610	Average Score: 231.42
Environment solved in 510 episodes!	Average Score: 231.42
Episode 611	Average Score: 231.20
Environment solved in 511 episodes!	Average Score: 231.20
Episode 612	Average Score: 230.49
Environment solved in 512 episodes!	Average Score: 230.49
Episode 613	Average Score: 230.33
Environment solved in 513 episodes!	Average Score: 230.33
Episode 614	Average Score: 231.48
Environment solved in 514 episodes!	Average Score: 231.48
Episode 615	Average Score: 231.57
Environment solved in 515 episodes!	Average Score: 231.57
Episode 616	Average Score: 231.62
Environment solved in 516 episodes!	Average Score: 231.62
Episode 617	Average Score: 234.70
Environment solved in 517 episodes!	Average Score: 234.70
Episode 618	Average Score: 234.39
Environment solved in 518 episodes!	Average Score: 234.39
Episode 619	Average Score: 233.89
Environment solved in 519 episodes!	Average Score: 233.89
Episode 620	Average Score: 234.60
Environment solved in 520 episodes!	Average Score: 234.60
Episode 621	Average Score: 234.29
Environment solved in 521 episodes!	Average Score: 234.29
Episode 622	Average Score: 233.95
Environment solved in 522 episodes!	Average Score: 233.95
Episode 623	Average Score: 236.12
Environment solved in 523 episodes!	Average Score: 236.12
Episode 624	Average Score: 236.33
Environment solved in 524 episodes!	Average Score: 236.33
Episode 625	Average Score: 239.28
Environment solved in 525 episodes!	Average Score: 239.28
Episode 626	Average Score: 239.95
Environment solved in 526 episodes!	Average Score: 239.95
Episode 627	Average Score: 242.02
Environment solved in 527 episodes!	Average Score: 242.02
Episode 628	Average Score: 243.08
Environment solved in 528 episodes!	Average Score: 243.08
Episode 629	Average Score: 240.80
Environment solved in 529 episodes!	Average Score: 240.80
Episode 630	Average Score: 241.09
Environment solved in 530 episodes!	Average Score: 241.09
Episode 631	Average Score: 241.18
Environment solved in 531 episodes!	Average Score: 241.18
Episode 632	Average Score: 240.90
Environment solved in 532 episodes!	Average Score: 240.90
Episode 633	Average Score: 240.79
Environment solved in 533 episodes!	Average Score: 240.79
Episode 634	Average Score: 240.89
Environment solved in 534 episodes!	Average Score: 240.89
Episode 635	Average Score: 241.22
Environment solved in 535 episodes!	Average Score: 241.22
Episode 636	Average Score: 242.01
Environment solved in 536 episodes!	Average Score: 242.01
Episode 637	Average Score: 241.40
Environment solved in 537 episodes!	Average Score: 241.40
Episode 638	Average Score: 240.75
Environment solved in 538 episodes!	Average Score: 240.75
Episode 639	Average Score: 240.64
Environment solved in 539 episodes!	Average Score: 240.64
Episode 640	Average Score: 240.39
Environment solved in 540 episodes!	Average Score: 240.39
Episode 641	Average Score: 237.64
Environment solved in 541 episodes!	Average Score: 237.64
Episode 642	Average Score: 237.60
Environment solved in 542 episodes!	Average Score: 237.60
Episode 643	Average Score: 237.67
Environment solved in 543 episodes!	Average Score: 237.67
Episode 644	Average Score: 239.08
Environment solved in 544 episodes!	Average Score: 239.08
Episode 645	Average Score: 236.44
Environment solved in 545 episodes!	Average Score: 236.44
Episode 646	Average Score: 236.00
Environment solved in 546 episodes!	Average Score: 236.00
Episode 647	Average Score: 236.27
Environment solved in 547 episodes!	Average Score: 236.27
Episode 648	Average Score: 236.18
Environment solved in 548 episodes!	Average Score: 236.18
Episode 649	Average Score: 238.55
Environment solved in 549 episodes!	Average Score: 238.55
Episode 650	Average Score: 238.29
Environment solved in 550 episodes!	Average Score: 238.29
Episode 651	Average Score: 235.76
Environment solved in 551 episodes!	Average Score: 235.76
Episode 652	Average Score: 235.67
Environment solved in 552 episodes!	Average Score: 235.67
Episode 653	Average Score: 238.96
Environment solved in 553 episodes!	Average Score: 238.96
Episode 654	Average Score: 239.21
Environment solved in 554 episodes!	Average Score: 239.21
Episode 655	Average Score: 238.88
Environment solved in 555 episodes!	Average Score: 238.88
Episode 656	Average Score: 238.54
Environment solved in 556 episodes!	Average Score: 238.54
Episode 657	Average Score: 237.99
Environment solved in 557 episodes!	Average Score: 237.99
Episode 658	Average Score: 238.10
Environment solved in 558 episodes!	Average Score: 238.10
Episode 659	Average Score: 238.27
Environment solved in 559 episodes!	Average Score: 238.27
Episode 660	Average Score: 238.56
Environment solved in 560 episodes!	Average Score: 238.56
Episode 661	Average Score: 238.85
Environment solved in 561 episodes!	Average Score: 238.85
Episode 662	Average Score: 238.97
Environment solved in 562 episodes!	Average Score: 238.97
Episode 663	Average Score: 239.06
Environment solved in 563 episodes!	Average Score: 239.06
Episode 664	Average Score: 239.36
Environment solved in 564 episodes!	Average Score: 239.36
Episode 665	Average Score: 239.55
Environment solved in 565 episodes!	Average Score: 239.55
Episode 666	Average Score: 239.80
Environment solved in 566 episodes!	Average Score: 239.80
Episode 667	Average Score: 239.53
Environment solved in 567 episodes!	Average Score: 239.53
Episode 668	Average Score: 239.77
Environment solved in 568 episodes!	Average Score: 239.77
Episode 669	Average Score: 239.63
Environment solved in 569 episodes!	Average Score: 239.63
Episode 670	Average Score: 239.79
Environment solved in 570 episodes!	Average Score: 239.79
Episode 671	Average Score: 241.10
Environment solved in 571 episodes!	Average Score: 241.10
Episode 672	Average Score: 241.21
Environment solved in 572 episodes!	Average Score: 241.21
Episode 673	Average Score: 241.16
Environment solved in 573 episodes!	Average Score: 241.16
Episode 674	Average Score: 240.86
Environment solved in 574 episodes!	Average Score: 240.86
Episode 675	Average Score: 241.29
Environment solved in 575 episodes!	Average Score: 241.29
Episode 676	Average Score: 241.09
Environment solved in 576 episodes!	Average Score: 241.09
Episode 677	Average Score: 239.25
Environment solved in 577 episodes!	Average Score: 239.25
Episode 678	Average Score: 239.44
Environment solved in 578 episodes!	Average Score: 239.44
Episode 679	Average Score: 239.19
Environment solved in 579 episodes!	Average Score: 239.19
Episode 680	Average Score: 236.52
Environment solved in 580 episodes!	Average Score: 236.52
Episode 681	Average Score: 236.25
Environment solved in 581 episodes!	Average Score: 236.25
Episode 682	Average Score: 236.31
Environment solved in 582 episodes!	Average Score: 236.31
Episode 683	Average Score: 239.34
Environment solved in 583 episodes!	Average Score: 239.34
Episode 684	Average Score: 236.87
Environment solved in 584 episodes!	Average Score: 236.87
Episode 685	Average Score: 234.38
Environment solved in 585 episodes!	Average Score: 234.38
Episode 686	Average Score: 234.38
Environment solved in 586 episodes!	Average Score: 234.38
Episode 687	Average Score: 234.21
Environment solved in 587 episodes!	Average Score: 234.21
Episode 688	Average Score: 233.90
Environment solved in 588 episodes!	Average Score: 233.90
Episode 689	Average Score: 233.56
Environment solved in 589 episodes!	Average Score: 233.56
Episode 690	Average Score: 231.13
Environment solved in 590 episodes!	Average Score: 231.13
Episode 691	Average Score: 231.97
Environment solved in 591 episodes!	Average Score: 231.97
Episode 692	Average Score: 231.95
Environment solved in 592 episodes!	Average Score: 231.95
Episode 693	Average Score: 231.81
Environment solved in 593 episodes!	Average Score: 231.81
Episode 694	Average Score: 231.83
Environment solved in 594 episodes!	Average Score: 231.83
Episode 695	Average Score: 232.09
Environment solved in 595 episodes!	Average Score: 232.09
Episode 696	Average Score: 232.27
Environment solved in 596 episodes!	Average Score: 232.27
Episode 697	Average Score: 232.71
Environment solved in 597 episodes!	Average Score: 232.71
Episode 698	Average Score: 232.56
Environment solved in 598 episodes!	Average Score: 232.56
Episode 699	Average Score: 232.65
Environment solved in 599 episodes!	Average Score: 232.65
Episode 700	Average Score: 232.69

Environment solved in 600 episodes!	Average Score: 232.69
Episode 701	Average Score: 233.23
Environment solved in 601 episodes!	Average Score: 233.23
Episode 702	Average Score: 233.70
Environment solved in 602 episodes!	Average Score: 233.70
Episode 703	Average Score: 233.73
Environment solved in 603 episodes!	Average Score: 233.73
Episode 704	Average Score: 233.89
Environment solved in 604 episodes!	Average Score: 233.89
Episode 705	Average Score: 234.22
Environment solved in 605 episodes!	Average Score: 234.22
Episode 706	Average Score: 234.27
Environment solved in 606 episodes!	Average Score: 234.27
Episode 707	Average Score: 234.96
Environment solved in 607 episodes!	Average Score: 234.96
Episode 708	Average Score: 234.88
Environment solved in 608 episodes!	Average Score: 234.88
Episode 709	Average Score: 234.59
Environment solved in 609 episodes!	Average Score: 234.59
Episode 710	Average Score: 234.50
Environment solved in 610 episodes!	Average Score: 234.50
Episode 711	Average Score: 234.19
Environment solved in 611 episodes!	Average Score: 234.19
Episode 712	Average Score: 234.50
Environment solved in 612 episodes!	Average Score: 234.50
Episode 713	Average Score: 232.46
Environment solved in 613 episodes!	Average Score: 232.46
Episode 714	Average Score: 232.41
Environment solved in 614 episodes!	Average Score: 232.41
Episode 715	Average Score: 231.57
Environment solved in 615 episodes!	Average Score: 231.57
Episode 716	Average Score: 232.32
Environment solved in 616 episodes!	Average Score: 232.32
Episode 717	Average Score: 231.98
Environment solved in 617 episodes!	Average Score: 231.98
Episode 718	Average Score: 232.04
Environment solved in 618 episodes!	Average Score: 232.04
Episode 719	Average Score: 233.24
Environment solved in 619 episodes!	Average Score: 233.24
Episode 720	Average Score: 232.83
Environment solved in 620 episodes!	Average Score: 232.83
Episode 721	Average Score: 232.71
Environment solved in 621 episodes!	Average Score: 232.71
Episode 722	Average Score: 232.98
Environment solved in 622 episodes!	Average Score: 232.98
Episode 723	Average Score: 233.16
Environment solved in 623 episodes!	Average Score: 233.16
Episode 724	Average Score: 232.73
Environment solved in 624 episodes!	Average Score: 232.73
Episode 725	Average Score: 232.60
Environment solved in 625 episodes!	Average Score: 232.60
Episode 726	Average Score: 233.09
Environment solved in 626 episodes!	Average Score: 233.09
Episode 727	Average Score: 232.80
Environment solved in 627 episodes!	Average Score: 232.80
Episode 728	Average Score: 232.96
Environment solved in 628 episodes!	Average Score: 232.96
Episode 729	Average Score: 235.11
Environment solved in 629 episodes!	Average Score: 235.11
Episode 730	Average Score: 232.72
Environment solved in 630 episodes!	Average Score: 232.72
Episode 731	Average Score: 232.75
Environment solved in 631 episodes!	Average Score: 232.75
Episode 732	Average Score: 233.26
Environment solved in 632 episodes!	Average Score: 233.26
Episode 733	Average Score: 233.79
Environment solved in 633 episodes!	Average Score: 233.79
Episode 734	Average Score: 231.59
Environment solved in 634 episodes!	Average Score: 231.59
Episode 735	Average Score: 231.52
Environment solved in 635 episodes!	Average Score: 231.52
Episode 736	Average Score: 232.21
Environment solved in 636 episodes!	Average Score: 232.21
Episode 737	Average Score: 232.52
Environment solved in 637 episodes!	Average Score: 232.52
Episode 738	Average Score: 232.86
Environment solved in 638 episodes!	Average Score: 232.86
Episode 739	Average Score: 233.27
Environment solved in 639 episodes!	Average Score: 233.27
Episode 740	Average Score: 233.72
Environment solved in 640 episodes!	Average Score: 233.72
Episode 741	Average Score: 236.56
Environment solved in 641 episodes!	Average Score: 236.56
Episode 742	Average Score: 236.40
Environment solved in 642 episodes!	Average Score: 236.40
Episode 743	Average Score: 236.16
Environment solved in 643 episodes!	Average Score: 236.16
Episode 744	Average Score: 236.61
Environment solved in 644 episodes!	Average Score: 236.61
Episode 745	Average Score: 239.24
Environment solved in 645 episodes!	Average Score: 239.24
Episode 746	Average Score: 239.61
Environment solved in 646 episodes!	Average Score: 239.61
Episode 747	Average Score: 237.94
Environment solved in 647 episodes!	Average Score: 237.94
Episode 748	Average Score: 238.31
Environment solved in 648 episodes!	Average Score: 238.31
Episode 749	Average Score: 238.58
Environment solved in 649 episodes!	Average Score: 238.58
Episode 750	Average Score: 238.74
Environment solved in 650 episodes!	Average Score: 238.74
Episode 751	Average Score: 241.29
Environment solved in 651 episodes!	Average Score: 241.29
Episode 752	Average Score: 241.38
Environment solved in 652 episodes!	Average Score: 241.38
Episode 753	Average Score: 240.83
Environment solved in 653 episodes!	Average Score: 240.83
Episode 754	Average Score: 240.81
Environment solved in 654 episodes!	Average Score: 240.81
Episode 755	Average Score: 241.14
Environment solved in 655 episodes!	Average Score: 241.14
Episode 756	Average Score: 240.89
Environment solved in 656 episodes!	Average Score: 240.89
Episode 757	Average Score: 239.01
Environment solved in 657 episodes!	Average Score: 239.01
Episode 758	Average Score: 235.84
Environment solved in 658 episodes!	Average Score: 235.84
Episode 759	Average Score: 233.78
Environment solved in 659 episodes!	Average Score: 233.78
Episode 760	Average Score: 233.86
Environment solved in 660 episodes!	Average Score: 233.86
Episode 761	Average Score: 233.65
Environment solved in 661 episodes!	Average Score: 233.65
Episode 762	Average Score: 233.38
Environment solved in 662 episodes!	Average Score: 233.38
Episode 763	Average Score: 233.46
Environment solved in 663 episodes!	Average Score: 233.46
Episode 764	Average Score: 232.29
Environment solved in 664 episodes!	Average Score: 232.29
Episode 765	Average Score: 232.29
Environment solved in 665 episodes!	Average Score: 232.29
Episode 766	Average Score: 232.21
Environment solved in 666 episodes!	Average Score: 232.21
Episode 767	Average Score: 232.31
Environment solved in 667 episodes!	Average Score: 232.31
Episode 768	Average Score: 232.32
Environment solved in 668 episodes!	Average Score: 232.32
Episode 769	Average Score: 232.53
Environment solved in 669 episodes!	Average Score: 232.53
Episode 770	Average Score: 232.85
Environment solved in 670 episodes!	Average Score: 232.85
Episode 771	Average Score: 230.41
Environment solved in 671 episodes!	Average Score: 230.41
Episode 772	Average Score: 229.96
Environment solved in 672 episodes!	Average Score: 229.96
Episode 773	Average Score: 229.97
Environment solved in 673 episodes!	Average Score: 229.97
Episode 774	Average Score: 230.44
Environment solved in 674 episodes!	Average Score: 230.44
Episode 775	Average Score: 230.02
Environment solved in 675 episodes!	Average Score: 230.02
Episode 776	Average Score: 230.06
Environment solved in 676 episodes!	Average Score: 230.06
Episode 777	Average Score: 231.02
Environment solved in 677 episodes!	Average Score: 231.02
Episode 778	Average Score: 230.39
Environment solved in 678 episodes!	Average Score: 230.39
Episode 779	Average Score: 230.54
Environment solved in 679 episodes!	Average Score: 230.54
Episode 780	Average Score: 233.26
Environment solved in 680 episodes!	Average Score: 233.26
Episode 781	Average Score: 233.65
Environment solved in 681 episodes!	Average Score: 233.65
Episode 782	Average Score: 233.39
Environment solved in 682 episodes!	Average Score: 233.39
Episode 783	Average Score: 234.07
Environment solved in 683 episodes!	Average Score: 234.07
Episode 784	Average Score: 236.44
Environment solved in 684 episodes!	Average Score: 236.44
Episode 785	Average Score: 238.70
Environment solved in 685 episodes!	Average Score: 238.70
Episode 786	Average Score: 237.85
Environment solved in 686 episodes!	Average Score: 237.85
Episode 787	Average Score: 237.48
Environment solved in 687 episodes!	Average Score: 237.48
Episode 788	Average Score: 237.62
Environment solved in 688 episodes!	Average Score: 237.62
Episode 789	Average Score: 237.63
Environment solved in 689 episodes!	Average Score: 237.63
Episode 790	Average Score: 240.08
Environment solved in 690 episodes!	Average Score: 240.08
Episode 791	Average Score: 240.00
Environment solved in 691 episodes!	Average Score: 240.00
Episode 792	Average Score: 239.80
Environment solved in 692 episodes!	Average Score: 239.80
Episode 793	Average Score: 239.94
Environment solved in 693 episodes!	Average Score: 239.94
Episode 794	Average Score: 239.65
Environment solved in 694 episodes!	Average Score: 239.65
Episode 795	Average Score: 239.10
Environment solved in 695 episodes!	Average Score: 239.10
Episode 796	Average Score: 238.66
Environment solved in 696 episodes!	Average Score: 238.66
Episode 797	Average Score: 238.11
Environment solved in 697 episodes!	Average Score: 238.11
Episode 798	Average Score: 238.36
Environment solved in 698 episodes!	Average Score: 238.36
Episode 799	Average Score: 238.58
Environment solved in 699 episodes!	Average Score: 238.58
Episode 800	Average Score: 238.36

Environment solved in 700 episodes!	Average Score: 238.36
Episode 801	Average Score: 235.07
Environment solved in 701 episodes!	Average Score: 235.07
Episode 802	Average Score: 234.84
Environment solved in 702 episodes!	Average Score: 234.84
Episode 803	Average Score: 235.03
Environment solved in 703 episodes!	Average Score: 235.03
Episode 804	Average Score: 235.30
Environment solved in 704 episodes!	Average Score: 235.30
Episode 805	Average Score: 235.31
Environment solved in 705 episodes!	Average Score: 235.31
Episode 806	Average Score: 235.54
Environment solved in 706 episodes!	Average Score: 235.54
Episode 807	Average Score: 235.20
Environment solved in 707 episodes!	Average Score: 235.20
Episode 808	Average Score: 235.15
Environment solved in 708 episodes!	Average Score: 235.15
Episode 809	Average Score: 235.06
Environment solved in 709 episodes!	Average Score: 235.06
Episode 810	Average Score: 234.90
Environment solved in 710 episodes!	Average Score: 234.90
Episode 811	Average Score: 235.12
Environment solved in 711 episodes!	Average Score: 235.12
Episode 812	Average Score: 235.06
Environment solved in 712 episodes!	Average Score: 235.06
Episode 813	Average Score: 237.26
Environment solved in 713 episodes!	Average Score: 237.26
Episode 814	Average Score: 235.23
Environment solved in 714 episodes!	Average Score: 235.23
Episode 815	Average Score: 235.85
Environment solved in 715 episodes!	Average Score: 235.85
Episode 816	Average Score: 235.24
Environment solved in 716 episodes!	Average Score: 235.24
Episode 817	Average Score: 232.34
Environment solved in 717 episodes!	Average Score: 232.34
Episode 818	Average Score: 232.36
Environment solved in 718 episodes!	Average Score: 232.36
Episode 819	Average Score: 227.75
Environment solved in 719 episodes!	Average Score: 227.75
Episode 820	Average Score: 228.06
Environment solved in 720 episodes!	Average Score: 228.06
Episode 821	Average Score: 228.37
Environment solved in 721 episodes!	Average Score: 228.37
Episode 822	Average Score: 228.02
Environment solved in 722 episodes!	Average Score: 228.02
Episode 823	Average Score: 228.34
Environment solved in 723 episodes!	Average Score: 228.34
Episode 824	Average Score: 228.67
Environment solved in 724 episodes!	Average Score: 228.67
Episode 825	Average Score: 228.75
Environment solved in 725 episodes!	Average Score: 228.75
Episode 826	Average Score: 228.30
Environment solved in 726 episodes!	Average Score: 228.30
Episode 827	Average Score: 228.07
Environment solved in 727 episodes!	Average Score: 228.07
Episode 828	Average Score: 228.24
Environment solved in 728 episodes!	Average Score: 228.24
Episode 829	Average Score: 228.61
Environment solved in 729 episodes!	Average Score: 228.61
Episode 830	Average Score: 230.83
Environment solved in 730 episodes!	Average Score: 230.83
Episode 831	Average Score: 230.26
Environment solved in 731 episodes!	Average Score: 230.26
Episode 832	Average Score: 229.93
Environment solved in 732 episodes!	Average Score: 229.93
Episode 833	Average Score: 229.82
Environment solved in 733 episodes!	Average Score: 229.82
Episode 834	Average Score: 231.84
Environment solved in 734 episodes!	Average Score: 231.84
Episode 835	Average Score: 232.00
Environment solved in 735 episodes!	Average Score: 232.00
Episode 836	Average Score: 229.05
Environment solved in 736 episodes!	Average Score: 229.05
Episode 837	Average Score: 228.83
Environment solved in 737 episodes!	Average Score: 228.83
Episode 838	Average Score: 226.12
Environment solved in 738 episodes!	Average Score: 226.12
Episode 839	Average Score: 225.27
Environment solved in 739 episodes!	Average Score: 225.27
Episode 840	Average Score: 224.87
Environment solved in 740 episodes!	Average Score: 224.87
Episode 841	Average Score: 220.74
Environment solved in 741 episodes!	Average Score: 220.74
Episode 842	Average Score: 220.72
Environment solved in 742 episodes!	Average Score: 220.72
Episode 843	Average Score: 220.95
Environment solved in 743 episodes!	Average Score: 220.95
Episode 844	Average Score: 220.59
Environment solved in 744 episodes!	Average Score: 220.59
Episode 845	Average Score: 216.41
Environment solved in 745 episodes!	Average Score: 216.41
Episode 846	Average Score: 216.44
Environment solved in 746 episodes!	Average Score: 216.44
Episode 847	Average Score: 215.87
Environment solved in 747 episodes!	Average Score: 215.87
Episode 848	Average Score: 215.90
Environment solved in 748 episodes!	Average Score: 215.90
Episode 849	Average Score: 215.65
Environment solved in 749 episodes!	Average Score: 215.65
Episode 850	Average Score: 215.06
Environment solved in 750 episodes!	Average Score: 215.06
Episode 851	Average Score: 214.95
Environment solved in 751 episodes!	Average Score: 214.95
Episode 852	Average Score: 214.97
Environment solved in 752 episodes!	Average Score: 214.97
Episode 853	Average Score: 212.91
Environment solved in 753 episodes!	Average Score: 212.91
Episode 854	Average Score: 212.42
Environment solved in 754 episodes!	Average Score: 212.42
Episode 855	Average Score: 212.58
Environment solved in 755 episodes!	Average Score: 212.58
Episode 856	Average Score: 212.81
Environment solved in 756 episodes!	Average Score: 212.81
Episode 857	Average Score: 215.26
Environment solved in 757 episodes!	Average Score: 215.26
Episode 858	Average Score: 216.85
Environment solved in 758 episodes!	Average Score: 216.85
Episode 859	Average Score: 218.35
Environment solved in 759 episodes!	Average Score: 218.35
Episode 860	Average Score: 218.41
Environment solved in 760 episodes!	Average Score: 218.41
Episode 861	Average Score: 218.62
Environment solved in 761 episodes!	Average Score: 218.62
Episode 862	Average Score: 218.08
Environment solved in 762 episodes!	Average Score: 218.08
Episode 863	Average Score: 217.97
Environment solved in 763 episodes!	Average Score: 217.97
Episode 864	Average Score: 219.37
Environment solved in 764 episodes!	Average Score: 219.37
Episode 865	Average Score: 219.24
Environment solved in 765 episodes!	Average Score: 219.24
Episode 866	Average Score: 218.67
Environment solved in 766 episodes!	Average Score: 218.67
Episode 867	Average Score: 218.40
Environment solved in 767 episodes!	Average Score: 218.40
Episode 868	Average Score: 217.35
Environment solved in 768 episodes!	Average Score: 217.35
Episode 869	Average Score: 217.22
Environment solved in 769 episodes!	Average Score: 217.22
Episode 870	Average Score: 216.89
Environment solved in 770 episodes!	Average Score: 216.89
Episode 871	Average Score: 219.41
Environment solved in 771 episodes!	Average Score: 219.41
Episode 872	Average Score: 219.96
Environment solved in 772 episodes!	Average Score: 219.96
Episode 873	Average Score: 220.37
Environment solved in 773 episodes!	Average Score: 220.37
Episode 874	Average Score: 216.84
Environment solved in 774 episodes!	Average Score: 216.84
Episode 875	Average Score: 217.15
Environment solved in 775 episodes!	Average Score: 217.15
Episode 876	Average Score: 217.69
Environment solved in 776 episodes!	Average Score: 217.69
Episode 877	Average Score: 217.79
Environment solved in 777 episodes!	Average Score: 217.79
Episode 878	Average Score: 217.91
Environment solved in 778 episodes!	Average Score: 217.91
Episode 879	Average Score: 213.64
Environment solved in 779 episodes!	Average Score: 213.64
Episode 880	Average Score: 213.52
Environment solved in 780 episodes!	Average Score: 213.52
Episode 881	Average Score: 211.39
Environment solved in 781 episodes!	Average Score: 211.39
Episode 882	Average Score: 211.57
Environment solved in 782 episodes!	Average Score: 211.57
Episode 883	Average Score: 211.39
Environment solved in 783 episodes!	Average Score: 211.39
Episode 884	Average Score: 211.47
Environment solved in 784 episodes!	Average Score: 211.47
Episode 885	Average Score: 211.59
Environment solved in 785 episodes!	Average Score: 211.59
Episode 886	Average Score: 212.34
Environment solved in 786 episodes!	Average Score: 212.34
Episode 887	Average Score: 212.81
Environment solved in 787 episodes!	Average Score: 212.81
Episode 888	Average Score: 213.05
Environment solved in 788 episodes!	Average Score: 213.05
Episode 889	Average Score: 213.25
Environment solved in 789 episodes!	Average Score: 213.25
Episode 890	Average Score: 213.05
Environment solved in 790 episodes!	Average Score: 213.05
Episode 891	Average Score: 212.85
Environment solved in 791 episodes!	Average Score: 212.85
Episode 892	Average Score: 211.51
Environment solved in 792 episodes!	Average Score: 211.51
Episode 893	Average Score: 211.83
Environment solved in 793 episodes!	Average Score: 211.83
Episode 894	Average Score: 208.46
Environment solved in 794 episodes!	Average Score: 208.46
Episode 895	Average Score: 208.56
Environment solved in 795 episodes!	Average Score: 208.56
Episode 896	Average Score: 208.75
Environment solved in 796 episodes!	Average Score: 208.75
Episode 897	Average Score: 209.29
Environment solved in 797 episodes!	Average Score: 209.29
Episode 898	Average Score: 208.79
Environment solved in 798 episodes!	Average Score: 208.79
Episode 899	Average Score: 208.69
Environment solved in 799 episodes!	Average Score: 208.69
Episode 900	Average Score: 209.04

Environment solved in 800 episodes!	Average Score: 209.04
Episode 901	Average Score: 212.05
Environment solved in 801 episodes!	Average Score: 212.05
Episode 902	Average Score: 211.59
Environment solved in 802 episodes!	Average Score: 211.59
Episode 903	Average Score: 211.52
Environment solved in 803 episodes!	Average Score: 211.52
Episode 904	Average Score: 211.46
Environment solved in 804 episodes!	Average Score: 211.46
Episode 905	Average Score: 210.98
Environment solved in 805 episodes!	Average Score: 210.98
Episode 906	Average Score: 210.61
Environment solved in 806 episodes!	Average Score: 210.61
Episode 907	Average Score: 210.87
Environment solved in 807 episodes!	Average Score: 210.87
Episode 908	Average Score: 210.87
Environment solved in 808 episodes!	Average Score: 210.87
Episode 909	Average Score: 210.85
Environment solved in 809 episodes!	Average Score: 210.85
Episode 910	Average Score: 210.90
Environment solved in 810 episodes!	Average Score: 210.90
Episode 911	Average Score: 210.89
Environment solved in 811 episodes!	Average Score: 210.89
Episode 912	Average Score: 210.72
Environment solved in 812 episodes!	Average Score: 210.72
Episode 913	Average Score: 210.75
Environment solved in 813 episodes!	Average Score: 210.75
Episode 914	Average Score: 212.62
Environment solved in 814 episodes!	Average Score: 212.62
Episode 915	Average Score: 212.85
Environment solved in 815 episodes!	Average Score: 212.85
Episode 916	Average Score: 213.07
Environment solved in 816 episodes!	Average Score: 213.07
Episode 917	Average Score: 216.14
Environment solved in 817 episodes!	Average Score: 216.14
Episode 918	Average Score: 216.27
Environment solved in 818 episodes!	Average Score: 216.27
Episode 919	Average Score: 220.42
Environment solved in 819 episodes!	Average Score: 220.42
Episode 920	Average Score: 220.00
Environment solved in 820 episodes!	Average Score: 220.00
Episode 921	Average Score: 219.99
Environment solved in 821 episodes!	Average Score: 219.99
Episode 922	Average Score: 220.23
Environment solved in 822 episodes!	Average Score: 220.23
Episode 923	Average Score: 219.96
Environment solved in 823 episodes!	Average Score: 219.96
Episode 924	Average Score: 220.19
Environment solved in 824 episodes!	Average Score: 220.19
Episode 925	Average Score: 220.09
Environment solved in 825 episodes!	Average Score: 220.09
Episode 926	Average Score: 217.81
Environment solved in 826 episodes!	Average Score: 217.81
Episode 927	Average Score: 216.65
Environment solved in 827 episodes!	Average Score: 216.65
Episode 928	Average Score: 216.35
Environment solved in 828 episodes!	Average Score: 216.35
Episode 929	Average Score: 216.31
Environment solved in 829 episodes!	Average Score: 216.31
Episode 930	Average Score: 216.50
Environment solved in 830 episodes!	Average Score: 216.50
Episode 931	Average Score: 216.78
Environment solved in 831 episodes!	Average Score: 216.78
Episode 932	Average Score: 216.85
Environment solved in 832 episodes!	Average Score: 216.85
Episode 933	Average Score: 216.48
Environment solved in 833 episodes!	Average Score: 216.48
Episode 934	Average Score: 216.31
Environment solved in 834 episodes!	Average Score: 216.31
Episode 935	Average Score: 215.96
Environment solved in 835 episodes!	Average Score: 215.96
Episode 936	Average Score: 218.82
Environment solved in 836 episodes!	Average Score: 218.82
Episode 937	Average Score: 219.18
Environment solved in 837 episodes!	Average Score: 219.18
Episode 938	Average Score: 221.65
Environment solved in 838 episodes!	Average Score: 221.65
Episode 939	Average Score: 222.17
Environment solved in 839 episodes!	Average Score: 222.17
Episode 940	Average Score: 222.48
Environment solved in 840 episodes!	Average Score: 222.48
Episode 941	Average Score: 225.96
Environment solved in 841 episodes!	Average Score: 225.96
Episode 942	Average Score: 226.23
Environment solved in 842 episodes!	Average Score: 226.23
Episode 943	Average Score: 226.30
Environment solved in 843 episodes!	Average Score: 226.30
Episode 944	Average Score: 226.29
Environment solved in 844 episodes!	Average Score: 226.29
Episode 945	Average Score: 230.86
Environment solved in 845 episodes!	Average Score: 230.86
Episode 946	Average Score: 231.13
Environment solved in 846 episodes!	Average Score: 231.13
Episode 947	Average Score: 231.11
Environment solved in 847 episodes!	Average Score: 231.11
Episode 948	Average Score: 230.84
Environment solved in 848 episodes!	Average Score: 230.84
Episode 949	Average Score: 231.38
Environment solved in 849 episodes!	Average Score: 231.38
Episode 950	Average Score: 231.73
Environment solved in 850 episodes!	Average Score: 231.73
Episode 951	Average Score: 231.96
Environment solved in 851 episodes!	Average Score: 231.96
Episode 952	Average Score: 232.15
Environment solved in 852 episodes!	Average Score: 232.15
Episode 953	Average Score: 234.34
Environment solved in 853 episodes!	Average Score: 234.34
Episode 954	Average Score: 234.68
Environment solved in 854 episodes!	Average Score: 234.68
Episode 955	Average Score: 231.93
Environment solved in 855 episodes!	Average Score: 231.93
Episode 956	Average Score: 232.03
Environment solved in 856 episodes!	Average Score: 232.03
Episode 957	Average Score: 231.94
Environment solved in 857 episodes!	Average Score: 231.94
Episode 958	Average Score: 233.39
Environment solved in 858 episodes!	Average Score: 233.39
Episode 959	Average Score: 233.89
Environment solved in 859 episodes!	Average Score: 233.89
Episode 960	Average Score: 233.82
Environment solved in 860 episodes!	Average Score: 233.82
Episode 961	Average Score: 231.40
Environment solved in 861 episodes!	Average Score: 231.40
Episode 962	Average Score: 232.31
Environment solved in 862 episodes!	Average Score: 232.31
Episode 963	Average Score: 232.53
Environment solved in 863 episodes!	Average Score: 232.53
Episode 964	Average Score: 232.17
Environment solved in 864 episodes!	Average Score: 232.17
Episode 965	Average Score: 232.53
Environment solved in 865 episodes!	Average Score: 232.53
Episode 966	Average Score: 232.69
Environment solved in 866 episodes!	Average Score: 232.69
Episode 967	Average Score: 232.85
Environment solved in 867 episodes!	Average Score: 232.85
Episode 968	Average Score: 232.97
Environment solved in 868 episodes!	Average Score: 232.97
Episode 969	Average Score: 232.94
Environment solved in 869 episodes!	Average Score: 232.94
Episode 970	Average Score: 233.01
Environment solved in 870 episodes!	Average Score: 233.01
Episode 971	Average Score: 232.27
Environment solved in 871 episodes!	Average Score: 232.27
Episode 972	Average Score: 232.22
Environment solved in 872 episodes!	Average Score: 232.22
Episode 973	Average Score: 232.16
Environment solved in 873 episodes!	Average Score: 232.16
Episode 974	Average Score: 235.11
Environment solved in 874 episodes!	Average Score: 235.11
Episode 975	Average Score: 234.81
Environment solved in 875 episodes!	Average Score: 234.81
Episode 976	Average Score: 234.36
Environment solved in 876 episodes!	Average Score: 234.36
Episode 977	Average Score: 234.47
Environment solved in 877 episodes!	Average Score: 234.47
Episode 978	Average Score: 235.11
Environment solved in 878 episodes!	Average Score: 235.11
Episode 979	Average Score: 239.40
Environment solved in 879 episodes!	Average Score: 239.40
Episode 980	Average Score: 239.40
Environment solved in 880 episodes!	Average Score: 239.40
Episode 981	Average Score: 241.94
Environment solved in 881 episodes!	Average Score: 241.94
Episode 982	Average Score: 242.01
Environment solved in 882 episodes!	Average Score: 242.01
Episode 983	Average Score: 241.85
Environment solved in 883 episodes!	Average Score: 241.85
Episode 984	Average Score: 239.46
Environment solved in 884 episodes!	Average Score: 239.46
Episode 985	Average Score: 239.56
Environment solved in 885 episodes!	Average Score: 239.56
Episode 986	Average Score: 237.43
Environment solved in 886 episodes!	Average Score: 237.43
Episode 987	Average Score: 237.23
Environment solved in 887 episodes!	Average Score: 237.23
Episode 988	Average Score: 234.46
Environment solved in 888 episodes!	Average Score: 234.46
Episode 989	Average Score: 233.90
Environment solved in 889 episodes!	Average Score: 233.90
Episode 990	Average Score: 234.23
Environment solved in 890 episodes!	Average Score: 234.23
Episode 991	Average Score: 234.75
Environment solved in 891 episodes!	Average Score: 234.75
Episode 992	Average Score: 235.60
Environment solved in 892 episodes!	Average Score: 235.60
Episode 993	Average Score: 235.57
Environment solved in 893 episodes!	Average Score: 235.57
Episode 994	Average Score: 239.29
Environment solved in 894 episodes!	Average Score: 239.29
Episode 995	Average Score: 239.67
Environment solved in 895 episodes!	Average Score: 239.67
Episode 996	Average Score: 240.01
Environment solved in 896 episodes!	Average Score: 240.01
Episode 997	Average Score: 239.80
Environment solved in 897 episodes!	Average Score: 239.80
Episode 998	Average Score: 239.65
Environment solved in 898 episodes!	Average Score: 239.65
Episode 999	Average Score: 239.77
Environment solved in 899 episodes!	Average Score: 239.77
Episode 1000	Average Score: 237.28

Environment solved in 900 episodes!	Average Score: 237.28

def moving_average(a, n=100) :
    ret = np.cumsum(a, dtype=float)
    ret[n:] = ret[n:] - ret[:-n]
    return ret[n - 1:] / n

scores_ma_d3qn=moving_average(scores_dddqn,n=100)

# plot the scores
plt.plot(np.arange(len(scores_ma_d3qn)), scores_ma_d3qn)
plt.ylabel('Score')
plt.xlabel('Episode')
plt.show()

Let’s compare with the DQN and DDQN?

scores_dqn_with_target=np.loadtxt("dqn_with_target/scores.txt")
scores_dqn_no_target=np.loadtxt("dqn_no_target/scores.txt")
scores_ddqn=np.loadtxt("double_dqn/scores.txt")

scores_ma_dqn_no_target=moving_average(scores_dqn_no_target,n=100)
scores_ma_dqn_with_target=moving_average(scores_dqn_with_target,n=100)
scores_ma_ddqn=moving_average(scores_ddqn,n=100)

plt.plot(np.arange(len(scores_ma_d3qn)), scores_ma_d3qn,label="D3QN")
plt.plot(np.arange(len(scores_ma_ddqn)), scores_ma_ddqn,label="DDQN")
plt.plot(np.arange(len(scores_ma_dqn_no_target)), scores_ma_dqn_no_target,label="DQN without target network")
plt.plot(np.arange(len(scores_ma_dqn_with_target)), scores_ma_dqn_with_target,label="DQN with target network")
plt.ylabel('Score')
plt.xlabel('Episode ')
plt.legend()

<matplotlib.legend.Legend at 0x7f6510342b70>

The END

This is the end of first DQN trilogy*! I am still learning so there might be many mistakes. if you find any , please feel free to contact me!

• email: sezan92[at]gmail[dot]com
• linkedin: www.linkedin.com/in/sezan92
• facebook: www.facebook.com/sezan92