The Game is On! The DQN Trilogy-Part 2: Double Deep Q-Network (DDQN)

Following up from article on DQN. It is mandatory to read that one first!

First, I have to give the proper credit to Udacity for making their Reinforcement Learning Nanodegree files open source in github. I used many of their codes

Index

Background
ReplayBuffer
QNetwork
Agent
Compare

Background

Double Deep Q Network was proposed by Hado van Hasselt, Arthur Guez, David Silver , in this paper . Let’s look at Bellman update equation again

\begin{equation} \theta_Q \gets \theta_Q - \alpha \frac{\partial}{\partial \theta}(r+ \gamma max_a(Q_{\theta}(s_{t+1},a))-Q_{\theta}(s_t,a))^2 \end{equation}

With target network $Q_{\theta}$ , the equation becomes \begin{equation} \theta_Q \gets \theta_Q - \alpha \frac{\partial}{\partial \theta}(r+ \gamma max_a(Q{\theta}(s{t+1},a))-Q_{\theta}(s_t,a))^2 \end{equation}

What we are doing in the update equation, the target is set according to the maximum action of target network qnetwork_target. This is a bit problematic. We are not using target network’s actions in the game! we are using local networks actions. So it is counter intuitive to update the action according to target network! We should fix the target of local network!

\begin{equation} \theta_Q \gets \theta_Q - \alpha \frac{\partial}{\partial \theta}(r+ \gamma (Q`{\theta}(s{t+1},argmax_aQ_{\theta}(s_{t+1},a))-Q_{\theta}(s_t,a))^2 \end{equation}

Enough of Background, lets start coding!

1. Import the Necessary Packages

import gym
import random
import torch
import numpy as np
from collections import namedtuple, deque
import matplotlib.pyplot as plt
from torch import nn
import torch
import torch.nn.functional as F
import torch.optim as optim
%matplotlib inline

initialize the environment

env = gym.make('LunarLander-v2')
env.seed(0)
print('State shape: ', env.observation_space.shape)
print('Number of actions: ', env.action_space.n)

State shape:  (8,)
Number of actions:  4


/floyd/home/gym/gym/logger.py:30: UserWarning: [33mWARN: Box bound precision lowered by casting to float32[0m
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))

state_shape= env.observation_space.shape[0]
action_size= env.action_space.n

QNetwork Class

As usual, QNetwork class, nothing changed from DQN blog!

class QNetwork(nn.Module):
    """Actor (Policy) Model."""

    def __init__(self, state_size, action_size, seed, fc1_units=150, fc2_units=120):
        """Initialize parameters and build model.
        Params
        ======
            state_size (int): Dimension of each state
            action_size (int): Dimension of each action
            seed (int): Random seed
            fc1_units (int): Number of nodes in first hidden layer
            fc2_units (int): Number of nodes in second hidden layer
        """
        super(QNetwork, self).__init__()
        self.seed = torch.manual_seed(seed)
        self.fc1 = nn.Linear(state_size, fc1_units)
        self.fc2 = nn.Linear(fc1_units, fc2_units)
        self.final = nn.Linear(fc2_units, action_size)

    def forward(self, state):
        """Build a network that maps state -> action values."""
        x = F.relu(self.fc1(state))
        x = F.relu(self.fc2(x))
        return self.final(x)

ReplayBuffer

class ReplayBuffer:
    """Fixed-size buffer to store experience tuples."""

    def __init__(self, action_size, buffer_size, batch_size, seed):
        """Initialize a ReplayBuffer object.

        Params
        ======
            action_size (int): dimension of each action
            buffer_size (int): maximum size of buffer
            batch_size (int): size of each training batch
            seed (int): random seed
        """
        self.action_size = action_size
        self.memory = deque(maxlen=buffer_size)  
        self.batch_size = batch_size
        self.experience = namedtuple("Experience", field_names=["state", "action", "reward", "next_state", "done"])
        self.seed = random.seed(seed)
    
    def add(self, state, action, reward, next_state, done):
        """Add a new experience to memory."""
        e = self.experience(state, action, reward, next_state, done)
        self.memory.append(e)
    
    def sample(self):
        """Randomly sample a batch of experiences from memory."""
        experiences = random.sample(self.memory, k=self.batch_size)

        states = torch.from_numpy(np.vstack([e.state for e in experiences if e is not None])).float().to(device)
        actions = torch.from_numpy(np.vstack([e.action for e in experiences if e is not None])).long().to(device)
        rewards = torch.from_numpy(np.vstack([e.reward for e in experiences if e is not None])).float().to(device)
        next_states = torch.from_numpy(np.vstack([e.next_state for e in experiences if e is not None])).float().to(device)
        dones = torch.from_numpy(np.vstack([e.done for e in experiences if e is not None]).astype(np.uint8)).float().to(device)
  
        return (states, actions, rewards, next_states, dones)

    def __len__(self):
        """Return the current size of internal memory."""
        return len(self.memory)

Agent

I changed a little bit in the Agent class. The change is discussed in the following section

BUFFER_SIZE = int(1e5)  # replay buffer size
BATCH_SIZE = 64         # minibatch size
GAMMA = 0.99            # discount factor
TAU = 0.01              # for soft update of target parameters
LR = 0.001               # learning rate 
UPDATE_EVERY = 4        # how often to update the network

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

class Agent():
    """Interacts with and learns from the environment."""

    def __init__(self, state_size, action_size, seed):
        """Initialize an Agent object.
        
        Params
        ======
            state_size (int): dimension of each state
            action_size (int): dimension of each action
            seed (int): random seed
        """
        self.state_size = state_size
        self.action_size = action_size
        self.seed = random.seed(seed)

        # Q-Network
        self.qnetwork_local = QNetwork(state_size, action_size, seed).to(device)
        self.qnetwork_target = QNetwork(state_size, action_size, seed).to(device)
        self.optimizer = optim.Adam(self.qnetwork_local.parameters(), lr=LR)

        # Replay memory
        self.memory = ReplayBuffer(action_size, BUFFER_SIZE, BATCH_SIZE, seed)
        # Initialize time step (for updating every UPDATE_EVERY steps)
        self.t_step = 0
        self.criterion=nn.MSELoss()
    
    def step(self, state, action, reward, next_state, done):
        # Save experience in replay memory
        self.memory.add(state, action, reward, next_state, done)
        
        # Learn every UPDATE_EVERY time steps.
        self.t_step = (self.t_step + 1) % UPDATE_EVERY
        if self.t_step == 0:
            # If enough samples are available in memory, get random subset and learn
            if len(self.memory) > BATCH_SIZE:
                experiences = self.memory.sample()
                self.learn(experiences, GAMMA)

    def act(self, state, eps=0.):
        """Returns actions for given state as per current policy.
        
        Params
        ======
            state (array_like): current state
            eps (float): epsilon, for epsilon-greedy action selection
        """
        state = torch.from_numpy(state).float().unsqueeze(0).to(device)
        self.qnetwork_local.eval()
        with torch.no_grad():
            action_values = self.qnetwork_local(state)
        self.qnetwork_local.train()

        # Epsilon-greedy action selection
        if random.random() > eps:
            return np.argmax(action_values.cpu().data.numpy())
        else:
            return random.choice(np.arange(self.action_size))

    def learn(self, experiences, gamma):
        """Update value parameters using given batch of experience tuples.

        Params
        ======
            experiences (Tuple[torch.Tensor]): tuple of (s, a, r, s', done) tuples 
            gamma (float): discount factor
        """
        states, actions, rewards, next_states, dones = experiences

        predicted_actions=torch.argmax(self.qnetwork_local(next_states),1)
        predicted_actions=predicted_actions.reshape(-1,1)
      
        maxQ=self.qnetwork_target(next_states).gather(1,predicted_actions)
        y=rewards+gamma*maxQ*(1-dones)

        loss=F.mse_loss(y,self.qnetwork_local(states).gather(1,actions))
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        
        

        # ------------------- update target network ------------------- #
        self.soft_update(self.qnetwork_local, self.qnetwork_target, TAU)                     

    def soft_update(self, local_model, target_model, tau):
        """Soft update model parameters.
        θ_target = τ*θ_local + (1 - τ)*θ_target

        Params
        ======
            local_model (PyTorch model): weights will be copied from
            target_model (PyTorch model): weights will be copied to
            tau (float): interpolation parameter 
        """
        for target_param, local_param in zip(target_model.parameters(), local_model.parameters()):
            target_param.data.copy_(tau*local_param.data + (1.0-tau)*target_param.data)

The change in the `agent` class

We dont need to read full code! I have changed line 79 according to the equation in background section predicted_actions=torch.argmax(self.qnetwork_local(next_states),1) predicted_actions=predicted_actions.reshape(-1,1) maxQ=self.qnetwork_target(next_states).gather(1,predicted_actions)

agent = Agent(state_size=state_shape, action_size=action_size, seed=0)

Training!

n_episodes=1000
max_t=1000
eps_start=1.0
eps_end=0.1
eps_decay=0.995

def train_agent(agent,env,target_folder,eps_start=eps_start,eps_decay=eps_decay,eps_end=eps_end,gamma=GAMMA,n_episodes=n_episodes,max_t=max_t):
    """Deep Q-Learning.

    Params
    ======
        n_episodes (int): maximum number of training episodes
        max_t (int): maximum number of timesteps per episode
        eps_start (float): starting value of epsilon, for epsilon-greedy action selection
        eps_end (float): minimum value of epsilon
        eps_decay (float): multiplicative factor (per episode) for decreasing epsilon
    """
    scores = []                        # list containing scores from each episode
    scores_window = deque(maxlen=100)  # last 100 scores
    eps = eps_start                    # initialize epsilon
    for i_episode in range(1, n_episodes+1):

        state = env.reset()
        score = 0
        for t in range(max_t):
            action = agent.act(state, eps)
            next_state, reward, done, _ = env.step(action)

            agent.step(state, action, reward, next_state,done)
            state = next_state
            score += reward
            if done:
                break 
        scores_window.append(score)       # save most recent score
        scores.append(score)              # save most recent score
        eps = max(eps_end, eps_decay*eps) # decrease epsilon
        print('\rEpisode {}\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end="")
        if i_episode%10==0:
            torch.save(agent.qnetwork_local.state_dict(),"{}/dqn_agent{}.pkl".format(target_folder,i_episode))
        if i_episode % 100 == 0:
            print('\rEpisode {}\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))
        if np.mean(scores_window)>=200.0:
            print('\nEnvironment solved in {:d} episodes!\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))
    return scores

scores_ddqn=train_agent(agent,env,target_folder="double_dqn/")

Episode 100	Average Score: -141.10
Episode 200	Average Score: -39.10
Episode 300	Average Score: 30.98
Episode 400	Average Score: 174.96
Episode 422	Average Score: 200.26
Environment solved in 322 episodes!	Average Score: 200.26
Episode 423	Average Score: 201.46
Environment solved in 323 episodes!	Average Score: 201.46
Episode 424	Average Score: 200.73
Environment solved in 324 episodes!	Average Score: 200.73
Episode 425	Average Score: 201.70
Environment solved in 325 episodes!	Average Score: 201.70
Episode 426	Average Score: 202.82
Environment solved in 326 episodes!	Average Score: 202.82
Episode 427	Average Score: 203.66
Environment solved in 327 episodes!	Average Score: 203.66
Episode 428	Average Score: 204.85
Environment solved in 328 episodes!	Average Score: 204.85
Episode 429	Average Score: 205.76
Environment solved in 329 episodes!	Average Score: 205.76
Episode 430	Average Score: 205.22
Environment solved in 330 episodes!	Average Score: 205.22
Episode 431	Average Score: 207.68
Environment solved in 331 episodes!	Average Score: 207.68
Episode 432	Average Score: 208.78
Environment solved in 332 episodes!	Average Score: 208.78
Episode 433	Average Score: 212.83
Environment solved in 333 episodes!	Average Score: 212.83
Episode 434	Average Score: 213.13
Environment solved in 334 episodes!	Average Score: 213.13
Episode 435	Average Score: 213.44
Environment solved in 335 episodes!	Average Score: 213.44
Episode 436	Average Score: 218.12
Environment solved in 336 episodes!	Average Score: 218.12
Episode 437	Average Score: 218.60
Environment solved in 337 episodes!	Average Score: 218.60
Episode 438	Average Score: 221.31
Environment solved in 338 episodes!	Average Score: 221.31
Episode 439	Average Score: 225.80
Environment solved in 339 episodes!	Average Score: 225.80
Episode 440	Average Score: 226.65
Environment solved in 340 episodes!	Average Score: 226.65
Episode 441	Average Score: 228.46
Environment solved in 341 episodes!	Average Score: 228.46
Episode 442	Average Score: 228.79
Environment solved in 342 episodes!	Average Score: 228.79
Episode 443	Average Score: 229.32
Environment solved in 343 episodes!	Average Score: 229.32
Episode 444	Average Score: 229.27
Environment solved in 344 episodes!	Average Score: 229.27
Episode 445	Average Score: 231.01
Environment solved in 345 episodes!	Average Score: 231.01
Episode 446	Average Score: 230.76
Environment solved in 346 episodes!	Average Score: 230.76
Episode 447	Average Score: 231.25
Environment solved in 347 episodes!	Average Score: 231.25
Episode 448	Average Score: 231.59
Environment solved in 348 episodes!	Average Score: 231.59
Episode 449	Average Score: 231.30
Environment solved in 349 episodes!	Average Score: 231.30
Episode 450	Average Score: 232.01
Environment solved in 350 episodes!	Average Score: 232.01
Episode 451	Average Score: 231.76
Environment solved in 351 episodes!	Average Score: 231.76
Episode 452	Average Score: 233.15
Environment solved in 352 episodes!	Average Score: 233.15
Episode 453	Average Score: 232.92
Environment solved in 353 episodes!	Average Score: 232.92
Episode 454	Average Score: 233.38
Environment solved in 354 episodes!	Average Score: 233.38
Episode 455	Average Score: 234.11
Environment solved in 355 episodes!	Average Score: 234.11
Episode 456	Average Score: 234.61
Environment solved in 356 episodes!	Average Score: 234.61
Episode 457	Average Score: 234.39
Environment solved in 357 episodes!	Average Score: 234.39
Episode 458	Average Score: 234.36
Environment solved in 358 episodes!	Average Score: 234.36
Episode 459	Average Score: 234.15
Environment solved in 359 episodes!	Average Score: 234.15
Episode 460	Average Score: 233.80
Environment solved in 360 episodes!	Average Score: 233.80
Episode 461	Average Score: 233.77
Environment solved in 361 episodes!	Average Score: 233.77
Episode 462	Average Score: 234.02
Environment solved in 362 episodes!	Average Score: 234.02
Episode 463	Average Score: 234.34
Environment solved in 363 episodes!	Average Score: 234.34
Episode 464	Average Score: 234.39
Environment solved in 364 episodes!	Average Score: 234.39
Episode 465	Average Score: 234.36
Environment solved in 365 episodes!	Average Score: 234.36
Episode 466	Average Score: 235.18
Environment solved in 366 episodes!	Average Score: 235.18
Episode 467	Average Score: 235.21
Environment solved in 367 episodes!	Average Score: 235.21
Episode 468	Average Score: 235.70
Environment solved in 368 episodes!	Average Score: 235.70
Episode 469	Average Score: 235.73
Environment solved in 369 episodes!	Average Score: 235.73
Episode 470	Average Score: 236.56
Environment solved in 370 episodes!	Average Score: 236.56
Episode 471	Average Score: 235.98
Environment solved in 371 episodes!	Average Score: 235.98
Episode 472	Average Score: 231.36
Environment solved in 372 episodes!	Average Score: 231.36
Episode 473	Average Score: 231.27
Environment solved in 373 episodes!	Average Score: 231.27
Episode 474	Average Score: 231.66
Environment solved in 374 episodes!	Average Score: 231.66
Episode 475	Average Score: 231.28
Environment solved in 375 episodes!	Average Score: 231.28
Episode 476	Average Score: 231.12
Environment solved in 376 episodes!	Average Score: 231.12
Episode 477	Average Score: 233.15
Environment solved in 377 episodes!	Average Score: 233.15
Episode 478	Average Score: 236.17
Environment solved in 378 episodes!	Average Score: 236.17
Episode 479	Average Score: 236.49
Environment solved in 379 episodes!	Average Score: 236.49
Episode 480	Average Score: 236.59
Environment solved in 380 episodes!	Average Score: 236.59
Episode 481	Average Score: 236.92
Environment solved in 381 episodes!	Average Score: 236.92
Episode 482	Average Score: 236.78
Environment solved in 382 episodes!	Average Score: 236.78
Episode 483	Average Score: 236.19
Environment solved in 383 episodes!	Average Score: 236.19
Episode 484	Average Score: 235.80
Environment solved in 384 episodes!	Average Score: 235.80
Episode 485	Average Score: 233.63
Environment solved in 385 episodes!	Average Score: 233.63
Episode 486	Average Score: 233.39
Environment solved in 386 episodes!	Average Score: 233.39
Episode 487	Average Score: 233.59
Environment solved in 387 episodes!	Average Score: 233.59
Episode 488	Average Score: 234.52
Environment solved in 388 episodes!	Average Score: 234.52
Episode 489	Average Score: 234.77
Environment solved in 389 episodes!	Average Score: 234.77
Episode 490	Average Score: 233.92
Environment solved in 390 episodes!	Average Score: 233.92
Episode 491	Average Score: 235.61
Environment solved in 391 episodes!	Average Score: 235.61
Episode 492	Average Score: 235.85
Environment solved in 392 episodes!	Average Score: 235.85
Episode 493	Average Score: 235.83
Environment solved in 393 episodes!	Average Score: 235.83
Episode 494	Average Score: 235.52
Environment solved in 394 episodes!	Average Score: 235.52
Episode 495	Average Score: 235.93
Environment solved in 395 episodes!	Average Score: 235.93
Episode 496	Average Score: 235.70
Environment solved in 396 episodes!	Average Score: 235.70
Episode 497	Average Score: 236.35
Environment solved in 397 episodes!	Average Score: 236.35
Episode 498	Average Score: 236.67
Environment solved in 398 episodes!	Average Score: 236.67
Episode 499	Average Score: 236.72
Environment solved in 399 episodes!	Average Score: 236.72
Episode 500	Average Score: 237.14

Environment solved in 400 episodes!	Average Score: 237.14
Episode 501	Average Score: 237.66
Environment solved in 401 episodes!	Average Score: 237.66
Episode 502	Average Score: 238.71
Environment solved in 402 episodes!	Average Score: 238.71
Episode 503	Average Score: 238.58
Environment solved in 403 episodes!	Average Score: 238.58
Episode 504	Average Score: 237.18
Environment solved in 404 episodes!	Average Score: 237.18
Episode 505	Average Score: 237.38
Environment solved in 405 episodes!	Average Score: 237.38
Episode 506	Average Score: 236.64
Environment solved in 406 episodes!	Average Score: 236.64
Episode 507	Average Score: 236.70
Environment solved in 407 episodes!	Average Score: 236.70
Episode 508	Average Score: 234.33
Environment solved in 408 episodes!	Average Score: 234.33
Episode 509	Average Score: 235.03
Environment solved in 409 episodes!	Average Score: 235.03
Episode 510	Average Score: 235.05
Environment solved in 410 episodes!	Average Score: 235.05
Episode 511	Average Score: 234.66
Environment solved in 411 episodes!	Average Score: 234.66
Episode 512	Average Score: 234.90
Environment solved in 412 episodes!	Average Score: 234.90
Episode 513	Average Score: 234.99
Environment solved in 413 episodes!	Average Score: 234.99
Episode 514	Average Score: 235.43
Environment solved in 414 episodes!	Average Score: 235.43
Episode 515	Average Score: 235.21
Environment solved in 415 episodes!	Average Score: 235.21
Episode 516	Average Score: 233.91
Environment solved in 416 episodes!	Average Score: 233.91
Episode 517	Average Score: 233.63
Environment solved in 417 episodes!	Average Score: 233.63
Episode 518	Average Score: 232.78
Environment solved in 418 episodes!	Average Score: 232.78
Episode 519	Average Score: 232.39
Environment solved in 419 episodes!	Average Score: 232.39
Episode 520	Average Score: 230.87
Environment solved in 420 episodes!	Average Score: 230.87
Episode 521	Average Score: 231.25
Environment solved in 421 episodes!	Average Score: 231.25
Episode 522	Average Score: 229.24
Environment solved in 422 episodes!	Average Score: 229.24
Episode 523	Average Score: 229.96
Environment solved in 423 episodes!	Average Score: 229.96
Episode 524	Average Score: 230.40
Environment solved in 424 episodes!	Average Score: 230.40
Episode 525	Average Score: 230.30
Environment solved in 425 episodes!	Average Score: 230.30
Episode 526	Average Score: 230.32
Environment solved in 426 episodes!	Average Score: 230.32
Episode 527	Average Score: 230.56
Environment solved in 427 episodes!	Average Score: 230.56
Episode 528	Average Score: 227.44
Environment solved in 428 episodes!	Average Score: 227.44
Episode 529	Average Score: 227.25
Environment solved in 429 episodes!	Average Score: 227.25
Episode 530	Average Score: 227.13
Environment solved in 430 episodes!	Average Score: 227.13
Episode 531	Average Score: 226.63
Environment solved in 431 episodes!	Average Score: 226.63
Episode 532	Average Score: 226.18
Environment solved in 432 episodes!	Average Score: 226.18
Episode 533	Average Score: 225.99
Environment solved in 433 episodes!	Average Score: 225.99
Episode 534	Average Score: 225.34
Environment solved in 434 episodes!	Average Score: 225.34
Episode 535	Average Score: 225.38
Environment solved in 435 episodes!	Average Score: 225.38
Episode 536	Average Score: 225.51
Environment solved in 436 episodes!	Average Score: 225.51
Episode 537	Average Score: 225.85
Environment solved in 437 episodes!	Average Score: 225.85
Episode 538	Average Score: 223.40
Environment solved in 438 episodes!	Average Score: 223.40
Episode 539	Average Score: 222.94
Environment solved in 439 episodes!	Average Score: 222.94
Episode 540	Average Score: 222.56
Environment solved in 440 episodes!	Average Score: 222.56
Episode 541	Average Score: 222.65
Environment solved in 441 episodes!	Average Score: 222.65
Episode 542	Average Score: 222.72
Environment solved in 442 episodes!	Average Score: 222.72
Episode 543	Average Score: 220.46
Environment solved in 443 episodes!	Average Score: 220.46
Episode 544	Average Score: 221.56
Environment solved in 444 episodes!	Average Score: 221.56
Episode 545	Average Score: 221.09
Environment solved in 445 episodes!	Average Score: 221.09
Episode 546	Average Score: 221.64
Environment solved in 446 episodes!	Average Score: 221.64
Episode 547	Average Score: 221.83
Environment solved in 447 episodes!	Average Score: 221.83
Episode 548	Average Score: 222.35
Environment solved in 448 episodes!	Average Score: 222.35
Episode 549	Average Score: 223.13
Environment solved in 449 episodes!	Average Score: 223.13
Episode 550	Average Score: 222.95
Environment solved in 450 episodes!	Average Score: 222.95
Episode 551	Average Score: 223.31
Environment solved in 451 episodes!	Average Score: 223.31
Episode 552	Average Score: 223.04
Environment solved in 452 episodes!	Average Score: 223.04
Episode 553	Average Score: 223.67
Environment solved in 453 episodes!	Average Score: 223.67
Episode 554	Average Score: 223.70
Environment solved in 454 episodes!	Average Score: 223.70
Episode 555	Average Score: 222.67
Environment solved in 455 episodes!	Average Score: 222.67
Episode 556	Average Score: 222.31
Environment solved in 456 episodes!	Average Score: 222.31
Episode 557	Average Score: 221.60
Environment solved in 457 episodes!	Average Score: 221.60
Episode 558	Average Score: 221.70
Environment solved in 458 episodes!	Average Score: 221.70
Episode 559	Average Score: 222.02
Environment solved in 459 episodes!	Average Score: 222.02
Episode 560	Average Score: 221.77
Environment solved in 460 episodes!	Average Score: 221.77
Episode 561	Average Score: 221.57
Environment solved in 461 episodes!	Average Score: 221.57
Episode 562	Average Score: 221.67
Environment solved in 462 episodes!	Average Score: 221.67
Episode 563	Average Score: 219.02
Environment solved in 463 episodes!	Average Score: 219.02
Episode 564	Average Score: 219.18
Environment solved in 464 episodes!	Average Score: 219.18
Episode 565	Average Score: 219.51
Environment solved in 465 episodes!	Average Score: 219.51
Episode 566	Average Score: 220.12
Environment solved in 466 episodes!	Average Score: 220.12
Episode 567	Average Score: 220.43
Environment solved in 467 episodes!	Average Score: 220.43
Episode 568	Average Score: 220.43
Environment solved in 468 episodes!	Average Score: 220.43
Episode 569	Average Score: 220.32
Environment solved in 469 episodes!	Average Score: 220.32
Episode 570	Average Score: 220.55
Environment solved in 470 episodes!	Average Score: 220.55
Episode 571	Average Score: 221.43
Environment solved in 471 episodes!	Average Score: 221.43
Episode 572	Average Score: 226.32
Environment solved in 472 episodes!	Average Score: 226.32
Episode 573	Average Score: 226.97
Environment solved in 473 episodes!	Average Score: 226.97
Episode 574	Average Score: 227.11
Environment solved in 474 episodes!	Average Score: 227.11
Episode 575	Average Score: 227.37
Environment solved in 475 episodes!	Average Score: 227.37
Episode 576	Average Score: 227.40
Environment solved in 476 episodes!	Average Score: 227.40
Episode 577	Average Score: 227.17
Environment solved in 477 episodes!	Average Score: 227.17
Episode 578	Average Score: 224.94
Environment solved in 478 episodes!	Average Score: 224.94
Episode 579	Average Score: 225.00
Environment solved in 479 episodes!	Average Score: 225.00
Episode 580	Average Score: 224.81
Environment solved in 480 episodes!	Average Score: 224.81
Episode 581	Average Score: 223.95
Environment solved in 481 episodes!	Average Score: 223.95
Episode 582	Average Score: 224.08
Environment solved in 482 episodes!	Average Score: 224.08
Episode 583	Average Score: 223.71
Environment solved in 483 episodes!	Average Score: 223.71
Episode 584	Average Score: 224.48
Environment solved in 484 episodes!	Average Score: 224.48
Episode 585	Average Score: 226.57
Environment solved in 485 episodes!	Average Score: 226.57
Episode 586	Average Score: 227.24
Environment solved in 486 episodes!	Average Score: 227.24
Episode 587	Average Score: 227.29
Environment solved in 487 episodes!	Average Score: 227.29
Episode 588	Average Score: 227.33
Environment solved in 488 episodes!	Average Score: 227.33
Episode 589	Average Score: 227.21
Environment solved in 489 episodes!	Average Score: 227.21
Episode 590	Average Score: 225.81
Environment solved in 490 episodes!	Average Score: 225.81
Episode 591	Average Score: 226.72
Environment solved in 491 episodes!	Average Score: 226.72
Episode 592	Average Score: 226.87
Environment solved in 492 episodes!	Average Score: 226.87
Episode 593	Average Score: 226.66
Environment solved in 493 episodes!	Average Score: 226.66
Episode 594	Average Score: 227.10
Environment solved in 494 episodes!	Average Score: 227.10
Episode 595	Average Score: 227.05
Environment solved in 495 episodes!	Average Score: 227.05
Episode 596	Average Score: 227.03
Environment solved in 496 episodes!	Average Score: 227.03
Episode 597	Average Score: 227.27
Environment solved in 497 episodes!	Average Score: 227.27
Episode 598	Average Score: 224.78
Environment solved in 498 episodes!	Average Score: 224.78
Episode 599	Average Score: 224.58
Environment solved in 499 episodes!	Average Score: 224.58
Episode 600	Average Score: 224.72

Environment solved in 500 episodes!	Average Score: 224.72
Episode 601	Average Score: 224.65
Environment solved in 501 episodes!	Average Score: 224.65
Episode 602	Average Score: 225.07
Environment solved in 502 episodes!	Average Score: 225.07
Episode 603	Average Score: 225.52
Environment solved in 503 episodes!	Average Score: 225.52
Episode 604	Average Score: 227.91
Environment solved in 504 episodes!	Average Score: 227.91
Episode 605	Average Score: 227.40
Environment solved in 505 episodes!	Average Score: 227.40
Episode 606	Average Score: 227.92
Environment solved in 506 episodes!	Average Score: 227.92
Episode 607	Average Score: 227.71
Environment solved in 507 episodes!	Average Score: 227.71
Episode 608	Average Score: 230.40
Environment solved in 508 episodes!	Average Score: 230.40
Episode 609	Average Score: 230.26
Environment solved in 509 episodes!	Average Score: 230.26
Episode 610	Average Score: 230.18
Environment solved in 510 episodes!	Average Score: 230.18
Episode 611	Average Score: 231.13
Environment solved in 511 episodes!	Average Score: 231.13
Episode 612	Average Score: 230.94
Environment solved in 512 episodes!	Average Score: 230.94
Episode 613	Average Score: 231.29
Environment solved in 513 episodes!	Average Score: 231.29
Episode 614	Average Score: 231.84
Environment solved in 514 episodes!	Average Score: 231.84
Episode 615	Average Score: 232.25
Environment solved in 515 episodes!	Average Score: 232.25
Episode 616	Average Score: 232.16
Environment solved in 516 episodes!	Average Score: 232.16
Episode 617	Average Score: 232.57
Environment solved in 517 episodes!	Average Score: 232.57
Episode 618	Average Score: 231.23
Environment solved in 518 episodes!	Average Score: 231.23
Episode 619	Average Score: 231.34
Environment solved in 519 episodes!	Average Score: 231.34
Episode 620	Average Score: 233.29
Environment solved in 520 episodes!	Average Score: 233.29
Episode 621	Average Score: 232.95
Environment solved in 521 episodes!	Average Score: 232.95
Episode 622	Average Score: 235.40
Environment solved in 522 episodes!	Average Score: 235.40
Episode 623	Average Score: 235.42
Environment solved in 523 episodes!	Average Score: 235.42
Episode 624	Average Score: 235.58
Environment solved in 524 episodes!	Average Score: 235.58
Episode 625	Average Score: 235.51
Environment solved in 525 episodes!	Average Score: 235.51
Episode 626	Average Score: 232.97
Environment solved in 526 episodes!	Average Score: 232.97
Episode 627	Average Score: 233.43
Environment solved in 527 episodes!	Average Score: 233.43
Episode 628	Average Score: 236.15
Environment solved in 528 episodes!	Average Score: 236.15
Episode 629	Average Score: 236.78
Environment solved in 529 episodes!	Average Score: 236.78
Episode 630	Average Score: 237.41
Environment solved in 530 episodes!	Average Score: 237.41
Episode 631	Average Score: 238.07
Environment solved in 531 episodes!	Average Score: 238.07
Episode 632	Average Score: 238.85
Environment solved in 532 episodes!	Average Score: 238.85
Episode 633	Average Score: 239.18
Environment solved in 533 episodes!	Average Score: 239.18
Episode 634	Average Score: 239.63
Environment solved in 534 episodes!	Average Score: 239.63
Episode 635	Average Score: 236.88
Environment solved in 535 episodes!	Average Score: 236.88
Episode 636	Average Score: 236.91
Environment solved in 536 episodes!	Average Score: 236.91
Episode 637	Average Score: 236.78
Environment solved in 537 episodes!	Average Score: 236.78
Episode 638	Average Score: 239.47
Environment solved in 538 episodes!	Average Score: 239.47
Episode 639	Average Score: 239.39
Environment solved in 539 episodes!	Average Score: 239.39
Episode 640	Average Score: 239.80
Environment solved in 540 episodes!	Average Score: 239.80
Episode 641	Average Score: 239.85
Environment solved in 541 episodes!	Average Score: 239.85
Episode 642	Average Score: 240.10
Environment solved in 542 episodes!	Average Score: 240.10
Episode 643	Average Score: 242.89
Environment solved in 543 episodes!	Average Score: 242.89
Episode 644	Average Score: 242.14
Environment solved in 544 episodes!	Average Score: 242.14
Episode 645	Average Score: 241.89
Environment solved in 545 episodes!	Average Score: 241.89
Episode 646	Average Score: 241.81
Environment solved in 546 episodes!	Average Score: 241.81
Episode 647	Average Score: 241.47
Environment solved in 547 episodes!	Average Score: 241.47
Episode 648	Average Score: 241.43
Environment solved in 548 episodes!	Average Score: 241.43
Episode 649	Average Score: 241.15
Environment solved in 549 episodes!	Average Score: 241.15
Episode 650	Average Score: 241.15
Environment solved in 550 episodes!	Average Score: 241.15
Episode 651	Average Score: 241.45
Environment solved in 551 episodes!	Average Score: 241.45
Episode 652	Average Score: 241.73
Environment solved in 552 episodes!	Average Score: 241.73
Episode 653	Average Score: 241.52
Environment solved in 553 episodes!	Average Score: 241.52
Episode 654	Average Score: 241.51
Environment solved in 554 episodes!	Average Score: 241.51
Episode 655	Average Score: 242.94
Environment solved in 555 episodes!	Average Score: 242.94
Episode 656	Average Score: 242.04
Environment solved in 556 episodes!	Average Score: 242.04
Episode 657	Average Score: 242.66
Environment solved in 557 episodes!	Average Score: 242.66
Episode 658	Average Score: 242.87
Environment solved in 558 episodes!	Average Score: 242.87
Episode 659	Average Score: 242.92
Environment solved in 559 episodes!	Average Score: 242.92
Episode 660	Average Score: 243.61
Environment solved in 560 episodes!	Average Score: 243.61
Episode 661	Average Score: 243.96
Environment solved in 561 episodes!	Average Score: 243.96
Episode 662	Average Score: 244.52
Environment solved in 562 episodes!	Average Score: 244.52
Episode 663	Average Score: 247.03
Environment solved in 563 episodes!	Average Score: 247.03
Episode 664	Average Score: 247.15
Environment solved in 564 episodes!	Average Score: 247.15
Episode 665	Average Score: 246.34
Environment solved in 565 episodes!	Average Score: 246.34
Episode 666	Average Score: 247.14
Environment solved in 566 episodes!	Average Score: 247.14
Episode 667	Average Score: 247.28
Environment solved in 567 episodes!	Average Score: 247.28
Episode 668	Average Score: 247.75
Environment solved in 568 episodes!	Average Score: 247.75
Episode 669	Average Score: 245.74
Environment solved in 569 episodes!	Average Score: 245.74
Episode 670	Average Score: 243.52
Environment solved in 570 episodes!	Average Score: 243.52
Episode 671	Average Score: 242.97
Environment solved in 571 episodes!	Average Score: 242.97
Episode 672	Average Score: 241.20
Environment solved in 572 episodes!	Average Score: 241.20
Episode 673	Average Score: 240.87
Environment solved in 573 episodes!	Average Score: 240.87
Episode 674	Average Score: 240.35
Environment solved in 574 episodes!	Average Score: 240.35
Episode 675	Average Score: 240.74
Environment solved in 575 episodes!	Average Score: 240.74
Episode 676	Average Score: 240.86
Environment solved in 576 episodes!	Average Score: 240.86
Episode 677	Average Score: 241.28
Environment solved in 577 episodes!	Average Score: 241.28
Episode 678	Average Score: 243.73
Environment solved in 578 episodes!	Average Score: 243.73
Episode 679	Average Score: 243.33
Environment solved in 579 episodes!	Average Score: 243.33
Episode 680	Average Score: 243.09
Environment solved in 580 episodes!	Average Score: 243.09
Episode 681	Average Score: 243.66
Environment solved in 581 episodes!	Average Score: 243.66
Episode 682	Average Score: 243.77
Environment solved in 582 episodes!	Average Score: 243.77
Episode 683	Average Score: 244.75
Environment solved in 583 episodes!	Average Score: 244.75
Episode 684	Average Score: 244.64
Environment solved in 584 episodes!	Average Score: 244.64
Episode 685	Average Score: 245.14
Environment solved in 585 episodes!	Average Score: 245.14
Episode 686	Average Score: 244.56
Environment solved in 586 episodes!	Average Score: 244.56
Episode 687	Average Score: 245.00
Environment solved in 587 episodes!	Average Score: 245.00
Episode 688	Average Score: 244.51
Environment solved in 588 episodes!	Average Score: 244.51
Episode 689	Average Score: 244.73
Environment solved in 589 episodes!	Average Score: 244.73
Episode 690	Average Score: 244.66
Environment solved in 590 episodes!	Average Score: 244.66
Episode 691	Average Score: 244.21
Environment solved in 591 episodes!	Average Score: 244.21
Episode 692	Average Score: 244.28
Environment solved in 592 episodes!	Average Score: 244.28
Episode 693	Average Score: 244.68
Environment solved in 593 episodes!	Average Score: 244.68
Episode 694	Average Score: 244.61
Environment solved in 594 episodes!	Average Score: 244.61
Episode 695	Average Score: 245.08
Environment solved in 595 episodes!	Average Score: 245.08
Episode 696	Average Score: 244.72
Environment solved in 596 episodes!	Average Score: 244.72
Episode 697	Average Score: 244.84
Environment solved in 597 episodes!	Average Score: 244.84
Episode 698	Average Score: 247.16
Environment solved in 598 episodes!	Average Score: 247.16
Episode 699	Average Score: 247.09
Environment solved in 599 episodes!	Average Score: 247.09
Episode 700	Average Score: 245.38

Environment solved in 600 episodes!	Average Score: 245.38
Episode 701	Average Score: 245.49
Environment solved in 601 episodes!	Average Score: 245.49
Episode 702	Average Score: 245.19
Environment solved in 602 episodes!	Average Score: 245.19
Episode 703	Average Score: 243.84
Environment solved in 603 episodes!	Average Score: 243.84
Episode 704	Average Score: 243.82
Environment solved in 604 episodes!	Average Score: 243.82
Episode 705	Average Score: 243.82
Environment solved in 605 episodes!	Average Score: 243.82
Episode 706	Average Score: 243.81
Environment solved in 606 episodes!	Average Score: 243.81
Episode 707	Average Score: 244.19
Environment solved in 607 episodes!	Average Score: 244.19
Episode 708	Average Score: 244.07
Environment solved in 608 episodes!	Average Score: 244.07
Episode 709	Average Score: 243.52
Environment solved in 609 episodes!	Average Score: 243.52
Episode 710	Average Score: 243.92
Environment solved in 610 episodes!	Average Score: 243.92
Episode 711	Average Score: 243.53
Environment solved in 611 episodes!	Average Score: 243.53
Episode 712	Average Score: 243.76
Environment solved in 612 episodes!	Average Score: 243.76
Episode 713	Average Score: 241.16
Environment solved in 613 episodes!	Average Score: 241.16
Episode 714	Average Score: 238.65
Environment solved in 614 episodes!	Average Score: 238.65
Episode 715	Average Score: 238.59
Environment solved in 615 episodes!	Average Score: 238.59
Episode 716	Average Score: 240.42
Environment solved in 616 episodes!	Average Score: 240.42
Episode 717	Average Score: 240.24
Environment solved in 617 episodes!	Average Score: 240.24
Episode 718	Average Score: 242.96
Environment solved in 618 episodes!	Average Score: 242.96
Episode 719	Average Score: 240.88
Environment solved in 619 episodes!	Average Score: 240.88
Episode 720	Average Score: 240.49
Environment solved in 620 episodes!	Average Score: 240.49
Episode 721	Average Score: 240.24
Environment solved in 621 episodes!	Average Score: 240.24
Episode 722	Average Score: 239.92
Environment solved in 622 episodes!	Average Score: 239.92
Episode 723	Average Score: 240.13
Environment solved in 623 episodes!	Average Score: 240.13
Episode 724	Average Score: 239.97
Environment solved in 624 episodes!	Average Score: 239.97
Episode 725	Average Score: 240.39
Environment solved in 625 episodes!	Average Score: 240.39
Episode 726	Average Score: 242.98
Environment solved in 626 episodes!	Average Score: 242.98
Episode 727	Average Score: 242.92
Environment solved in 627 episodes!	Average Score: 242.92
Episode 728	Average Score: 243.34
Environment solved in 628 episodes!	Average Score: 243.34
Episode 729	Average Score: 243.21
Environment solved in 629 episodes!	Average Score: 243.21
Episode 730	Average Score: 243.31
Environment solved in 630 episodes!	Average Score: 243.31
Episode 731	Average Score: 243.32
Environment solved in 631 episodes!	Average Score: 243.32
Episode 732	Average Score: 243.15
Environment solved in 632 episodes!	Average Score: 243.15
Episode 733	Average Score: 243.51
Environment solved in 633 episodes!	Average Score: 243.51
Episode 734	Average Score: 244.03
Environment solved in 634 episodes!	Average Score: 244.03
Episode 735	Average Score: 246.78
Environment solved in 635 episodes!	Average Score: 246.78
Episode 736	Average Score: 246.93
Environment solved in 636 episodes!	Average Score: 246.93
Episode 737	Average Score: 244.65
Environment solved in 637 episodes!	Average Score: 244.65
Episode 738	Average Score: 244.24
Environment solved in 638 episodes!	Average Score: 244.24
Episode 739	Average Score: 245.02
Environment solved in 639 episodes!	Average Score: 245.02
Episode 740	Average Score: 245.22
Environment solved in 640 episodes!	Average Score: 245.22
Episode 741	Average Score: 245.38
Environment solved in 641 episodes!	Average Score: 245.38
Episode 742	Average Score: 245.47
Environment solved in 642 episodes!	Average Score: 245.47
Episode 743	Average Score: 242.96
Environment solved in 643 episodes!	Average Score: 242.96
Episode 744	Average Score: 240.51
Environment solved in 644 episodes!	Average Score: 240.51
Episode 745	Average Score: 241.07
Environment solved in 645 episodes!	Average Score: 241.07
Episode 746	Average Score: 241.03
Environment solved in 646 episodes!	Average Score: 241.03
Episode 747	Average Score: 241.53
Environment solved in 647 episodes!	Average Score: 241.53
Episode 748	Average Score: 241.28
Environment solved in 648 episodes!	Average Score: 241.28
Episode 749	Average Score: 239.21
Environment solved in 649 episodes!	Average Score: 239.21
Episode 750	Average Score: 239.19
Environment solved in 650 episodes!	Average Score: 239.19
Episode 751	Average Score: 238.64
Environment solved in 651 episodes!	Average Score: 238.64
Episode 752	Average Score: 238.80
Environment solved in 652 episodes!	Average Score: 238.80
Episode 753	Average Score: 238.52
Environment solved in 653 episodes!	Average Score: 238.52
Episode 754	Average Score: 238.31
Environment solved in 654 episodes!	Average Score: 238.31
Episode 755	Average Score: 238.07
Environment solved in 655 episodes!	Average Score: 238.07
Episode 756	Average Score: 239.37
Environment solved in 656 episodes!	Average Score: 239.37
Episode 757	Average Score: 239.90
Environment solved in 657 episodes!	Average Score: 239.90
Episode 758	Average Score: 239.81
Environment solved in 658 episodes!	Average Score: 239.81
Episode 759	Average Score: 239.85
Environment solved in 659 episodes!	Average Score: 239.85
Episode 760	Average Score: 239.76
Environment solved in 660 episodes!	Average Score: 239.76
Episode 761	Average Score: 239.96
Environment solved in 661 episodes!	Average Score: 239.96
Episode 762	Average Score: 239.20
Environment solved in 662 episodes!	Average Score: 239.20
Episode 763	Average Score: 239.01
Environment solved in 663 episodes!	Average Score: 239.01
Episode 764	Average Score: 238.91
Environment solved in 664 episodes!	Average Score: 238.91
Episode 765	Average Score: 236.32
Environment solved in 665 episodes!	Average Score: 236.32
Episode 766	Average Score: 235.92
Environment solved in 666 episodes!	Average Score: 235.92
Episode 767	Average Score: 235.44
Environment solved in 667 episodes!	Average Score: 235.44
Episode 768	Average Score: 234.85
Environment solved in 668 episodes!	Average Score: 234.85
Episode 769	Average Score: 237.04
Environment solved in 669 episodes!	Average Score: 237.04
Episode 770	Average Score: 238.99
Environment solved in 670 episodes!	Average Score: 238.99
Episode 771	Average Score: 239.38
Environment solved in 671 episodes!	Average Score: 239.38
Episode 772	Average Score: 241.36
Environment solved in 672 episodes!	Average Score: 241.36
Episode 773	Average Score: 241.49
Environment solved in 673 episodes!	Average Score: 241.49
Episode 774	Average Score: 239.49
Environment solved in 674 episodes!	Average Score: 239.49
Episode 775	Average Score: 239.79
Environment solved in 675 episodes!	Average Score: 239.79
Episode 776	Average Score: 240.04
Environment solved in 676 episodes!	Average Score: 240.04
Episode 777	Average Score: 236.55
Environment solved in 677 episodes!	Average Score: 236.55
Episode 778	Average Score: 236.51
Environment solved in 678 episodes!	Average Score: 236.51
Episode 779	Average Score: 234.28
Environment solved in 679 episodes!	Average Score: 234.28
Episode 780	Average Score: 234.86
Environment solved in 680 episodes!	Average Score: 234.86
Episode 781	Average Score: 235.12
Environment solved in 681 episodes!	Average Score: 235.12
Episode 782	Average Score: 235.12
Environment solved in 682 episodes!	Average Score: 235.12
Episode 783	Average Score: 234.33
Environment solved in 683 episodes!	Average Score: 234.33
Episode 784	Average Score: 231.66
Environment solved in 684 episodes!	Average Score: 231.66
Episode 785	Average Score: 231.42
Environment solved in 685 episodes!	Average Score: 231.42
Episode 786	Average Score: 231.79
Environment solved in 686 episodes!	Average Score: 231.79
Episode 787	Average Score: 231.41
Environment solved in 687 episodes!	Average Score: 231.41
Episode 788	Average Score: 231.73
Environment solved in 688 episodes!	Average Score: 231.73
Episode 789	Average Score: 231.58
Environment solved in 689 episodes!	Average Score: 231.58
Episode 790	Average Score: 233.63
Environment solved in 690 episodes!	Average Score: 233.63
Episode 791	Average Score: 231.11
Environment solved in 691 episodes!	Average Score: 231.11
Episode 792	Average Score: 230.89
Environment solved in 692 episodes!	Average Score: 230.89
Episode 793	Average Score: 230.58
Environment solved in 693 episodes!	Average Score: 230.58
Episode 794	Average Score: 230.56
Environment solved in 694 episodes!	Average Score: 230.56
Episode 795	Average Score: 229.39
Environment solved in 695 episodes!	Average Score: 229.39
Episode 796	Average Score: 229.98
Environment solved in 696 episodes!	Average Score: 229.98
Episode 797	Average Score: 229.44
Environment solved in 697 episodes!	Average Score: 229.44
Episode 798	Average Score: 229.19
Environment solved in 698 episodes!	Average Score: 229.19
Episode 799	Average Score: 229.37
Environment solved in 699 episodes!	Average Score: 229.37
Episode 800	Average Score: 231.36

Environment solved in 700 episodes!	Average Score: 231.36
Episode 801	Average Score: 231.10
Environment solved in 701 episodes!	Average Score: 231.10
Episode 802	Average Score: 230.93
Environment solved in 702 episodes!	Average Score: 230.93
Episode 803	Average Score: 231.64
Environment solved in 703 episodes!	Average Score: 231.64
Episode 804	Average Score: 231.55
Environment solved in 704 episodes!	Average Score: 231.55
Episode 805	Average Score: 231.95
Environment solved in 705 episodes!	Average Score: 231.95
Episode 806	Average Score: 231.97
Environment solved in 706 episodes!	Average Score: 231.97
Episode 807	Average Score: 232.34
Environment solved in 707 episodes!	Average Score: 232.34
Episode 808	Average Score: 229.78
Environment solved in 708 episodes!	Average Score: 229.78
Episode 809	Average Score: 229.94
Environment solved in 709 episodes!	Average Score: 229.94
Episode 810	Average Score: 229.80
Environment solved in 710 episodes!	Average Score: 229.80
Episode 811	Average Score: 229.89
Environment solved in 711 episodes!	Average Score: 229.89
Episode 812	Average Score: 230.02
Environment solved in 712 episodes!	Average Score: 230.02
Episode 813	Average Score: 232.20
Environment solved in 713 episodes!	Average Score: 232.20
Episode 814	Average Score: 234.81
Environment solved in 714 episodes!	Average Score: 234.81
Episode 815	Average Score: 234.51
Environment solved in 715 episodes!	Average Score: 234.51
Episode 816	Average Score: 234.26
Environment solved in 716 episodes!	Average Score: 234.26
Episode 817	Average Score: 234.06
Environment solved in 717 episodes!	Average Score: 234.06
Episode 818	Average Score: 233.66
Environment solved in 718 episodes!	Average Score: 233.66
Episode 819	Average Score: 235.65
Environment solved in 719 episodes!	Average Score: 235.65
Episode 820	Average Score: 235.93
Environment solved in 720 episodes!	Average Score: 235.93
Episode 821	Average Score: 236.06
Environment solved in 721 episodes!	Average Score: 236.06
Episode 822	Average Score: 236.12
Environment solved in 722 episodes!	Average Score: 236.12
Episode 823	Average Score: 236.05
Environment solved in 723 episodes!	Average Score: 236.05
Episode 824	Average Score: 236.60
Environment solved in 724 episodes!	Average Score: 236.60
Episode 825	Average Score: 236.59
Environment solved in 725 episodes!	Average Score: 236.59
Episode 826	Average Score: 236.25
Environment solved in 726 episodes!	Average Score: 236.25
Episode 827	Average Score: 235.19
Environment solved in 727 episodes!	Average Score: 235.19
Episode 828	Average Score: 235.25
Environment solved in 728 episodes!	Average Score: 235.25
Episode 829	Average Score: 235.34
Environment solved in 729 episodes!	Average Score: 235.34
Episode 830	Average Score: 235.36
Environment solved in 730 episodes!	Average Score: 235.36
Episode 831	Average Score: 235.08
Environment solved in 731 episodes!	Average Score: 235.08
Episode 832	Average Score: 234.64
Environment solved in 732 episodes!	Average Score: 234.64
Episode 833	Average Score: 234.62
Environment solved in 733 episodes!	Average Score: 234.62
Episode 834	Average Score: 234.07
Environment solved in 734 episodes!	Average Score: 234.07
Episode 835	Average Score: 234.28
Environment solved in 735 episodes!	Average Score: 234.28
Episode 836	Average Score: 234.08
Environment solved in 736 episodes!	Average Score: 234.08
Episode 837	Average Score: 236.58
Environment solved in 737 episodes!	Average Score: 236.58
Episode 838	Average Score: 236.59
Environment solved in 738 episodes!	Average Score: 236.59
Episode 839	Average Score: 236.39
Environment solved in 739 episodes!	Average Score: 236.39
Episode 840	Average Score: 236.63
Environment solved in 740 episodes!	Average Score: 236.63
Episode 841	Average Score: 236.29
Environment solved in 741 episodes!	Average Score: 236.29
Episode 842	Average Score: 236.12
Environment solved in 742 episodes!	Average Score: 236.12
Episode 843	Average Score: 238.20
Environment solved in 743 episodes!	Average Score: 238.20
Episode 844	Average Score: 240.64
Environment solved in 744 episodes!	Average Score: 240.64
Episode 845	Average Score: 240.49
Environment solved in 745 episodes!	Average Score: 240.49
Episode 846	Average Score: 240.44
Environment solved in 746 episodes!	Average Score: 240.44
Episode 847	Average Score: 239.69
Environment solved in 747 episodes!	Average Score: 239.69
Episode 848	Average Score: 239.97
Environment solved in 748 episodes!	Average Score: 239.97
Episode 849	Average Score: 242.03
Environment solved in 749 episodes!	Average Score: 242.03
Episode 850	Average Score: 241.93
Environment solved in 750 episodes!	Average Score: 241.93
Episode 851	Average Score: 242.20
Environment solved in 751 episodes!	Average Score: 242.20
Episode 852	Average Score: 239.71
Environment solved in 752 episodes!	Average Score: 239.71
Episode 853	Average Score: 239.92
Environment solved in 753 episodes!	Average Score: 239.92
Episode 854	Average Score: 240.02
Environment solved in 754 episodes!	Average Score: 240.02
Episode 855	Average Score: 239.94
Environment solved in 755 episodes!	Average Score: 239.94
Episode 856	Average Score: 239.74
Environment solved in 756 episodes!	Average Score: 239.74
Episode 857	Average Score: 239.24
Environment solved in 757 episodes!	Average Score: 239.24
Episode 858	Average Score: 239.20
Environment solved in 758 episodes!	Average Score: 239.20
Episode 859	Average Score: 239.01
Environment solved in 759 episodes!	Average Score: 239.01
Episode 860	Average Score: 238.92
Environment solved in 760 episodes!	Average Score: 238.92
Episode 861	Average Score: 238.25
Environment solved in 761 episodes!	Average Score: 238.25
Episode 862	Average Score: 238.76
Environment solved in 762 episodes!	Average Score: 238.76
Episode 863	Average Score: 238.86
Environment solved in 763 episodes!	Average Score: 238.86
Episode 864	Average Score: 235.36
Environment solved in 764 episodes!	Average Score: 235.36
Episode 865	Average Score: 238.71
Environment solved in 765 episodes!	Average Score: 238.71
Episode 866	Average Score: 238.17
Environment solved in 766 episodes!	Average Score: 238.17
Episode 867	Average Score: 238.59
Environment solved in 767 episodes!	Average Score: 238.59
Episode 868	Average Score: 238.76
Environment solved in 768 episodes!	Average Score: 238.76
Episode 869	Average Score: 238.98
Environment solved in 769 episodes!	Average Score: 238.98
Episode 870	Average Score: 234.84
Environment solved in 770 episodes!	Average Score: 234.84
Episode 871	Average Score: 234.65
Environment solved in 771 episodes!	Average Score: 234.65
Episode 872	Average Score: 234.70
Environment solved in 772 episodes!	Average Score: 234.70
Episode 873	Average Score: 234.63
Environment solved in 773 episodes!	Average Score: 234.63
Episode 874	Average Score: 233.75
Environment solved in 774 episodes!	Average Score: 233.75
Episode 875	Average Score: 233.00
Environment solved in 775 episodes!	Average Score: 233.00
Episode 876	Average Score: 231.33
Environment solved in 776 episodes!	Average Score: 231.33
Episode 877	Average Score: 234.34
Environment solved in 777 episodes!	Average Score: 234.34
Episode 878	Average Score: 234.77
Environment solved in 778 episodes!	Average Score: 234.77
Episode 879	Average Score: 237.52
Environment solved in 779 episodes!	Average Score: 237.52
Episode 880	Average Score: 237.53
Environment solved in 780 episodes!	Average Score: 237.53
Episode 881	Average Score: 235.54
Environment solved in 781 episodes!	Average Score: 235.54
Episode 882	Average Score: 235.23
Environment solved in 782 episodes!	Average Score: 235.23
Episode 883	Average Score: 235.89
Environment solved in 783 episodes!	Average Score: 235.89
Episode 884	Average Score: 236.36
Environment solved in 784 episodes!	Average Score: 236.36
Episode 885	Average Score: 235.00
Environment solved in 785 episodes!	Average Score: 235.00
Episode 886	Average Score: 234.99
Environment solved in 786 episodes!	Average Score: 234.99
Episode 887	Average Score: 235.28
Environment solved in 787 episodes!	Average Score: 235.28
Episode 888	Average Score: 235.05
Environment solved in 788 episodes!	Average Score: 235.05
Episode 889	Average Score: 235.07
Environment solved in 789 episodes!	Average Score: 235.07
Episode 890	Average Score: 234.85
Environment solved in 790 episodes!	Average Score: 234.85
Episode 891	Average Score: 237.76
Environment solved in 791 episodes!	Average Score: 237.76
Episode 892	Average Score: 238.06
Environment solved in 792 episodes!	Average Score: 238.06
Episode 893	Average Score: 238.09
Environment solved in 793 episodes!	Average Score: 238.09
Episode 894	Average Score: 238.13
Environment solved in 794 episodes!	Average Score: 238.13
Episode 895	Average Score: 239.16
Environment solved in 795 episodes!	Average Score: 239.16
Episode 896	Average Score: 238.96
Environment solved in 796 episodes!	Average Score: 238.96
Episode 897	Average Score: 239.13
Environment solved in 797 episodes!	Average Score: 239.13
Episode 898	Average Score: 239.82
Environment solved in 798 episodes!	Average Score: 239.82
Episode 899	Average Score: 237.29
Environment solved in 799 episodes!	Average Score: 237.29
Episode 900	Average Score: 237.02

Environment solved in 800 episodes!	Average Score: 237.02
Episode 901	Average Score: 237.54
Environment solved in 801 episodes!	Average Score: 237.54
Episode 902	Average Score: 237.71
Environment solved in 802 episodes!	Average Score: 237.71
Episode 903	Average Score: 237.89
Environment solved in 803 episodes!	Average Score: 237.89
Episode 904	Average Score: 238.19
Environment solved in 804 episodes!	Average Score: 238.19
Episode 905	Average Score: 237.80
Environment solved in 805 episodes!	Average Score: 237.80
Episode 906	Average Score: 237.74
Environment solved in 806 episodes!	Average Score: 237.74
Episode 907	Average Score: 237.69
Environment solved in 807 episodes!	Average Score: 237.69
Episode 908	Average Score: 240.28
Environment solved in 808 episodes!	Average Score: 240.28
Episode 909	Average Score: 240.53
Environment solved in 809 episodes!	Average Score: 240.53
Episode 910	Average Score: 240.20
Environment solved in 810 episodes!	Average Score: 240.20
Episode 911	Average Score: 239.97
Environment solved in 811 episodes!	Average Score: 239.97
Episode 912	Average Score: 239.90
Environment solved in 812 episodes!	Average Score: 239.90
Episode 913	Average Score: 238.85
Environment solved in 813 episodes!	Average Score: 238.85
Episode 914	Average Score: 236.68
Environment solved in 814 episodes!	Average Score: 236.68
Episode 915	Average Score: 234.39
Environment solved in 815 episodes!	Average Score: 234.39
Episode 916	Average Score: 234.59
Environment solved in 816 episodes!	Average Score: 234.59
Episode 917	Average Score: 234.79
Environment solved in 817 episodes!	Average Score: 234.79
Episode 918	Average Score: 234.77
Environment solved in 818 episodes!	Average Score: 234.77
Episode 919	Average Score: 235.05
Environment solved in 819 episodes!	Average Score: 235.05
Episode 920	Average Score: 234.59
Environment solved in 820 episodes!	Average Score: 234.59
Episode 921	Average Score: 234.51
Environment solved in 821 episodes!	Average Score: 234.51
Episode 922	Average Score: 234.76
Environment solved in 822 episodes!	Average Score: 234.76
Episode 923	Average Score: 234.99
Environment solved in 823 episodes!	Average Score: 234.99
Episode 924	Average Score: 234.82
Environment solved in 824 episodes!	Average Score: 234.82
Episode 925	Average Score: 234.81
Environment solved in 825 episodes!	Average Score: 234.81
Episode 926	Average Score: 235.18
Environment solved in 826 episodes!	Average Score: 235.18
Episode 927	Average Score: 236.36
Environment solved in 827 episodes!	Average Score: 236.36
Episode 928	Average Score: 233.74
Environment solved in 828 episodes!	Average Score: 233.74
Episode 929	Average Score: 233.54
Environment solved in 829 episodes!	Average Score: 233.54
Episode 930	Average Score: 233.18
Environment solved in 830 episodes!	Average Score: 233.18
Episode 931	Average Score: 233.12
Environment solved in 831 episodes!	Average Score: 233.12
Episode 932	Average Score: 233.35
Environment solved in 832 episodes!	Average Score: 233.35
Episode 933	Average Score: 233.34
Environment solved in 833 episodes!	Average Score: 233.34
Episode 934	Average Score: 233.78
Environment solved in 834 episodes!	Average Score: 233.78
Episode 935	Average Score: 233.24
Environment solved in 835 episodes!	Average Score: 233.24
Episode 936	Average Score: 230.74
Environment solved in 836 episodes!	Average Score: 230.74
Episode 937	Average Score: 228.01
Environment solved in 837 episodes!	Average Score: 228.01
Episode 938	Average Score: 227.65
Environment solved in 838 episodes!	Average Score: 227.65
Episode 939	Average Score: 227.56
Environment solved in 839 episodes!	Average Score: 227.56
Episode 940	Average Score: 227.22
Environment solved in 840 episodes!	Average Score: 227.22
Episode 941	Average Score: 227.49
Environment solved in 841 episodes!	Average Score: 227.49
Episode 942	Average Score: 227.77
Environment solved in 842 episodes!	Average Score: 227.77
Episode 943	Average Score: 227.94
Environment solved in 843 episodes!	Average Score: 227.94
Episode 944	Average Score: 228.43
Environment solved in 844 episodes!	Average Score: 228.43
Episode 945	Average Score: 228.88
Environment solved in 845 episodes!	Average Score: 228.88
Episode 946	Average Score: 226.45
Environment solved in 846 episodes!	Average Score: 226.45
Episode 947	Average Score: 226.73
Environment solved in 847 episodes!	Average Score: 226.73
Episode 948	Average Score: 226.33
Environment solved in 848 episodes!	Average Score: 226.33
Episode 949	Average Score: 224.24
Environment solved in 849 episodes!	Average Score: 224.24
Episode 950	Average Score: 224.45
Environment solved in 850 episodes!	Average Score: 224.45
Episode 951	Average Score: 224.23
Environment solved in 851 episodes!	Average Score: 224.23
Episode 952	Average Score: 226.00
Environment solved in 852 episodes!	Average Score: 226.00
Episode 953	Average Score: 224.66
Environment solved in 853 episodes!	Average Score: 224.66
Episode 954	Average Score: 224.69
Environment solved in 854 episodes!	Average Score: 224.69
Episode 955	Average Score: 224.81
Environment solved in 855 episodes!	Average Score: 224.81
Episode 956	Average Score: 224.86
Environment solved in 856 episodes!	Average Score: 224.86
Episode 957	Average Score: 224.97
Environment solved in 857 episodes!	Average Score: 224.97
Episode 958	Average Score: 225.05
Environment solved in 858 episodes!	Average Score: 225.05
Episode 959	Average Score: 225.25
Environment solved in 859 episodes!	Average Score: 225.25
Episode 960	Average Score: 225.24
Environment solved in 860 episodes!	Average Score: 225.24
Episode 961	Average Score: 225.78
Environment solved in 861 episodes!	Average Score: 225.78
Episode 962	Average Score: 225.35
Environment solved in 862 episodes!	Average Score: 225.35
Episode 963	Average Score: 225.20
Environment solved in 863 episodes!	Average Score: 225.20
Episode 964	Average Score: 228.78
Environment solved in 864 episodes!	Average Score: 228.78
Episode 965	Average Score: 228.73
Environment solved in 865 episodes!	Average Score: 228.73
Episode 966	Average Score: 229.22
Environment solved in 866 episodes!	Average Score: 229.22
Episode 967	Average Score: 229.29
Environment solved in 867 episodes!	Average Score: 229.29
Episode 968	Average Score: 229.61
Environment solved in 868 episodes!	Average Score: 229.61
Episode 969	Average Score: 229.22
Environment solved in 869 episodes!	Average Score: 229.22
Episode 970	Average Score: 233.51
Environment solved in 870 episodes!	Average Score: 233.51
Episode 971	Average Score: 233.63
Environment solved in 871 episodes!	Average Score: 233.63
Episode 972	Average Score: 233.91
Environment solved in 872 episodes!	Average Score: 233.91
Episode 973	Average Score: 233.87
Environment solved in 873 episodes!	Average Score: 233.87
Episode 974	Average Score: 236.77
Environment solved in 874 episodes!	Average Score: 236.77
Episode 975	Average Score: 237.12
Environment solved in 875 episodes!	Average Score: 237.12
Episode 976	Average Score: 238.56
Environment solved in 876 episodes!	Average Score: 238.56
Episode 977	Average Score: 232.96
Environment solved in 877 episodes!	Average Score: 232.96
Episode 978	Average Score: 232.54
Environment solved in 878 episodes!	Average Score: 232.54
Episode 979	Average Score: 232.63
Environment solved in 879 episodes!	Average Score: 232.63
Episode 980	Average Score: 232.67
Environment solved in 880 episodes!	Average Score: 232.67
Episode 981	Average Score: 234.53
Environment solved in 881 episodes!	Average Score: 234.53
Episode 982	Average Score: 234.60
Environment solved in 882 episodes!	Average Score: 234.60
Episode 983	Average Score: 234.49
Environment solved in 883 episodes!	Average Score: 234.49
Episode 984	Average Score: 236.67
Environment solved in 884 episodes!	Average Score: 236.67
Episode 985	Average Score: 237.43
Environment solved in 885 episodes!	Average Score: 237.43
Episode 986	Average Score: 237.31
Environment solved in 886 episodes!	Average Score: 237.31
Episode 987	Average Score: 237.07
Environment solved in 887 episodes!	Average Score: 237.07
Episode 988	Average Score: 237.36
Environment solved in 888 episodes!	Average Score: 237.36
Episode 989	Average Score: 237.42
Environment solved in 889 episodes!	Average Score: 237.42
Episode 990	Average Score: 237.64
Environment solved in 890 episodes!	Average Score: 237.64
Episode 991	Average Score: 237.14
Environment solved in 891 episodes!	Average Score: 237.14
Episode 992	Average Score: 236.83
Environment solved in 892 episodes!	Average Score: 236.83
Episode 993	Average Score: 236.46
Environment solved in 893 episodes!	Average Score: 236.46
Episode 994	Average Score: 236.22
Environment solved in 894 episodes!	Average Score: 236.22
Episode 995	Average Score: 235.79
Environment solved in 895 episodes!	Average Score: 235.79
Episode 996	Average Score: 235.55
Environment solved in 896 episodes!	Average Score: 235.55
Episode 997	Average Score: 235.79
Environment solved in 897 episodes!	Average Score: 235.79
Episode 998	Average Score: 235.44
Environment solved in 898 episodes!	Average Score: 235.44
Episode 999	Average Score: 237.33
Environment solved in 899 episodes!	Average Score: 237.33
Episode 1000	Average Score: 237.13

Environment solved in 900 episodes!	Average Score: 237.13

scores_ddqn_np = np.array(scores_ddqn)
np.savetxt("double_dqn/scores.txt",scores_ddqn_np)

def moving_average(a, n=100) :
    ret = np.cumsum(a, dtype=float)
    ret[n:] = ret[n:] - ret[:-n]
    return ret[n - 1:] / n

scores_ma_ddqn=moving_average(scores_ddqn,n=100)

# plot the scores
fig = plt.figure()
ax = fig.add_subplot(111)
plt.plot(np.arange(len(scores_ma_ddqn)), scores_ma_ddqn)
plt.ylabel('Score')
plt.xlabel('Episode')
plt.show()

png

Let’s compare with the DQN?

scores_dqn_with_target=np.loadtxt("dqn_with_target/scores.txt")
scores_dqn_no_target=np.loadtxt("dqn_no_target/scores.txt")

scores_ma_dqn_no_target=moving_average(scores_dqn_no_target,n=100)
scores_ma_dqn_with_target=moving_average(scores_dqn_with_target,n=100)

# plot the scores
plt.plot(np.arange(len(scores_ma_ddqn)), scores_ma_ddqn,label="DDQN")
plt.plot(np.arange(len(scores_ma_dqn_no_target)), scores_ma_dqn_no_target,label="DQN without target network")
plt.plot(np.arange(len(scores_ma_dqn_with_target)), scores_ma_dqn_with_target,label="DQN with target network")
plt.ylabel('Score')
plt.xlabel('Episode ')
plt.legend()

<matplotlib.legend.Legend at 0x7f1347666630>

png

We see, Double Deep Q Network outscores other two!!! But is there still a place to improve?!! Obviously! Next blog comes on Duelling Double Deep Q Network!