深度强化学习：Deep Q-Learning
在前两篇文章强化学习基础：基本概念和动态规划和强化学习基础：蒙特卡罗和时序差分中介绍的强化学习的三种经典方法（动态规划、蒙特卡罗以及时序差分）适用于有限的状态集合$\mathcal{S}$，以时序差分中的Q-Learning算法为例，一般来说使用n行(n = number of states)和m列(m= number of actions)的矩阵(Q table)来储存action-value function的值，如下图所示：
对于连续的状态集合$\mathcal{S}$，上述方法就不能适用了，这时可以引入神经网络来估计Q的值，即Deep Q-Learning，如下图所示：
接下来介绍Deep Q-Learning中常用的几种技巧，用于提升学习效果：
- Stack States：对于连续的状态集合，单个状态不能很好地描述整体的状况。例如下图所示，要判断黑色方块的移动方向，仅凭一副图像是无法判断的，需要连续的多幅图像才能判断出黑色方块在向右移动
- Experience Replay：如下图所示，防止算法在训练过程中忘记了之前场景获得的经验，创建一个Replay Buffer，不断回放之前的场景对算法进行训练；另一方面，相邻的场景之间(例如$[S_{t},A_{t},R_{t+1},S_{t+1}]$与$[S_{t+1},A_{t+1},R_{t+2},S_{t+2}]$)有着一定的相关性,为了防止算法被固定在某些特定的状态空间，从Replay Buffer中随机抽样选取场景进行训练（打乱场景之间的顺序，减少相邻场景的相关性）
- Fixed Q-targets：针对Deep Q-Learning中计算Q值的神经网络的权重系数的更新，有公式如左图所示，此时将TD target近似为了$q_{\pi}(S,A)$的真值，但是当不断更新权重系数时TD target也是不断变化的，这就会使得在训练过程中$q_{\pi}(S,A)$的估计值要接近一个不断变化的值，加大了训练难度，减小了训练效率。解决方案如右图所示，使用相对固定的参数来估计TD target
- Double DQNs：解决TD target对$q_{\pi}(S,A)$的真值可能高估的问题，方案是在计算TD target时使用两个不同的神经网络，将动作$a$的选择过程与TD target的计算过程进行分割。如果和Fixed Q-targets结合起来，可以直接使用$w$和$w^{-}$这两组参数（如下图所示）
- Dueling DQN：如下图所示，相对于直接计算action-value function $Q(s,a)$，将$Q(s,a)$分解为state-value function $V(s)$与advantage-value function $A(s,a)$之和，这样做的原因是多数状态下采取何种行动对$Q(s,a)$的值影响不大，适合直接对状态函数$V(s)$进行估计，再叠加上不同行动对其的影响。在实际计算中，为了使得从$Q(s,a)$能够唯一确定$V(s)$与$A(s,a)$，可以令$A(s,a)$的均值（即$\frac{1}{|\mathcal{A}|}\sum_{a^{\prime}} A(s,a^{\prime})$）为0
-
Prioritized Experience Replay：如下图所示，对Replay Buffer中的每个场景加入一个优先级，一个场景的TD error越大，它对应的优先级就越大。其中$e$是一个大于0的常数，防止抽样概率为0；$a$控制按优先级抽样和均匀抽样的比重，$a=1$时完全按优先级抽样，$a=0$时退化为均匀抽样；另外由于是按照优先级进行抽样，还需要改写神经网络中权重系数的更新规则，针对优先级高的场景减少权重更新的步长，使权重更新的过程与场景出现的真实概率一致（特别是在训练的后期，通过参数$b$控制），避免过拟合高优先级的场景
代码实现
使用vizdoom强化学习环境，以其中的一个任务为基础进行训练，该任务要确保玩家活着走到目标位置（要做到这一点，路上要躲避敌人的射击或者杀死敌人，否则不可能成功）。该任务每步的奖励与玩家和目标的距离变化有关（+dX for getting closer, -dX for getting further），此外若玩家死亡会有-100的惩罚。
值得注意的是在代码中使用SumTree(二叉树，每个父节点的值是两个子节点的值的和)这一数据结构来存储Replay Buffer中的Priority，具体结构如下图所示，这样做的目的是方便按优先级进行抽样，可以使得每次抽样以及更新优先级的计算均为$O(\ln{n})$。在抽样过程中抽得的叶子节点$x$满足$P(x\leq{k})=\frac{\sum_{i=1}^{k}Priority_i}{\sum_{i=1}^{n}Priority_i},\text{ }k\in\{1,2,\cdots,n\}$
import tensorflow as tf # Deep Learning library import numpy as np # Handle matrices from vizdoom import * # Doom Environment import random # Handling random number generation import time # Handling time calculation from skimage import transform# Help us to preprocess the frames from collections import deque# Ordered collection with ends import matplotlib.pyplot as plt # Display graphs import warnings # This ignore all the warning messages that are normally printed during the training because of skiimage warnings.filterwarnings('ignore') ### Here we create our environment def create_environment(): game = DoomGame() # Load the correct configuration game.load_config("deadly_corridor.cfg") # Load the correct scenario (in our case deadly_corridor scenario) game.set_doom_scenario_path("deadly_corridor.wad") game.init() # Create an hot encoded version of our actions (7 possible actions) possible_actions = np.identity(7,dtype=int).tolist() return game, possible_actions game, possible_actions = create_environment() ### Preprocess(reduce the complexity of states and the training time) def preprocess_frame(frame): # Grayscale frame(color not add important information, already done by the config file) # Crop the screen (remove part that contains no information) cropped_frame = frame[15:-5,20:-20] #[Up: Down, Left: right] # Normalize Pixel Values normalized_frame = cropped_frame/255.0 # Resize preprocessed_frame = transform.resize(normalized_frame, [100,120]) return preprocessed_frame # 100x120x1 frame ### Stack frames stack_size = 4 #stack 4 frames # Initialize deque with zero-images, one array for each image stacked_frames = deque([np.zeros((100,120), dtype=np.int) for i in range(stack_size)], maxlen=4) def stack_frames(stacked_frames, state, is_new_episode): frame = preprocess_frame(state) #preprocess frame if is_new_episode: # Clear our stacked_frames stacked_frames = deque([np.zeros((100,120), dtype=np.int) for i in range(stack_size)], maxlen=4) # Because we're in a new episode, copy the same frame 4x stacked_frames.append(frame) stacked_frames.append(frame) stacked_frames.append(frame) stacked_frames.append(frame) # Stack the frames stacked_state = np.stack(stacked_frames, axis=2) else: # Append frame to deque, automatically removes the oldest frame stacked_frames.append(frame) # Build the stacked state stacked_state = np.stack(stacked_frames, axis=2) return stacked_state, stacked_frames ### Set the hyperparameters # MODEL HYPERPARAMETERS state_size = [100,120,4] # Our input is a stack of 4 frames hence 100x120x4 (Width, height, channels) action_size = game.get_available_buttons_size() # 7 possible actions learning_rate = 0.00025 # Alpha (i.e., learning rate) # TRAINING HYPERPARAMETERS total_episodes = 5000 # Total episodes for training max_steps = 5000 # Max possible steps in an episode batch_size = 64 # FIXED Q TARGETS HYPERPARAMETERS max_tau = 10000 # The number of steps where we update our target network # EXPLORATION HYPERPARAMETERS for epsilon greedy strategy explore_start = 1.0 # exploration probability at start explore_stop = 0.01 # minimum exploration probability decay_rate = 0.00005 # exponential decay rate for exploration prob # Q LEARNING hyperparameters gamma = 0.95 # Discounting rate # MEMORY HYPERPARAMETERS(If you have GPU change to 1 million) pretrain_length = 100000 # Number of experiences stored in the Memory when initialized for the first time memory_size = 100000 # Number of experiences the Memory can keep # MODIFY THIS TO FALSE IF YOU JUST WANT TO SEE THE TRAINED AGENT training = True ### Set up Deep Q network and Target network (both are Dueling Network) class DDDQNNet: def __init__(self, state_size, action_size, learning_rate, name): self.state_size = state_size self.action_size = action_size self.learning_rate = learning_rate self.name = name # use tf.variable_scope to know which network we're using (DQN or target_net) # it will be useful when we will update our w- parameters (by copy the DQN parameters) with tf.variable_scope(self.name): # create the placeholders self.inputs_ = tf.placeholder(tf.float32, [None, *state_size], name="inputs") #[None,100,120,4] self.ISWeights_ = tf.placeholder(tf.float32, [None,1], name='IS_weights') self.actions_ = tf.placeholder(tf.float32, [None, action_size], name="actions_") # Remember that target_Q is the R(s,a) + max Qhat(s', a') self.target_Q = tf.placeholder(tf.float32, [None], name="target") # first conv layer self.conv1 = tf.layers.conv2d(inputs = self.inputs_, filters = 32, kernel_size = [8,8], \ strides = [4,4], padding = "VALID", name = "conv1", \ kernel_initializer=tf.contrib.layers.xavier_initializer_conv2d()) self.conv1_out = tf.nn.elu(self.conv1, name="conv1_out") # second conv layer self.conv2 = tf.layers.conv2d(inputs = self.conv1_out, filters = 64, kernel_size = [4,4], \ strides = [2,2], padding = "VALID", name = "conv2", \ kernel_initializer=tf.contrib.layers.xavier_initializer_conv2d()) self.conv2_out = tf.nn.elu(self.conv2, name="conv2_out") # third conv layer self.conv3 = tf.layers.conv2d(inputs = self.conv2_out, filters = 128, kernel_size = [4,4], \ strides = [2,2], padding = "VALID", name = "conv3", \ kernel_initializer=tf.contrib.layers.xavier_initializer_conv2d()) self.conv3_out = tf.nn.elu(self.conv3, name="conv3_out") self.flatten = tf.layers.flatten(self.conv3_out) # Here we separate into two streams (Dueling Network) # The one that calculate V(s) self.value_fc = tf.layers.dense(inputs = self.flatten, units = 512, activation = tf.nn.elu, \ kernel_initializer=tf.contrib.layers.xavier_initializer(), \ name="value_fc") self.value = tf.layers.dense(inputs = self.value_fc, units = 1, activation = None, \ kernel_initializer=tf.contrib.layers.xavier_initializer(), \ name="value") # The one that calculate A(s,a) self.advantage_fc = tf.layers.dense(inputs = self.flatten, units = 512, activation = tf.nn.elu, \ kernel_initializer=tf.contrib.layers.xavier_initializer(), \ name="advantage_fc") self.advantage = tf.layers.dense(inputs = self.advantage_fc, units = self.action_size, activation = None, \ kernel_initializer=tf.contrib.layers.xavier_initializer(), \ name="advantages") # Agregating layer # Q(s,a) = V(s) + (A(s,a) - 1/|A| * sum A(s,a')) self.output = self.value + tf.subtract(self.advantage, tf.reduce_mean(self.advantage, axis=1, keepdims=True)) # Predicted Q value self.Q = tf.reduce_sum(tf.multiply(self.output, self.actions_), axis=1) # For computing priority and updating Sumtree self.absolute_errors = tf.abs(self.target_Q - self.Q) # The loss is modified because of Priority Experience Replay self.loss = tf.reduce_mean(self.ISWeights_ * tf.squared_difference(self.target_Q, self.Q)) self.optimizer = tf.train.RMSPropOptimizer(self.learning_rate).minimize(self.loss) # Reset the graph tf.reset_default_graph() # Instantiate the DQNetwork DQNetwork = DDDQNNet(state_size, action_size, learning_rate, name="DQNetwork") # Instantiate the target network TargetNetwork = DDDQNNet(state_size, action_size, learning_rate, name="TargetNetwork") ### Data Struture to store experience and priority(SumTree) class SumTree(object): data_pointer = 0 # Here we initialize the tree with all nodes = 0, and initialize the data with all values = 0 def __init__(self, capacity): self.capacity = capacity # Number of leaf nodes (final nodes) that contains experiences # Generate the tree with all nodes values = 0 # Parent nodes = capacity - 1, Leaf nodes = capacity self.tree = np.zeros(2 * capacity - 1) # Contains the experiences (so the size of data is capacity) self.data = np.zeros(capacity, dtype=object) # Here we add our priority score in the sumtree leaf and add the experience in data def add(self, priority, data): # Look at what index we want to put the experience tree_index = self.data_pointer + self.capacity - 1 # the leaves from left to right self.data[self.data_pointer] = data # Update data frames self.update(tree_index, priority) # Update the leaf self.data_pointer += 1 # Add 1 to data_pointer # If we're above the capacity, you go back to first index (we overwrite) if self.data_pointer >= self.capacity: self.data_pointer = 0 # Update the leaf priority score and propagate the change through tree def update(self, tree_index, priority): # Change = new priority score - former priority score change = priority - self.tree[tree_index] self.tree[tree_index] = priority # Propagate the change through tree while tree_index != 0: tree_index = (tree_index - 1) // 2 self.tree[tree_index] += change # Here we get the leaf and associated experience # the returned index is the smallest index satisfying: sum(leaf priority) >= v for leaf index <= returned index def get_leaf(self, v): parent_index = 0 while True: left_child_index = 2 * parent_index + 1 right_child_index = left_child_index + 1 # If we reach bottom, end the search if left_child_index >= len(self.tree): leaf_index = parent_index break else: # downward search if v <= self.tree[left_child_index]: parent_index = left_child_index else: v -= self.tree[left_child_index] parent_index = right_child_index data_index = leaf_index - self.capacity + 1 return leaf_index, self.tree[leaf_index], self.data[data_index] @property def total_priority(self): return self.tree[0] # Returns the root node ### Create Replay Buffer and Prioritized Experience Replay class Memory(object): PER_e = 0.01 # Hyperparameter that we use to avoid some experiences to have 0 probability of being taken PER_a = 0.6 # Hyperparameter that we use to make a tradeoff between taking only exp with high priority and sampling randomly PER_b = 0.4 # importance-sampling, from initial value increasing to 1 PER_b_increment_per_sampling = 0.001 absolute_error_upper = 1. # clipped abs error def __init__(self, capacity): # Making the tree self.tree = SumTree(capacity) # Store a new experience in our tree # Each new experience have a score of max_prority (it will be then improved when we use this exp to train our DDQN) def store(self, experience): # Find the max priority max_priority = np.max(self.tree.tree[-self.tree.capacity:]) # If the max priority = 0 we can't put priority = 0 since this exp will never have a chance to be selected # So we use an upper limit if max_priority == 0: max_priority = self.absolute_error_upper self.tree.add(max_priority, experience) # set the max p for new exp # First, to sample a minibatch of n size, the range [0, priority_total] is split into n ranges. # Then a value is uniformly sampled from each range # We search in the sumtree, the experience where priority score correspond to sample values are retrieved from # Finally, we calculate IS weights for each minibatch element def sample(self, n): memory_b = [] # Create a sample array that will contains the minibatch b_idx, b_ISWeights = np.empty((n,), dtype=np.int32), np.empty((n, 1), dtype=np.float32) priority_segment = self.tree.total_priority / n # priority segment # Here we increasing the PER_b each time we sample a new minibatch self.PER_b = np.min([1., self.PER_b + self.PER_b_increment_per_sampling]) # max = 1 # Calculating the max_weight p_min = np.min(self.tree.tree[-self.tree.capacity:]) / self.tree.total_priority max_weight = (p_min * n) ** (-self.PER_b) for i in range(n): # A value is uniformly sample from each range a, b = priority_segment * i, priority_segment * (i + 1) value = np.random.uniform(a, b) # Experience that correspond to each value is retrieved index, priority, data = self.tree.get_leaf(value) sampling_probabilities = priority / self.tree.total_priority # P(j) # IS = (1/N * 1/P(i))**b /max wi == (N*P(i))**-b /max wi b_ISWeights[i, 0] = np.power(n * sampling_probabilities, -self.PER_b)/ max_weight b_idx[i]= index experience = [data] memory_b.append(experience) return b_idx, memory_b, b_ISWeights # Update the priorities on the tree def batch_update(self, tree_idx, abs_errors): abs_errors += self.PER_e # convert to abs and avoid 0 clipped_errors = np.minimum(abs_errors, self.absolute_error_upper) ps = np.power(clipped_errors, self.PER_a) for ti, p in zip(tree_idx, ps): self.tree.update(ti, p) ### Deal with the empty memory problem (pre-populate memory by taking random actions and storing the experience) memory = Memory(memory_size) # Instantiate memory game.new_episode() # Render the environment for i in range(pretrain_length): # If it's the first step if i == 0: # First we need a state state = game.get_state().screen_buffer state, stacked_frames = stack_frames(stacked_frames, state, True) action = random.choice(possible_actions) # Random action reward = game.make_action(action) # Get the rewards done = game.is_episode_finished() # Look if the episode is finished # If the player is dead if done: # the episode ends so no next state next_state = np.zeros((120,140), dtype=np.int) next_state, stacked_frames = stack_frames(stacked_frames, next_state, False) # Add experience to memory experience = state, action, reward, next_state, done memory.store(experience) # Start a new episode game.new_episode() # First we need a state state = game.get_state().screen_buffer # Stack the frames state, stacked_frames = stack_frames(stacked_frames, state, True) else: # Get the next state next_state = game.get_state().screen_buffer next_state, stacked_frames = stack_frames(stacked_frames, next_state, False) # Add experience to memory experience = state, action, reward, next_state, done memory.store(experience) # Our state is now the next_state state = next_state ### Choose action from Q (use ϵ-greedy strategy) def predict_action(explore_start, explore_stop, decay_rate, decay_step, state, actions): exp_exp_tradeoff = np.random.rand() # First we randomize a number explore_probability = explore_stop + (explore_start - explore_stop) * np.exp(-decay_rate * decay_step) if (explore_probability > exp_exp_tradeoff): # Make a random action (exploration) action = random.choice(actions) else: # Get action from Q-network (exploitation) # Estimate the Qs values state Qs = sess.run(DQNetwork.output, feed_dict = {DQNetwork.inputs_: state.reshape((1, *state.shape))}) # Take the biggest Q value (= the best action) choice = np.argmax(Qs) action = actions[int(choice)] return action, explore_probability ### Copy the parameters of DQN to Target_network (used for Fixed Q-target and Double DQN) def update_target_graph(): # Get the parameters of our DQNNetwork from_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, "DQNetwork") # Get the parameters of our Target_network to_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, "TargetNetwork") op_holder = [] # Update our target_network parameters with DQNNetwork parameters for from_var,to_var in zip(from_vars,to_vars): op_holder.append(to_var.assign(from_var)) return op_holder ### Train the agent saver = tf.train.Saver() # Saver will help us to save our model if training == True: with tf.Session() as sess: sess.run(tf.global_variables_initializer()) # Initialize the variables decay_step = 0 # Initialize the decay step tau = 0 # Set tau = 0 game.init() # Init the game # Update the parameters of our TargetNetwork with DQN_weights update_target = update_target_graph() sess.run(update_target) for episode in range(total_episodes): step = 0 # Set step to 0 episode_rewards = [] # Initialize the rewards of the episode game.new_episode() # Make a new episode and observe the first state state = game.get_state().screen_buffer state, stacked_frames = stack_frames(stacked_frames, state, True) while step < max_steps: step += 1 tau += 1 decay_step +=1 # ϵ-greedy stragety action, explore_probability = predict_action(explore_start, explore_stop, decay_rate, decay_step, state, possible_actions) reward = game.make_action(action) # Do the action done = game.is_episode_finished() # Look if the episode is finished episode_rewards.append(reward) # Add the reward to total reward # If the game is finished if done: # the episode ends so no next state next_state = np.zeros((120,140), dtype=np.int) next_state, stacked_frames = stack_frames(stacked_frames, next_state, False) # Set step = max_steps to end the episode step = max_steps # Get the total reward of the episode total_reward = np.sum(episode_rewards) print('Episode: {}'.format(episode), 'Total reward: {}'.format(total_reward), \ 'Training loss: {:.4f}'.format(loss), 'Explore P: {:.4f}'.format(explore_probability)) # Add experience to memory experience = state, action, reward, next_state, done memory.store(experience) else: next_state = game.get_state().screen_buffer # Get the next state # Stack the frame of the next_state next_state, stacked_frames = stack_frames(stacked_frames, next_state, False) # Add experience to memory experience = state, action, reward, next_state, done memory.store(experience) state = next_state # LEARNING PART tree_idx, batch, ISWeights_mb = memory.sample(batch_size) # Obtain random mini-batch from memory states_mb = np.array([each[0][0] for each in batch], ndmin=3) actions_mb = np.array([each[0][1] for each in batch]) rewards_mb = np.array([each[0][2] for each in batch]) next_states_mb = np.array([each[0][3] for each in batch], ndmin=3) dones_mb = np.array([each[0][4] for each in batch]) target_Qs_batch = [] # DOUBLE DQN # Use DQNNetwork to select the action a' to take at next_state s' (action with the highest Q-value) # Use TargetNetwork to calculate the Q_val of Q(s',a') # Get Q values for next_state q_next_state = sess.run(DQNetwork.output, feed_dict = {DQNetwork.inputs_: next_states_mb}) # Calculate Qtarget for all actions at that state q_target_next_state = sess.run(TargetNetwork.output, feed_dict = {TargetNetwork.inputs_: next_states_mb}) # Set Q_target = r if the episode ends at s+1, otherwise set Q_target = r + gamma * Qtarget(s',a') for i in range(0, len(batch)): terminal = dones_mb[i] # We got a' action = np.argmax(q_next_state[i]) # If we are in a terminal state, only equals reward if terminal: target_Qs_batch.append(rewards_mb[i]) else: # Take the Qtarget for action a' target = rewards_mb[i] + gamma * q_target_next_state[i][action] target_Qs_batch.append(target) targets_mb = np.array([each for each in target_Qs_batch]) # Optimize _, loss, absolute_errors = sess.run([DQNetwork.optimizer, DQNetwork.loss, DQNetwork.absolute_errors], \ feed_dict={DQNetwork.inputs_: states_mb, DQNetwork.target_Q: targets_mb, \ DQNetwork.actions_: actions_mb, DQNetwork.ISWeights_: ISWeights_mb}) # Update priority memory.batch_update(tree_idx, absolute_errors) # Fixed Q target if tau > max_tau: # Update the parameters of our TargetNetwork with DQN_weights update_target = update_target_graph() sess.run(update_target) tau = 0 print("Model updated") # Save model every 5 episodes if episode % 5 == 0: save_path = saver.save(sess, "./models/model.ckpt") print("Model Saved") ### Watch the agent play with tf.Session() as sess: game = DoomGame() # Load the correct configuration (TESTING) game.load_config("deadly_corridor_testing.cfg") # Load the correct scenario (in our case deadly_corridor scenario) game.set_doom_scenario_path("deadly_corridor.wad") game.init() # Load the model saver.restore(sess, "./models/model.ckpt") for i in range(10): game.new_episode() state = game.get_state().screen_buffer state, stacked_frames = stack_frames(stacked_frames, state, True) while not game.is_episode_finished(): # EPSILON GREEDY STRATEGY exp_exp_tradeoff = np.random.rand() explore_probability = 0.01 if (explore_probability > exp_exp_tradeoff): # Make a random action (exploration) action = random.choice(possible_actions) else: # Get action from Q-network (exploitation) Qs = sess.run(DQNetwork.output, feed_dict = {DQNetwork.inputs_: state.reshape((1, *state.shape))}) choice = np.argmax(Qs) # Take the biggest Q value (= the best action) action = possible_actions[int(choice)] game.make_action(action) done = game.is_episode_finished() if done: break else: next_state = game.get_state().screen_buffer next_state, stacked_frames = stack_frames(stacked_frames, next_state, False) state = next_state score = game.get_total_reward() print("Score: ", score) game.close()
