cs20_3-1

0. Review

  1. Computation graph

    • TensorFlow separates definition of computations from their execution

    • Phase 1: assemble a graph

      Phase 2: use a session to execute operations in the graph.

  2. TensorBoard

  3. tf.constant and tf.Variable

    1. Constant values are stored in the graph definition

      Sessions allocate memory to store variable values

    2. tf.placeholder and feed_dict

      Feed values into placeholders with a dictionary (feed_dict)

      Easy to use but poor performance

  4. Avoid lazy loading

    1. Separate the assembling of graph and executing ops
    2. Use Python attribute to ensure a function is only loaded the first time it’s called

1. Linear regression on birth_life data

  1. Phase 1: Assemble our graph

    • Read in data
    • Create placeholders for inputs and labels
    • Create weight and bias
    • Inference
    • Specify loss function
    • Create optimizer
  2. Phase 2: Train our model

    • Initialize variables
    • Run optimizer
  3. 在完成phase 1之后,使用tf.summary创建graph的log文件,以备tensorboard可视化

    Write log files using a FileWriter: writer = tf.summary.FileWriter('./graphs/linear_reg', sess.graph)

  4. See it on TensorBoard

    • run code
    • execute tensorboard command

2. Control Flow

  1. e.g. Implementing Huber loss

    • description:

    • code:

      def huber_loss(labels, predictions, delta=14.0):
          residual = tf.abs(labels - predictions)
          def f1(): return 0.5 * tf.square(residual)
          def f2(): return delta * residual - 0.5 * tf.square(delta)
          return tf.cond(residual < delta, f1, f2)
      
  2. TF Control Flow-table

    Control Flow Ops tf.group, tf.count_up_to, tf.cond, tf.case, tf.while_loop, ...
    Comparison Ops tf.equal, tf.not_equal, tf.less, tf.greater, tf.where, ...
    Logical Ops tf.logical_and, tf.logical_not, tf.logical_or, tf.logical_xor
    Debugging Ops tf.is_finite, tf.is_inf, tf.is_nan, tf.Assert, tf.Print, ...

3. tf.data

  1. Placeholder

    • Pro: put the data processing outside TensorFlow, making it easy to do in Python

    • Cons: users often end up processing their data in a single thread and creating data bottleneck that slows execution down.

  2. tf.data:

    • Instead of doing inference with placeholders and feeding in data later, do inference directly with data

    • tf.data.Dataset

      tf.data.Iterator

    • tf.data.Dataset

      1. from variables

        tf.data.Dataset.from_tensor_slices((features, labels))
        tf.data.Dataset.from_generator(gen, output_types, output_shapes)
        dataset = tf.data.Dataset.from_tensor_slices((data[:,0], data[:,1]))
        
      2. from file

        tf.data.TextLineDataset(filenames)
        tf.data.FixedLengthRecordDataset(filenames)
        tf.data.TFRecordDataset(filenames)
        
    • tf.data.Iterator

      • Create an iterator to iterate through samples in Dataset

      • e.g.

        iterator = dataset.make_one_shot_iterator()
        # Iterates through the dataset exactly once. No need to initialization.
        
        iterator = dataset.make_initializable_iterator()
        Iterates through the dataset as many times as we want. Need to initialize with each epoch.
        #
        iterator = dataset.make_one_shot_iterator()
        X, Y = iterator.get_next()         # X is the birth rate, Y is the life expectancy
        with tf.Session() as sess:
        	print(sess.run([X, Y]))		# >> [1.822, 74.82825]
        	print(sess.run([X, Y]))		# >> [3.869, 70.81949]
        	print(sess.run([X, Y]))		# >> [3.911, 72.15066]
        #
        iterator = dataset.make_initializable_iterator()
        ...
        for i in range(100): 
                sess.run(iterator.initializer) 
                total_loss = 0
                try:
                    while True:
                        sess.run([optimizer]) 
                except tf.errors.OutOfRangeError:
                    pass
        
    • Handling data in TensorFlow

      • e.g.

        dataset = dataset.shuffle(1000)
        dataset = dataset.repeat(100)
        dataset = dataset.batch(128)
        dataset = dataset.map(lambda x: tf.one_hot(x, 10)) 
        # convert each elem of dataset to one_hot vector
        
    • Does tf.data really perform better?

    • Should we always use tf.data?

      • For prototyping, feed dict can be faster and easier to write (pythonic)
      • tf.data is tricky to use when you have complicated preprocessing or multiple data sources
      • NLP data is normally just a sequence of integers. In this case, transferring the data over to GPU is pretty quick, so the speedup of tf.data isn't that large
    • How does TensorFlow know what variables to update? Optimizer

4. Optimizers, gradients

  • e.g.

    optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01).minimize(loss)
    _, l = sess.run([optimizer, loss], feed_dict={X: x, Y:y})
    # Session looks at all trainable variables that loss depends on and update them
    # 比如weight,bias,gradiets_1丢到GradientDescent(),但是只要weights和bias是trainable
    
  • Trainable variables

    tf.Variable(initial_value=None, trainable=True,...)
    # Specify if a variable should be trained or not
    # By default, all variables are trainable
    
  • List of optimizers in TF

    tf.train.GradientDescentOptimizer
    tf.train.AdagradOptimizer
    tf.train.MomentumOptimizer
    tf.train.AdamOptimizer
    tf.train.FtrlOptimizer
    tf.train.RMSPropOptimizer
    ...
    #
    #“Advanced” optimizers work better when tuned, but are generally harder to tune
    
  • question:

    1. How to know that our model is correct?
    2. How to improve our model?

5. Logistic regression on MNIST

  • MNIST Database: Each image is a 28x28 array, flattened out to be a 1-d tensor of size 784.

  • X: image of a handwritten digit

    Y: the digit value

    Recognize the digit in the image

  • Model: Inference:

    Y_predicted = softmax(X * w + b)

    Cross entropy loss: -log(Y_predicted)

  • Process data:

    from tensorflow.examples.tutorials.mnist import input_data
    MNIST = input_data.read_data_sets('data/mnist', one_hot=True) 
    
    # MNIST.train: 55,000 examples
    # MNIST.validation: 5,000 examples
    # MNIST.test: 10,000 examples
    # No immediate way to convert Python generators to tf.data.Dataset
    #
    mnist_folder = 'data/mnist'
    utils.download_mnist(mnist_folder)
    train, val, test = utils.read_mnist(mnist_folder, flatten=True)
    #
    #
    train_data = tf.data.Dataset.from_tensor_slices(train)
    train_data = train_data.shuffle(10000) # optional
    test_data = tf.data.Dataset.from_tensor_slices(test)
    #
    #
    iterator = train_data.make_initializable_iterator()
    img, label = iterator.get_next()
    # problem:
    # Can only do inference with train_data. 
    # Need to build another subgraph with another iterator for test_data!!!
    # so improvement。。。
    #
    # 因为这里的train_data,test_data的type及shape都是相同,同一个dataset来的,只是有不同用途,所以可以共用一个iterator的base class
    iterator = tf.data.Iterator.from_structure(train_data.output_types, 
                                               train_data.output_shapes)
    img, label = iterator.get_next()
    
    train_init = iterator.make_initializer(train_data)  # initializer for train_data
    test_init = iterator.make_initializer(test_data)	# initializer for train_data
    #
    #
    # Initialize iterator with the dataset you want 
    with tf.Session() as sess:
        ...
        for i in range(n_epochs):       
            sess.run(train_init)  # use train_init during training loop
            try:
                while True:
                    _, l = sess.run([optimizer, loss])
            except tf.errors.OutOfRangeError:
                pass
       #
       # test the model
        sess.run(test_init)	    # use test_init during testing
        try:
            while True:
                sess.run(accuracy)
        except tf.errors.OutOfRangeError:
            pass
    
  • Phase 1: Assemble our graph

    • Step 1: Read in data

    • Step 2: Create datasets and iterator

      train_data = tf.data.Dataset.from_tensor_slices(train)
      train_data = train_data.shuffle(10000) # optional(train必须置乱D_train)
      train_data = train_data.batch(batch_size)
      test_data = tf.data.Dataset.from_tensor_slices(test)
      test_data = test_data.batch(batch_size)
      
      iterator = tf.data.Iterator.from_structure(train_data.output_types, 
                                        train_data.output_shapes)
      img, label = iterator.get_next()
      train_init = iterator.make_initializer(train_data)
      test_init = iterator.make_initializer(test_data)
      
    • Step 3: Create weights and biases

      use tf.get_variable()
      
      
    • Step 4: Build model to predict Y

      logits = tf.matmul(img, w) + b
      # We don’t do softmax here, as we’ll do softmax together with cross_entropy loss.
      # It’s more efficient to compute gradients w.r.t. logits than w.r.t. softmax
      
    • Step 5: Specify loss function

      entropy = tf.nn.softmax_cross_entropy_with_logits(labels=label, logits=logits)
      # 等价于先做softmax,再对概率化后的logits和本身就是0或1的labels做cross entropy
      loss = tf.reduce_mean(entropy)
      
    • Step 6: Create optimizer

      tf.train.AdamOptimizer(learning_rate=0.01).minimize(loss)
      
  • Phase 2: Train our model

    • Step 1: Initialize variables
    • Step 2: Run optimizer op
  • TensorBoard it

6. Loss functions

  • cross entropy: 写上我写的一个link
posted @ 2019-02-12 17:50  hzhang_NJU  阅读(289)  评论(0编辑  收藏  举报