Tensorflow 踩的坑（一）

上午，准备将一个数据集编码成TFrecord 格式。然后，总是报错，下面这个bug一直无法解决，无论是Google，还是github。出现乱码，提示：

Invalid argument: Could not parse example input, value ‘#######’

这个好像牛头不对马嘴，出现在控制台上最后的提示是：

OutOfRangeError (see above for traceback): RandomShuffleQueue '_2_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0) [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_UINT8, DT_UINT8],

这个博客OutOfRangeError : RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed说，可能是tf.train.shuffle_batch()函数中参数num_threads的原因。于是，我就将这个改成了64，128。线程多了，电脑一下子死机了，中间还不断输出乱码。

这个给我提醒了一下，可能是因为文件输入来不及读取完全的缘故。而我处理的文件是遥感图片tiff，一个图片（5000，5000，3），一张图片大概是72M大小。于是我裁剪了图片（900，900，3），然后程序运行良好。

算是一个坑吧，网上没找到相关记录，记录下，看到的人少走弯路吧~

输入大文件的时候，完整报错如下：

2018-06-03 14:50:15.765894: W c:\l\tensorflow_1501907206084\work\tensorflow-1.2.1\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE instructions, but these are available on your machine and could speed up CPU computations.
2018-06-03 14:50:15.766269: W c:\l\tensorflow_1501907206084\work\tensorflow-1.2.1\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE2 instructions, but these are available on your machine and could speed up CPU computations.
2018-06-03 14:50:15.766605: W c:\l\tensorflow_1501907206084\work\tensorflow-1.2.1\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
2018-06-03 14:50:15.767003: W c:\l\tensorflow_1501907206084\work\tensorflow-1.2.1\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-06-03 14:50:15.767453: W c:\l\tensorflow_1501907206084\work\tensorflow-1.2.1\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2018-06-03 14:50:17.411339: W c:\l\tensorflow_1501907206084\work\tensorflow-1.2.1\tensorflow\core\framework\op_kernel.cc:1158] Invalid argument: Could not parse example input, value: '
���/
���
	label_raw���
���
��������������������������������������
Traceback (most recent call last):
  File "D:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1139, in _do_call
    return fn(*args)
  File "D:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1121, in _run_fn
    status, run_metadata)
  File "D:\Anaconda3\lib\contextlib.py", line 66, in __exit__
    next(self.gen)
  File "D:\Anaconda3\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_2_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0)
	 [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_UINT8, DT_UINT8], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:/experiment/01_try/MLStudy/CAN_Seg/makedataset.py", line 130, in <module>
    i, l = sess.run([images, labels])  # 在会话中取出image和label
  File "D:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 789, in run
    run_metadata_ptr)
  File "D:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 997, in _run
    feed_dict_string, options, run_metadata)
  File "D:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1132, in _do_run
    target_list, options, run_metadata)
  File "D:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _do_call
    raise type(e)(node_def, op, message)
2018-06-03 14:50:18.943971: W c:\l\tensorflow_1501907206084\work\tensorflow-1.2.1\tensorflow\core\framework\op_kernel.cc:1158] Invalid argument: Could not parse example input, value: '
���/
���
	label_raw���
���
���
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_2_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0)
	 [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_UINT8, DT_UINT8], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]

Caused by op 'shuffle_batch', defined at:
  File "E:/experiment/01_try/MLStudy/CAN_Seg/makedataset.py", line 123, in <module>
    images, labels = get_batch(filename_queue, batch_size=1)
  File "E:/experiment/01_try/MLStudy/CAN_Seg/makedataset.py", line 82, in get_batch
    capacity=10, min_after_dequeue=7)
  File "D:\Anaconda3\lib\site-packages\tensorflow\python\training\input.py", line 1217, in shuffle_batch
    name=name)
  File "D:\Anaconda3\lib\site-packages\tensorflow\python\training\input.py", line 788, in _shuffle_batch
    dequeued = queue.dequeue_many(batch_size, name=name)
  File "D:\Anaconda3\lib\site-packages\tensorflow\python\ops\data_flow_ops.py", line 457, in dequeue_many
    self._queue_ref, n=n, component_types=self._dtypes, name=name)
  File "D:\Anaconda3\lib\site-packages\tensorflow\python\ops\gen_data_flow_ops.py", line 946, in _queue_dequeue_many_v2
    timeout_ms=timeout_ms, name=name)
  File "D:\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "D:\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "D:\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

OutOfRangeError (see above for traceback): RandomShuffleQueue '_2_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0)
	 [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_UINT8, DT_UINT8], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]