训练日志

creating index...
index created!
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.69s).
Accumulating evaluation results...
DONE (t=0.44s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.010
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.005
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.025
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
2020-04-14 03:33:45.485659: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1318] OP_REQUIRES failed at conv_ops.cc:398 : Resource exhausted: OOM when allocating tensor with shape[1,38,50,512] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
Traceback (most recent call last):
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\client\session.py", line 1322, in _do_call
return fn(*args)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\client\session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\client\session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,38,50,512] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
[[Node: Conv/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block3/unit_23/bottleneck_v1/Relu, Conv/weights/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "object_detection/training_20200410.py", line 109, in <module>
tf.app.run()
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
_sys.exit(main(argv))
File "object_detection/training_20200410.py", line 105, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\estimator\training.py", line 447, in train_and_evaluate
return executor.run()
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\estimator\training.py", line 531, in run
return self.run_local()
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\estimator\training.py", line 669, in run_local
hooks=train_hooks)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\estimator\estimator.py", line 366, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1119, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1135, in _train_model_default
saving_listeners)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1336, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\training\monitored_session.py", line 577, in run
run_metadata=run_metadata)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1053, in run
run_metadata=run_metadata)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1144, in run
raise six.reraise(*original_exc_info)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\six.py", line 703, in reraise
raise value
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1129, in run
return self._sess.run(*args, **kwargs)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1201, in run
run_metadata=run_metadata)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\training\monitored_session.py", line 981, in run
return self._sess.run(*args, **kwargs)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\client\session.py", line 900, in run
run_metadata_ptr)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\client\session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\client\session.py", line 1316, in _do_run
run_metadata)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\client\session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,38,50,512] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
[[Node: Conv/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block3/unit_23/bottleneck_v1/Relu, Conv/weights/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'Conv/Conv2D', defined at:
File "object_detection/training_20200410.py", line 109, in <module>
tf.app.run()
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
_sys.exit(main(argv))
File "object_detection/training_20200410.py", line 105, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\estimator\training.py", line 447, in train_and_evaluate
return executor.run()
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\estimator\training.py", line 531, in run
return self.run_local()
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\estimator\training.py", line 669, in run_local
hooks=train_hooks)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\estimator\estimator.py", line 366, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1119, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1132, in _train_model_default
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1107, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\object_detection-0.1-py3.6.egg\object_detection\model_lib.py", line 269, in model_fn
features[fields.InputDataFields.true_image_shape])
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\object_detection-0.1-py3.6.egg\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 647, in predict
image_shape) = self._extract_rpn_feature_maps(preprocessed_inputs)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\object_detection-0.1-py3.6.egg\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 994, in _extract_rpn_feature_maps
reuse=reuse)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 183, in func_with_args
return func(*args, **current_args)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1154, in convolution2d
conv_dims=2)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 183, in func_with_args
return func(*args, **current_args)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1057, in convolution
outputs = layer.apply(inputs)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 774, in apply
return self.__call__(inputs, *args, **kwargs)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\layers\base.py", line 329, in __call__
outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 703, in __call__
outputs = self.call(inputs, *args, **kwargs)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\keras\layers\convolutional.py", line 184, in call
outputs = self._convolution_op(inputs, self.kernel)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 868, in __call__
return self.conv_op(inp, filter)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 520, in __call__
return self.call(inp, filter)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 204, in __call__
name=self.name)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 1042, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\framework\ops.py", line 3414, in create_op
op_def=op_def)
File "N:\Anaconda3\install\envs\zhouying2\lib\site-packages\tensorflow\python\framework\ops.py", line 1740, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1,38,50,512] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
[[Node: Conv/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block3/unit_23/bottleneck_v1/Relu, Conv/weights/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 

(zhouying2) F:\TensorflowProject\ObjectDetection\models-1.12.0\research>
(zhouying2) F:\TensorflowProject\ObjectDetection\models-1.12.0\research>
(zhouying2) F:\TensorflowProject\ObjectDetection\models-1.12.0\research>
(zhouying2) F:\TensorflowProject\ObjectDetection\models-1.12.0\research>

 

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"