The Explanation: TensorFlow consumed all memory of all GPUs
When runing a TensorFlow or Keras script (as script 1) on a computer with multiple GPUs, the nvidia-smi -l command shows all the memory resource of all GPUs is consumed, as:
But, there is only one GPU is doing the calculation. In fact, the TensorFlow just made use of 1 GPU. Running script 2, the resource monitor shows
It’s easy to obtain that conclusion.
Inferred from python - How to prevent tensorflow from allocating the totality of a GPU memory? - Stack Overflow
Currently, this fraction is applied uniformly to all of the GPUs on the same machine; there is no way to set this on a per-GPU basis.
If tf.ConfigProto().gpu_options.allow_growth = False ( by default), TensorFlow is going to declare to consume the same percentage (the upper band) of memory of all visible GPUs.
As a summary, if the task is not written for multiple GPUs, it would be helpful to apply import setGPU to distribute the Tensorflow task to a certain GPU. In that scenario, there would be a free GPU, one could create another process for another task on the free GPU.
script 1
#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
# @Time : 2018/8/3 15:24
# @File : explore_tensorflow_gpu_usage.py
# @Author : yusisc (yusisc@gmail.com)
# import setGPU
import numpy as np
import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, LSTM
from keras.optimizers import SGD
# import tensorflow as tf
# from keras.backend.tensorflow_backend import set_session
# config = tf.ConfigProto()
# config.gpu_options.allow_growth = True
# # config.gpu_options.per_process_gpu_memory_fraction = 0.3
# set_session(tf.Session(config=config))
# Generate dummy data
x_train = np.random.random((1000, 20))
y_train = keras.utils.to_categorical(np.random.randint(10, size=(1000,)), num_classes=10)
# build model
model = Sequential()
model.add(Dense(1024*16, activation='relu', input_dim=20))
model.add(Dense(1024*16, activation='relu'))
model.add(Dense(10, activation='softmax'))
sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy',
optimizer=sgd,
metrics=['accuracy'])
# train model
model.fit(x_train, y_train,
epochs=20,
batch_size=128)
script 2
#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
# @Time : 2018/8/3 15:24
# @File : explore_tensorflow_gpu_usage.py
# @Author : yusisc (yusisc@gmail.com)
# import setGPU
import numpy as np
import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, LSTM
from keras.optimizers import SGD
import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
# config.gpu_options.per_process_gpu_memory_fraction = 0.3
set_session(tf.Session(config=config))
# Generate dummy data
x_train = np.random.random((1000, 20))
y_train = keras.utils.to_categorical(np.random.randint(10, size=(1000,)), num_classes=10)
# build model
model = Sequential()
model.add(Dense(1024*16, activation='relu', input_dim=20))
model.add(Dense(1024*16, activation='relu'))
model.add(Dense(10, activation='softmax'))
sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy',
optimizer=sgd,
metrics=['accuracy'])
# train model
model.fit(x_train, y_train,
epochs=20,
batch_size=128)ref
python - How to prevent tensorflow from allocating the totality of a GPU memory? - Stack Overflow
https://stackoverflow.com/questions/34199233/how-to-prevent-tensorflow-from-allocating-the-totality-of-a-gpu-memory
Limit the resource usage for tensorflow backend · Issue #1538 · keras-team/keras
https://github.com/keras-team/keras/issues/1538
