Why is Keras Running So Slow?

More notes for myself… so it may not be helpful for you who bumped into here. 😉

Setting Theano correctly is not enough to ensure you can run deep learning software correctly. In our case, it will be Keras, and it can slow to a crawl if not setup properly.

Again, there could be many causes but I try to outline a clean step what I did, the performance I run a good setup, so you can compare. Hopefully you can glean some places where you did wrong.

My server has the following specifications finished running the steps outlined here.

  • OS: Ubuntu 14.04 LTS, X64

  • GPU: Nvidia Geforce GTX 780

  • Ubuntu 14.04 LTS

  • CUDA 7.5

  • Theano 0.7.0

  • Numpy 1.8.2

  • Kera 0.2.0

  • Scipy 0.13.3

  • NVIDIA-SMI 352.39

  • Graphics Driver Version: 352.39

  1. Make sure your Theano configuration file, located at ~/.theanorc, is correct:
1
2
3
4
5
# Run this python code
from theano.sandbox.cuda.dnn import *
print(dnn_available())
print(dnn_available.msg)

  1. Install CuDNN if you are using ConvNet. The basic implementations of convolution in Theano are significantly slower.

I also captured the environment variables, replace echio with your username:

Googling the error message doesn’t help too much.

You need to add the location of the 3. into CPATH, LD_LIBRARY_PATH and LIBRARY_PATH. This is what my .bashrc looks like this (replace echio with your username):

If you instead see this error message:

You probably didn’t have CUDA environment variables setup properly. See the above ~/.bashrc lines for correct setup.

Run this code again.

Run this python code

from theano.sandbox.cuda.dnn import * print(dnn_available()) print(dnn_available.msg)

You should see below when executed in a Python REPL.

This is good! Re-run your Keras code and hopefully it will be fast this time…

This may or may not solve your problem, but it certainly solved some of my problems. You will probably have to learn to debug things a bit to figure out how to get it to run well.

References