6 May 2020

Install caffe with python3 support in Ubuntu

In this blog post I would guide you through the installation process of BVLC Caffe deep learning framework with python3 support. I felt the necessity of writing this blog post because the installation instructions available here did not work perfectly for me. I encountered several errors during installation which were due to old version of python-dateutil.

So let’s dive into the installation instructions.

I. Install caffe with GPU and python support using `make`

Fire up ubuntu terminal with CTRL + ALT + T then update and upgrade ubuntu packages list.

$ sudo apt-get -y update
$ sudo apt-get -y upgrade

Install the following dependencies of caffe framework.

$ sudo apt-get -y install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
$ sudo apt-get -y install --no-install-recommends libboost-all-dev
$ sudo apt-get -y install libatlas-base-dev libopenblas-dev
$ sudo apt-get -y install libgflags-dev libgoogle-glog-dev liblmdb-dev

Now let’s change directory to where we want keep our caffe code. And clone the repository. I pull the code to Programs directory inside home directory.

$ cd ~/Programs
$ git clone https://github.com/BVLC/caffe.git

Once the code is pulled, go to python directory inside caffe and install the python requirements.

$ cd caffe
$ cd python

Make sure you use following versions of numpy and python-dateutil libraries. Edit the requirements.txt file. Replace python-dateutil and numpy with the following.

python-dateutil==2.8.1
numpy==1.16

Let’s upgrade pip to the latest version and install python requirements.

$ pip3 install --upgrade pip // Upgrade pip
$ pip3 install -r requirements.txt
$ cd ..

Now we need to add caffe python path to the $PYTHONPATH environment variable.

$ export PYTHONPATH=/home/avl/Programs/caffe/python:$PYTHONPATH

You can verify if the path is added correctly using $ echo $PYTHONPATH. Also retrieve Python USER_SITE with the help of following command.

$ python3 -m site

Output will be something like this:

sys.path = [
    '/usr/local',
    '/usr/local/openpose/python',
    '/usr/local/caffe-1.0/python',
    '/usr/local/opencv-3.4.4/python/cv2/python-3.5',
    '/usr/lib/python35.zip',
    '/usr/lib/python3.5',
    '/usr/lib/python3.5/plat-x86_64-linux-gnu',
    '/usr/lib/python3.5/lib-dynload',
    '/home/avl/.local/lib/python3.5/site-packages',
    '/usr/local/lib/python3.5/dist-packages',
    '/usr/lib/python3/dist-packages',
    '/usr/lib/python3.5/dist-packages',
]
USER_BASE: '/home/avl/.local' (exists)
USER_SITE: '/home/avl/.local/lib/python3.5/site-packages' (exists)
ENABLE_USER_SITE: True

If USER_SITE doesn’t exist, then use the one that ends with /dist-packages

'/usr/local/lib/python3.5/dist-packages'

Make another copy of Makefile configuration.

$ cp Makefile.config.example Makefile.config

This file consists of all the configurations to install caffe. We edit this file to install caffe with GPU and python3 support. Edit this Makefile.config file using your favourite editor.

To build with GPU and cuDNN, uncomment the line USE_CUDNN and comment the line CPU_ONLY.

USE_CUDNN := 1
# CPU_ONLY := 1

Since I am using OPENCV version 3.4.4, I uncomment the following and set to 1 to use OPENCV:

USE_OPENCV := 1

I want to build caffe with python3 so I comment the PYTHON_INCLUDE lines that point to python2.7

# PYTHON_INCLUDE := /usr/include/python2.7
# /usr/lib/python2.7/dist-packages/numpy/core/include

And uncomment PYTHON_INCLUDE lines that point to python3. Add the path referred to by USER_SITE that we checked earlier to PYTHON_INCLUDE. PYTHON_LIBRARIES and PYTHON_INCLUDE should look like the following.

PYTHON_LIBRARIES := boost_python-py35 python3.5m
PYTHON_INCLUDE := /usr/include/python3.5m /usr/lib/python3.5/dist-packages/numpy/core/include /home/avl/.local/lib/python3.5/site-packages/numpy/core/include

Uncomment the line WITH_PYTHON_LAYER := 1.

In the line INCLUDE_DIRS, add the path of hdf5/serial. Now INCLUDE_DIRS should look like:

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial

Again add the path of hdf5 after appending serial which you get from the following command to LIBRARY_DIRS.

$ find /usr/lib -name hdf5

By now LIBRARY_DIRS should look like:

LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial

Next, run the following commands to get the path of libpython3.5m $ find /usr -name libpython3.5*. You will see multiple paths to the library but add /usr/lib/x86_64-linux-gnu/ to PYTHON_LIB and LIBRARY_DIRS.

PYTHON_LIB := /usr/lib /usr/lib/x86_64-linux-gnu/
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial /usr/lib/x86_64-linux-gnu/

Similarly, find the path to libboost_python-py35.

find /usr -name libboost_python-py35*

Generally, the path is /usr/lib/x86_64-linux-gnu/ which has already been added to the PYTHON_LIB and LIBRARY_DIRS above. However, if it is different then add it to both of these variables.

Don’t forget to rename boost_python3 in PYTHON_LIBRARIES to boost_python-py35.

PYTHON_LIBRARIES := boost_python-py35 python3.5m

We have completed editing the Makefile.config. Now run the following commands to install caffe. This will take a while to complete.

$ make all -j8
$ make runtest
$ make pycaffe

You can verify caffe installation by importing caffe in python3 interpreter.

$ python3
Python 3.5.2 (default, Apr 16 2020, 17:47:17) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import caffe
>>>  

If the installation is successful, import caffe should work without any issues.

II. Install caffe with GPU and python support using `cmake-gui`

You can also edit configurations using cmake-gui. Follow following instructions to build caffe using cmake.

Once you are inside the caffe directory, make a new build directory where you’ll build all the necessary files to install.

$ cd caffe
$ mkdir build
$ cd build
$ cmake-gui ..

Now you should get the cmake GUI window. Check the Advanced checkbox to load more options.

Search for python in the search bar and make sure you set the values as shown in the image. Set all the variables referring to python3.

Cmake gui

Click on Configure. Once configuration is successful, click Generate. This will generate necessary files inside the build folder.

Now we are ready to run make and install caffe.

$ make all -j8
$ make runtest
$ make pycaffe
$ make install

Verify the installation by running the following command.

$ python3
Python 3.5.2 (default, Apr 16 2020, 17:47:17) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>> 

After the installation is successful to use caffe we need to set certain environment variables. Add the following lines in the ~/.bashrc file.

# Caffe
export LD_LIBRARY_PATH=/home/avl/Programs/caffe/build/install/lib:$LD_LIBRARY_PATH
export PYTHONPATH=/home/avl/Programs/caffe/build/install/python:$PYTHONPATH
export CAFFE_ROOT=/home/avl/Programs/caffe

To use the caffe right away, we need to source the bashrc file.

$ source ~/.bashrc

Now, you should be able to import caffe in python and use it.

Tags:

0 comments

Install caffe with python3 support in Ubuntu

I. Install caffe with GPU and python support using make

II. Install caffe with GPU and python support using cmake-gui

Tags:

I. Install caffe with GPU and python support using `make`

II. Install caffe with GPU and python support using `cmake-gui`