Amazon Ubuntu Deep Learning Instance Configuring Steps

These are the basic steps I used when configuring an Ubuntu deep learning instance on Amazon EC2: set up a Git deployment target, install Bazel, prepare TensorFlow, install Python modules, and expose Jupyter for interactive work.

1. Build a Git Deployment Server

Generate an SSH key on the client machine if one does not already exist:

ssh-keygen -t rsa -C "user.email"

Create or use a bare Git repository on the instance, then edit its post-receive hook:

vim sample.git/hooks/post-receive

Example post-receive hook:

#!/bin/sh

GIT_WORK_TREE=/home/ubuntu/Deployment/sample git checkout -f
chmod -R 777 /home/ubuntu/Deployment/sample

Make the hook executable:

chmod +x sample.git/hooks/post-receive

After this, pushing to the bare repository checks the latest files out into /home/ubuntu/Deployment/sample.

2. Install Bazel

Add the Bazel APT repository as a package source. This is a one-time setup step:

echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add -

If you want to install the testing version of Bazel, replace stable with testing in the repository line.

Install Bazel:

sudo apt-get update
sudo apt-get install bazel

Once installed, Bazel can be upgraded with:

sudo apt-get upgrade bazel

For newer Ubuntu releases, the exact Bazel installation method may have changed. Check the current Bazel documentation and verify the installed version locally:

bazel version

3. Compile TensorFlow

TensorFlow compilation depends heavily on the TensorFlow version, CUDA version, cuDNN version, Python version, and GPU driver installed on the instance. Before compiling, verify the local environment:

python --version
python3 --version
nvcc --version
nvidia-smi
bazel version

A typical source build workflow is:

git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow
./configure
bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package

Then build a Python wheel:

./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg

Install the generated wheel with pip or pip3, depending on the Python environment you configured:

pip install /tmp/tensorflow_pkg/tensorflow-*.whl

Verify the installation:

python -c "import tensorflow as tf; print(tf.__version__)"

4. Install Python Modules

Install the commonly used Python packages:

pip install tensorflow numpy pandas

If the instance uses Python 3 by default, use:

pip3 install tensorflow numpy pandas

It is usually safer to install packages inside a virtual environment:

python3 -m venv ~/venvs/deeplearning
source ~/venvs/deeplearning/bin/activate
pip install --upgrade pip
pip install tensorflow numpy pandas

Check that the modules import correctly:

python - <<'PY'
import tensorflow as tf
import numpy as np
import pandas as pd

print('tensorflow', tf.__version__)
print('numpy', np.__version__)
print('pandas', pd.__version__)
PY

5. Configure Jupyter

Install Jupyter:

pip install jupyter

Generate the configuration file:

jupyter notebook --generate-config

Set a password:

jupyter notebook password

Start Jupyter on the instance:

jupyter notebook --ip=0.0.0.0 --no-browser

If the notebook needs to be accessed from a browser, open the correct EC2 security group port or use an SSH tunnel. An SSH tunnel is usually safer:

ssh -L 8888:localhost:8888 ubuntu@your-ec2-host

Then open this local URL:

http://localhost:8888

Reference: <https://punchagan.muse-amuse.in/posts/create-a-public-jupyter-server-quickly.html>

Leave a Reply