TensorFlow Source Build Failing on Amazon Ubuntu: A Memory-First Diagnostic Guide

I first wrote this note in 2017 after a TensorFlow build failed on a small Amazon Ubuntu instance. At the time I installed SWIG, suspected an out-of-memory failure, reduced Bazel’s parallelism, and added swap. The useful part of that diagnosis still holds: a compiler process that disappears without a useful error is often being killed because the machine ran out of memory.

The commands around that lesson have aged. TensorFlow’s build target, Bazel flags, supported Python versions, compiler requirements, and GPU setup have all changed. This revised guide keeps the original experience while separating today’s recommended installation path from the more demanding source-build path.

Table of Contents

Start with an official wheel

Unless you need to change TensorFlow itself, enable a nonstandard instruction set, or produce a custom package, use an official wheel. It is faster, repeatable, and avoids most compiler and Bazel compatibility problems.

On a supported Linux system:

sudo apt update
sudo apt install -y python3-pip python3-venv

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install tensorflow

cd /tmp
python -c 'import tensorflow as tf; print(tf.__version__)'

For an NVIDIA GPU, follow TensorFlow’s current pip guide and check its support matrix before installing. The current Linux package route is:

python -m pip install 'tensorflow[and-cuda]'
nvidia-smi
python -c 'import tensorflow as tf; print(tf.config.list_physical_devices("GPU"))'

If pip reports that no matching distribution exists, do not switch immediately to a source build. First compare the machine architecture and Python version with TensorFlow’s current supported configurations.

Confirm whether memory is actually the problem

Before changing the build, record the environment:

uname -a
uname -m
python3 --version
free -h
swapon --show
df -h /

An exit status of 137, a lone Killed message, or a compiler process that vanishes can indicate an out-of-memory kill. On a systemd-based Ubuntu instance, inspect the current boot’s kernel log:

sudo journalctl -k -b | grep -Ei 'out of memory|oom|killed process' || true

That evidence matters. A real compiler error needs a toolchain or source fix; an OOM kill needs lower concurrency, more memory, or swap.

Pin one TensorFlow release and its toolchain

Do not build an arbitrary checkout with whatever Bazel and Python happen to be installed. Choose a TensorFlow tag, then use the versions listed for that release in TensorFlow’s official source-build matrix. TensorFlow also records its expected Bazel version in .bazelversion, which Bazelisk can select automatically.

git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow

TF_TAG=v2.21.0  # Example only: choose a release supported by your environment.
git checkout "$TF_TAG"
cat .bazelversion

Create a clean Python environment and run the configuration step from the selected checkout:

sudo apt update
sudo apt install -y git python3-dev python3-pip python3-venv

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
./configure

Install the exact compiler, Bazel/Bazelisk, and any CUDA or cuDNN versions required by that tag. Since TensorFlow 2.13, Clang is the default compiler in the official instructions, but the tested version still depends on the release.

In my 2017 note I installed SWIG before retrying. That was a speculative troubleshooting step, not proof of the root cause, and SWIG is not part of the current official prerequisites for this build path.

Build conservatively on a small CPU instance

For a current CPU wheel, the official target is //tensorflow/tools/pip_package:wheel. On a memory-constrained machine, start with one job and a conservative Bazel RAM budget:

bazel build \
  --config=opt \
  --jobs=1 \
  --local_ram_resources=2048 \
  --verbose_failures \
  --repo_env=USE_PYWRAP_RULES=1 \
  --repo_env=WHEEL_NAME=tensorflow_cpu \
  //tensorflow/tools/pip_package:wheel

--jobs=1 reduces concurrent work. --local_ram_resources=2048 tells Bazel’s scheduler to budget roughly 2 GB for local actions; it is not a hard operating-system memory limit. Adjust it to the instance rather than copying the number blindly.

The 2017 command used --local_resources and the old //tensorflow/tools/pip_package:build_pip_package target. Keep it only as historical context: neither is the current documented command.

For a GPU wheel, TensorFlow’s current source-build guide uses additional CUDA configuration, including --config=cuda, --config=cuda_wheel, and WHEEL_NAME=tensorflow. Follow the guide for the selected release instead of mixing CPU and GPU flags from different eras.

Add swap only as a fallback

Reducing concurrency may be enough. If the machine still runs out of memory, resizing the instance is usually faster and more reliable than compiling heavily into swap. For a temporary or low-cost machine, a swap file can still help, but inspect the host first:

free -h
swapon --show
df -h /
test ! -e /swapfile || { echo '/swapfile already exists; stop and inspect it'; exit 1; }

Ubuntu’s swapon documentation notes that files created with fallocate may be rejected on filesystems that expose holes, and Btrfs needs special handling. A zero-filled file is the more portable choice for a conventional filesystem:

sudo dd if=/dev/zero of=/swapfile bs=1MiB count=4096 status=progress
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
swapon --show

Do not run mkswap against an existing or unidentified path. If this swap file should survive reboot, add exactly one entry to /etc/fstab:

/swapfile none swap sw 0 0

Then validate the file before rebooting:

sudo findmnt --verify --verbose

If the root filesystem is Btrfs, encrypted in an unusual way, or managed by an image policy, use the platform’s documented swap procedure instead.

Install and verify the wheel

The current build writes wheels under wheel_house. Install the generated package into the active virtual environment, then test it outside the source checkout so that local source files do not shadow the installed package:

python -m pip install bazel-bin/tensorflow/tools/pip_package/wheel_house/*.whl

(
  cd /tmp
  python -c 'import tensorflow as tf; print(tf.__version__)'
)

A practical failure map

Symptom	Check first	Likely next action
`Killed`, exit code 137, or OOM entry in the kernel log	`free -h`, `swapon --show`, and `journalctl -k`	Reduce `--jobs`, set a realistic RAM budget, resize the instance, or add carefully configured swap
`No matching distribution found` from pip	Python version, CPU architecture, OS, and TensorFlow’s support matrix	Use a supported Python/platform combination; a source build is not the first fix
Bazel version error	The selected tag’s `.bazelversion` and official build matrix	Use Bazelisk or install the exact tested Bazel version
Compiler or header error	Full `--verbose_failures` output and the release’s tested compiler	Align the compiler and dependencies with the selected TensorFlow tag
CUDA is not found or the GPU list is empty	Driver visibility, `nvidia-smi`, and the current CUDA/cuDNN requirements	Follow one release’s official GPU path end to end
Import works only inside the checkout, or behaves strangely there	Current working directory and `python -m pip show tensorflow`	Test from `/tmp` or another directory outside the source tree

What I learned from the original failure

My useful 2017 observation was not “install SWIG.” It was that source-build failures on small cloud instances need to be classified before packages and flags are changed. Prove whether the kernel killed the compiler, keep the build toolchain aligned with one TensorFlow release, and use an official wheel whenever a custom build has no clear benefit.

Official references

For the historical trail, the early TensorFlow issue discussions around memory pressure during compilation and source-build failures show why this was a common problem on small machines.