How to Fix Docker Out of Memory (OOM) on AWS EC2


As a Senior DevOps Engineer for WebToolsWiz.com, addressing Docker Out of Memory (OOM) issues on AWS EC2 is a critical skill. This guide provides a direct, no-fluff approach to diagnose and fix this common problem.


Fixing Docker Out of Memory (OOM) on AWS EC2

1. The Root Cause Docker Out of Memory (OOM) on AWS EC2 instances primarily occurs when the collective memory footprint of running containers exceeds the EC2 instance’s available RAM and configured swap space. This deficit forces the Linux OOM killer to terminate processes, often targeting memory-hungry containers or even the Docker daemon itself, to restore system stability.

2. Quick Fix (CLI) The fastest way to alleviate OOM issues, especially on undersized EC2 instances, is to increase the system’s swap space.

First, check current memory and swap usage:

free -h

Next, create and activate a 2GB swap file (adjust size as needed for your workload):

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Make the swap persistent across reboots:

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Optionally, restart the Docker daemon to ensure it recognizes new system resources:

sudo systemctl restart docker

3. Configuration Check To protect the Docker daemon itself from being targeted by the OOM killer and to manage container resource usage effectively, configure daemon.json and container limits.

Edit or create the Docker daemon configuration file: /etc/docker/daemon.json

Add or modify the oom-score-adj entry to make the Docker daemon less prone to being killed by the OOM killer:

{
  "oom-score-adj": -500
}

Setting oom-score-adj to a negative value (e.g., -500 to -1000) reduces the likelihood of the Docker daemon being terminated in low-memory situations. Save the file and restart the Docker daemon (sudo systemctl restart docker) for changes to take effect.

Important: For preventing individual containers from causing OOM, always define explicit memory limits (e.g., --memory=1g, --memory-swap=1g, --cpus=1) in your docker run commands or docker-compose.yml files.

4. Verification After applying the fixes, verify the changes:

Check the new swap space is active:

free -h

Verify the Docker daemon’s OOM score adjustment (ensure it reflects the negative value you set):

cat /proc/$(pgrep dockerd)/oom_score_adj

Confirm the Docker service is running without issues:

sudo systemctl status docker