How to Fix Terraform Too Many Open Files on Debian 11

As a Senior DevOps Engineer, encountering the “Terraform Too Many Open Files” error, particularly on a stable environment like Debian 11, is a clear signal that a fundamental system limit is being hit. While Terraform is the application reporting the error, the root cause lies within the operating system’s configuration. This guide will walk you through diagnosing and resolving this common issue.

Troubleshooting Guide: Terraform “Too Many Open Files” on Debian 11

1. The Root Cause: Why This Happens on Debian 11

The “Too Many Open Files” error indicates that Terraform, or more precisely, the process running Terraform, has exhausted its allotted number of file descriptors. A file descriptor is an abstract indicator used to access a file or other input/output resource, such as:

  • Actual files on disk (state files, configuration files, provider binaries).
  • Network sockets (connections to cloud provider APIs, remote state backends).
  • Pipes, FIFOs, and other internal communication channels.

On Debian 11 (and most Linux distributions), the default per-process limit for file descriptors (ulimit -n) is often set conservatively, typically to 1024.

Why Terraform Hits This Limit: Terraform, especially with complex configurations, multiple providers, numerous resources, or during operations like terraform refresh, terraform plan, or terraform apply on large infrastructure, can quickly consume a significant number of file descriptors. Each provider plugin, each resource interaction (API call), each file read/write contributes to this count. When the total number of open file descriptors for the Terraform process (and its child processes, like provider plugins) exceeds the configured ulimit -n value, the operation fails with the “Too Many Open Files” error.

It’s not a Terraform bug, but rather Terraform exposing a system-level constraint under heavy workload.

2. Quick Fix (CLI): Temporary Session Adjustment

For immediate relief and testing, you can temporarily increase the file descriptor limit for your current shell session. This change will only apply to the current shell and any processes launched from it, and will be lost once the shell session is closed.

  1. Check Current Limit: First, check your current limit:

    ulimit -n

    You’ll likely see 1024 or a similar low value.

  2. Increase Limit for Current Session: Set a higher limit. A common starting point for DevOps tools is 65536.

    ulimit -n 65536

    Note: If you’re using sudo to run Terraform, you might need to use sudo su - first to get a root shell, then set the ulimit, or use sudo sh -c "ulimit -n 65536 && terraform apply". However, it’s generally better to run Terraform as a regular user with appropriate permissions.

  3. Run Terraform: Now, execute your Terraform command from the same shell:

    terraform plan
# or
terraform apply

If the issue disappears, it confirms that the file descriptor limit was indeed the problem. Proceed to the “Configuration Check” section to make this change persistent.

3. Configuration Check: Making Changes Persistent

For a permanent solution, you need to modify system-wide configuration files. There are a few key places to check and adjust on Debian 11:

3.1. User Limits with /etc/security/limits.conf

This is the primary file for configuring resource limits for users and groups. Changes here require logging out and back in, or a reboot, to take effect for user sessions.

  1. Edit limits.conf: Open the file with sudo:

    sudo nano /etc/security/limits.conf

  2. Add/Modify Entries: Add or modify the following lines at the end of the file. We’ll set both soft and hard limits for all users (*).

    • soft limit: The default limit that processes get. This can be temporarily overridden by the user up to the hard limit.
    • hard limit: The absolute ceiling that a user or process cannot exceed.
    # Increase file descriptor limit for all users
*  soft  nofile  65536
*  hard  nofile  131072
    • Explanation:
      • *: Applies to all users. You can replace * with a specific username (e.g., devopsuser) or group (e.g., @devopsgroup) if you want to restrict the change.
      • soft: The soft limit for file descriptors.
      • hard: The hard limit for file descriptors.
      • nofile: Specifies the maximum number of open files.
      • 65536: A commonly recommended value for systems running heavy I/O applications.
      • 131072: A higher hard limit to allow for temporary increases if needed.

  3. Save and Exit: Save the file (Ctrl+O, Enter) and exit Nano (Ctrl+X).

  4. Important Note for limits.conf: For limits.conf changes to take effect for login shells, ensure that the pam_limits.so module is correctly enabled. On Debian, this is typically done by default via /etc/pam.d/common-session or /etc/pam.d/login. You might see a line like:

    session required pam_limits.so

    If it’s commented out or missing, you may need to uncomment/add it.

3.2. Kernel-wide File Descriptor Limit with /etc/sysctl.conf

While limits.conf sets per-user/per-process limits, the kernel itself has a maximum number of file descriptors it can manage globally. This limit (fs.file-max) is usually very high by default, but it’s good to be aware of it and adjust if your hard nofile limit in limits.conf approaches or exceeds it.

  1. Edit sysctl.conf: Open the file with sudo:

    sudo nano /etc/sysctl.conf

  2. Add/Modify Entry: Add or modify the following line:

    # Increase kernel-wide file descriptor limit
fs.file-max = 200000
    • Explanation: This sets the maximum number of file handles the kernel can allocate. This should always be higher than any hard nofile limit you set in limits.conf.

  3. Save and Exit: Save the file (Ctrl+O, Enter) and exit Nano (Ctrl+X).

  4. Apply Changes Immediately: For sysctl.conf changes to take effect without a reboot:

    sudo sysctl -p

3.3. Systemd Service Units (If Terraform Runs as a Service/CI Agent)

If Terraform is executed by a CI/CD agent (e.g., Jenkins, GitLab Runner) or a custom systemd service, the limits defined in limits.conf might not apply directly to the service daemon. In such cases, you need to set the LimitNOFILE directive within the relevant systemd service unit file.

  1. Identify the Service: Find the .service file for your CI agent or custom Terraform service (e.g., /etc/systemd/system/jenkins.service).

  2. Edit the Service File:

    sudo systemctl edit --full <your-service-name.service>
# e.g., sudo systemctl edit --full jenkins.service

  3. Add/Modify LimitNOFILE: Under the [Service] section, add or modify the LimitNOFILE directive:

    [Service]
# ... other directives ...
LimitNOFILE=65536

  4. Reload Systemd and Restart Service:

    sudo systemctl daemon-reload
sudo systemctl restart <your-service-name.service>

4. Verification: How to Test Your Changes

After making persistent configuration changes, it’s crucial to verify that they have been applied correctly.

  1. Log Out and Log In (or Reboot): For changes in /etc/security/limits.conf to take effect for your user, you must log out of your current session and log back in. A full system reboot is also an option if you prefer to ensure all changes across the board are applied.

  2. Check Your User’s Limit: After logging back in, open a new terminal and check your ulimit:

    ulimit -n

    This should now reflect your new soft nofile limit (e.g., 65536).

  3. Check a Running Process’s Limit (Optional): If you have a long-running process that might be hitting the limit (e.g., a CI agent), you can check its specific limits:

    • Find the Process ID (PID): 
      pgrep -a terraform # or pgrep -a jenkins
    • Check its limits using the PID: 
      cat /proc/<PID>/limits
      Look for Max open files. It should show your new limits.

  4. Run Terraform Again: Execute your problematic Terraform command once more.

    terraform plan
# or
terraform apply

    With the increased file descriptor limits, Terraform should now complete its operation without encountering the “Too Many Open Files” error.

By systematically addressing the file descriptor limits at the appropriate system levels, you can effectively resolve this common issue and ensure your Terraform operations run smoothly on Debian 11.

⚡ Related Troubleshooting Guides