Troubleshooting “Next.js Broken Pipe” on CentOS 7

As Senior DevOps Engineers, we often encounter subtle system-level issues manifesting as application errors. One such common problem, particularly when deploying Node.js-based applications like Next.js on CentOS 7, is the dreaded “Broken Pipe” error (EPIPE). This guide will help you diagnose and resolve this issue.

---

1. The Root Cause: CentOS 7’s System Defaults

The “Broken Pipe” error (EPIPE) in the context of a Next.js application on CentOS 7 almost invariably points to insufficient system resource limits, specifically the number of open file descriptors (FDs).

Why this happens on CentOS 7:

• Default ulimit -n: CentOS 7, by default, often has a relatively low ulimit -n (number of open files) set for non-root users, frequently 1024 or 4096.
• Next.js Resource Usage:
  • During next build: The build process for complex Next.js applications can involve a significant number of file operations – reading source files, writing compiled assets, traversing dependency trees, and potentially spawning child processes. Each file or network socket opened consumes a file descriptor.
  • During Runtime: A running Next.js server (especially under load or serving many concurrent requests) will open numerous network sockets (for client connections, database connections, API calls, etc.), static assets, and log files. Each active connection or file stream consumes an FD.
• The Breakdown: When the Next.js application (or one of its child processes) attempts to open a file or create a socket, and the number of currently open FDs for that user/process exceeds its ulimit -n soft limit, the operation fails. If the application then tries to write to a resource whose underlying FD was closed or never opened successfully due to this limit, you get an EPIPE error, indicating that the write failed because the receiving end of the pipe (or socket) is no longer available.

While less common for a single application, other limits like nproc (number of processes) or the system-wide fs.file-max could also contribute if the issue is severe or involves many concurrent services.

---

2. Quick Fix (CLI - Temporary)

To immediately alleviate the issue for your current session and test if resource limits are indeed the problem, you can temporarily raise the file descriptor limit for the user running the Next.js application.

1. Identify the User: Determine which user account your Next.js application runs under (e.g., webuser, node, nginx, or a dedicated service user).

2. Switch to the User (if necessary):

   sudo su - <username> # Replace <username> with your application's user

3. Check Current Limits:

   ulimit -n

   You will likely see a value like 1024 or 4096.

4. Set New Temporary Limit: Choose a significantly higher value. 65536 is a common safe starting point.

   ulimit -n 65536

5. Verify New Limit:

   ulimit -n

6. Restart Next.js Application: Stop and restart your Next.js application within this same shell session where you set the ulimit. If you’re using PM2, ensure you restart it as the user with the elevated limits.

   # Example for a Next.js app started directly
   # (replace with your actual start command, e.g., 'npm start', 'yarn start', 'pm2 start index.js')
   npm run build # if the error occurs during build
   npm start

If the “Broken Pipe” error disappears, you’ve confirmed that file descriptor limits were the root cause. Now, you need to make these changes persistent.

---

3. Configuration Check (Permanent Fix)

For a robust and persistent solution, you need to modify system configuration files.

a. For All Sessions / Specific Users (/etc/security/limits.conf)

This file manages resource limits for users logging in via PAM (Pluggable Authentication Modules), which includes direct SSH logins and services started without systemd unit files (e.g., PM2 started by a user).

1. Edit limits.conf:

   sudo vi /etc/security/limits.conf

2. Add or Modify Entries: Add the following lines at the end of the file, replacing <username> with the actual user running your Next.js application. If you want this to apply to all non-root users, use *.

   # <domain> <type> <item> <value>
   <username>      soft    nofile  65536
   <username>      hard    nofile  65536
   # Optional: If you also suspect process limits
   # <username>      soft    nproc   4096
   # <username>      hard    nproc   4096

   • soft limit: The limit that is enforced for a user at login. It can be increased by the user up to the hard limit.
   • hard limit: The maximum limit that the user can ever set. Only root can increase a hard limit.
   • nofile: Refers to the maximum number of open file descriptors.
   • nproc: Refers to the maximum number of processes a user can own.

3. Save and Exit.

4. Reboot or Re-login: For these changes to take effect, the user needs to log out and log back in, or the system needs to be rebooted.

b. For systemd Managed Services

If your Next.js application is managed by a systemd service unit (highly recommended for production), you should specify the limits directly in its unit file.

1. Identify Service File: Your service file might be located at /etc/systemd/system/nextjs.service or /usr/lib/systemd/system/nextjs.service.

2. Edit the Service File:

   sudo vi /etc/systemd/system/nextjs.service

3. Add Limits to [Service] Section: Locate the [Service] section and add the LimitNOFILE and LimitNPROC directives.

   [Service]
   # ... existing configurations ...
   User=<username>           # Ensure this matches your app's user
   Group=<groupname>         # Ensure this matches your app's group
   LimitNOFILE=65536         # Set the max open files for this service
   LimitNPROC=4096           # Optional: Set max processes if needed
   # ... more configurations ...

4. Reload systemd and Restart Service:

   sudo systemctl daemon-reload
   sudo systemctl restart nextjs.service # Replace 'nextjs.service' with your actual service name

c. System-Wide File Descriptor Limit (/etc/sysctl.conf)

While less frequently the primary cause than ulimit -n, it’s good practice to ensure the system-wide limit for file descriptors is also sufficiently high, especially on busy servers.

1. Check Current System Limit:

   sysctl fs.file-max

2. Edit sysctl.conf:

   sudo vi /etc/sysctl.conf

3. Add or Modify Entry: Add or update the following line. A common value for fs.file-max on a production server is between 250000 and 1000000.

   fs.file-max = 500000

4. Save and Exit.
5. Apply Changes:

   sudo sysctl -p

---

4. Verification

After applying the permanent fixes, it’s crucial to verify that the changes have taken effect and that your Next.js application is stable.

1. Verify Limits for the Running Process: This is the most direct way to confirm your application is running with the desired limits.

   • Find Next.js Process ID (PID):

     pgrep -a node | grep nextjs # Adjust 'grep nextjs' if your app has a different identifier
     # Or more generally:
     ps aux | grep node

     Note down the PID of your main Next.js process.
   • Check Process Limits:

     cat /proc/<PID>/limits # Replace <PID> with the actual process ID

     Look for the “Max open files” line. It should reflect the new 65536 soft and hard limits.

2. Monitor Logs: Check your application’s logs for any recurring “Broken Pipe” or resource exhaustion errors.

   • If using systemd:

     sudo journalctl -u nextjs.service -f # Replace 'nextjs.service'

   • If logging to a file:

     tail -f /path/to/your/nextjs/app.log

3. Simulate Load (Stress Test): If the error primarily occurred under load or during build, simulate those conditions to confirm stability.

   • For build issues: Rerun npm run build or yarn build multiple times.
   • For runtime issues: Use a load testing tool like ApacheBench (ab), wrk, or k6 to send a high volume of requests to your Next.js server.

     # Example using ApacheBench
     sudo yum install httpd-tools -y # If not already installed
     ab -n 10000 -c 100 http://localhost:3000/ # 10,000 requests, 100 concurrent

     Observe both the load test results and your application logs for any errors.

By systematically applying these steps, you should successfully resolve the “Next.js Broken Pipe” error on CentOS 7, ensuring your application runs reliably even under demanding conditions.