How to Fix MongoDB Broken Pipe on CentOS 7

As Senior DevOps Engineers at WebToolsWiz.com, we frequently encounter and resolve critical issues. The “MongoDB Broken Pipe” error on CentOS 7 is a common hurdle, often signaling an underlying resource constraint rather than a direct MongoDB malfunction. This guide will walk you through diagnosing and resolving it professionally.

Troubleshooting Guide: MongoDB Broken Pipe on CentOS 7

The “MongoDB Broken Pipe” error, often seen in client applications or within MongoDB’s logs, typically indicates that a connection to the MongoDB server was unexpectedly terminated. On CentOS 7, this issue is most frequently rooted in system resource limits.

1. The Root Cause: Why this happens on CentOS 7

On CentOS 7, the most prevalent cause for “MongoDB Broken Pipe” errors is insufficient file descriptor limits (ulimit -n). MongoDB, being a high-performance database, requires a substantial number of open file descriptors for several reasons:

  • Data Files and Journaling: MongoDB opens numerous files for its data, indexes, and journaling (WiredTiger storage engine primarily).
  • Network Sockets: Each incoming client connection, as well as internal connections (e.g., between replica set members), consumes a file descriptor.
  • Log Files: Standard log files also require descriptors.

When the mongod process attempts to open a new file or establish a new socket connection and hits the operating system’s configured nofile (number of open files) limit, the operation fails. This can lead to clients experiencing “broken pipe” errors because their connection is abruptly closed or cannot be established. Default CentOS 7 ulimit values are often conservative (e.g., 1024 open files) and are insufficient for production MongoDB instances, especially under load.

Less commonly, but still relevant, nproc (number of processes/threads) limits can also play a role, as MongoDB utilizes multiple threads for various operations.

2. Quick Fix (CLI)

To temporarily alleviate the issue and confirm the problem is related to ulimit, we can modify the running MongoDB service’s limits and restart it.

  1. Check Current mongod Process Limits: First, find the Process ID (PID) of the running mongod instance:

    pgrep mongod

    Let’s assume the PID is 12345. Now, check its current limits:

    sudo cat /proc/12345/limits | grep "Max open files"
sudo cat /proc/12345/limits | grep "Max processes"

    This will show you the “Soft Limit” and “Hard Limit” for the running process. If these are low (e.g., 1024), they are likely the culprit.

  2. Temporarily Increase Limits for the mongod Service: CentOS 7 uses systemd to manage services. The most direct way to temporarily apply a higher limit is to create a systemd override.

    sudo systemctl edit mongod.service

    This command will open a temporary editor (usually vi or nano). Add the following lines to increase the limits. A good starting point is 64000 or 65535 for nofile and nproc.

    [Service]
LimitNOFILE=64000
LimitNPROC=64000

    Save and exit the editor. systemd will automatically create an override file (/etc/systemd/system/mongod.service.d/override.conf) and reload the daemon.

  3. Restart MongoDB Service:

    sudo systemctl restart mongod

    Note: This is a quick fix. For persistent changes, follow the “Configuration Check” section below.

3. Configuration Check

For a robust and persistent solution, you need to configure ulimit values at both the system level and within the systemd service definition for MongoDB.

  1. System-Wide Limits (/etc/security/limits.conf): This file defines resource limits for users and groups. Add or modify the following lines, typically at the end of the file. Replace mongodb with the actual user MongoDB runs as if different (e.g., root or a custom user).

    sudo vi /etc/security/limits.conf

    Add:

    # MongoDB limits
mongodb        soft    nofile         64000
mongodb        hard    nofile         64000
mongodb        soft    nproc          64000
mongodb        hard    nproc          64000
    • soft limit: The current limit that the system enforces.
    • hard limit: The maximum value that the soft limit can be increased to by an unprivileged user.
    • nofile: Number of open file descriptors.
    • nproc: Number of processes/threads.

    Note: For limits.conf to take effect, the system’s PAM (Pluggable Authentication Modules) configuration must be set up to read it. On CentOS 7, this is typically handled by default via /etc/pam.d/system-auth and common-session including pam_limits.so. A reboot or a re-login of the user running mongod (if not systemd-managed) might be required, though systemd offers a more direct approach for services.

  2. Systemd Service Specific Limits (/etc/systemd/system/mongod.service.d/override.conf): Even with limits.conf set, systemd can sometimes override or ignore these for services. The most reliable way to set limits for mongod is directly within its systemd unit configuration. Use systemctl edit to create or modify an override file, which is the recommended practice:

    sudo systemctl edit mongod.service

    Add or ensure the following lines are present within the [Service] section:

    [Service]
LimitNOFILE=64000
LimitNPROC=64000

    Save and exit. systemd will automatically reload the daemon when you exit the editor.

    Alternatively, you could directly edit /lib/systemd/system/mongod.service, but this is discouraged as it might be overwritten by package updates. Creating an override file is safer.

  3. Restart MongoDB Service: After making persistent configuration changes, restart the MongoDB service to apply them:

    sudo systemctl restart mongod

4. Verification

After applying the fixes, it’s crucial to verify that the changes have taken effect and that MongoDB is operating correctly.

  1. Check MongoDB Service Status: Confirm the service is running without errors:

    sudo systemctl status mongod

    Look for Active: active (running) and check recent log entries for any new error messages.

  2. Verify mongod Process Limits: Get the new PID of the mongod process after restart:

    pgrep mongod

    Then check its limits again. Replace NEW_PID with the actual PID:

    sudo cat /proc/NEW_PID/limits | grep "Max open files"
sudo cat /proc/NEW_PID/limits | grep "Max processes"

    You should now see 64000 (or your configured value) for both “Soft Limit” and “Hard Limit.”

  3. Check MongoDB Logs: Examine MongoDB’s logs for any recurring “Broken Pipe” errors or new warnings/errors.

    sudo journalctl -u mongod --since "5 minutes ago"
# Or, if using a dedicated log file:
sudo tail -f /var/log/mongodb/mongod.log

  4. Test MongoDB Connectivity and Functionality:

    • Try connecting from a client application or the mongo shell.
    • Perform typical database operations (insert, query, update, delete).
    • If you have a monitoring system, observe the number of open connections and file descriptors used by the mongod process to ensure it’s operating within the new limits and not hitting ceilings.

By systematically addressing the ulimit configuration on your CentOS 7 system, you can effectively resolve the “MongoDB Broken Pipe” error and ensure the stability and performance of your database.

⚡ Related Troubleshooting Guides