How to Fix Ansible 403 Forbidden on CentOS 7

Troubleshooting “Ansible 403 Forbidden” on CentOS 7

As a Senior DevOps Engineer, encountering a “403 Forbidden” error when Ansible attempts to connect to a target host is a common roadblock. While the HTTP status code “403 Forbidden” typically signifies that a web server understands the request but refuses to authorize it, in the context of Ansible, it almost universally points to an SSH authentication or authorization failure on the target CentOS 7 machine. Ansible primarily relies on SSH for communication, and a “403 Forbidden” message from Ansible’s perspective means the SSH connection was established, but the remote host denied the subsequent operations, usually due to insufficient permissions or system-level security restrictions.

Let’s break down the common causes and how to resolve them on CentOS 7.

1. The Root Cause: Why this happens on CentOS 7

On CentOS 7, the “403 Forbidden” (or more accurately, an SSH permission denied error that Ansible interprets similarly) typically stems from one or more of the following security mechanisms and configurations:

  • SELinux (Security-Enhanced Linux): This is by far the most frequent culprit on CentOS. SELinux might be preventing the sshd daemon from accessing user home directories, .ssh directories, or authorized_keys files, even if standard file permissions appear correct.
  • FirewallD: The default firewall on CentOS 7 (firewalld) may be blocking SSH connections (port 22) from the Ansible controller.
  • Incorrect File/Directory Permissions: The SSH daemon is very strict about the permissions of the user’s home directory, the .ssh directory, and the authorized_keys file on the target host. Any misconfiguration here will lead to a rejection.
  • SSH Daemon Configuration (sshd_config): Restrictive settings in /etc/ssh/sshd_config on the target host (e.g., PermitRootLogin no, PasswordAuthentication no, AllowUsers, DenyUsers, incorrect AuthorizedKeysFile paths) can prevent successful authentication.
  • Missing or Incorrect SSH Key: The public key for the Ansible user might not be present in ~/.ssh/authorized_keys on the target host, or the private key on the Ansible controller might be incorrect or missing.

2. Quick Fix (CLI)

These commands provide immediate, though sometimes temporary or less secure, workarounds. Always understand the security implications before applying them in production.

On the Target CentOS 7 Machine:

  1. Temporarily Disable SELinux: This is often the quickest way to confirm if SELinux is the issue.

    sudo setenforce 0
# Verify status
sestatus
# Output should show "Current mode: permissive"

    Note: setenforce 0 changes the mode to permissive until the next reboot or until setenforce 1 is run. This is NOT a permanent solution.

  2. Ensure SSH Service is Running and Enabled:

    sudo systemctl start sshd
sudo systemctl enable sshd
sudo systemctl status sshd

  3. Allow SSH through Firewalld:

    sudo firewall-cmd --zone=public --add-service=ssh --permanent
sudo firewall-cmd --reload
# Verify the service is active
sudo firewall-cmd --list-services --zone=public

    This is a relatively safe and persistent fix for the firewall.

  4. Correct SSH Directory and File Permissions: Assuming the user is ansible_user and their home directory is /home/ansible_user:

    sudo chmod 700 /home/ansible_user/.ssh
sudo chmod 600 /home/ansible_user/.ssh/authorized_keys
sudo chown -R ansible_user:ansible_user /home/ansible_user/.ssh

    The .ssh directory should only be writable by the owner. The authorized_keys file must only be readable by the owner.

On the Ansible Controller (if key transfer is the issue):

  • Copy SSH Public Key to Target: This assumes you have password authentication enabled temporarily or the target has already been provisioned with a working SSH key for another user. 
    ssh-copy-id ansible_user@your_target_ip
    You’ll be prompted for ansible_user’s password on the target.

3. Configuration Check (Permanent Solutions)

For a robust and secure environment, directly editing configuration files provides permanent solutions.

On the Target CentOS 7 Machine:

  1. SELinux Configuration (/etc/selinux/config and Contexts):

    • Preferred Method (Auditing & Contexts): If setenforce 0 resolves the issue, SELinux is the cause. Instead of disabling it, identify and resolve the context issue.
      • Check audit logs: sudo ausearch -c sshd | audit2allow -M my-ssh-policy (This command will suggest a policy to allow the blocked action. Review my-ssh-policy.te and my-ssh-policy.sh before running the sh script to install the policy.)
      • Restore default SELinux contexts: This is often needed after manually moving/creating files. 
        sudo restorecon -Rv /home/ansible_user/.ssh
      • Set correct SELinux context manually (less common for .ssh): 
        sudo semanage fcontext -a -t ssh_home_t "/home/ansible_user/.ssh(/.*)?"
sudo restorecon -Rv /home/ansible_user/.ssh
    • Permanent Permissive Mode (Less Secure, for specific cases): If you must disable SELinux for specific non-production environments, edit: 
      sudo vi /etc/selinux/config
      Change SELINUX=enforcing to SELINUX=permissive. Reboot for this change to take full effect. Warning: Disabling SELinux significantly reduces system security.

  2. FirewallD Configuration (/etc/firewalld/zones/public.xml): The firewall-cmd commands in the “Quick Fix” section are the recommended way to configure Firewalld permanently. You can verify the configuration by checking /etc/firewalld/zones/public.xml (or the relevant zone file) for a line like <service name="ssh"/>.

  3. SSH Daemon Configuration (/etc/ssh/sshd_config): Ensure these critical settings are configured correctly for key-based authentication:

    sudo vi /etc/ssh/sshd_config
    • Port 22 (or your custom SSH port)
    • PubkeyAuthentication yes (ensure this is enabled)
    • AuthorizedKeysFile .ssh/authorized_keys (ensure this points to the correct location relative to the user’s home directory)
    • PasswordAuthentication no (recommended for production, but means you MUST use keys)
    • PermitRootLogin no (recommended for production; if trying to connect as root, you’ll need PermitRootLogin yes or prohibit-password with keys, but using a non-root user and sudo is preferred).
    • Ensure any AllowUsers or DenyUsers directives do not restrict your Ansible user.

    After making changes, restart the SSH service:

    sudo systemctl restart sshd

4. Verification

After applying any fixes, it’s crucial to verify connectivity from your Ansible controller.

  1. Direct SSH Connection (Verbose for Debugging): This is your primary tool for diagnosing SSH issues outside of Ansible.

    ssh -v ansible_user@your_target_ip
    • Look for output indicating successful authentication.
    • If it fails, the verbose output (-v, you can add -vvv for even more detail) will often explicitly state why (e.g., “Permission denied (publickey,gssapi-keyex,gssapi-with-mic).”, “bad ownership or modes”, “SELinux policy prevented”).

  2. Ansible ping Module: Once ssh -v works, try the Ansible ping module.

    ansible all -m ping -u ansible_user --private-key /path/to/your/private_key.pem -i /path/to/your/inventory.ini

    (Replace all with your target host pattern, -u with your user, and -i with your inventory if not using the default.)

    A successful output will look like:

    your_target_ip | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}

  3. Check Logs on Target Host: If SSH or Ansible still fails, review the authentication logs on the target host:

    sudo journalctl -u sshd -n 50 --no-pager
# Or for older systems/more detail sometimes
sudo tail -f /var/log/secure

    Look for specific error messages related to authentication failures, permission denied, or SELinux AVC denials. These logs will provide the most precise reason for the 403 Forbidden (SSH denial) error.

By systematically working through these steps, you’ll be able to identify and resolve the root cause of “Ansible 403 Forbidden” errors on your CentOS 7 systems, ensuring smooth and secure automation.

⚡ Related Troubleshooting Guides