How to Fix Ansible 403 Forbidden on AWS Lambda

Troubleshooting Guide: Ansible 403 Forbidden on AWS Lambda

As Senior DevOps Engineers, we’ve all encountered the dreaded “403 Forbidden” error. When it pops up while Ansible is trying to interact with AWS Lambda, it points directly to an access control problem within the AWS ecosystem. This guide will help you diagnose and resolve this issue efficiently.

1. The Root Cause: Why this happens on AWS Lambda

A “403 Forbidden” error from the AWS API, when Ansible attempts to manage Lambda resources, indicates that the IAM principal (the IAM user, IAM role, or temporary credentials) that Ansible is using lacks the necessary permissions to perform the requested action on the target Lambda resource.

It’s crucial to understand that this is not an issue with your Lambda function’s execution role (the role Lambda assumes to run your code and access other AWS services). Instead, it’s about the permissions of the entity running Ansible to interact with the AWS Lambda API itself. Ansible acts as a client making API calls to AWS; a 403 means AWS denied that specific call based on the credentials provided.

Common scenarios include:

  • Trying to create, update, or delete a function without lambda:CreateFunction, lambda:UpdateFunctionConfiguration, or lambda:DeleteFunction permissions.
  • Attempting to invoke a function without lambda:InvokeFunction permission.
  • Listing functions without lambda:ListFunctions or lambda:GetFunction permissions.
  • Accessing a resource in a different region or account for which the principal has no permissions.

2. Quick Fix (CLI)

For immediate testing or if you need to quickly establish baseline functionality, you can attach a broad AWS managed policy. Remember, this is for quick diagnosis, not a long-term production solution due to the principle of least privilege.

  1. Identify the IAM Principal: First, determine which IAM principal Ansible is using. If running locally, it’s often linked to your default AWS CLI profile or environment variables. If on an EC2 instance, it’s the instance profile’s role.

    To check your current AWS CLI identity (which Ansible often inherits if not explicitly configured otherwise):

    aws sts get-caller-identity

    This will output details like UserId, Account, and Arn. The Arn will tell you if you’re an IAM User or an assumed role.

  2. Attach a Broad Policy (e.g., AWSLambda_FullAccess):

    • If your Ansible principal is an IAM User:

      aws iam attach-user-policy --user-name <YourIAMUserName> --policy-arn arn:aws:iam::aws:policy/AWSLambda_FullAccess

      Replace <YourIAMUserName> with the name from your sts get-caller-identity output (e.g., arn:aws:iam::<AccountID>:user/<YourIAMUserName>).

    • If your Ansible principal is an IAM Role (e.g., EC2 Instance Profile):

      aws iam attach-role-policy --role-name <YourIAMRoleName> --policy-arn arn:aws:iam::aws:policy/AWSLambda_FullAccess

      Replace <YourIAMRoleName> with the role name (e.g., arn:aws:iam::<AccountID>:role/<YourIAMRoleName>).

    After attaching, wait a minute or two for policy propagation, then re-run your Ansible playbook. If the 403 error disappears, you’ve confirmed it’s an IAM permissions issue. You can then proceed to refine the policy.

3. Configuration Check

To implement a secure and robust solution, you’ll need to verify and potentially modify your IAM policies and Ansible’s AWS credential configuration.

3.1. IAM Policy Refinement (Principle of Least Privilege)

If the quick fix worked, your next step is to create a custom IAM policy that grants only the necessary permissions.

  1. Identify Required Actions: Review your Ansible playbook or the failing command. What specific Lambda actions is it trying to perform?

    • Creating/Updating functions: lambda:CreateFunction, lambda:UpdateFunctionConfiguration, lambda:DeleteFunction
    • Invoking functions: lambda:InvokeFunction
    • Reading function details: lambda:GetFunction, lambda:ListFunctions
    • Managing aliases/versions: lambda:CreateAlias, lambda:UpdateAlias, lambda:PublishVersion
    • Managing permissions: lambda:AddPermission, lambda:RemovePermission

  2. Craft the IAM Policy Document: Create a JSON policy document with specific actions and resource ARNs.

    Example: Policy to allow updating a specific Lambda function:

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "lambda:UpdateFunctionCode",
                "lambda:UpdateFunctionConfiguration",
                "lambda:GetFunction",
                "lambda:ListFunctions"
            ],
            "Resource": "arn:aws:lambda:<region>:<account-id>:function:<your-function-name>"
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::<account-id>:role/<your-lambda-execution-role-name>"
        }
    ]
}
    • iam:PassRole: This permission is crucial if your Ansible playbook is creating or updating Lambda functions and specifies an IAM role for the function’s execution. Without iam:PassRole on the execution role, Ansible cannot assign that role to the Lambda function, resulting in a 403.
    • Replace <region>, <account-id>, <your-function-name>, and <your-lambda-execution-role-name> with your specific values. Use * for resources if you need to manage all Lambda functions, but always prefer specific ARNs.

  3. Attach the Custom Policy: Create the policy (if it doesn’t exist) and attach it to your Ansible principal (IAM User or Role) using the AWS CLI or IAM console.

    # Create the policy (if new)
aws iam create-policy --policy-name AnsibleLambdaManager --policy-document file://ansible-lambda-policy.json

# Attach to user
aws iam attach-user-policy --user-name <YourIAMUserName> --policy-arn arn:aws:iam::<account-id>:policy/AnsibleLambdaManager

# OR attach to role
aws iam attach-role-policy --role-name <YourIAMRoleName> --policy-arn arn:aws:iam::<account-id>:policy/AnsibleLambdaManager

3.2. Ansible AWS Credentials and Region

Ensure Ansible is using the correct credentials and targeting the correct region.

  • Environment Variables: 
    export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="wJalr..."
export AWS_SESSION_TOKEN="FQoG..." # If using temporary credentials
export AWS_REGION="us-east-1"
  • AWS CLI Profile (most common for local execution): Ensure your ~/.aws/credentials and ~/.aws/config files are set up correctly, and Ansible is configured to use the right profile (either via AWS_PROFILE env var or profile parameter in Ansible modules).
  • Ansible Playbook Parameters: Many community.aws modules allow specifying aws_access_key, aws_secret_key, aws_session_token, and region directly. While convenient for testing, hardcoding credentials in playbooks is highly discouraged for security reasons. 
    - name: Deploy Lambda function
  community.aws.lambda_function:
    state: present
    name: my-lambda-function
    runtime: python3.9
    handler: index.handler
    role: arn:aws:iam::{{ account_id }}:role/lambda_exec_role
    zip_file: /path/to/my_function.zip
    region: us-east-1 # Explicitly set region
    # aws_access_key: "{{ lookup('env', 'AWS_ACCESS_KEY_ID') }}" # Discouraged practice
    # ...

4. Verification

Once you’ve adjusted policies or credentials, verify the fix.

  1. AWS CLI Test: Before re-running Ansible, try the equivalent AWS CLI command that Ansible was attempting, using the same credentials/profile. This helps isolate if the issue is with Ansible’s interaction or purely an IAM permission problem.

    • Example (if Ansible was failing on lambda_function update): 
      aws lambda get-function --function-name <your-function-name> --region <your-region>
# If this works, try an update
# aws lambda update-function-configuration --function-name <your-function-name> --timeout 60 --region <your-region>

    If the AWS CLI command succeeds, your IAM permissions are likely correct, and you might need to re-examine Ansible’s specific module parameters or environment. If it still fails with 403, the IAM policy needs further refinement.

  2. Run Ansible Playbook: Execute your Ansible playbook or specific task that was previously failing. Observe the output for success or any new errors.

  3. Check AWS CloudTrail: For definitive proof of what happened, go to the AWS CloudTrail console. Filter events by:

    • Event Name: *Lambda* (or the specific action, e.g., UpdateFunctionConfiguration)
    • Event Source: lambda.amazonaws.com
    • Read-Only/Write-Only: Filter to Write if you’re trying to modify resources.
    • Look for AccessDenied errors. The eventSource, eventName, and errorMessage within the event details will provide precise information about which permission was missing for which resource. This is your ultimate source of truth for IAM debugging.

By systematically addressing the IAM permissions of your Ansible principal, ensuring correct credential configuration, and verifying with both CLI and CloudTrail, you can effectively resolve “Ansible 403 Forbidden” errors when managing AWS Lambda.

⚡ Related Troubleshooting Guides