Bug: Docker image-based Lambda failures

[jack-e-tabaska]

Description:

sam deploy intermittently fails while creating Docker image-based Lambdas.

Steps to reproduce:

sam deploy
--stack-name ${STACK_NAME}
--capabilities CAPABILITY_IAM
--no-fail-on-empty-changeset
--resolve-s3
--parameter-overrides REDACTED
--image-repositories bigDumperLambda=${bigDumperRepoUri}
--image-repositories bqLoaderLambda=${bqLoaderRepoUri}
--image-repositories littleCheckerLambda=${littleCheckerRepoUri}
--image-repositories littleDumperLambda=${littleDumperRepoUri}
--image-repositories publisherLambda=${publisherRepoUri}
--image-repositories jobCheckerLambda=${jobCheckerRepoUri}
--image-repositories tableMakerLambda=${tableMakerRepoUri}
--tags exd_version=${EXD_VERSION}

Observed result:

Error message from CloudFormation: Resource handler returned message: "Lambda does not have permission to access the ECR image. Check the ECR permissions. (Service: Lambda, Status Code: 403, Request ID: 3afd69aa-201d-4f73-a500-e739b9bee696) (SDK Attempt Count: 1)" (RequestToken: 35c52deb-ee89-080f-0a66-e94ef9fb4f8e, HandlerErrorCode: AccessDenied)

When I check the ECR repository in question, I find that any preexisting permissions document has been removed.

I can usually resolve this by rerunning the sam deploy command.

Expected result:

Lambdas deployed without errors, on the first try

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Linux
2. sam --version: 1.142.1
3. AWS region: us-east-1