Updating python version 3.10 Runtime.ImportModuleError for Aurora PostgreSQL

[jypil-scout24]

Since python v3.9 is nearing EOL, the lambda runtime automatically updates to v3.10

This caused function to fail with error:
[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named '_pg' Traceback (most recent call last):

[davemeekin]

Also seeing this on

Runtime Version: python:3.9.v101 Runtime Version ARN: arn:aws:lambda:eu-central-1::runtime:af29e10439856e364100c5dec1ce9c55d44feb2772258f7a7e480a95474aa18f

[rstevens011]

How were these rotation lambdas deployed?

[davemeekin]

We took the cloud formation template provided by AWS and amended it to add a permission boundary

We deploy using CDK and create a cloudformation stack from our SAM template as we need to pass in a permission boundary (We're waiting on #27).

We've just done a change set against the aws template to generate a new template. This has updated the runtime to Python 3.10. This seems to be working for us now.

New Template

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Parameters:
  endpoint:
    Type: String
    Description: The Secrets Manager endpoint to use.
  functionName:
    Type: String
    Description: The name of the Lambda function.
  invokingServicePrincipal:
    Type: String
    Description: The service principal for the invoking service.
    Default: "secretsmanager.amazonaws.com"
  vpcSubnetIds:
    Type: CommaDelimitedList
    Description: A comma-separated list of VPC subnet IDs applied to the database network.
    Default: ""
  vpcSecurityGroupIds:
    Type: CommaDelimitedList
    Description: A comma-separated list of security group IDs applied to the database.
    Default: ""
  kmsKeyArn:
    Type: String
    Description: The ARN of the KMS key that Secrets Manager uses to encrypt the secret.
    Default: ""
  excludeCharacters:
    Type: String
    Description: A string of the characters that you don't want in the password.
    Default: ":/@\"'\\"
  passwordLength:
    Type: Number
    Description: The length of the password.
    Default: 32
    MinValue: 1
    MaxValue: 4096
  excludeNumbers:
    Type: String
    Description: Specifies whether to exclude numbers from the password.
    Default: false
    AllowedValues:
      - true
      - false
  excludePunctuation:
    Type: String
    Description: "Specifies whether to exclude the following punctuation characters from the password: ! \" # $ % & ' ( ) * + , - . / : ; < = > ? @ [  ] ^ _ ` { | } ~."
    Default: false
    AllowedValues:
      - true
      - false
  excludeUppercase:
    Type: String
    Description: Specifies whether to exclude uppercase letters from the password.
    Default: false
    AllowedValues:
      - true
      - false
  excludeLowercase:
    Type: String
    Description: Specifies whether to exclude lowercase letters from the password.
    Default: false
    AllowedValues:
      - true
      - false
  requireEachIncludedType:
    Type: String
    Description: Specifies whether to include at least one upper and lowercase letter, one number, and one punctuation.
    Default: true
    AllowedValues:
      - true
      - false
  runtime:
    Type: String
    Description: The python runtime associated with the Lambda function
    Default: "python3.10"
  customPermissionBoundary:
    Type: String
    Description: Permission Boundary to attach to role
    Default: ""
  rotationRoleArn:
    Type: String
    Description: Role to perform the rotation
    Default: ""
Conditions:
  AddVpcConfig: !And
    - !Not [!Equals ["", !Join ["", !Ref vpcSubnetIds]]]
    - !Not [!Equals ["", !Join ["", !Ref vpcSecurityGroupIds]]]
  KmsKeyArnExists: !Not [!Equals ["", !Ref kmsKeyArn]]
Resources:
  SecretsManagerRDSPostgreSQLRotationSingleUser:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName:
        Ref: functionName
      Description: Rotates a Secrets Manager secret for Amazon RDS PostgreSQL credentials using the single user rotation strategy.
      Handler: lambda_function.lambda_handler
      Runtime: !Ref "runtime"
      CodeUri: s3://secrets-manager-rotation-apps-9e825f5413b8ec477b47f17943b55cee/SecretsManagerRDSPostgreSQLRotationSingleUser/SecretsManagerRDSPostgreSQLRotationSingleUser.zip
      Timeout: 30
      Policies:
        - VPCAccessPolicy: {}
        - AWSSecretsManagerRotationPolicy:
            FunctionName:
              Ref: functionName
        - !If
          - KmsKeyArnExists
          - Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - kms:Decrypt
                  - kms:DescribeKey
                  - kms:GenerateDataKey
                Resource: !Ref kmsKeyArn
          - !Ref "AWS::NoValue"
      Environment:
        Variables:
          SECRETS_MANAGER_ENDPOINT:
            Ref: endpoint
          EXCLUDE_CHARACTERS:
            Ref: excludeCharacters
          PASSWORD_LENGTH:
            Ref: passwordLength
          EXCLUDE_NUMBERS:
            Ref: excludeNumbers
          EXCLUDE_PUNCTUATION:
            Ref: excludePunctuation
          EXCLUDE_UPPERCASE:
            Ref: excludeUppercase
          EXCLUDE_LOWERCASE:
            Ref: excludeLowercase
          REQUIRE_EACH_INCLUDED_TYPE:
            Ref: requireEachIncludedType
      VpcConfig: !If
        - AddVpcConfig
        - SubnetIds: !Ref vpcSubnetIds
          SecurityGroupIds: !Ref vpcSecurityGroupIds
        - !Ref "AWS::NoValue"
      Tags:
        SecretsManagerLambda: Rotation
      PermissionsBoundary:
        Ref: customPermissionBoundary
      Role:
        Ref: rotationRoleArn
  LambdaPermission:
    Type: 'AWS::Lambda::Permission'
    Properties:
      Action: 'lambda:InvokeFunction'
      FunctionName: !GetAtt SecretsManagerRDSPostgreSQLRotationSingleUser.Arn
      Principal: !Ref invokingServicePrincipal
      SourceAccount: !Ref "AWS::AccountId"
Outputs:
  RotationLambdaARN:
    Description: The ARN of the rotation lambda
    Value: !GetAtt SecretsManagerRDSPostgreSQLRotationSingleUser.Arn

Our Previous Template:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Parameters:
  endpoint:
    Type: String
    Description: The Secrets Manager endpoint to use.
  functionName:
    Type: String
    Description: The name of the Lambda function.
  invokingServicePrincipal:
    Type: String
    Description: The service principal for the invoking service.
    Default: "secretsmanager.amazonaws.com"
  vpcSubnetIds:
    Type: CommaDelimitedList
    Description: A comma-separated list of VPC subnet IDs applied to the database network.
    Default: ""
  vpcSecurityGroupIds:
    Type: CommaDelimitedList
    Description: A comma-separated list of security group IDs applied to the database.
    Default: ""
  kmsKeyArn:
    Type: String
    Description: The ARN of the KMS key that Secrets Manager uses to encrypt the secret.
    Default: ""
  excludeCharacters:
    Type: String
    Description: A string of the characters that you don't want in the password.
    Default: ":/@\"'\\"
  runtime:
    Type: String
    Description: The python runtime associated with the Lambda function
    Default: "python3.9"
  customPermissionBoundary:
    Type: String
    Description: Permission Boundary to attach to role
    Default: ""
  rotationRoleArn:
    Type: String
    Description: Role to perform the rotation
    Default: ""
Conditions:
  AddVpcConfig: !And
    - !Not [!Equals ["", !Join ["", !Ref vpcSubnetIds]]]
    - !Not [!Equals ["", !Join ["", !Ref vpcSecurityGroupIds]]]
  KmsKeyArnExists: !Not [!Equals ["", !Ref kmsKeyArn]]
Resources:
  SecretsManagerRDSPostgreSQLRotationSingleUser:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName:
        Ref: functionName
      Description: Rotates a Secrets Manager secret for Amazon RDS PostgreSQL credentials using the single user rotation strategy.
      Handler: lambda_function.lambda_handler
      Runtime: !Ref "runtime"
      CodeUri: s3://secrets-manager-rotation-apps-9e825f5413b8ec477b47f17943b55cee/SecretsManagerRDSPostgreSQLRotationSingleUser/SecretsManagerRDSPostgreSQLRotationSingleUser.zip
      AutoPublishCodeSha256: c7d45d7e762dd829be5cf96c4deef0532b97d3e87274fcde5704a490d22eb233
      Timeout: 30
      Policies:
        - VPCAccessPolicy: {}
        - AWSSecretsManagerRotationPolicy:
            FunctionName:
              Ref: functionName
        - !If
          - KmsKeyArnExists
          - Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - kms:Decrypt
                  - kms:DescribeKey
                  - kms:GenerateDataKey
                Resource: !Ref kmsKeyArn
          - !Ref "AWS::NoValue"
      Environment:
        Variables:
          SECRETS_MANAGER_ENDPOINT:
            Ref: endpoint
          EXCLUDE_CHARACTERS:
            Ref: excludeCharacters
      VpcConfig: !If
        - AddVpcConfig
        - SubnetIds: !Ref vpcSubnetIds
          SecurityGroupIds: !Ref vpcSecurityGroupIds
        - !Ref "AWS::NoValue"
      Tags:
        SecretsManagerLambda: Rotation
      PermissionsBoundary:
        Ref: customPermissionBoundary
      Role:
        Ref: rotationRoleArn
  LambdaPermission:
    Type: 'AWS::Lambda::Permission'
    Properties:
      Action: 'lambda:InvokeFunction'
      FunctionName: !GetAtt SecretsManagerRDSPostgreSQLRotationSingleUser.Arn
      Principal: !Ref invokingServicePrincipal
      SourceAccount: !Ref "AWS::AccountId"
Outputs:
  RotationLambdaARN:
    Description: The ARN of the rotation lambda
    Value: !GetAtt SecretsManagerRDSPostgreSQLRotationSingleUser.Arn

[nithyasri-lakshminarasimhan]

Hi, can you clarify how the lambda was originally deployed and how exactly your lambda runtime automatically updates to v3.10?
For instance, the common deployment strategies include console, SAR, AWS::SecretsManager-2020-07-23 or AWS::SecretsManager-2024-09-16. Each of these deployment strategies has a specific update path as listed here which updates the dependent libraries as well
The reason for the import failures could be that while the python runtime was updated, the corresponding dependencies were not.

[jypil-scout24]

Hi, @nithyasri-lakshminarasimhan , Sorry for the late reply. We are using AWS::SecretsManager-2020-07-23. We don't explicitly define the Runtime in HostedRotationLambda. From the document, I understand that it is recommended to migrate to AWS::SecretsManager-2024-09-16. Does this mean AWS::SecretsManager-2020-07-23 does not support v3.10 runtime?

[nithyasri-lakshminarasimhan]

Hi, while it is incorrect to say that theAWS::SecretsManager-2020-07-23 cannot support v3.10 runtime, there is no direct update path to update to v3.10. The AWS::SecretsManager-2024-09-16 was introduced to make updates like this easier. So it would be the recommended path to upgrade to v3.10.