Skip to content

Unable to rotate Aurora Postgres secret if using the ReadOnly endpoint as Host #146

@LukaHedtSV

Description

@LukaHedtSV

Hi there,

I'm attempting to set up an Aurora v2 Cluster with the following users with Auto-Rotate password turned on for all of them:

  • postgres (admin)
  • lambda_writer (connects to the Aurora Writer endpoint)
  • lambda_reader (connects to the Aurora Reader endpoint)

lambda_writer's setup is the default, and it seems to work properly.

I've set up the Database Secret so that lambda_reader's host value is the ReadOnly endpoint for Aurora, as I don't want any users connecting with this secret to tax a potentially stressed Writer instance.

Unfortunately, Aurora v2 Reader instances aren't consider by the RDS client to be "Read Replicas", so when the Multi-User Rotation lambda goes to set the secret, and sees that the hostnames for the master and child secrets differ, the "isReadReplica" (see below) function fails:

def is_rds_replica_database(replica_dict, master_dict):
"""Validates that the database of a secret is a replica of the database of the master secret
This helper function validates that the database of a secret is a replica of the database of the master secret.
Args:
replica_dict (dictionary): The secret dictionary containing the replica database
primary_dict (dictionary): The secret dictionary containing the primary database
Returns:
isReplica : whether or not the database is a replica
Raises:
ValueError: If the new username length would exceed the maximum allowed
"""
# Setup the client
rds_client = boto3.client('rds')
# Get instance identifiers from endpoints
replica_instance_id = replica_dict['host'].split(".")[0]
master_instance_id = master_dict['host'].split(".")[0]
try:
describe_response = rds_client.describe_db_instances(DBInstanceIdentifier=replica_instance_id)
except Exception as err:
logger.warning("Encountered error while verifying rds replica status: %s" % err)
return False
instances = describe_response['DBInstances']
# Host from current secret cannot be found
if not instances:
logger.info("Cannot verify replica status - no RDS instance found with identifier: %s" % replica_instance_id)
return False
# DB Instance identifiers are unique - can only be one result
current_instance = instances[0]
return master_instance_id == current_instance.get('ReadReplicaSourceDBInstanceIdentifier')

I've checked as to why with the following setup:

  • An Aurora v2 DbCluster with:
    • 1 Writer instance
    • 1 Reader instance

And then I got the following in the CLI:

aws rds describe-db-instances --region ap-southeast-2 | grep Replica
            "ReadReplicaDBInstanceIdentifiers": [],
            "ReadReplicaDBInstanceIdentifiers": [],

I think the potential solution is to check that the host value is an Endpoint for the given cluster potentially, since Secret.attach adds the dbClusterIdentifier to the Secret Value for Aurora DB Connections, but I'm not sure tbh.

Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions