feat(rs): python udf deprecation scaffolding

[vdelacruzb]

Description

Shortcut

• Story: https://app.shortcut.com/cartoteam/story/509829/research-lambda-distribution-package-models
• Autolink: [sc-509829]

Also including:

• Integrate lambda deployment in AT CI CD [sc-509837]
• Migrate all the existing Redshift Python [sc-509838]
• Generate installable lambda packages [sc-509836]

Some changes to take into account:

• Notice that in order to deploy gateway and sql we have to deploy from the root passing cloud=redshift.
• There seems to be a bug in redshift that avoid us to include long enough lambda names. For that reason short RS_LAMBDA_PREFIX (10 characters or less) are expected. Also each lambda gets a simplified name.
• For simple gateway functions, the function.yaml defines the parameters and the .sql to deploy the external function is generated dynamically,
• There is a folder gateway/functions/_shared with shared code in order to avoid duplication. It's relevant specially in tilers, data... Please notice that due to it and including handlers some old code has been reorganised. I've tried to avoid as many changes as possible comparing with the original.
• Some fixtures like processing were updated as we are now using a newer version of scipy.
• A CLAUDE.md has been added on each repository. It's an autogenerated file just to give context to CLAUDE about the repo avoiding having to do it on every console that we open.
• The CI environment is really cleaned in case of error in deployment of tests. This might have to be fixed for others clouds.

Type of change

• Feature

Acceptance

Tests done:

Repository rules

• make lint, make lint-fix
• make test, make test-unit, make test-integration
• make deploy cloud=redshift (from the root of the repo and gateway). Also with filters.
• make create-package cloud=redshift. Also with filters. Depending of it including lds, import, etc. it will slightly change the readme generated and add steps to SETUP the cloud native gateway connection.

Created package installation

• Install (interactive) with different authentication methods. (Didn't try the 4th for EC2).
• Install (interactive) leaving empty the Execution and Invoke roles so they get created and assigned automatically.
• Install (interactive) keeping the same role names and schemas.

Others

• When Installing through the different methods gave a try to execute few functions.
• Checked that errors are propagated to redshift.
• Tested that lambda functions are overridden.
• Check that the versions in the README and package change with the clouds/redshift/version.txt
• Adding and removing dedicateds from PR.
• Checking that the CI artefacts are removed.

[github-actions]

Dedicated environment removed

[github-actions]

Dedicated dedicated_563_carto environment deployed in Redshift host redshift-serverless-shared.515242991920.us-east-1.redshift-serverless.amazonaws.com and carto_dev_data database

• Gateway functions (Lambda): ded-563-*
• Clouds functions (SQL UDFs): deployed

[github-actions]

Dedicated environment removed

[github-actions]

Dedicated dedicated_563_carto environment deployed in Redshift host redshift-serverless-shared.515242991920.us-east-1.redshift-serverless.amazonaws.com and carto_dev_data database

• Gateway functions (Lambda): ded-563-*
• Clouds functions (SQL UDFs): deployed

[Jesus89]

Good job!

I have one question. Are the placekey, quadbin, quadkey, random and s2 public functions explicitly removed?

[vdelacruzb]

Good job!

I have one question. Are the placekey, quadbin, quadkey, random and s2 public functions explicitly removed?

Regarding this point. I've not really removed these modules, but moved most of the implementations to the gateway folder. I've done it that in functions that doesn't have multiple of complex wrappers on top.

An example, core/gateway/functions/quadbin/quadbin_fromlonglat will deploy the function QUADBIN_FROMLONGLAT. Another thing that I've moved to the gateway are the integrations test for these functions . So in this case: clouds/redshift/modules/test/quadbin/test_quadbin_fromlonglat got move to core/gateway/functions/quadbin/quadbin_fromlonglat/tests/integration/test_quadbin_fromlonglat.py. I move them because was weird to me having the whole implementation in the gateway and then only the integration test in clouds.

In other words the gateway folder structure should be able to take care of everything: deploying lambdas, external functions SQL (or full function sql implementations), unit tests and integrations tests. I've just migrated the minimum necessary (internal functions and functions with no or small wrappers) but the whole redshift (or others clouds) could be migrated.

[Jesus89]

OK, I see. Thanks for confirming.