Serverless Data Lake Framework 2.11.2

What's Changed

• [sdlf-cicd] extract create_repository into new file, add API-support gitlab by @dlpzx in #543

• [sdlf-cicd] fix: add conditional on empty custom identifier SSM param by @dlpzx in #542

Full Changelog: 2.11.1...2.11.2

Serverless Data Lake Framework 2.11.1

What's Changed

• [sdlf-datalakelibrary] simplified datalake-library by @dlpzx in #540
• [sdlf-utils] fix legislators example script by @dlpzx in #541

Full Changelog: 2.11.0...2.11.1

Serverless Data Lake Framework 2.11.0

What's Changed

⚠️ Please review the pull request and its documentation before upgrading.

• [sdlf-cicd] blue/green support for upgrade from v1 to v2 + docs by @dlpzx in #537

New Contributors

• @dlpzx made their first contribution in #537

Full Changelog: 2.10.1...2.11.0

Serverless Data Lake Framework 2.10.1

What's Changed

• [sdlf-cicd] artifacts buckets by @cnfait in #534
• [sdlf-cicd] bootstrap reusability for data accounts by @cnfait in #535

Full Changelog: 2.10.0...2.10.1

Serverless Data Lake Framework 2.10.0

This is a new release for SDLF 2.x.

The repository has been moved to a new name, under a new GiHub organization (from awslabs to aws-solutions-library-samples). Everything else remains the same - licence, code, workshop, documentation.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• [sdlf-cicd] generic: pin seed-farmer version by @cnfait in #511
• improve architecture doc by @cnfait in #512
• Omnibus PR around CICD by @cnfait in #518
• [sdlf-stage-lambda/glue] shell script check by @cnfait in #519
• [sdlf-cicd] simpler retrieval of CICD role by @cnfait in #520
• [sdlf-stage-*] seedfarmer removal for emrserverless/ecsfargate by @cnfait in #521
• [sdlf-*] stack names use deployment type by @cnfait in #522
• [sdlf-cicd] use pcodebuildsuffix in artifacts bucket name by @cnfait in #523
• [sdlf-datalakeLibrary] update ruff noqa directive by @cnfait in #524
• [sdlf-utils] workshop cleanup script by @cnfait in #525
• [sdlf-*] don't set executable bit when unnecessary by @cnfait in #526
• [sdlf-cicd] handle workshop workflow by @cnfait in #527
• [sdlf-cicd] handle crossaccount role trust relationship by @cnfait in #528
• [sdlf-cicd] add missing sdlf-datalakeLibrary construct by @cnfait in #529
• [sdlf-utils] better handling of aws profile argument in legislators example by @cnfait in #530
• [sdlf-stage-lambda/glue] sync with workshop build by @cnfait in #531
• [docs] doc update for sdlf-cicd by @cnfait in #532
• update version to 2.10.0 by @cnfait in #533

Full Changelog: 2.9.0...2.10.0

Serverless Data Lake Framework 2.9.0

This is a new release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

The repository has been moved to a new name, under a new GiHub organization. Everything else remains the same - licence, code, workshop, documentation.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• remove pDataset/pPipeline/pStage parameters by @cnfait in #506
• [sdlf-cicd] generic: loop over list of prefixes instead of duplicating code by @cnfait in #507
• add sdlf-cicd generic manifest to cfnlint exclude list by @cnfait in #508
• update version to 2.9.0 by @cnfait in #509

Full Changelog: 2.8.0...2.9.0

Serverless Data Lake Framework 2.8.0

This is a new release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• [sdlf-cicd] simple cicd without codecommit - first take by @cnfait in #498
• [sdlf-utils] workshop update by @cnfait in #499
• Revert "remove replicate lambda" by @cnfait in #501
• [sdlf-cicd] rewrite deploy-generic.sh by @cnfait in #502
• AWS Solutions README.md by @cnfait in #503
• [sdlf-cicd] 2.8.0 doc update by @cnfait in #504

Full Changelog: 2.7.0...2.8.0

Serverless Data Lake Framework 2.7.0

This is a new release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• fix datalakeLibrary case in workshop files by @cnfait in #403
• update version to 2.7.0-alpha.0 by @cnfait in #404
• redesign sdlf-team and sdlf-dataset by @cnfait in #405
• add glue database for raw bucket by @gwenika in #406
• add glue database for analytics bucket by @cnfait in #407
• add glue crawler for raw bucket by @cnfait in #408
• add glue crawler for analytics bucket by @cnfait in #409
• sdlf-dataset can be deployed without a team by @cnfait in #410
• workshop examples: add stage- prefix to stages by @cnfait in #411
• remove replicate lambda by @cnfait in #412
• [sdlf-foundations][cdk] remove legacy resources by @cnfait in #413
• [sdlf-dataset][cdk] add peh, manifests tables by @cnfait in #414
• cdk: sdlf-dataset/sdlf-team design changes by @cnfait in #415
• [sdlf-team][sdlf-dataset][cdk] fix imports by @cnfait in #416
• [cdk] Enable key rotation for all KMS keys by @cnfait in #417
• [sdlf-dataset] deployment fixes by @cnfait in #418
• [sdlf-foundations][cdk] remove CentralBucket reference by @cnfait in #419
• [sdlf-foundations][cdk] CfnDataLakeSettings by @cnfait in #420
• [sdlf-foundations][cdk] data_bucket method to handle raw/stage/analytics buckets creation by @cnfait in #421
• rework sdlf-foundations (cdk) deployment with seedfarmer by @cnfait in #422
• rework sdlf-dataset (cdk) deployment with seedfarmer by @cnfait in #423
• [sdlf-dataset][cdk] data_catalog method to handle raw/stage/analytics catalogs creation by @cnfait in #424
• [sdlf-foundations][cdk] avoid using a hardcoded stack name by @cnfait in #425
• [sdlf-dataset][cdk] avoid using a hardcoded stack name by @cnfait in #426
• [sdlf-dataset][cdk] update dynamo table names by @cnfait in #427
• [cdk] remove pEnvironment parameter by @cnfait in #428
• [sdlf-dataset][cdk] remove pPipelineDetails parameter by @cnfait in #429
• [sdlf-foundations][cdk] replace octagon with sdlf in dynamo table name by @cnfait in #430
• [cdk] ssm parameters naming change by @cnfait in #431
• [sdlf-dataset][cdk] add sdlf:dataset lf-tag association for raw and analytics databases by @cnfait in #432
• [sdlf-dataset][cdk] add EMR Security Configuration SSM parameter by @cnfait in #433
• [sdlf-dataset][cdk] ruff linting by @cnfait in #434
• [sdlf-foundations][cdk] dynamo table: let cloudformation do the naming by @cnfait in #435
• [sdlf-foundations][cdk] ruff linting by @cnfait in #436
• [sdlf-team][cdk] ruff linting by @cnfait in #437
• [sdlf-pipeline][sdlf-stage-*][cdk] remove sdlf-team dependency by @cnfait in #438
• Ruff linting for all CDK constructs by @cnfait in #439
• [sdlf-foundations][cdk] ruff formatting by @cnfait in #440
• [sdlf-foundations][cdk] metadata for seedfarmer by @cnfait in #441
• [sdlf-dataset][cdk] metadata for seedfarmer by @cnfait in #442
• add seedfarmer.gitmodules to gitignore by @cnfait in #443
• [sdlf-foundations][cdk] ability to provide parameter values without --parameters by @cnfait in #444
• [sdlf-dataset][cdk] ability to provide parameter values without CfnParameter by @cnfait in #445
• [sdlf-pipeline][cdk] remove pEnv parameter by @cnfait in #446
• [sdlf-dataset][cdk] org and domain parameters by @cnfait in #447
• [cdk] rework sdlf-pipeline deployment with seedfarmer by @cnfait in #448
• [sdlf-datalakeLibrary] remove legacy octagon support by @cnfait in #449
• [sdlf-pipeline][sdlf-stage-lambda][cdk] rewrite by @cnfait in #450
• [sdlf-foundations/team/dataset/stage-lambda] cloudformation version without legacy resources by @tomaszwrzonski in #451
• [sdlf-stage-lambda] remove pEnvironment by @cnfait in #452
• [sdlf-foundations] use new ssm parameter format by @cnfait in #453
• [sdlf-foundations] remove object metadata dynamo table by @cnfait in #454
• [sdlf-team] remove unused parameters by @cnfait in #455
• [sdlf-dataset] use new ssm parameter format by @cnfait in #456
• [sdlf-stage-glue] cfn without legacy by @cnfait in #457
• [sdlf-team][cdk] remove unused parameters by @cnfait in #458
• [sdlf-stage-lambda/glue] new ssm parameter format, remove subscription filters by @cnfait in #459
• [sdlf-stage-lambda/glue] remove unused ELK parameters by @cnfait in #460
• [sdlf-foundations] support for multiple instances by @cnfait in #461
• [sdlf-foundations] support for 'anycompany' org name by @cnfait in #462
• [sdlf-foundations] add Deployment Instance output, rename pOrg/pDomain SSM by @cnfait in #463
• [sdlf-dataset] support for multiple instances by @cnfait in #464
• [sdlf-dataset] remove unused parameter by @cnfait in #465
• [sdlf-pipeline] cfn without legacy by @cnfait in #466
• [workshop build] sdlf-utils manifest files by @cnfait in #467
• [sdlf-stage-lambda/glue] update input and output ssm parameters by @cnfait in #468
• [sdlf-datalakeLibrary] support for multiple instances by @cnfait in #469
• [sdlf-stage-lambda] pass instance parameters to datalakeLibrary by @cnfait in https://github.com/awslabs/aws-serverless-data-lake-framewor...

Serverless Data Lake Framework 2.7.0-alpha.1

Work is ongoing on version 2.7.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.7.0 is officially available.

This alpha release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• [sdlf-dataset][cdk] data_catalog method to handle raw/stage/analytics catalogs creation by @cnfait in #424
• [sdlf-foundations][cdk] avoid using a hardcoded stack name by @cnfait in #425
• [sdlf-dataset][cdk] avoid using a hardcoded stack name by @cnfait in #426
• [sdlf-dataset][cdk] update dynamo table names by @cnfait in #427
• [cdk] remove pEnvironment parameter by @cnfait in #428
• [sdlf-dataset][cdk] remove pPipelineDetails parameter by @cnfait in #429
• [sdlf-foundations][cdk] replace octagon with sdlf in dynamo table name by @cnfait in #430
• [cdk] ssm parameters naming change by @cnfait in #431
• [sdlf-dataset][cdk] add sdlf:dataset lf-tag association for raw and analytics databases by @cnfait in #432
• [sdlf-dataset][cdk] add EMR Security Configuration SSM parameter by @cnfait in #433
• [sdlf-dataset][cdk] ruff linting by @cnfait in #434
• [sdlf-foundations][cdk] dynamo table: let cloudformation do the naming by @cnfait in #435
• [sdlf-foundations][cdk] ruff linting by @cnfait in #436
• [sdlf-team][cdk] ruff linting by @cnfait in #437
• [sdlf-pipeline][sdlf-stage-*][cdk] remove sdlf-team dependency by @cnfait in #438
• Ruff linting for all CDK constructs by @cnfait in #439
• [sdlf-foundations][cdk] ruff formatting by @cnfait in #440
• [sdlf-foundations][cdk] metadata for seedfarmer by @cnfait in #441
• [sdlf-dataset][cdk] metadata for seedfarmer by @cnfait in #442
• add seedfarmer.gitmodules to gitignore by @cnfait in #443
• [sdlf-foundations][cdk] ability to provide parameter values without --parameters by @cnfait in #444
• [sdlf-dataset][cdk] ability to provide parameter values without CfnParameter by @cnfait in #445

Full Changelog: 2.7.0-alpha.0...2.7.0-alpha.1

Serverless Data Lake Framework 2.7.0-alpha.0

Work is ongoing on version 2.7.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.7.0 is officially available.

This alpha release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• fix datalakeLibrary case in workshop files by @cnfait in #403
• update version to 2.7.0-alpha.0 by @cnfait in #404
• redesign sdlf-team and sdlf-dataset by @cnfait in #405
• add glue database for raw bucket by @gwenika in #406
• add glue database for analytics bucket by @cnfait in #407
• add glue crawler for raw bucket by @cnfait in #408
• add glue crawler for analytics bucket by @cnfait in #409
• sdlf-dataset can be deployed without a team by @cnfait in #410
• workshop examples: add stage- prefix to stages by @cnfait in #411
• remove replicate lambda by @cnfait in #412
• [sdlf-foundations][cdk] remove legacy resources by @cnfait in #413
• [sdlf-dataset][cdk] add peh, manifests tables by @cnfait in #414
• cdk: sdlf-dataset/sdlf-team design changes by @cnfait in #415
• [sdlf-team][sdlf-dataset][cdk] fix imports by @cnfait in #416
• [cdk] Enable key rotation for all KMS keys by @cnfait in #417
• [sdlf-dataset] deployment fixes by @cnfait in #418
• [sdlf-foundations][cdk] remove CentralBucket reference by @cnfait in #419
• [sdlf-foundations][cdk] CfnDataLakeSettings by @cnfait in #420
• [sdlf-foundations][cdk] data_bucket method to handle raw/stage/analytics buckets creation by @cnfait in #421
• rework sdlf-foundations (cdk) deployment with seedfarmer by @cnfait in #422
• rework sdlf-dataset (cdk) deployment with seedfarmer by @cnfait in #423

Full Changelog: 2.6.0...2.7.0-alpha.0