Skip to content

Commit 6dca61a

Browse files
committed
[docs] doc update for sdlf-cicd
1 parent fb87cc7 commit 6dca61a

File tree

2 files changed

+34
-20
lines changed

2 files changed

+34
-20
lines changed

README.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -66,19 +66,19 @@ The [SDLF workshop](https://sdlf.workshop.aws/) walks you through the deployment
6666

6767
1. Get the latest stable release of SDLF, unarchive it and cd to the new folder:
6868
```
69-
curl -L -O https://github.com/awslabs/aws-serverless-data-lake-framework/archive/refs/tags/2.8.0.tar.gz
70-
tar xzf 2.8.0.tar.gz
71-
cd ./aws-serverless-data-lake-framework-2.8.0/
69+
curl -L -O https://github.com/awslabs/aws-serverless-data-lake-framework/archive/refs/tags/2.9.0.tar.gz
70+
tar xzf 2.9.0.tar.gz
71+
cd ./aws-serverless-data-lake-framework-2.9.0/
7272
```
7373

7474
2. Deploy the CodeBuild projects for bootstrapping the rest of the infrastructure:
7575
```
7676
cd sdlf-cicd/
77-
./deploy-cicd.sh -p aws_profile_name datalake
7877
./deploy-role.sh -p aws_profile_name datalake
78+
./deploy-cicd.sh -p aws_profile_name datalake
7979
```
8080

81-
`./deploy-cicd.sh --help` and `./deploy-role.sh --help` give more details about available options.
81+
`./deploy-role.sh --help` and `./deploy-cicd.sh --help` give more details about available options.
8282

8383
3. Start the `sdlf-cicd-datalake` project and wait for it to complete. It creates an end-to-end data lake infrastructure, including data processing and consumption services.
8484

docs/constructs/cicd.md

Lines changed: 29 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -3,46 +3,60 @@
33
!!! note
44
`sdlf-cicd` is defined in the [sdlf-cicd](https://github.com/awslabs/aws-serverless-data-lake-framework/tree/main/sdlf-cicd) folder of the [SDLF repository](https://github.com/awslabs/aws-serverless-data-lake-framework).
55

6-
## Simplified CICD with SDLF > 2.8.0
6+
## Simplified CICD with SDLF 2.10.0
77

8-
**SDLF > 2.8.0** has a simplified CICD setup that can be used with any [source provider](https://docs.aws.amazon.com/codebuild/latest/userguide/access-tokens.html) supported by AWS CodeBuild. SDLF creates two CodeBuild projects able to deploy SDLF constructs (a third is created during the deployment process, but the user is never expected to interact directly with it), leaving the configuration of the source step to the user:
8+
!!! note
9+
Also check the [SDLF public workshop](https://catalog.us-east-1.prod.workshops.aws/workshops/501cb14c-91b3-455c-a2a9-d0a21ce68114/en-US/10-demo).
10+
11+
**SDLF ≥ 2.10.0** has a simplified CICD setup that can be used with any [source provider](https://docs.aws.amazon.com/codebuild/latest/userguide/access-tokens.html) supported by AWS CodeBuild. SDLF creates two CodeBuild projects able to deploy SDLF constructs, leaving the configuration of the source step to the user:
912

10-
* `sdlf-cicd-bootstrap`, publishing SDLF constructs in the private CloudFormation registry,
11-
* `sdlf-cicd-{name provided by the user}`, deploying the actual infrastructure,
12-
* and the third appearing during deployment, that should not be modified: `codeseeder-sdlf`.
13+
* `sdlf-cicd-bootstrap`, uploading SDLF constructs on S3 ready to be used by the CloudFormation service
14+
* they can also be published as CloudFormation modules on the private CloudFormation registry (`DEPLOYMENT_TYPE` set to `cfn-module`)
15+
* `sdlf-cicd-{name provided by the user}`, deploying the actual infrastructure.
1316

14-
These CodeBuild projects are required in every account SDLF is used, and can be deployed with the `./deploy-cicd.sh` script in the `sdlf-cicd` module:
17+
They can be deployed with the `./deploy-cicd.sh` script in the `sdlf-cicd` module:
1518

1619
```
1720
./deploy-cicd.sh --help # provides details on the available options
1821
19-
./deploy-cicd.sh -p aws_profile sdlf-main
22+
# `main` can be changed to a more sensible name
23+
./deploy-cicd.sh -p aws_profile main
24+
```
25+
26+
Replace `aws_profile` with the name of the AWS profile giving access to the account SDLF CICD infrastructure will be deployed in. `main` is a user-provided name - feel free to use a different one, within constraints.
27+
28+
These CodeBuild projects requires an IAM role in the AWS account data infrastructure will be deployed (even if the account is the same the CodeBuild projects are in). This role can be deployed with the `./deploy-role.sh` script in the `sdlf-cicd` module:
29+
30+
```
31+
./deploy-role.sh --help # provides details on the available options
32+
33+
# the name must match the name used with ./deploy-cicd.sh, in this case `main`
34+
./deploy-role.sh -p aws_profile main
2035
```
2136

22-
Replace `aws_profile` with the name of the AWS profile giving access to the account SDLF will be used in. `sdlf-main` is a user-provided name - feel free to use a different one, within constraints.
37+
Replace `aws_profile` with the name of the AWS profile giving access to the account SDLF data infrastructure will be deployed in. `main` is a user-provided name - it **must match** the name used with ./deploy-cicd.sh, in this case `main`.
2338

2439
Once the CodeBuild projects are created, the user can edit them manually to configure the source provider. Please refer to the [AWS CodeBuild documentation](https://docs.aws.amazon.com/codebuild/latest/userguide/access-tokens.html) for details on how to do this. `sdlf-cicd-bootstrap` should have a git repository containing a copy of the entire SDLF repository as source. `sdlf-cicd-{name provided by the user}` should use an empty git repository as source - this is the repository end users are going to work in, pushing CloudFormation templates referencing SDLF constructs, as seen in the [public workshop](https://catalog.us-east-1.prod.workshops.aws/workshops/501cb14c-91b3-455c-a2a9-d0a21ce68114/en-US/10-demo/200-foundations).
2540

26-
There is no SDLF-specific DevOps account in this setup.
2741

28-
`./deploy-cicd.sh` can be run multiple times in the same account if required, with different names. This can be useful if teams with different scopes are responsible for the data architectures. For example:
42+
`./deploy-cicd.sh` can be run multiple times in the same account if required, with different names. This can be useful if teams with different scopes are responsible for the data architecture in an account. For example:
2943

3044
```
31-
./deploy-cicd.sh -p aws_profile sdlf-main
32-
./deploy-cicd.sh -p aws_profile sdlf-engineering
45+
./deploy-cicd.sh -p aws_profile platform
46+
./deploy-cicd.sh -p aws_profile engineers
3347
```
3448

3549
This would create the following:
3650

3751
* `sdlf-cicd-bootstrap`
3852
* this is created only once, no matter how many times `./deploy-cicd.sh` is used in a given account.
39-
* `sdlf-cicd-sdlf-main`
53+
* `sdlf-cicd-platform`
4054
* this CodeBuild project could be linked to a git repository owned by a data platform team, creating centralized storage layers with `sdlf-foundations`.
41-
* `sdlf-cicd-sdlf-engineering`
55+
* `sdlf-cicd-engineers`
4256
* this CodeBuild project could be linked to a git repository owned by a data engineers team, creating technical catalogs and data processing pipelines with `sdlf-dataset` and the various `sdlf-stage-*`.
4357

4458

45-
## Infrastructure
59+
## Infrastructure - SDLF ≤ 2.10.0
4660

4761
!!! note
4862
This section and the rest of the page does not apply to the setup described above.

0 commit comments

Comments
 (0)