|
3 | 3 | !!! note |
4 | 4 | `sdlf-cicd` is defined in the [sdlf-cicd](https://github.com/awslabs/aws-serverless-data-lake-framework/tree/main/sdlf-cicd) folder of the [SDLF repository](https://github.com/awslabs/aws-serverless-data-lake-framework). |
5 | 5 |
|
6 | | -## Simplified CICD with SDLF > 2.8.0 |
| 6 | +## Simplified CICD with SDLF ≥ 2.10.0 |
7 | 7 |
|
8 | | -**SDLF > 2.8.0** has a simplified CICD setup that can be used with any [source provider](https://docs.aws.amazon.com/codebuild/latest/userguide/access-tokens.html) supported by AWS CodeBuild. SDLF creates two CodeBuild projects able to deploy SDLF constructs (a third is created during the deployment process, but the user is never expected to interact directly with it), leaving the configuration of the source step to the user: |
| 8 | +!!! note |
| 9 | + Also check the [SDLF public workshop](https://catalog.us-east-1.prod.workshops.aws/workshops/501cb14c-91b3-455c-a2a9-d0a21ce68114/en-US/10-demo). |
| 10 | + |
| 11 | +**SDLF ≥ 2.10.0** has a simplified CICD setup that can be used with any [source provider](https://docs.aws.amazon.com/codebuild/latest/userguide/access-tokens.html) supported by AWS CodeBuild. SDLF creates two CodeBuild projects able to deploy SDLF constructs, leaving the configuration of the source step to the user: |
9 | 12 |
|
10 | | -* `sdlf-cicd-bootstrap`, publishing SDLF constructs in the private CloudFormation registry, |
11 | | -* `sdlf-cicd-{name provided by the user}`, deploying the actual infrastructure, |
12 | | -* and the third appearing during deployment, that should not be modified: `codeseeder-sdlf`. |
| 13 | +* `sdlf-cicd-bootstrap`, uploading SDLF constructs on S3 ready to be used by the CloudFormation service |
| 14 | + * they can also be published as CloudFormation modules on the private CloudFormation registry (`DEPLOYMENT_TYPE` set to `cfn-module`) |
| 15 | +* `sdlf-cicd-{name provided by the user}`, deploying the actual infrastructure. |
13 | 16 |
|
14 | | -These CodeBuild projects are required in every account SDLF is used, and can be deployed with the `./deploy-cicd.sh` script in the `sdlf-cicd` module: |
| 17 | +They can be deployed with the `./deploy-cicd.sh` script in the `sdlf-cicd` module: |
15 | 18 |
|
16 | 19 | ``` |
17 | 20 | ./deploy-cicd.sh --help # provides details on the available options |
18 | 21 |
|
19 | | -./deploy-cicd.sh -p aws_profile sdlf-main |
| 22 | +# `main` can be changed to a more sensible name |
| 23 | +./deploy-cicd.sh -p aws_profile main |
| 24 | +``` |
| 25 | + |
| 26 | +Replace `aws_profile` with the name of the AWS profile giving access to the account SDLF CICD infrastructure will be deployed in. `main` is a user-provided name - feel free to use a different one, within constraints. |
| 27 | + |
| 28 | +These CodeBuild projects requires an IAM role in the AWS account data infrastructure will be deployed (even if the account is the same the CodeBuild projects are in). This role can be deployed with the `./deploy-role.sh` script in the `sdlf-cicd` module: |
| 29 | + |
| 30 | +``` |
| 31 | +./deploy-role.sh --help # provides details on the available options |
| 32 | +
|
| 33 | +# the name must match the name used with ./deploy-cicd.sh, in this case `main` |
| 34 | +./deploy-role.sh -p aws_profile main |
20 | 35 | ``` |
21 | 36 |
|
22 | | -Replace `aws_profile` with the name of the AWS profile giving access to the account SDLF will be used in. `sdlf-main` is a user-provided name - feel free to use a different one, within constraints. |
| 37 | +Replace `aws_profile` with the name of the AWS profile giving access to the account SDLF data infrastructure will be deployed in. `main` is a user-provided name - it **must match** the name used with ./deploy-cicd.sh, in this case `main`. |
23 | 38 |
|
24 | 39 | Once the CodeBuild projects are created, the user can edit them manually to configure the source provider. Please refer to the [AWS CodeBuild documentation](https://docs.aws.amazon.com/codebuild/latest/userguide/access-tokens.html) for details on how to do this. `sdlf-cicd-bootstrap` should have a git repository containing a copy of the entire SDLF repository as source. `sdlf-cicd-{name provided by the user}` should use an empty git repository as source - this is the repository end users are going to work in, pushing CloudFormation templates referencing SDLF constructs, as seen in the [public workshop](https://catalog.us-east-1.prod.workshops.aws/workshops/501cb14c-91b3-455c-a2a9-d0a21ce68114/en-US/10-demo/200-foundations). |
25 | 40 |
|
26 | | -There is no SDLF-specific DevOps account in this setup. |
27 | 41 |
|
28 | | -`./deploy-cicd.sh` can be run multiple times in the same account if required, with different names. This can be useful if teams with different scopes are responsible for the data architectures. For example: |
| 42 | +`./deploy-cicd.sh` can be run multiple times in the same account if required, with different names. This can be useful if teams with different scopes are responsible for the data architecture in an account. For example: |
29 | 43 |
|
30 | 44 | ``` |
31 | | -./deploy-cicd.sh -p aws_profile sdlf-main |
32 | | -./deploy-cicd.sh -p aws_profile sdlf-engineering |
| 45 | +./deploy-cicd.sh -p aws_profile platform |
| 46 | +./deploy-cicd.sh -p aws_profile engineers |
33 | 47 | ``` |
34 | 48 |
|
35 | 49 | This would create the following: |
36 | 50 |
|
37 | 51 | * `sdlf-cicd-bootstrap` |
38 | 52 | * this is created only once, no matter how many times `./deploy-cicd.sh` is used in a given account. |
39 | | -* `sdlf-cicd-sdlf-main` |
| 53 | +* `sdlf-cicd-platform` |
40 | 54 | * this CodeBuild project could be linked to a git repository owned by a data platform team, creating centralized storage layers with `sdlf-foundations`. |
41 | | -* `sdlf-cicd-sdlf-engineering` |
| 55 | +* `sdlf-cicd-engineers` |
42 | 56 | * this CodeBuild project could be linked to a git repository owned by a data engineers team, creating technical catalogs and data processing pipelines with `sdlf-dataset` and the various `sdlf-stage-*`. |
43 | 57 |
|
44 | 58 |
|
45 | | -## Infrastructure |
| 59 | +## Infrastructure - SDLF ≤ 2.10.0 |
46 | 60 |
|
47 | 61 | !!! note |
48 | 62 | This section and the rest of the page does not apply to the setup described above. |
|
0 commit comments