docs: add cicd multi stages docs (#754)

ran-isenberg · Ran Isenberg · web-flow · commit 7c5dd89bd233 · 2023-10-19T23:00:34.000+03:00
* docs: add cicd multi stages docs

* add time check

---------

Co-authored-by: Ran Isenberg &lt;ran.isenberg@ranthebuilder.cloud&gt;
diff --git a/Makefile b/Makefile
@@ -56,7 +56,7 @@ integration:
 e2e:
 	poetry run pytest tests/e2e  --cov-config=.coveragerc --cov=service --cov-report xml
 
-pr: deps pre-commit complex lint unit deploy integration e2e
+pr: deps pre-commit complex lint lint-docs unit deploy coverage-tests e2e
 
 coverage-tests:
 	poetry run pytest tests/unit tests/integration  --cov-config=.coveragerc --cov=service --cov-report xml
diff --git a/README.md b/README.md
@@ -54,6 +54,7 @@ This project aims to reduce cognitive load and answer these questions for you by
 - Python Serverless service with a recommended file structure.
 - CDK infrastructure with infrastructure tests and security tests.
 - CI/CD pipelines based on Github actions that deploys to AWS with python linters, complexity checks and style formatters.
+- CI/CD pipeline deploys to dev/staging and production environment with different gates between each environment
 - Makefile for simple developer experience.
 - The AWS Lambda handler embodies Serverless best practices and has all the bells and whistles for a proper production ready handler.
 - AWS Lambda handler uses [AWS Lambda Powertools](https://docs.powertools.aws.dev/lambda-python/).
diff --git a/docs/best_practices/environment_variables.md b/docs/best_practices/environment_variables.md
@@ -17,11 +17,11 @@ The best practice for handling environment variables is to validate & parse them
 In case of misconfiguration, a validation exception is raised with all the relevant exception details.
 
 ## **Open source**
+
 The code in this post has been moved to an open source project you can use:
 
 The [AWS Lambda environment variables modeler](https://github.com/ran-isenberg/aws-lambda-env-modeler){:target="_blank" rel="noopener"}
 
-
 ## **Blog Reference**
 
 Read more about the importance of validating environment variables and how this utility works. Click [**HERE**](https://www.ranthebuilder.cloud/post/aws-lambda-cookbook-environment-variables){:target="_blank" rel="noopener"}
diff --git a/docs/best_practices/monitoring.md b/docs/best_practices/monitoring.md
@@ -8,7 +8,8 @@ description: Monitoring
 
 ## **Key Concepts**
 
-Utilizing AWS CloudWatch dashboards enables centralized monitoring of API Gateway, Lambda functions, and DynamoDB, providing real-time insights into their performance and operational health. By aggregating metrics, logs, and alarms, CloudWatch facilitates swift issue diagnosis and analysis across your serverless applications. Additionally, setting up alarms ensures immediate alerts during anomalous activities, enabling proactive issue mitigation.
+Utilizing AWS CloudWatch dashboards enables centralized monitoring of API Gateway, Lambda functions, and DynamoDB, providing real-time insights into their performance and operational health.
+By aggregating metrics, logs, and alarms, CloudWatch facilitates swift issue diagnosis and analysis across your serverless applications. Additionally, setting up alarms ensures immediate alerts during anomalous activities, enabling proactive issue mitigation.
 
 ## **Service Architecture**
 
@@ -54,7 +55,6 @@ As for DynamoDB tables, we have the primary database and the idempotency table f
 
 Personas that use this dashboard: developers, SREs.
 
-
 ### **Alarms**
 
 Having visibility and information is one thing, but being proactive and knowing beforehand that a significant error is looming is another. A CloudWatch
@@ -74,7 +74,6 @@ For latency-related issues, we define the following alarm:
 
 For P90, P50 metrics, follow this [explanation.](https://www.dnv.com/article/terminology-explained-p10-p50-and-p90-202611#:~:text=Proved%20(P90)%3A%20The%20lowest,equal%20or%20exceed%20P10%20estimate.){:target="_blank" rel="noopener"}
 
-
 For internal server errors rate, we define the following alarm:
 ![5xx](../media/monitoring/alarm_5xx.png)
 
@@ -89,7 +88,6 @@ We use the open-source [cdk-monitoring-constructs](https://github.com/cdklabs/cd
 
 You can view find the monitoring CDK construct [here](https://github.com/ran-isenberg/aws-lambda-handler-cookbook/blob/main/cdk/service/monitoring.py).
 
-
 ## **Further Reading**
 
 If you wish to learn more about this concept and go over details on the CDK code, check out my [blog post](https://www.ranthebuilder.cloud/post/how-to-effortlessly-monitor-serverless-applications-with-cloudwatch-part-one).
diff --git a/docs/getting_started.md b/docs/getting_started.md
@@ -10,7 +10,6 @@ description: AWS Lambda Cookbook Project Getting started
 * [poetry](https://pypi.org/project/poetry/){target="_blank"} - Make sure to run ``poetry config --local virtualenvs.in-project true`` so all dependencies are installed in the project '.venv' folder.
 * For Windows based machines, use the Makefile_windows version (rename to Makefile). Default Makefile is for Mac/Linux.
 
-
 ## **Creating a Developer Environment**
 
 1. Run ``make dev``
diff --git a/docs/index.md b/docs/index.md
@@ -32,6 +32,7 @@ This project aims to reduce cognitive load and answer these questions for you by
 - Python Serverless service with a recommended file structure.
 - CDK infrastructure with infrastructure tests and security tests.
 - CI/CD pipelines based on Github actions that deploys to AWS with python linters, complexity checks and style formatters.
+- CI/CD pipeline deploys to dev/staging and production environment with different gates between each environment
 - Makefile for simple developer experience.
 - The AWS Lambda handler embodies Serverless best practices and has all the bells and whistles for a proper production ready handler.
 - AWS Lambda handler uses [AWS Lambda Powertools](https://docs.powertools.aws.dev/lambda-python/){:target="_blank" rel="noopener"}.
@@ -59,6 +60,7 @@ The utilities cover multiple aspects of a production-ready service, including:
 - [**Dynamic configuration & features flags**](best_practices/dynamic_configuration.md)
 - [**Serverless Monitoring**](https://www.ranthebuilder.cloud/post/how-to-effortlessly-monitor-serverless-applications-with-cloudwatch-part-one)
 - [**API Idempotency**](https://www.ranthebuilder.cloud/post/serverless-api-idempotency-with-aws-lambda-powertools-and-cdk){:target="_blank" rel="noopener"}
+- [**Learn How to Write AWS Lambda Functions with Three Architecture Layers**](https://www.ranthebuilder.cloud/post/learn-how-to-write-aws-lambda-functions-with-architecture-layers){:target="_blank" rel="noopener"}
 
 While the code examples are written in Python, the principles are valid to any supported AWS Lambda handler programming language.
 
diff --git a/docs/media/cicd_main.png b/docs/media/cicd_main.png
diff --git a/docs/media/cicd_pr.png b/docs/media/cicd_pr.png
diff --git a/docs/pipeline.md b/docs/pipeline.md
@@ -3,18 +3,17 @@ title: CI/CD Pipeline
 description: AWS Lambda Cookbook - Elevate Your Handler's Code Python pipeline
 ---
 
-## **CI/CD Pipeline**
+## **Getting Started**
 
 The GitHub CI/CD pipeline includes the following steps.
 
 The pipelines uses environment secrets (under the defined environment 'dev', 'staging' and 'production') for code coverage and for the role to deploy to AWS.
 
-When you clone this repository be sure to define the environments in your [repo settings](https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment) and add two environment secrets:
+When you clone this repository be sure to define the environments in your [repo settings](https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment) and create a secret per environment:
 
-1. AWS_ROLE - to role to assume for your GitHub worker as defined [here](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) .
-2. CODECOV_TOKEN - for [code coverage integration](https://app.codecov.io/).
+- AWS_ROLE - to role to assume for your GitHub worker as defined [here](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services)
 
-### Makefile Commands
+### **Makefile Commands**
 
 All steps can be run locally using the makefile. See details below:
 
@@ -34,9 +33,40 @@ All steps can be run locally using the makefile. See details below:
 - Code coverage tests  - run `make coverage-tests` in the IDE after CDK dep
 - Update GitHub documentation branch
 
-### Other Capabilities
+### **Other Capabilities**
 
 - Automatic Python dependencies update with Dependabot
 - Easy to use makefile allows to run locally all commands in the GitHub actions
 - Run local docs server, prior to push in pipeline - run `make docs`  in the IDE
 - Prepare PR, run all checks with one command - run `make pr` in the IDE
+
+## **Environments & Pipelines**
+
+All GitHub workflows are stored under `.github/workflows` folder.
+
+The two most important ones are `pr-serverless-service`  and `main-serverless-service`.
+
+### **pr-serverless-service**
+
+<img alt="alt_text" src="../media/cicd_pr.png" />
+
+`pr-serverless-service` runs for every pull request you open. It expects you defined a GitHub environment by the name `dev` and that it includes a secret by the name of `AWS_ROLE`.
+
+It includes two jobs: 'quality_standards' and 'tests' where a failure in 'quality_standards' does not trigger 'tests'. Both jobs MUST pass in order to to be able to merge.
+
+'quality_standards' includes all linters, pre-commit checks and units tests and 'tests' deploys the service to AWS, runs code coverage checks, security checks and E2E tests. Stack is destroyed at the end. Stack has a 'dev' prefix as part of its name.
+
+Once merged, `main-serverless-service` will run.
+
+### **main-serverless-service**
+
+<img alt="alt_text" src="../media/cicd_main.png" />
+
+`main-serverless-service` runs for every MERGED pull request that runs on the main branch. It expects you defined a GitHub environments by the name `staging` and `production` and that both includes a secret by the name of `AWS_ROLE`.
+
+It includes three jobs: 'staging', 'production' and 'publish_github_pages'.
+
+'staging' does not run any of the 'quality_standards' checks, since they were already checked before the code was merged. It runs just coverage tests and E2E tests. Stack is not deleted. Stack has a 'staging' prefix as part of its name.
+Any failure in staging will stop the pipeline and production environment will not get updated with the new code.
+
+'production' does not run any of the 'quality_standards' checks, since they were already checked before the code was merged. It does not run any test at the moment. Stack is not deleted. Stack has a 'production' prefix as part of its name.
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -14,6 +14,7 @@ nav:
       - best_practices/tracer.md
       - best_practices/metrics.md
       - best_practices/environment_variables.md
+      - Architecture Layers: https://www.ranthebuilder.cloud/post/learn-how-to-write-aws-lambda-functions-with-architecture-layers" target="_blank"
       - best_practices/input_validation.md
       - best_practices/dynamic_configuration.md
       - best_practices/monitoring.md
diff --git a/service/dal/dynamo_dal_handler.py b/service/dal/dynamo_dal_handler.py
@@ -1,4 +1,5 @@
 import uuid
+from datetime import datetime
 
 import boto3
 from botocore.exceptions import ClientError
@@ -26,13 +27,21 @@ def _get_db_handler(self, table_name: str) -> Table:
         dynamodb: DynamoDBServiceResource = boto3.resource('dynamodb')
         return dynamodb.Table(table_name)
 
+    def _get_unix_time(self) -> int:
+        return int(datetime.utcnow().timestamp())
+
     @tracer.capture_method(capture_response=False)
     def create_order_in_db(self, customer_name: str, order_item_count: int) -> Order:
         order_id = str(uuid.uuid4())
         logger.append_keys(order_id=order_id)
         logger.info('trying to save order', customer_name=customer_name, order_item_count=order_item_count)
         try:
-            entry = OrderEntry(id=order_id, name=customer_name, item_count=order_item_count)
+            entry = OrderEntry(
+                id=order_id,
+                name=customer_name,
+                item_count=order_item_count,
+                created_at=self._get_unix_time(),
+            )
             table: Table = self._get_db_handler(self.table_name)
             table.put_item(Item=entry.model_dump())
         except (ClientError, ValidationError) as exc:  # pragma: no cover
@@ -41,4 +50,4 @@ def create_order_in_db(self, customer_name: str, order_item_count: int) -> Order
             raise InternalServerException(error_msg) from exc
 
         logger.info('finished create order successfully', order_item_count=order_item_count, customer_name=customer_name)
-        return Order(id=order_id, name=customer_name, item_count=order_item_count)
+        return Order(id=entry.id, name=entry.name, item_count=entry.item_count)
diff --git a/service/dal/models/db.py b/service/dal/models/db.py
@@ -9,3 +9,4 @@ class OrderEntry(BaseModel):
     id: OrderId  # primary key
     name: Annotated[str, Field(min_length=1, max_length=20)]
     item_count: PositiveInt
+    created_at: PositiveInt
diff --git a/tests/integration/test_create_order.py b/tests/integration/test_create_order.py
@@ -1,4 +1,5 @@
 import json
+from datetime import datetime
 from http import HTTPStatus
 from typing import Any
 
@@ -74,6 +75,8 @@ def test_handler_200_ok(mocker, table_name: str):
     assert 'Item' in response
     assert response['Item']['name'] == customer_name
     assert response['Item']['item_count'] == order_item_count
+    now = int(datetime.utcnow().timestamp())
+    assert now - int(response['Item']['created_at']) <= 60  # assume item was created in last minute, check that utc time calc is correct
 
 
 def test_internal_server_error(mocker, table_name: str):
diff --git a/tests/unit/test_dal_schema.py b/tests/unit/test_dal_schema.py
@@ -1,11 +1,13 @@
 import uuid
+from datetime import datetime
 
 import pytest
 from aws_lambda_powertools.utilities.parser import ValidationError
 
 from service.dal.models.db import OrderEntry
 
 order_id = str(uuid.uuid4())
+created_at = int(datetime.utcnow().timestamp())
 
 
 def test_invalid_items_type():
@@ -15,7 +17,7 @@ def test_invalid_items_type():
 
     # When & Then: OrderEntry is initialized, expect a ValidationError
     with pytest.raises(ValidationError):
-        OrderEntry(id=order_id, name=customer_name, item_count=order_item_count)
+        OrderEntry(id=order_id, name=customer_name, item_count=order_item_count, created_at=created_at)
 
 
 def test_invalid_items_negative():
@@ -25,7 +27,7 @@ def test_invalid_items_negative():
 
     # When & Then: OrderEntry is initialized, expect a ValidationError
     with pytest.raises(ValidationError):
-        OrderEntry(id=order_id, name=customer_name, item_count=order_item_count)
+        OrderEntry(id=order_id, name=customer_name, item_count=order_item_count, created_at=created_at)
 
 
 def test_invalid_items_zero():
@@ -35,7 +37,7 @@ def test_invalid_items_zero():
 
     # When & Then: OrderEntry is initialized, expect a ValidationError
     with pytest.raises(ValidationError):
-        OrderEntry(id=order_id, name=customer_name, item_count=order_item_count)
+        OrderEntry(id=order_id, name=customer_name, item_count=order_item_count, created_at=created_at)
 
 
 def test_invalid_order_id():
@@ -46,7 +48,7 @@ def test_invalid_order_id():
 
     # When & Then: OrderEntry is initialized, expect a ValidationError
     with pytest.raises(ValidationError):
-        OrderEntry(id=order_id_invalid, name=customer_name, item_count=order_item_count)
+        OrderEntry(id=order_id_invalid, name=customer_name, item_count=order_item_count, created_at=created_at)
 
 
 def test_valid_output():
@@ -56,4 +58,4 @@ def test_valid_output():
 
     # When: OrderEntry is initialized
     # Then: No exception should be raised
-    OrderEntry(id=order_id, name=customer_name, item_count=order_item_count)
+    OrderEntry(id=order_id, name=customer_name, item_count=order_item_count, created_at=created_at)
