Merge branch 'master' into fix/update-lsql-and-ldms-instructions

switch180 · web-flow · commit dfb4ffcf4698 · 2025-07-09T15:02:44.000-05:00
diff --git a/README.md b/README.md
@@ -23,5 +23,10 @@ Make a pull request with changes. PRs will be automatically checked to make sure
 
 On merge to master, a GitHub action will deploy the assets to amazon-dynamodb-labs.com and verify the build to ensure the markdown and other files are correctly formatted. From there, a maintainer must manually pull the changes and push to https://catalog.workshops.aws/dynamodb-labs/en-US
 
+#### Internal maintainer: sync changes to internal copy of this repo
+This public repo is pushed to the internal repo in workshop studio using a combination of rsync (we assume you are on macOS) and git. The file `sync.sh` copies the source files to the WS repo folder, and after that you follow the internal README.md to complete the sync.
+1. Run sync.sh to sync the public repo to the amazon-dynamodb-immersion-day repo. e.g. `./sync.sh  -d /Users/$USER/workspace/amazon-dynamodb-immersion-day` . Choose y to sync the files.
+2. Change into the directory for amazon-dynamodb-immersion-day (the workshop studio version), open README.md, and follow the instructions there to git add and push the changes internally. Note that some assets, specifically the LEDA central account resources, are only authored on the internal repo, and they have a separate set of commands to push updates because they are assets that must be in a special S3 bucket owned by WS.
+
 ## License
 This project is licensed under the Apache-2.0 License.
diff --git a/content/rdbms-migration/migration-chapter00.en.md b/content/rdbms-migration/migration-chapter00.en.md
@@ -6,17 +6,17 @@ weight: 10
 ---
 In this module, you will create an environment to host the MySQL database on Amazon EC2. This instance will be used to host source database and simulate on-premise side of migration architecture.
 All the resources to configure source infrastructure are deployed via [Amazon CloudFormation](https://aws.amazon.com/cloudformation/) template.
-There are two CloudFormation templates used in this exercise which will deploy following resources.
+There are two CloudFormation templates used in this exercise which deploy the following resources.
 
-CloudFormation MySQL Template Resources:
-  - OnPrem VPC: Source VPC will represent an on-premise source environment in the N. Virginia region. This VPC will host source MySQL database on Amazon EC2
-  - Amazon EC2 MySQL Database: Amazon EC2 Amazon Linux 2 AMI with MySQL installed and running
-  - Load IMDb dataset: The template will create IMDb database on MySQL and load IMDb public dataset files into database. You can learn more about IMDb dataset inside [Explore Source Model](/hands-on-labs/rdbms-migration/migration-chapter03)
+CloudFormation MySQL Template Resources (**Already deployed**):
+  - **OnPrem VPC**: Source VPC will represent an on-premise source environment in the workshop region. This VPC will host source MySQL database on Amazon EC2
+  - **Amazon EC2 MySQL Database**: Amazon EC2 Amazon Linux 2 AMI with MySQL installed and running
+  - **Load IMDb dataset**: The template will create IMDb database on MySQL and load IMDb public dataset files into database. You can learn more about IMDb dataset inside [Explore Source Model](/hands-on-labs/rdbms-migration/migration-chapter03)
 
 
 
-CloudFormation DMS Instance Resources:
-  - DMS VPC:  Migration VPC on in the N. Virginia region. This VPC will host DMS replication instance.
-  - Replication Instance: DMS Replication instance that will facilitate database migration from source MySQL server on EC2 to Amazon DynamoDB
+CloudFormation DMS Instance Resources (**Needs deploying**):
+  - **DMS VPC**:  Migration VPC in the workshop region. This VPC will host DMS replication instance.
+  - **Replication Instance**: DMS Replication instance that will facilitate database migration from source MySQL server on EC2 to Amazon DynamoDB
 
 ![Final Deployment Architecture](/static/images/migration-environment.png)
diff --git a/content/rdbms-migration/migration-chapter04.en.md b/content/rdbms-migration/migration-chapter04.en.md
@@ -10,6 +10,7 @@ You can often query the data from multiple tables and assemble at the presentati
 To support high-traffic queries with ultra-low latency, designing a schema to take advantage of a NoSQL system generally makes technical and economic sense.
 
 To start designing a target data model in Amazon DynamoDB that will scale efficiently, you must identify the common access patterns. For the IMDb use case we have identified a set of access patterns as described below:
+
 ![Final Deployment Architecture](/static/images/migration32.png)
 
 A common approach to DynamoDB schema design is to identify application layer entities and use denormalization and composite key aggregation to reduce query complexity.
diff --git a/content/relational-migration/setup/index2.en.md b/content/relational-migration/setup/index2.en.md
@@ -25,7 +25,7 @@ The Lambda source code project has been setup as follows
   * **chalicelib/dynamodb_calls.py**
 
 
-1. Next, let's deploy the Chalice application stack. This step may take a few minutes to complete.
+1. Next, let's deploy the Chalice application stack. This step may take a few minutes to complete. 
 ```bash
 chalice deploy --stage relational
 ```
diff --git a/design-patterns/cloudformation/C9.yaml b/design-patterns/cloudformation/C9.yaml
@@ -85,8 +85,45 @@ Mappings:
     options:
       UserDataURL: "https://amazon-dynamodb-labs.com/assets/UserDataC9.sh"
       version: "1"
+  # AWS Managed Prefix Lists for EC2 InstanceConnect
+  AWSRegions2PrefixListID:
+    ap-south-1:
+      PrefixList: pl-0fa83cebf909345ca
+    eu-north-1:
+      PrefixList: pl-0bd77a95ba8e317a6
+    eu-west-3:
+      PrefixList: pl-0f2a97ab210dbbae1
+    eu-west-2:
+      PrefixList: pl-067eefa539e593d55
+    eu-west-1:
+      PrefixList: pl-0839cc4c195a4e751
+    ap-northeast-3:
+      PrefixList: pl-086543b458dc7add9
+    ap-northeast-2:
+      PrefixList: pl-00ec8fd779e5b4175
+    ap-northeast-1:
+      PrefixList: pl-08d491d20eebc3b95
+    ca-central-1:
+      PrefixList: pl-0beea00ad1821f2ef
+    sa-east-1:
+      PrefixList: pl-029debe66aa9d13b3
+    ap-southeast-1:
+      PrefixList: pl-073f7512b7b9a2450
+    ap-southeast-2:
+      PrefixList: pl-0e1bc5673b8a57acc
+    eu-central-1:
+      PrefixList: pl-03384955215625250
+    us-east-1:
+      PrefixList: pl-0e4bcff02b13bef1e
+    us-east-2:
+      PrefixList: pl-03915406641cb1f53
+    us-west-1:
+      PrefixList: pl-0e99958a47b22d6ab
+    us-west-2:
+      PrefixList: pl-047d464325e7bf465
+
 Resources:
-#LADV Role
+  #LADV Role
   DDBReplicationRole:
       Type: AWS::IAM::Role
       Properties:
@@ -742,6 +779,11 @@ Resources:
           IpProtocol: tcp
           FromPort: 3306
           ToPort: 3306
+        - Description: "Allow Instance Connect"
+          FromPort: 22
+          ToPort: 22
+          IpProtocol: tcp
+          SourcePrefixListId: !FindInMap [AWSRegions2PrefixListID, !Ref 'AWS::Region', PrefixList]
       Tags:
         - Key: Name
           Value: MySQL-SecurityGroup
@@ -802,6 +844,24 @@ Resources:
           mysql -u root "-p${DbMasterPassword}" -e "GRANT ALL PRIVILEGES ON *.* TO '${DbMasterUsername}'"
           mysql -u root "-p${DbMasterPassword}" -e "FLUSH PRIVILEGES"
           mysql -u root "-p${DbMasterPassword}" -e "CREATE DATABASE app_db;"
+          ## Setup MySQL Tables
+          cd /var/lib/mysql-files/
+          curl -O https://www.amazondynamodblabs.com/static/rdbms-migration/rdbms-migration.zip
+          unzip -q rdbms-migration.zip
+          chmod 775 *.*
+          mysql -u root "-p${DbMasterPassword}" -e "CREATE DATABASE imdb;"
+          mysql -u root "-p${DbMasterPassword}" -e "CREATE TABLE imdb.title_akas (titleId VARCHAR(200), ordering VARCHAR(200),title VARCHAR(1000), region VARCHAR(1000), language VARCHAR(1000), types VARCHAR(1000),attributes VARCHAR(1000),isOriginalTitle VARCHAR(5),primary key (titleId, ordering));"
+          mysql -u root "-p${DbMasterPassword}" -e "CREATE TABLE imdb.title_basics (tconst  VARCHAR(200), titleType  VARCHAR(1000),primaryTitle  VARCHAR(1000), originalTitle  VARCHAR(1000), isAdult  VARCHAR(1000), startYear  VARCHAR(1000),endYear  VARCHAR(1000),runtimeMinutes  VARCHAR(1000),genres  VARCHAR(1000),primary key (tconst));"
+          mysql -u root "-p${DbMasterPassword}" -e "CREATE TABLE imdb.title_crew (tconst  VARCHAR(200), directors  VARCHAR(1000),writers  VARCHAR(1000),primary key (tconst));"
+          mysql -u root "-p${DbMasterPassword}" -e "CREATE TABLE imdb.title_principals (tconst  VARCHAR(200), ordering  VARCHAR(200),nconst  VARCHAR(200), category  VARCHAR(1000), job  VARCHAR(1000), characters  VARCHAR(1000),primary key (tconst,ordering,nconst));"
+          mysql -u root "-p${DbMasterPassword}" -e "CREATE TABLE imdb.title_ratings (tconst  VARCHAR(200), averageRating float,numVotes  integer,primary key (tconst));"
+          mysql -u root "-p${DbMasterPassword}" -e "CREATE TABLE imdb.name_basics (nconst  VARCHAR(200), primaryName  VARCHAR(1000),birthYear  VARCHAR(1000), deathYear  VARCHAR(1000), primaryProfession  VARCHAR(1000), knownForTitles VARCHAR(1000),primary key (nconst));"
+          mysql -u root "-p${DbMasterPassword}" -e "LOAD DATA INFILE '/var/lib/mysql-files/title_ratings.tsv' IGNORE INTO TABLE imdb.title_ratings FIELDS TERMINATED BY '\t';"
+          mysql -u root "-p${DbMasterPassword}" -e "LOAD DATA INFILE '/var/lib/mysql-files/title_basics.tsv'  IGNORE INTO TABLE imdb.title_basics FIELDS TERMINATED BY '\t';"
+          mysql -u root "-p${DbMasterPassword}" -e "LOAD DATA INFILE '/var/lib/mysql-files/title_crew.tsv' IGNORE INTO TABLE imdb.title_crew FIELDS TERMINATED BY '\t';"
+          mysql -u root "-p${DbMasterPassword}" -e "LOAD DATA INFILE '/var/lib/mysql-files/title_principals.tsv' IGNORE INTO TABLE imdb.title_principals FIELDS TERMINATED BY '\t';"
+          mysql -u root "-p${DbMasterPassword}" -e "LOAD DATA INFILE '/var/lib/mysql-files/name_basics.tsv' IGNORE INTO TABLE imdb.name_basics FIELDS TERMINATED BY '\t';"
+          mysql -u root "-p${DbMasterPassword}" -e "LOAD DATA INFILE '/var/lib/mysql-files/title_akas.tsv' IGNORE INTO  TABLE imdb.title_akas FIELDS TERMINATED BY '\t';"
       Tags:
         - Key: Name
           Value: MySQL-Instance
diff --git a/sync.sh b/sync.sh
@@ -0,0 +1,110 @@
+#!/bin/bash
+
+usage() {
+    echo "Usage: $0 [--dry-run] -d DEST_REPO"
+    echo "  --dry-run    Show what would be synced without making changes"
+    echo "  -d, --dest   Destination repository path"
+    echo
+    echo "Example:"
+    echo "  $0 --dry-run -d /Users/$USER/workspace/amazon-dynamodb-immersion-day"
+    exit 1
+}
+
+# Get the directory where the script is located
+SOURCE_REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# Parse command line arguments
+DRY_RUN=false
+DEST_REPO=""
+
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        --dry-run)
+            DRY_RUN=true
+            shift
+            ;;
+        -d|--dest)
+            DEST_REPO="$2"
+            shift 2
+            ;;
+        *)
+            usage
+            ;;
+    esac
+done
+
+# Validate required parameters
+if [ -z "$DEST_REPO" ]; then
+    usage
+fi
+
+# Define source and destination directory pairs
+src_dirs=(
+    "content"
+    "static"
+)
+
+dest_dirs=(
+    "content"
+    "static"
+)
+
+# Define source and destination file pairs
+src_files=(
+    "design-patterns/cloudformation/C9.yaml"
+)
+
+dest_files=(
+    "static/ddb.yaml"
+)
+
+# Function to perform sync
+perform_sync() {
+    local rsync_options="-avz"
+
+    if [ "$DRY_RUN" = true ]; then
+        rsync_options="$rsync_options --dry-run"
+        echo "Performing dry run..."
+    else
+        echo "Performing actual sync..."
+    fi
+
+    echo "Source repository: $SOURCE_REPO"
+    echo "Destination repository: $DEST_REPO"
+
+    # Sync directories
+    for i in "${!src_dirs[@]}"; do
+        echo "Syncing directory: ${src_dirs[$i]}/ -> ${dest_dirs[$i]}/"
+        mkdir -p "$DEST_REPO/${dest_dirs[$i]}"
+        rsync $rsync_options "$SOURCE_REPO/${src_dirs[$i]}/" "$DEST_REPO/${dest_dirs[$i]}/"
+    done
+
+    # Sync individual files
+    for i in "${!src_files[@]}"; do
+        echo "Syncing file: ${src_files[$i]} -> ${dest_files[$i]}"
+        dest_dir=$(dirname "$DEST_REPO/${dest_files[$i]}")
+        mkdir -p "$dest_dir"
+        rsync $rsync_options "$SOURCE_REPO/${src_files[$i]}" "$DEST_REPO/${dest_files[$i]}"
+    done
+    echo "Great! Now follow instructions in the amazon-dynamodb-immersion-day README.md document to complete the sync."
+}
+
+# Verify destination repository exists
+if [ ! -d "$DEST_REPO" ]; then
+    echo "Error: Destination repository does not exist: $DEST_REPO"
+    exit 1
+fi
+
+# Execute sync
+if [ "$DRY_RUN" = true ]; then
+    perform_sync
+else
+    read -p "This will perform an actual sync. Are you sure? (y/n) " -n 1 -r
+    echo
+    if [[ $REPLY =~ ^[Yy]$ ]]; then
+        perform_sync
+    else
+        echo "Sync cancelled."
+        exit 1
+    fi
+fi
