Deployment Errors - Master Issue

[neebs12]

Overview

Errors and Resolution

• lambda-role was already made: had to be deleted and reran
  • Note: this happens when initial home deployment fails and a destroy is not run
• In init-process of tester engine - IF the s3 bucket is not reached to get script the tester immediately executes - ie: skips the synchronization process.
  • Requirement: Tester engine contains a "default" testing script, thus synchronization still needs to occur.
• test comp message is sent but containers remain running
  • Note: line 45 of aws/lambda/orchestrator/index.js is commented out while we are still developing - uncomment out if you want to test
• difficult to know if services are left running after destroy: resource group tags? (prob future work)
  • cdk s3 buckets and cloudformation stack remain from bootstrapping process

Resolved

• failed to create 1/4 timestream tables: RateLimitExceeded error
  • see PR @local - 33
  • Solution: used for loop instead of .forEach for a delayed loop. Also added a further 500ms delay for each table creation write to ensure RateLimitError is not hit. Implemented a solution in a soon to be made PR - specific file is /scripts/utils/clearTimestream.js. @A-Thresher feel free to test again. I have tested with 4 regions (thus 8 tables in current schema) and it works as required.
  • Note: attempted to solve problem by initializing tables in CDK. However, we cannot guarantee that the database is created BEFORE the tables are created within the CDK without changing a significant workflow of current deployment. This version of the solution was tried but errored out - thus not implemented
• tester count is hardcoded: to be determined based on VU numbers
  • see PR @local - 33
  • This can be solved using a .ts script that has a fn which specifies max VU per container. - see solution in comments.
• failed to delete timestream in parallel mode
  • see PR @local - 33
  • 
  • Note: this may be because there are now "many" tables and await timestreamWriteClient.send( new DeleteTableCommand({...})) in /scripts/utils/clearTimestream.js is not guaranteed to delete a table but to only request for it. Consider applying a recursive fn with base-case of noOfTables to undefined with a sleep of 500 to 1000ms to also avoid and any potential rate limit errors given queries to the client.
  • Note: Implemented a solution in a soon to be made PR - specific file is /scripts/utils/clearTimestream.js. @A-Thresher feel free to test again. I have tested with 4 regions (thus 8 tables in current schema) and it works as required.
• Failed to create a S3 bucket a@home even if home deployment ran normally.
  • see PR @local - 33. Added resiliency to S3 bucket creation and teardown - is tested when E2E (2 remote regions) when S3 bucket was deliberately commented out from CDK - although S3 bucket deployment is still included in CDK in final production version
  • 
  • Note: Destroyed home and re-ran again. No issue, error did not come up again.
• Failed to build one s3 bucket: removed CDKToolkit in cloudformation and reran.
  • Note: This S3 bucket was the home region's bootstrap bucket. Remote region buckets were created fine.
  • Solution: This has been resolved momentarily. A work around is made where the s3 bucket is made if the CDK Bootstrapping process fails to make it. However this relies on the bucket having a consistent name (potentially tied to the cdk version used)

[neebs12]

For issue: tester count is hardcoded: to be determined based on VU numbers

import * as config from './config.json';

/**
 * This function will require straight from config, this function outputs VU count and desired count per region
 * @param region string
 * @returns {vu: number, desiredCount: number}
 */
const getVUAndDesiredCountByRegion = (region: keyof typeof config.REMOTE_REGIONS): { VU: number, desiredCount: number } => {
  const MAX_VU_PER_CONTAINER = 200;
  const regionVU = config.REMOTE_REGIONS[region];

  // first case: MAX_VU_PER_CONTAINER >= regionVU, thus all VU will be in one container
  if (MAX_VU_PER_CONTAINER >= regionVU) {
    return {
      VU: regionVU,
      desiredCount: 1,
    };
  }

  // second case: MAX_VU_PER_CONTAINER < regionVU, thus we need to split VU EVENLY into multiple containers, will result in some VU error
  const desiredCount = Math.ceil(regionVU / MAX_VU_PER_CONTAINER);
  const specificVU = Math.ceil(regionVU / desiredCount);

  return {
    VU: specificVU,
    desiredCount: desiredCount
  }
}

See tests:

/*
for given config.json ...
{
  "DURATION": 2000,
  "HOME_REGION": "us-east-1",
  "REMOTE_REGIONS": {
    "ap-northeast-1": 5,
    "us-east-2": 200,
    "ca-central-2": 201,
    "ca-central-3": 203,
    "ca-central-4": 204,
    "us-west-1": 199,
    "ca-central-1": 1000,
    "eu-west-3": 2219
  }
}
*/

const regions = Object.keys(config.REMOTE_REGIONS) as Array<keyof typeof config.REMOTE_REGIONS>;

regions.forEach(region => {
  const { VU, desiredCount } = getVUAndDesiredCountByRegion(region);
  const finalVU = VU * desiredCount;
  const actualVU = config.REMOTE_REGIONS[region];
  console.log("------------------");
  console.log(`Region: ${region}, VU per container: ${VU}, desiredCount: ${desiredCount}, finalVU: ${finalVU}, actualVU: ${actualVU}, error: ${finalVU - actualVU}`);
});

Outputs are - note on "error". I think this is acceptable. I think error could reach desiredCount - 1 at most. Considering size of MAX_VU_PER_CONTAINER , I think, proportionally, this is acceptable, but we would need to take note of this.

------------------
Region: ap-northeast-1, VU per container: 5, desiredCount: 1, finalVU: 5, actualVU: 5, error: 0
------------------
Region: us-east-2, VU per container: 200, desiredCount: 1, finalVU: 200, actualVU: 200, error: 0
------------------
Region: ca-central-2, VU per container: 101, desiredCount: 2, finalVU: 202, actualVU: 201, error: 1
------------------
Region: ca-central-3, VU per container: 102, desiredCount: 2, finalVU: 204, actualVU: 203, error: 1
------------------
Region: ca-central-4, VU per container: 102, desiredCount: 2, finalVU: 204, actualVU: 204, error: 0
------------------
Region: us-west-1, VU per container: 199, desiredCount: 1, finalVU: 199, actualVU: 199, error: 0
------------------
Region: ca-central-1, VU per container: 200, desiredCount: 5, finalVU: 1000, actualVU: 1000, error: 0
------------------
Region: eu-west-3, VU per container: 185, desiredCount: 12, finalVU: 2220, actualVU: 2219, error: 1

[sni153]

Ran into the following issue when running npm run deploy:parallel:all

Solved the issue by going to the Home Regions -> cloudformation and deleting the CDK Toolkit.

[sni153]

CDK bootstrapping also fails sometimes for remote regions even though it may appear to be successful upon initial deployment.

Ran into the following error message after re-running npm run deploy:parallel:all

To resolve this issue, run npm run destroy:parallel:all
and go into cloudformation and the CDKToolkit for the region that failed the bootstrapping process.

[sni153]

Throttling error when running node ./src/index.js teardown-all

[neebs12]

Appears to not allow my account to deploy to eu-central-2 😅 no idea why

[neebs12]

Problem:

• Encountered Throttling from polling on the lambda side.

Solutions:

• Data aggregator (init) initial send.
• Load generator polling: Need to increase the polling duration from 1000ms to -> 10000ms (plus randomized in these ranges for spreading out?) in the init process of the load generator.
• Load Generator (post) final send.

Note: Error name is: "TooManyRequestsException"

See config file (culprit 😆)

[A-Thresher]

Outside of just reducing the polling rate (which should be a future work topic itself, replace with server side events or pub/sub), we can probably put finding and expanding performance bounds for just about everything in future work