Revert "No public description"

This reverts commit 42153eb.

Author: gericdong

.cloud-build/Notebooks.txt

@@ -0,0 +1,15 @@
+generative-ai/gemini/getting-started/intro_gemini_2_0_flash.ipynb
+generative-ai/gemini/getting-started/intro_gemini_2_0_flash_lite.ipynb
+generative-ai/gemini/getting-started/intro_gemini_2_0_image_gen.ipynb
+generative-ai/gemini/getting-started/intro_gemini_2_5_flash.ipynb
+generative-ai/gemini/getting-started/intro_gemini_2_5_pro.ipynb
+generative-ai/gemini/getting-started/intro_gemini_chat.ipynb
+generative-ai/gemini/getting-started/intro_gemini_curl.ipynb
+generative-ai/gemini/chat-completions/intro_chat_completions_api.ipynb
+generative-ai/gemini/code-execution/intro_code_execution.ipynb
+generative-ai/gemini/controlled-generation/intro_controlled_generation.ipynb
+generative-ai/gemini/function-calling/intro_function_calling.ipynb
+generative-ai/gemini/global-endpoint/intro_global_endpoint.ipynb
+generative-ai/gemini/prompts/intro_prompt_design.ipynb
+generative-ai/gemini/use-cases/spatial-understanding/spatial_understanding.ipynb
+generative-ai/gemini/use-cases/intro_multimodal_use_cases.ipynb

.cloud-build/README.md

@@ -0,0 +1,61 @@
+# Notebook Testing
+
+This script is designed to automate the uploading of Jupyter Notebook files (`.ipynb`) from a local directory to a Google Cloud Storage (GCS) bucket
+
+## Purpose
+
+The script simplifies the process of transferring multiple notebook files to GCS, making it easier to manage and deploy your notebooks in a cloud environment.
+
+## Script Description
+
+The script performs the following actions:
+
+1. **Reads Output URI variable:**
+   - It reads the destination GCS bucket URI from a variable named `OUTPUT_URI` injected from secret manager. This allows for easy configuration of the destination.
+2. **Iterates Through Notebooks:**
+   - It loops through all `.ipynb` files located in the `Notebooks.txt` file.
+3. **Copies Notebooks to GCS:**
+   - For each notebook file, it extracts the filename using `basename`.
+   - It then uses the `gcloud storage cp` command to copy the notebook file to the specified GCS bucket, maintaining the directory structure.
+
+## Prerequisites
+
+Before running this script, ensure you have the following:
+
+- **Google Cloud SDK (gcloud):** The Google Cloud SDK must be installed and configured with appropriate credentials. You should be authenticated to the Google Cloud project where you want to upload the notebooks.
+- **GCS Bucket:** The destination GCS bucket must exist.
+- **OUTPUT_URI:** A secret named `OUTPUT_URI` must be injected for this step in your pipeline. This variable should contain the full GCS URI of the destination bucket (e.g., `gs://your-bucket-name`).
+- **Jupyter Notebooks:** The Jupyter Notebook files (`.ipynb`) should be located in the `/workspace/generative-ai/gemini/getting-started/` directory.
+
+## How to Use
+
+1. **Set Up `OUTPUT_URI`:**
+
+   - Create a variable named `OUTPUT_URI` in the same directory as your script.
+   - Add the full GCS URI of your destination bucket to this file. For example:
+
+     ```none
+     gs://your-bucket-name
+     ```
+
+2. **Place Notebooks:**
+
+   - Ensure the names of the `.ipynb` files are located in the `Notebooks.txt` file.
+
+3. **Add the script as a step in your pipeline:**
+
+## Example
+
+Assuming you have a GCS bucket named `my-notebooks-bucket` and a notebook file named `example.ipynb` in the specified directory:
+
+1. Create `OUTPUT_URI` variable the content: `gs://my-notebooks-bucket`
+2. Place `example.ipynb` in `/workspace/generative-ai/gemini/getting-started/`
+3. Run the script.
+4. The `example.ipynb` will be copied to `gs://my-notebooks-bucket/generative-ai/gemini/getting-started/example.ipynb`.
+
+## Notes
+
+- Ensure that the service account used by `gcloud` has the necessary permissions to write to the GCS bucket.
+- The script assumes that the directory `/workspace/generative-ai/gemini/getting-started/` exists. If it doesn't, the script will not find any notebooks.
+- This script will overwrite files with the same name in the GCS bucket.
+- If you have a large number of notebooks, consider adding error handling and logging to the script.

.cloud-build/clone-repo/README.md

@@ -0,0 +1,17 @@
+# Script for Copying Gemini Getting Started Files to GCS
+
+This script automates the process of cloning your notebook repository and will copy the notebook files to a specified Google Cloud Storage (GCS) location.
+
+## Prerequisites
+
+- **Google Cloud SDK (gcloud) installed and configured:** You need to have the `gcloud` command-line tool installed and authenticated with your Google Cloud account.
+- **gsutil installed:** `gsutil` is part of the Google Cloud SDK and is used for interacting with GCS.
+- **Git installed:** Git is required to clone the repository.
+- **OUTPUT_URI:** A secret inject from secret manager Google Cloud in your account.
+
+## How to use
+
+1. **Create `OUTPUT_URI` secret:**
+
+2. **Run the script:**
+   Add script to pipeline

.cloud-build/clone-repo/clone_repo.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# Clones the generative-ai repository from GitHub.
+git clone --depth 1 -b main https://github.com/GoogleCloudPlatform/generative-ai.git
+
+# Changes the current directory to the cloned repository.
+cd generative-ai || exit 1

.cloud-build/cloudbuild.yaml

@@ -0,0 +1,43 @@
+timeout: 7200s
+
+steps:
+  # [START copy-repo]
+  - name: google/cloud-sdk
+    id: "copy-repo"
+    entrypoint: bash
+    args:
+      - "-c"
+      - |
+        . .cloud-build/clone-repo/clone_repo.sh
+  # [END copy-repo]
+  # [START execute-notebooks]
+  - name: google/cloud-sdk
+    id: "execute-notebooks"
+    entrypoint: bash
+    timeout: 7200s # Because this step takes the longest. Avg runtime is 1 hour and 20 minutes.
+    args:
+      - "-c"
+      - |
+        gcloud secrets versions access latest --secret=NOTEBOOK_RUNTIME_TEMPLATE > NOTEBOOK_RUNTIME_TEMPLATE
+        gcloud secrets versions access latest --secret=PROJECT_ID > PROJECT_ID
+        gcloud secrets versions access latest --secret=REGION > REGION
+        gcloud secrets versions access latest --secret=SA_EMAIL > SA
+        gcloud secrets versions access latest --secret=GCS_BUCKET > OUTPUT_URI
+        gcloud secrets versions access latest --secret=PS_TOPIC > PS_TOPIC
+        . .cloud-build/executor/test_notebooks.sh
+# [END execute-notebooks]
+# [START github-notification]
+#- name: google/cloud-sdk
+#  id: 'github-notification'
+#  entrypoint: bash
+#  args:
+#  - '-c'
+#  - |
+#    gcloud secrets versions access latest --secret=GH_token > GH_token.txt
+
+#    . .cloud-build/github-issue/ghcli-install.sh
+#    . .cloud-build/github-issue/auto_issue_create.sh
+
+# [END github-notification]
+
+logsBucket: ${_LOG_BUCKET}

.cloud-build/executor/README.md

@@ -0,0 +1,33 @@
+# Automated Vertex AI Colab Notebook Execution
+
+This Bash script automates the execution of Jupyter Notebooks using Vertex AI Colab executions. It iterates through notebooks in a specified directory, triggers executions, and monitors their status.
+
+## Purpose
+
+The script aims to:
+
+- Automate the execution of all `.ipynb` files located in `/workspace/generative-ai/gemini/getting-started`.
+- Use a pre-configured Notebook Runtime Template.
+- Store execution outputs in a Google Cloud Storage (GCS) bucket.
+- Monitor execution status and identify failed notebooks.
+- Output a list of failed notebooks to both the console and a file (`/workspace/Failure.txt`).
+
+## Prerequisites
+
+- Google Cloud SDK (`gcloud`) installed and configured.
+- Authentication set up with `gcloud auth login`.
+- Vertex AI API enabled.
+- Necessary variables (PROJECT_ID, REGION, SA, OUTPUT_URI, NOTEBOOK_RUNTIME_TEMPLATE) stored in separate secret manager in the same directory as the script.
+- A Google Cloud Storage bucket accessible for storing notebook outputs.
+- Notebook Runtime Template created in Google Cloud Vertex AI.
+- Notebooks present in the `/workspace/generative-ai/gemini/getting-started` directory.
+
+## Usage
+
+1. **Prepare Variable in secret manager:** Create the following values in secret manager
+   - `PROJECT_ID`: Your Google Cloud Project ID.
+   - `REGION`: The Google Cloud region to use.
+   - `SA`: The service account to use for the executions.
+   - `OUTPUT_URI`: The GCS URI for storing execution outputs (e.g., `gs://your-bucket`).
+   - `NOTEBOOK_RUNTIME_TEMPLATE`: The full resource name of your Notebook Runtime Template.
+2. **Add to your pipeline:**

.cloud-build/executor/test_notebooks.sh

@@ -0,0 +1,144 @@
+#!/bin/bash
+
+alias gcurl='curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json"'
+
+TARGET=$(cat .cloud-build/Notebooks.txt)
+
+current_date=$(date +%Y-%m-%d)
+current_time=$(date +%H-%M-%S)
+current_time_readable=$(date "+%B %d %Y %H:%M:%S")
+
+NOTEBOOK_RUNTIME_TEMPLATE=$(cat NOTEBOOK_RUNTIME_TEMPLATE)
+OUTPUT_URI=$(cat OUTPUT_URI)
+SA=$(cat SA)
+PROJECT_ID=$(cat PROJECT_ID)
+REGION=$(cat REGION)
+PUBSUB_TOPIC=$(cat PS_TOPIC)
+
+failed_count=0
+failed_notebooks=()
+total_count=0
+successful_notebooks=()
+successful_count=0
+
+for x in $TARGET; do
+  total_count=$((total_count + 1))
+  # Use the full path from the repository for display name
+  DISPLAY_NAME="${x##generative-ai/}"
+  DISPLAY_NAME="${DISPLAY_NAME%.ipynb}-$current_date-$current_time"
+  echo "Starting execution for ${x}"
+
+  # Execute and get the operation ID
+  OPERATION_ID=$(gcloud colab executions create \
+    --display-name="$DISPLAY_NAME" \
+    --notebook-runtime-template="$NOTEBOOK_RUNTIME_TEMPLATE" \
+    --direct-content="$x" \
+    --gcs-output-uri="$OUTPUT_URI" \
+    --project="$PROJECT_ID" \
+    --region="$REGION" \
+    --service-account="$SA" \
+    --verbosity=debug \
+    --execution-timeout="1h30m" \
+    --format="value(name)")
+
+  echo "Operation ID: $OPERATION_ID"
+  TRUNCATED_OPERATION_ID=$(echo "$OPERATION_ID" | cut -c 67-85)
+
+  # check job status
+  echo "Waiting for execution to complete..."
+  if ! EXECUTION_DETAILS=$(gcloud colab executions describe "$TRUNCATED_OPERATION_ID" --region="$REGION"); then
+    echo "Error describing execution for ${x}. See logs for details."
+    failed_count=$((failed_count + 1))
+    failed_notebooks+=("${x}")
+    continue
+  else
+    echo "Execution completed for ${x}"
+  fi
+
+  # Check the jobState
+  JOB_STATE=$(echo "$EXECUTION_DETAILS" | grep "jobState:" | awk '{print $2}')
+  if [[ "$JOB_STATE" == "JOB_STATE_SUCCEEDED" ]]; then
+    echo "Notebook execution succeeded."
+    successful_count=$((successful_count + 1))
+    successful_notebooks+=("${x}")
+  else
+    echo "Notebook execution failed. Job state: $JOB_STATE. Please use id $TRUNCATED_OPERATION_ID to troubleshoot notebook ${x}. See log for details."
+    failed_count=$((failed_count + 1))
+    failed_notebooks+=("${x}")
+    continue
+  fi
+
+done
+
+# Print the final list of failed notebooks
+if [[ ${#failed_notebooks[@]} -gt 0 ]]; then
+  echo "Failed Notebooks:"
+  for notebook in "${failed_notebooks[@]}"; do
+    echo "- $notebook" | tee -a /workspace/Failure.txt
+  done
+fi
+
+if [[ $failed_count -gt 0 ]]; then
+  echo "Total failed notebook executions: $failed_count"
+fi
+
+if [[ $successful_count -gt 0 ]]; then
+  echo "Total successful notebook executions: $successful_count"
+fi
+
+# Prep pub/sub message
+failed_notebooks_str=$(
+  IFS=', '
+  echo "${failed_notebooks[*]}"
+)
+
+# prep notebook name for pub/sub message
+failed_notebooks_str=$(
+  IFS=', '
+  echo "${failed_notebooks[*]}"
+)
+
+if [[ -n "$failed_notebooks_str" ]]; then
+  IFS=',' read -ra failed_notebooks_array <<<"$failed_notebooks_str"
+  trimmed_notebooks=()
+  for notebook in "${failed_notebooks_array[@]}"; do
+    trimmed_notebooks+=("$(echo -n "$notebook" | sed 's/ *$//')")
+  done
+  failed_notebooks_str=$(
+    IFS=', '
+    echo "${trimmed_notebooks[*]}"
+  )
+else
+  failed_notebooks_str=""
+fi
+
+successful_notebooks_str=$(
+  IFS=', '
+  echo "${successful_notebooks[*]}"
+)
+
+if [[ -n "$successful_notebooks_str" ]]; then
+  IFS=',' read -ra successful_notebooks_array <<<"$successful_notebooks_str"
+  trimmed_successful_notebooks=()
+  for notebook in "${successful_notebooks_array[@]}"; do
+    trimmed_successful_notebooks+=("$(echo -n "$notebook" | sed 's/ *$//')")
+  done
+  successful_notebooks_str=$(
+    IFS=', '
+    echo "${trimmed_successful_notebooks[*]}"
+  )
+else
+  successful_notebooks_str=""
+fi
+
+# Construct the message to send to pub/sub topic
+message_data="{\"total_count\":$((total_count + 0)),\"failed_count\":$((failed_count + 0)),\"failed_notebooks\":\"${failed_notebooks_str}\",\"successful_notebooks\":\"${successful_notebooks_str}\",\"successful_count\":$((successful_count + 0)),\"execution_date\":\"${current_time_readable}\"}"
+
+# Publish to Pub/Sub
+echo "$(date) - INFO - Publishing to Pub/Sub topic: $PUBSUB_TOPIC"
+if ! gcloud pubsub topics publish "$PUBSUB_TOPIC" --message="$message_data" --project="$PROJECT_ID"; then
+  echo "$(date) - ERROR - Failed to publish to Pub/Sub topic $PUBSUB_TOPIC. Check permissions and topic configuration."
+  #exit 1
+fi
+
+echo "All notebook executions completed."

.cloud-build/github-issue/README.md

@@ -0,0 +1,24 @@
+# Create GitHub Issue on Notebook Test Failure
+
+These Bash scripts installs GitHub CLI and automates the creation of a GitHub issue when automated notebook tests fail. It reads the error output from a specified file and uses the `gh` CLI to create a new issue in a designated repository.
+
+## Purpose
+
+The script aims to:
+
+- Detect the presence of a failure file (`/workspace/Failure.txt`).
+- If the file exists, create a GitHub issue with the file's content as the issue body.
+- Include a predefined title, labels, and assignees in the issue.
+- Provide informative messages about the success or failure of issue creation.
+
+## Prerequisites
+
+- GitHub CLI (`gh`) installed and authenticated.
+- A GitHub repository with write access for the user running the script.
+- A failure file (`/workspace/Failure.txt`) containing the error output from failed notebook tests.
+
+## Usage
+
+1. **Save the script:** Save the provided script as `create_gh_issue.sh`.
+2. **Set Repository Details:** Modify the variables `REPO_OWNER`, `REPO_NAME`, `ISSUE_ASSIGNEES` and `FAILURE_FILE` within the script to match your GitHub repository details and the location of the failure file.
+3. **Add script to pipeline:**

.cloud-build/github-issue/auto_issue_create.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+ISSUE_TITLE="Failed automated Notebook Testing"
+ISSUE_LABELS="bug"
+# ISSUE_ASSIGNEES="CadillacBurgess1"
+REPO_OWNER="GoogleCloudPlatform"
+REPO_NAME="generative-ai"
+FAILURE_FILE="/workspace/Failure.txt"
+
+if [ -f "$FAILURE_FILE" ]; then
+  ISSUE_BODY=$(cat "$FAILURE_FILE")
+
+  if ! gh issue create \
+    -t "$ISSUE_TITLE" \
+    -b "$ISSUE_BODY" \
+    -l "$ISSUE_LABELS" \
+    -a "$ISSUE_ASSIGNEES" \
+    -R "$REPO_OWNER/$REPO_NAME"; then
+    echo "Error creating issue."
+    exit 1
+  fi
+
+  echo "Issue created with output of failed notebooks."
+else
+  echo "No notebooks failed. Issue creation skipped."
+fi

.cloud-build/github-issue/ghcli-install.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+(type -p wget >/dev/null || (apt update && apt-get install wget -y)) &&
+  mkdir -p /etc/apt/keyrings &&
+  chmod 0755 /etc/apt/keyrings &&
+  out=$(mktemp) && wget -nv -O$out https://cli.github.com/packages/githubcli-archive-keyring.gpg &&
+  cat $out | tee /etc/apt/keyrings/githubcli-archive-keyring.gpg >/dev/null &&
+  rm -f "$out" &&
+  chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg &&
+  echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list >/dev/null &&
+  apt update &&
+  apt install gh -y
+
+apt update -y
+apt install gh -y
+gh auth login --with-token <GH_token.txt