Add API kind overview docs (#2230)

(cherry picked from commit 6a7d6f6)

Author: deliahu

docs/overview.md

@@ -0,0 +1,34 @@
+# Overview
+
+## Cluster
+
+The Cortex cluster is an EKS (Kubernetes) cluster in a dedicated VPC on your AWS account.
+
+### Worker node groups
+
+The kubernetes cluster uses EC2 autoscaling groups for its worker node groups. Cortex supports most EC2 instance types, and the necessary device drivers are installed to expose GPUs and Inferentia chips to your workloads. Reserved and spot instances can be used to help reduce costs.
+
+Cortex uses the Kubernetes Cluster Autoscaler to scale the appropriate node groups to satisfy the compute demands of your workloads.
+
+### Networking
+
+By default, a new dedicated VPC is created for the cluster during installation.
+
+Two network loadbalancers (NLBs) are created to route traffic to the cluster. One loadbalancer is dedicated for traffic to your APIs, and the other loadbalancer is dedicated for API management requests to Cortex from your CLI or Python client. Traffic to the loadbalancers can be secured and restricted based on your cluster configuration.
+
+### Observability
+
+All logs from the Cortex cluster are pushed to a CloudWatch log group using FluentBit. An in-cluster Prometheus installation is used to collect metrics for observability and autoscaling purposes. Metrics and dashboards pertaining to your APIs and instance usage can be viewed and modified via Grafana.
+
+## Deploying to the cluster
+
+After a successful Cortex cluster installation, you can use the Cortex CLI or Python Client to deploy different types of workloads. The clients use AWS credentials to authenticate to the Cortex cluster.
+
+Cortex uses a collection of containers, referred to as a pod, as the atomic unit; scaling and replication occurs at the pod level. The orchestration and scaling of pods is unique to the different types of workloads:
+
+* Realtime
+* Async
+* Batch
+* Task
+
+Visit the workload-specific documentation for more details.

docs/summary.md

@@ -1,6 +1,7 @@
 # Summary
 
 * [Get started](start.md)
+* [Overview](overview.md)
 
 ## Clusters
 
@@ -29,7 +30,7 @@
 
 ## Workloads
 
-* [Realtime APIs](workloads/realtime/realtime-apis.md)
+* [Realtime](workloads/realtime/realtime.md)
   * [Example](workloads/realtime/example.md)
   * [Configuration](workloads/realtime/configuration.md)
   * [Containers](workloads/realtime/containers.md)
@@ -38,18 +39,18 @@
   * [Metrics](workloads/realtime/metrics.md)
   * [Statuses](workloads/realtime/statuses.md)
   * [Troubleshooting](workloads/realtime/troubleshooting.md)
-* [Async APIs](workloads/async/async-apis.md)
+* [Async](workloads/async/async.md)
   * [Example](workloads/async/example.md)
   * [Configuration](workloads/async/configuration.md)
   * [Containers](workloads/async/containers.md)
   * [Statuses](workloads/async/statuses.md)
-* [Batch APIs](workloads/batch/batch-apis.md)
+* [Batch](workloads/batch/batch.md)
   * [Example](workloads/batch/example.md)
   * [Configuration](workloads/batch/configuration.md)
   * [Containers](workloads/batch/containers.md)
   * [Jobs](workloads/batch/jobs.md)
   * [Statuses](workloads/batch/statuses.md)
-* [Task APIs](workloads/task/task-apis.md)
+* [Task](workloads/task/task.md)
   * [Example](workloads/task/example.md)
   * [Configuration](workloads/task/configuration.md)
   * [Containers](workloads/task/containers.md)

docs/workloads/async/async-apis.md

@@ -0,0 +1,15 @@
+# Async
+
+Async APIs are designed for asynchronous workloads in which the user submits an asynchronous request and retrieves the result later (either by polling or through a webhook).
+
+Async APIs are a good fit for users who want to submit longer workloads (such as video, audio or document processing), and do not need the result immediately or synchronously.
+
+## How it works
+
+When you deploy an AsyncAPI, Cortex creates an SQS queue, a pool of Async Gateway workers, and a pool of workers running your containers.
+
+The Async Gateway is responsible for submitting the workloads to the queue and for retrieving workload statuses and results. Cortex fully implements and manages the Async Gateway and the queue.
+
+The pool of workers running your containers autoscales based on the average number of messages in the queue and can scale down to 0 (if configured to do so).
+
+![](https://user-images.githubusercontent.com/7456627/111491999-9b67f100-873c-11eb-87f0-effcf4aab01b.png)

docs/workloads/async/async.md

@@ -0,0 +1,19 @@
+# Batch
+
+Batch APIs run distributed and fault-tolerant batch processing jobs on demand.
+
+Batch APIs are a good fit for users who want to break up their workloads and distribute them across a dedicated pool of workers (for example, running inference on a set of images).
+
+## How it works
+
+When you deploy a Batch API, Cortex creates an endpoint to receive job submissions.
+
+Upon job submission, Cortex responds with a Job ID, and asynchronously triggers a Batch Job.
+
+First, Cortex deploys an enqueuer, which breaks up the data in the job into batches and pushes them onto an SQS FIFO queue.
+
+After enqueuing is complete, Cortex initializes the requested number of worker pods and attaches a dequeuer sidecar to each pod. The dequeuer is responsible for retrieving batches from the queue and making an http request to your pod for each batch.
+
+After the worker pods have emptied the queue, the job is marked as complete, and Cortex will terminate the worker pods and delete the SQS queue.
+
+You can make GET requests to the BatchAPI endpoint to get the status of the Job and metrics such as the number of batches completed and failed.

docs/workloads/batch/batch-apis.md

@@ -0,0 +1,11 @@
+# Realtime
+
+Realtime APIs respond to requests synchronously and autoscale based on in-flight request volumes.
+
+Realtime APIs are a good fit for users who want to run stateless containers as a scalable microservice (for example, deploying machine learning models as APIs).
+
+## How it works
+
+When you deploy a Realtime API, Cortex initializes a pool of worker pods and attaches a proxy sidecar to each of the pods.
+
+The proxy is responsible for receiving incoming requests, queueing them (if necessary), and forwarding them to your pod when it is ready. Autoscaling is based on aggregate in-flight request volume, which is published by the proxy sidecars.

docs/workloads/batch/batch.md

@@ -0,0 +1,15 @@
+# Task
+
+Task APIs provide a lambda-style execution of containers. They are useful for running your containers on demand.
+
+Task APIs are a good fit when you need to trigger container execution via an HTTP request. They can be used to run tasks (e.g. training models), and can be configured as task runners for orchestrators (such as airflow).
+
+## How it works
+
+When you deploy a Task API, an endpoint is created to receive task submissions.
+
+Upon submitting a Task, Cortex will respond with a Task ID and will asynchronously trigger the execution of a Task.
+
+Cortex will initialize one or more worker pods based on your API specification. After the worker pod(s) run to completion, the Task is marked as completed and the worker pod(s) are terminated.
+
+You can make GET requests to the Task API endpoint to retreive the status of the Task.