ON-15464: Troubleshooting guide

* sfnettest example now uses latency profile by default rather than recommended
* Removed trailing slash on `$REGISTRY_BASE` causing invalid duplicate slashes
* Added built-in syntax validation to user-modifiable commands
* Documented containers of Onload Device Plugin pod
* Updated README to reflect recent `setPreload` changes
* Symlinks to existing docs in `docs/`

Author: pcolledg-amd

DEVELOPING.md

@@ -13,38 +13,39 @@ and [Kubebuilder](https://kubebuilder.io/).
 
 The Onload Operator and Onload Device Plugin consume Onload container images (`onload-user` and either `onload-source` or `onload-module`). You may wish to pre-populate your cluster's container image registry, either with the [official images provided](README.md#provided-images) or [your own builds](README.md#build).
 
-## Build and deploy Onload Operator from source
+## Build and deploy from source
 
 Configure a development registry and configure cluster for [insecure registries](README.md#insecure-registries)
-if required. Specify the base of the following images:
+if required. Specify the following image locations:
 
 ```sh
-export REGISTRY_BASE=image-registry.openshift-image-registry.svc:5000/onload-clusterlocal/
+export REGISTRY_BASE=image-registry.openshift-image-registry.svc:5000/onload-clusterlocal
+export IMG=$REGISTRY_BASE/onload-operator:latest
+export DEVICE_IMG=$REGISTRY_BASE/onload-device-plugin:latest
 ```
 
 Create and push the Onload Operator controller image:
 
 ```sh
-make docker-build docker-push IMG=$REGISTRY_BASE/onload-operator:latest
+make docker-build docker-push
 ```
 
 Create and push the Onload Device Plugin image:
 
 ```sh
-export DEVICE_IMG=$REGISTRY_BASE/onload-device-plugin:latest
 make device-plugin-docker-build device-plugin-docker-push
 ```
 
 Deploy the Onload Operator:
 
 ```sh
-make deploy IMG=$REGISTRY_BASE/operator:latest
+make deploy
 ```
+
 Ensure that `$DEVICE_IMG` is exported when deploying the operator, or append `DEVICE_IMG=...` to the make invocation.
 
 Continue with [deploying the Onload CR](README.md#onload-custom-resource-cr).
 
-
 ## Footnotes
 
 Copyright (c) 2023 Advanced Micro Devices, Inc.

Makefile

@@ -165,7 +165,7 @@ sfc-mc-build: ## Build sfc mc yaml
 
 .PHONY: sfc-mc-deploy
 sfc-mc-deploy: ## Deploy sfc mc yaml
-	oc apply -f ./scripts/machineconfig/output/99-sfc-machineconfig.yaml
+	oc apply --validate=true -f ./scripts/machineconfig/output/99-sfc-machineconfig.yaml
 
 .PHONY: sfc-mc-undeploy
 sfc-mc-undeploy:

README.md

@@ -63,6 +63,8 @@ When using [in-cluster builds](#onload-module-in-cluster-builds), other dependen
 the method selected. These may include `ubi-minimal` container image and
 [UBI RPM repositories](https://access.redhat.com/articles/4238681).
 
+Nodes require 60MB of root-writable local storage, by default in `/opt`.
+
 ### Provided Images
 
 This repository's YAML configuration uses the following images by default:
@@ -130,19 +132,33 @@ git clone -b v3.0 https://github.com/Xilinx-CNS/kubernetes-onload && cd kubernet
 
 cp -r config/samples/default-clusterlocal config/samples/my-operator
 $EDITOR config/samples/my-operator/kustomization.yaml
-kubectl apply -k config/samples/my-operator
+kubectl apply --validate=true -k config/samples/my-operator
 ```
 
+> [!TIP]
+> Replacing `kubectl apply` with `kubectl kustomize` will output a complete YAML manifest file which can be copied to a
+> network that does not have access to this repository.
+
 ### Onload Device Plugin
 
 The Onload Device Plugin implements the [Kubernetes Device Plugin API](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)
 to expose a [Kubernetes Resource](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
 named `amd.com/onload`.
 
-It is distributed as the container image `onload-device-plugin` and is deployed and configured entirely by
-the Onload Operator. Its image location is configured as an environment variable within the Onload Operator deployment
-([see above](#local-onload-operator-images-in-restricted-networks)) and its ImagePullPolicy as part of
-[Onload Custom Resource (CR)](#onload-custom-resource-cr) along with its other customisation properties.
+It is distributed as the container image `onload-device-plugin`. The image location is configured as an environment
+variable within the Onload Operator deployment ([see above](#local-onload-operator-images-in-restricted-networks)) and
+its ImagePullPolicy as part of [Onload Custom Resource (CR)](#onload-custom-resource-cr), along with its other
+customisation properties.
+
+The Onload Operator manages an Onload Device Plugin DaemonSet which deploys, to each node selected for acceleration,
+a pod consisting of 3 containers:
+
+* Init (`init` container, `onload-user` image)
+  -- for copying Onload files to host filesystem and Onload Worker volume.
+* Onload Worker (`onload-worker` container, `onload-device-plugin` image)
+  -- provides Onload Control Plane environment; privileged access to network namespaces.
+* Onload Device Plugin (`device-plugin` container, `onload-device-plugin` image)
+  -- for Kubernetes Device Plugin API; privileged access to Kubernetes API.
 
 ### Onload Custom Resource (CR)
 
@@ -270,7 +286,8 @@ spec:
         amd.com/onload: 1
 ```
 
-All applications started within the pod environment will be accelerated due to the `LD_PRELOAD` environment variable.
+All applications started within the pod environment will be accelerated due to the `LD_PRELOAD` environment variable
+unless `setPreload: false` is configured in Onload CR.
 
 ### Resource `amd.com/onload`
 
@@ -298,6 +315,17 @@ Binary mounts (if `mountOnload` is true, by default in `/opt/onload/usr/bin/`)
 If you wish to customise where files are mounted in the container's filesystem this can be configured with the fields
 of `spec.devicePlugin` in an Onload CR.
 
+> [!IMPORTANT]
+> Kubernetes Device Plugin only affects initial pod scheduling
+>
+> Kubernetes Device Plugin is designed to configure pods once only, at creation time. If the Onload CR is re-applied to
+> the cluster with settings that would change pod environment -- for example, changing the value of `setPreload` --
+> then running pods must be recreated before using these changes.
+>
+> Additionally, Kubernetes does not evict pods when node resources are removed; pods do not automatically have a formal
+> dependency on Onload Device Plugin or Onload Module. This has the advantage that minor Onload Operator behaviour
+> does not affect the workloads its components pre-configured.
+
 ### Example client-server with sfnettest
 
 Please see [config/samples/sfnettest](config/samples/sfnettest).
@@ -348,6 +376,10 @@ Currently the script produces ConfigMaps with a fixed naming structure,
 for example if you want to create a ConfigMap from a profile called
 `name.opf` the generated name will be `onload-name-profile`.
 
+## Troubleshooting
+
+Please see dedicated [troubleshooting guide](docs/troubleshooting.md).
+
 ## Build
 
 ### Onload Module pre-built images
@@ -373,6 +405,9 @@ Please see [DEVELOPING](DEVELOPING.md) documentation.
 Developing Onload Operator does not require building these images as official images are available.
 
 If you wish to build these images, please follow ['Distributing as container image' in Onload repository's DEVELOPING](https://github.com/Xilinx-CNS/onload/blob/master/DEVELOPING.md#distributing-as-container-image).
+This includes building debug versions. All Onload images in use must be consistent, in exact commit and build
+parameters. For example, a debug build of `onload-user` must be used with a debug build of `onload-module`. Build
+parameter specification is provided in the sample Onload CRs for the in-cluster build method.
 
 ### Insecure registries
 

config/samples/sfnettest/README.md

@@ -1,35 +1,55 @@
 # Onload application example: sfnettest
 
-Here, we run a small utility, [`sfnt-pingpong` from sfnettest](https://github.com/Xilinx-CNS/cns-sfnettest), in a client and server pair to demonstrate Onload acceleration.
+Here, we run a small utility, [`sfnt-pingpong` from sfnettest](https://github.com/Xilinx-CNS/cns-sfnettest), in a
+client and server pair to demonstrate Onload acceleration.
+
+The sfnettest image is solely focused on its performance utils and thus has a micro shell environment; in-depth network
+inspection should be performed using dedicated software or Onload tools available on the node's host filesystem.
 
 ## Deploy
 
-The example will require customisation to your environment. By default, this will deploy two pods running on nodes named `compute-0` and `compute-1`:
+The following example [client-server.yaml](client-server.yaml) will require customisation to your environment. The
+manifest utilises two separately deployed resources which are recommended as part of a full Onload Operator deployment:
+
+* A [Multus network](../../../docs/nad.md)
+  -- connects the pods to hardware that supports acceleration.
+* An [Onload profile](../../../README.md#using-onload-profiles)
+  -- sets environment variables for the pod which are then consumed by userland Onload running in the container(s).
+
+Review the defaults and apply:
+
+* Node names: `compute-0` and `compute-1`
+* Network: `ipvlan-bond0` (Multus)
+* Namespace: `default`
+* Onload accelerated: `amd.com/onload` resource
+* Onload profile: `onload-latency-profile`
 
 ```sh
-kubectl apply -f client-server.yaml
+kubectl apply --validate=true -f client-server.yaml
+kubectl describe pods -l app.kubernetes.io/part-of=sfnettest
 ```
 
 ## Interactive test
 
-Obtain the SFC interface's IP address of the `onload-sfnettest-server` pod:
+Obtain the SFC interface's IP address of the `onload-sfnettest-server` pod, here `198.19.0.1`:
 
 ```sh
-$ kubectl describe pod onload-sfnettest-server | grep AddedInterface
-  Normal  AddedInterface  24s   multus   Add eth0 [192.168.8.114/23] from openshift-sdn
-  Normal  AddedInterface  24s   multus   Add net1 [198.19.0.1/16] from default/ipvlan-bond0
+$ kubectl get events --field-selector involvedObject.name=sfnettest --field-selector reason=AddedInterface
+LAST SEEN   TYPE     REASON           OBJECT                        MESSAGE
+24s         Normal   AddedInterface   pod/onload-sfnettest-client   Add eth0 [192.168.6.203/23] from openshift-sdn
+24s         Normal   AddedInterface   pod/onload-sfnettest-client   Add net1 [198.19.1.1/16] from default/ipvlan-bond0
+24s         Normal   AddedInterface   pod/onload-sfnettest-server   Add eth0 [192.168.8.143/23] from openshift-sdn
+24s         Normal   AddedInterface   pod/onload-sfnettest-server   Add net1 [198.19.0.1/16] from default/ipvlan-bond0
 ```
 
 The server pod is already running the accelerated `sfnt-pingpong` instance.
 
-Run the client from the `onload-sfnettest-client` pod:
+Run `sfnt-pingpong` as a client within the `onload-sfnettest-client` pod, which has an accelerated environment:
 
 ```sh
 kubectl exec onload-sfnettest-client -- sfnt-pingpong udp 198.19.0.1
 ```
 
-You will likely want to [use an Onload profile](../../../README.md#using-onload-profiles).
-
 ---
 
-Copyright (c) 2023 Advanced Micro Devices, Inc.
+Copyright (c) 2023-2024 Advanced Micro Devices, Inc.

config/samples/sfnettest/client-server.yaml

@@ -1,10 +1,13 @@
 # SPDX-License-Identifier: MIT
-# SPDX-FileCopyrightText: (c) Copyright 2023 Advanced Micro Devices, Inc.
+# SPDX-FileCopyrightText: (c) Copyright 2023-2024 Advanced Micro Devices, Inc.
 apiVersion: v1
 kind: Pod
 metadata:
   name: onload-sfnettest-server
   namespace: default
+  labels:
+    app.kubernetes.io/part-of: sfnettest
+    app.kubernetes.io/component: server
   annotations:
     k8s.v1.cni.cncf.io/networks: ipvlan-bond0
 spec:
@@ -21,6 +24,9 @@ spec:
     resources:
       limits:
         amd.com/onload: 1
+    envFrom:
+      - configMapRef:
+          name: onload-latency-profile
     securityContext:
       allowPrivilegeEscalation: false
       capabilities:
@@ -33,6 +39,9 @@ kind: Pod
 metadata:
   name: onload-sfnettest-client
   namespace: default
+  labels:
+    app.kubernetes.io/part-of: sfnettest
+    app.kubernetes.io/component: client
   annotations:
     k8s.v1.cni.cncf.io/networks: ipvlan-bond0
 spec:
@@ -54,6 +63,9 @@ spec:
     resources:
       limits:
         amd.com/onload: 1
+    envFrom:
+      - configMapRef:
+          name: onload-latency-profile
     securityContext:
       allowPrivilegeEscalation: false
       capabilities:

docs/MachineConfig-sfc.md

@@ -1 +1 @@
-scripts/machineconfig/README.md
+../scripts/machineconfig/README.md

docs/sfnettest.md

@@ -0,0 +1 @@
+../config/samples/sfnettest/README.md

docs/sfptpd.md

@@ -0,0 +1 @@
+../config/samples/sfptpd/README.md