rootless DRA deployment

[vsoch]

Hi Folks!

Has anyone tested in a rootless context? I'm testing with user space kubernetes usernetes which uses the kind base image with docker compose. I'm using podman compose, specifically. I've enabled the feature gates, and added the equivalent parameters for the kubelet to start. The kubelet starts OK, but then on the dranet install I'm not sure we can get around this:

We are running the rootless container with "privileged" and can see /dev/infiniband

# ls -l /dev/infiniband/
total 0
crw------- 1 nobody nogroup 231,  64 Jun 26 16:18 issm0
crw-rw-rw- 1 nobody nogroup  10, 121 Jun 26 16:18 rdma_cm
crw------- 1 nobody nogroup 231,   0 Jun 26 16:18 umad0
crw-rw-rw- 1 nobody nogroup 231, 192 Jun 26 16:18 uverbs0

And that works to interact with UCX for Infiniband, however I suspect there is an underlying mknod call that goes to the kernel, is seen as my actual user (and the call would require real root) and then it fails. Are there any workarounds for this, or any success to deploy DRA/dranet in a rootless context?

Thank you!

[vsoch]

Making progress (maybe 😆) I added the dranet container to be part of podman compose, ensuring to share the /etc volume to see the kubeconfig.

---
services:
  node:
    image: usernetes_node
    hostname: ${NODE_NAME}
    privileged: true
    restart: always
    networks:
      corona192:
        ipv4_address: ${NODE_IP}
    ports:
      # <host>:<container>
      # etcd (default: 2379)
      - ${PORT_ETCD}:${PORT_ETCD}
      # kube-apiserver (default: 6443)
      - ${PORT_KUBE_APISERVER}:${PORT_KUBE_APISERVER}
      # kubelet (default: 10250)
      - ${PORT_KUBELET}:${PORT_KUBELET}
      # flannel (default: 8472)
      - ${PORT_FLANNEL}:${PORT_FLANNEL}/udp
    volumes:
      - kubelet-plugins:/var/lib/kubelet/plugins_registry
      - .:/usernetes:ro
      - /boot:/boot:ro
      - /lib/modules:/lib/modules:ro
      - node-var:/var
      - node-opt:/opt
      - node-etc:/etc
      - type: tmpfs
        target: /run
      - type: tmpfs
        target: /tmp
    working_dir: /usernetes
    environment:
      KUBECONFIG: /etc/kubernetes/admin.conf
      HOST_IP: ${HOST_IP}
    sysctls:
      - net.ipv4.ip_forward=1
      # In addition, `net.ipv4.conf.default.rp_filter`
      # has to be set to 0 (disabled) or 2 (loose)
      # in the daemon's network namespace.
    annotations:
      # Accelerate network for nerdctl >= 2.0.0-beta.4 with bypass4netns >= 0.4.1
      "nerdctl/bypass4netns": "${BYPASS4NETNS:-false}"
      "nerdctl/bypass4netns-ignore-bind": "true"
      "nerdctl/bypass4netns-ignore-subnets": "${BYPASS4NETNS_IGNORE_SUBNETS:-}"

  dranet:
    image: ghcr.io/google/dranet:v0.4.0
    container_name: dranet-driver
    privileged: true
    restart: always
    # DraNet should see the host network to manage its devices
    network_mode: "host"
    volumes:
      - kubelet-plugins:/var/lib/kubelet/plugins_registry
      - /sys:/sys:ro
      - /dev:/dev
      - node-etc:/etc:ro 
    command:
      - /dranet
      - --kubeconfig=/etc/kubernetes/admin.conf

networks:
  corona192:
    ipam:
      config:
        # Each of the nodes has to have a different IP.
        - subnet: ${NODE_SUBNET}
volumes:
  node-var: {}
  node-opt: {}
  node-etc: {}
  kubelet-plugins: {}

When that first comes up, we haven't run kubeadm-init so we get a bit of a race...

0704 19:25:52.655821       1 app.go:75] FLAG: --v="0"
I0704 19:25:52.655827       1 app.go:75] FLAG: --vmodule=""
F0704 19:25:52.655886       1 app.go:99] can not create client-go configuration: stat /etc/kubernetes/admin.conf: no such file or directory

of course with restart since the next command to the Makefile is make kubadm-init (which populates that directory) we get beyond that. This isn't working yet, but it's a tiny bit of progress. I need to understand next how to create the NRI service (an equivalent device plugin / sock I think).

I0704 19:28:55.907914       1 app.go:75] FLAG: --v="0"
I0704 19:28:55.907917       1 app.go:75] FLAG: --vmodule=""
I0704 19:28:55.911705       1 driver.go:80] RDMA subsystem in mode: shared
time="2025-07-04T19:28:57Z" level=info msg="Created plugin 00-dra.net (dranet, handles RunPodSandbox,StopPodSandbox,RemovePodSandbox,CreateContainer)"
I0704 19:28:57.912345       1 app.go:157] driver started
I0704 19:28:57.912468       1 driver.go:142] NRI plugin failed with error failed to connect to NRI service: dial unix /var/run/nri/nri.sock: connect: no such file or directory
I0704 19:28:57.912499       1 driver.go:148] Restarting NRI plugin 0 out of 5
I0704 19:28:57.912535       1 driver.go:142] NRI plugin failed with error failed to connect to NRI service: dial unix /var/run/nri/nri.sock: connect: no such file or directory
I0704 19:28:57.912542       1 driver.go:148] Restarting NRI plugin 1 out of 5
I0704 19:28:57.912573       1 driver.go:142] NRI plugin failed with error failed to connect to NRI service: dial unix /var/run/nri/nri.sock: connect: no such file or directory
I0704 19:28:57.912581       1 driver.go:148] Restarting NRI plugin 2 out of 5
I0704 19:28:57.912614       1 driver.go:142] NRI plugin failed with error failed to connect to NRI service: dial unix /var/run/nri/nri.sock: connect: no such file or directory
I0704 19:28:57.912621       1 driver.go:148] Restarting NRI plugin 3 out of 5
I0704 19:28:57.912654       1 driver.go:142] NRI plugin failed with error failed to connect to NRI service: dial unix /var/run/nri/nri.sock: connect: no such file or directory
I0704 19:28:57.912662       1 driver.go:148] Restarting NRI plugin 4 out of 5
F0704 19:28:57.912669       1 driver.go:151] NRI plugin failed for 5 times to be restarted

[vsoch]

Ah, I think NRI needs to be enabled, this bit here, for the container runtime.

dranet/install.yaml

Lines 112 to 120 in a15ec9b

	if grep -q "io.containerd.nri.v1.nri" /etc/containerd/config.toml
	then
	echo "containerd config contains NRI reference already; taking no action"
	else
	echo "containerd config does not mention NRI, thus enabling it";
	printf '%s\n' "[plugins.\"io.containerd.nri.v1.nri\"]" " disable = false" " disable_connections = false" " plugin_config_path = \"/etc/nri/conf.d\"" " plugin_path = \"/opt/nri/plugins\"" " plugin_registration_timeout = \"5s\"" " plugin_request_timeout = \"5s\"" " socket_path = \"/var/run/nri/nri.sock\"" >> /etc/containerd/config.toml
	echo "restarting containerd"
	nsenter -t 1 -m -u -i -n -p -- systemctl restart containerd
	fi

[vsoch]

Next step of progress - I'm reporting here because my allocation will likely end, and usernetes is deployed to a temporary space and I'll lose all my work. I got nri enabled and viewable by the dranet container:

I0704 19:55:33.171887       1 driver.go:80] RDMA subsystem in mode: shared
F0704 19:55:33.173098       1 app.go:153] driver failed to start: start kubelet plugin: start registrar: start gRPC server: listen on "": listen unix /var/lib/kubelet/plugins_registry/dra.net-reg.sock: bind: no such file or directory
I0704 19:55:33.645123       1 app.go:182] dranet go go1.24.3 build: 000e9c8ccc6e91abe9893b036699ff96ba7cea17 time: 2025-05-25T16:43:52Z
I0704 19:55:33.645181       1 app.go:75] FLAG: --add_dir_header="false"
I0704 19:55:33.645191       1 app.go:75] FLAG: --alsologtostderr="false"
I0704 19:55:33.645194       1 app.go:75] FLAG: --bind-address=":9177"
I0704 19:55:33.645198       1 app.go:75] FLAG: --filter="attributes[\"dra.net/type\"].StringValue  != \"veth\""
I0704 19:55:33.645201       1 app.go:75] FLAG: --hostname-override=""
I0704 19:55:33.645205       1 app.go:75] FLAG: --kubeconfig="/etc/kubernetes/admin.conf"
I0704 19:55:33.645207       1 app.go:75] FLAG: --log_backtrace_at=":0"
I0704 19:55:33.645213       1 app.go:75] FLAG: --log_dir=""
I0704 19:55:33.645216       1 app.go:75] FLAG: --log_file=""
I0704 19:55:33.645219       1 app.go:75] FLAG: --log_file_max_size="1800"
I0704 19:55:33.645222       1 app.go:75] FLAG: --logtostderr="true"
I0704 19:55:33.645224       1 app.go:75] FLAG: --one_output="false"
I0704 19:55:33.645227       1 app.go:75] FLAG: --skip_headers="false"
I0704 19:55:33.645229       1 app.go:75] FLAG: --skip_log_headers="false"
I0704 19:55:33.645232       1 app.go:75] FLAG: --stderrthreshold="2"
I0704 19:55:33.645239       1 app.go:75] FLAG: --v="0"
I0704 19:55:33.645242       1 app.go:75] FLAG: --vmodule=""
I0704 19:55:33.649276       1 driver.go:80] RDMA subsystem in mode: shared
time="2025-07-04T19:55:34Z" level=info msg="Created plugin 00-dra.net (dranet, handles RunPodSandbox,StopPodSandbox,RemovePodSandbox,CreateContainer)"
I0704 19:55:34.651839       1 app.go:157] driver started
time="2025-07-04T19:55:34Z" level=info msg="Registering plugin 00-dra.net..."
time="2025-07-04T19:55:34Z" level=info msg="Configuring plugin 00-dra.net for runtime containerd/v2.0.5..."
time="2025-07-04T19:55:34Z" level=info msg="Started plugin 00-dra.net..."
I0704 19:55:34.662297       1 db.go:312] could not get pci vendor information : open /sys/class/net/enp41s0f3u1u2c2/device/vendor: no such file or directory
E0704 19:55:34.775111       1 reflector.go:200] "Failed to watch" err="failed to list *v1beta1.ResourceSlice: Get \"https://u7s-corona192:6443/apis/resource.k8s.io/v1beta1/resourceslices?fieldSelector=spec.driver%3Ddra.net%2Cspec.nodeName%3Dcorona192&limit=500&resourceVersion=0\": dial tcp: lookup u7s-corona192 on 192.168.255.1:53: no such host" logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1beta1.ResourceSlice"
E0704 19:55:35.906790       1 reflector.go:200] "Failed to watch" err="failed to list *v1beta1.ResourceSlice: Get \"https://u7s-corona192:6443/apis/resource.k8s.io/v1beta1/resourceslices?fieldSelector=spec.driver%3Ddra.net%2Cspec.nodeName%3Dcorona192&limit=500&resourceVersion=0\": dial tcp: lookup u7s-corona192 on 192.168.255.1:53: no such host" logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1beta1.ResourceSlice"
E0704 19:55:38.093476       1 reflector.go:200] "Failed to watch" err="failed to list *v1beta1.ResourceSlice: Get \"https://u7s-corona192:6443/apis/resource.k8s.io/v1beta1/resourceslices?fieldSelector=spec.driver%3Ddra.net%2Cspec.nodeName%3Dcorona192&limit=500&resourceVersion=0\": dial tcp: lookup u7s-corona192 on 192.168.255.1:53: no such host" logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1beta1.ResourceSlice"
I0704 19:55:42.252239       1 nri_hooks.go:43] Synchronized state with the runtime (4 pods, 4 containers)...
I0704 19:55:42.252272       1 nri_hooks.go:47] Synchronize Pod kube-system/kube-scheduler-u7s-corona192 UID e56e8da4b647b6f4effcc725c0289c22
I0704 19:55:42.252280       1 nri_hooks.go:47] Synchronize Pod kube-system/etcd-u7s-corona192 UID a6834263b3686c77e54bcc3d8c070206
I0704 19:55:42.252285       1 nri_hooks.go:47] Synchronize Pod kube-system/kube-apiserver-u7s-corona192 UID c4c785b79e08b54089d4c68507e34ff6
I0704 19:55:42.252288       1 nri_hooks.go:47] Synchronize Pod kube-system/kube-controller-manager-u7s-corona192 UID 9b6b392241d287ebfb205493596229a9
E0704 19:55:43.513183       1 reflector.go:200] "Failed to watch" err="failed to list *v1beta1.ResourceSlice: Get \"https://u7s-corona192:6443/apis/resource.k8s.io/v1beta1/resourceslices?fieldSelector=spec.driver%3Ddra.net%2Cspec.nodeName%3Dcorona192&limit=500&resourceVersion=0\": dial tcp: lookup u7s-corona192 on 192.168.255.1:53: no such host" logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1beta1.ResourceSlice"
E0704 19:55:53.061082       1 reflector.go:200] "Failed to watch" err="failed to list *v1beta1.ResourceSlice: Get \"https://u7s-corona192:6443/apis/resource.k8s.io/v1beta1/resourceslices?fieldSelector=spec.driver%3Ddra.net%2Cspec.nodeName%3Dcorona192&limit=500&resourceVersion=0\": dial tcp: lookup u7s-corona192 on 192.168.255.1:53: no such host" logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1beta1.ResourceSlice"

Since we don't have init containers, I added a Makefile.d script for usernetes ./Makefile.d/dra.sh

#!/bin/bash
set -o errexit
set -o pipefail
set -o nounset
set -x
if grep -q "io.containerd.nri.v1.nri" /etc/containerd/config.toml
  then
    echo "containerd config contains NRI reference already; taking no action"
  else
    echo "containerd config does not mention NRI, thus enabling it";
    printf '%s\n' "[plugins.\"io.containerd.nri.v1.nri\"]" "  disable = false" "  disable_connections = false" "  plugin_config_path = \"/etc/nri/conf.d\"" "  plugin_path = \"/opt/nri/plugins\"" "  plugin_registration_timeout = \"5s\"" "  plugin_request_timeout = \"5s\"" "  socket_path = \"/var/run/nri/nri.sock\"" >> /etc/containerd/config.toml
    echo "restarting containerd"
    systemctl restart containerd
fi

We of course don't need nsenter because we aren't running in a pod. It gets run immediately after the container (with the kubelet) is created to tweak and restart containerd.

.PHONY: up
up: check-preflight
	sed -i "s/default_network/$(HOSTNAME)/g" docker-compose.yaml
	$(COMPOSE) up --build -d
	$(NODE_SHELL) /bin/bash /usernetes/Makefile.d/dra.sh

The harder bit was the docker-compose - the nri.sock is stored in /var/run but I don't think you can just do a shared /var (a volume) because when we create the tmpfs /run it is technically just a symbolic link, so it clobbers it.

$ ls -l /var/run
lrwxrwxrwx 1 root root 6 Jun 26 01:01 /var/run -> ../run

So this is the docker-compose to generate the above, with the noted changes.

services:
  node:
    image: usernetes_node
    hostname: ${NODE_NAME}
    privileged: true
    restart: always
    networks:
      corona192:
        ipv4_address: ${NODE_IP}
    ports:
      # <host>:<container>
      # etcd (default: 2379)
      - ${PORT_ETCD}:${PORT_ETCD}
      # kube-apiserver (default: 6443)
      - ${PORT_KUBE_APISERVER}:${PORT_KUBE_APISERVER}
      # kubelet (default: 10250)
      - ${PORT_KUBELET}:${PORT_KUBELET}
      # flannel (default: 8472)
      - ${PORT_FLANNEL}:${PORT_FLANNEL}/udp
    volumes:
      - .:/usernetes:ro
      - /boot:/boot:ro
      - /lib/modules:/lib/modules:ro
      - node-var:/var
      - node-opt:/opt
      - node-etc:/etc
      - node-run:/run
      - type: tmpfs
        target: /tmp
    working_dir: /usernetes
    environment:
      KUBECONFIG: /etc/kubernetes/admin.conf
      HOST_IP: ${HOST_IP}
    sysctls:
      - net.ipv4.ip_forward=1
      # In addition, `net.ipv4.conf.default.rp_filter`
      # has to be set to 0 (disabled) or 2 (loose)
      # in the daemon's network namespace.
    annotations:
      # Accelerate network for nerdctl >= 2.0.0-beta.4 with bypass4netns >= 0.4.1
      "nerdctl/bypass4netns": "${BYPASS4NETNS:-false}"
      "nerdctl/bypass4netns-ignore-bind": "true"
      "nerdctl/bypass4netns-ignore-subnets": "${BYPASS4NETNS_IGNORE_SUBNETS:-}"

  dranet:
    image: ghcr.io/google/dranet:v0.4.0
    container_name: dranet-driver
    privileged: true
    restart: always
    # DraNet needs to see the host network to manage its devices
    network_mode: "host"
    volumes:
      - /sys:/sys:ro
      - /dev:/dev
      - node-var:/var
      - node-run:/run:ro
      - node-etc:/etc:ro 
    command:
      - /dranet
      - --kubeconfig=/etc/kubernetes/admin.conf

networks:
  corona192:
    ipam:
      config:
        # Each of the nodes has to have a different IP.
        # The node IP here is not accessible from other nodes.
        - subnet: ${NODE_SUBNET}
volumes:
  node-run: {}
  node-var: {}
  node-opt: {}
  node-etc: {}

And just a note - the network is named based on the node because podman stores cni configs for rootless in the user's home:

$ ls ~/.config/cni/net.d/
cni.lock  default  usernetes_corona192.conflist

That of course is a path shared across nodes. If it is just named usernetes.conflist there is a race and all the worker nodes / control plane clobber one another. I opened a simple fix here that would work for both cases:

rootless-containers/usernetes#375

But it looks unlikely to be merged, so we are maintaining on our fork for the HPC use case.

Looking into the next layer of this bug onion! 🐛 🧅

[vsoch]

okay I... don't see any bugs? I think we are communicating with the api server, and there aren't errors about the hostname anymore!

But I made a lot of changes.

The following issues came up:

• We couldn't see the kube-apiserver
• If we provided extra_hosts, we faced the catch22 that we are using the host network (corona192) but the kubelet only knows u7s-corona192.
• the rbac permissions didn't exist yet

services:
  node:
    image: usernetes_node
    hostname: ${NODE_NAME}
    privileged: true
    restart: always
    networks:
      corona192:
        ipv4_address: ${NODE_IP}
    expose:
      - "${PORT_KUBE_APISERVER}"
    ports:
      # <host>:<container>
      # etcd (default: 2379)
      - ${PORT_ETCD}:${PORT_ETCD}
      # kube-apiserver (default: 6443)
      - ${PORT_KUBE_APISERVER}:${PORT_KUBE_APISERVER}
      # kubelet (default: 10250)
      - ${PORT_KUBELET}:${PORT_KUBELET}
      # flannel (default: 8472)
      - ${PORT_FLANNEL}:${PORT_FLANNEL}/udp
    volumes:
      - .:/usernetes:ro
      - /boot:/boot:ro
      - /lib/modules:/lib/modules:ro
      - node-var:/var
      - node-opt:/opt
      - node-etc:/etc
      - node-run:/run
      - type: tmpfs
        target: /tmp
    working_dir: /usernetes
    environment:
      KUBECONFIG: /etc/kubernetes/admin.conf
      HOST_IP: ${HOST_IP}
    sysctls:
      - net.ipv4.ip_forward=1
      # In addition, `net.ipv4.conf.default.rp_filter`
      # has to be set to 0 (disabled) or 2 (loose)
      # in the daemon's network namespace.
    annotations:
      # Accelerate network for nerdctl >= 2.0.0-beta.4 with bypass4netns >= 0.4.1
      "nerdctl/bypass4netns": "${BYPASS4NETNS:-false}"
      "nerdctl/bypass4netns-ignore-bind": "true"
      "nerdctl/bypass4netns-ignore-subnets": "${BYPASS4NETNS_IGNORE_SUBNETS:-}"

  dranet:
    image: ghcr.io/google/dranet:v0.4.0
    container_name: dranet-driver
    privileged: true
    restart: always
    # DraNet needs to see the host network to manage its devices
    network_mode: "host"
    volumes:
      - /sys:/sys:ro
      - /dev:/dev
      - node-var:/var
      - node-run:/run:ro
      - node-etc:/etc:ro 
    extra_hosts:
      # We need to be able to see the api server
      - "${NODE_NAME}:${HOST_IP}"
    command:
      - /dranet
      - --kubeconfig=/etc/kubernetes/admin.conf
      - --hostname-override=${NODE_NAME}

networks:
  corona192:
    ipam:
      config:
        # Each of the nodes has to have a different IP.
        # The node IP here is not accessible from other nodes.
        - subnet: ${NODE_SUBNET}
volumes:
  node-run: {}
  node-var: {}
  node-opt: {}
  node-etc: {}

The extra_hosts tells us how to lookup the address we find in the kubeconfig for the apiserver. But since it's on the physical host, we provide that address instead of the private network for usernetes. Then we need to also do a --hostname-override for dranet because it then tries to use the actual host, and we want it to switch back to u7s-corona192 Finally, the rbac application seems to only take with a restart to the container - another race-y thing, but it seems to work. I'll move this orchestration outside of usernetes and to be with our cluster service.

This seems to be working, at least to the point I don't see error messages. I'm going to see the extent to which I can create something for Inifiniband.

[vsoch]

I don't see that it discovers the Infiniband devices:

kubectl get resourceslice -o yaml

apiVersion: v1
items:
- apiVersion: resource.k8s.io/v1beta2
  kind: ResourceSlice
  metadata:
    creationTimestamp: "2025-07-04T20:34:37Z"
    generateName: u7s-corona192-dra.net-
    generation: 1
    name: u7s-corona192-dra.net-bgvh4
    ownerReferences:
    - apiVersion: v1
      controller: true
      kind: Node
      name: u7s-corona192
      uid: cc334c97-06da-4854-b684-61b5bcfeef0e
    resourceVersion: "506"
    uid: 31952bf2-599e-47bd-860e-364c7fed1b71
  spec:
    devices:
    - attributes:
        dra.net/alias:
          string: ""
        dra.net/encapsulation:
          string: ether
        dra.net/ifName:
          string: enp41s0f3u1u2c2
        dra.net/pciAddressBus:
          string: "20"
        dra.net/pciAddressDevice:
          string: "08"
        dra.net/pciAddressDomain:
          string: "0000"
        dra.net/pciAddressFunction:
          string: "1"
        dra.net/rdma:
          bool: false
        dra.net/sriov:
          bool: false
        dra.net/state:
          string: unknown
        dra.net/type:
          string: device
        dra.net/virtual:
          bool: false
      name: enp41s0f3u1u2c2
    - attributes:
        dra.net/alias:
          string: ""
        dra.net/encapsulation:
          string: ether
        dra.net/ifName:
          string: eno1
        dra.net/ipv4:
          string: 192.168.64.192
        dra.net/mac:
          string: ac:1f:6b:48:8d:c8
        dra.net/mtu:
          int: 1500
        dra.net/numaNode:
          int: 1
        dra.net/pciAddressBus:
          string: e0
        dra.net/pciAddressDevice:
          string: "05"
        dra.net/pciAddressDomain:
          string: "0000"
        dra.net/pciAddressFunction:
          string: "1"
        dra.net/pciVendor:
          string: Intel Corporation
        dra.net/rdma:
          bool: false
        dra.net/sriov:
          bool: true
        dra.net/sriovVfs:
          int: 0
        dra.net/state:
          string: up
        dra.net/type:
          string: device
        dra.net/virtual:
          bool: false
      name: eno1
    - attributes:
        dra.net/alias:
          string: ""
        dra.net/encapsulation:
          string: ether
        dra.net/ifName:
          string: eno2
        dra.net/numaNode:
          int: 1
        dra.net/pciAddressBus:
          string: e0
        dra.net/pciAddressDevice:
          string: "05"
        dra.net/pciAddressDomain:
          string: "0000"
        dra.net/pciAddressFunction:
          string: "1"
        dra.net/pciVendor:
          string: Intel Corporation
        dra.net/rdma:
          bool: false
        dra.net/sriov:
          bool: true
        dra.net/sriovVfs:
          int: 0
        dra.net/state:
          string: down
        dra.net/type:
          string: device
        dra.net/virtual:
          bool: false
      name: eno2
    driver: dra.net
    nodeName: u7s-corona192
    pool:
      generation: 1
      name: u7s-corona192
      resourceSliceCount: 1
kind: List
metadata:
  resourceVersion: ""

I am going to naively assume this is something I can control (and not a rootless limitation) - maybe the container needs to have rdma-core and similar to query for info, or maybe it is being mis-classified as veth (and filtered out). I'll explore both. It looks like this is a build with debian and just dranet added. I'll try a custom build.

[vsoch]

OK I think I see why it's not being seen. It would need to be seen based on a net device?

https://github.com/Mellanox/rdmamap/blob/37bd11cc4c57da931b7b117f829fb663d46ce480/rdma_map.go#L348-L361

But in our setup, we only have the driver exposed - we interact with it via UCX. The ib0ip (that is on the host) is not possible to get in the rootless environment. I think I know what I need to test next - using other functions in that library to detect the device in a different way., e.g., there are a few other options in this file: https://github.com/Mellanox/rdmamap/blob/37bd11cc4c57da931b7b117f829fb663d46ce480/rdma_map.go#L60

I need to save the state of my changes - it was a huge adventure getting a dranet base built that would actually have ibv_devinfo working. I'm going to do that first.

[vsoch]

OK, found the devices with other functions!

[TEST 1] Calling rdmamap.GetRdmaDeviceList()
  [SUCCESS] Found 1 RDMA devices:
    - mlx5_0
      - Checking for character devices...
        [SUCCESS] Found char devices: [/dev/infiniband/issm0 /dev/infiniband/umad0 /dev/infiniband/uverbs0 /dev/infiniband/rdma_cm]

[TEST 2] Calling rdmamap.GetRdmaDevicesForPcidev()
  [INFO] Found PCI address for mlx5_0: 0000:c4:00.0
  [SUCCESS] Found RDMA devices for PCI addr 0000:c4:00.0: [mlx5_0]

Here is how I ran that to test:

package main

import (
	"fmt"
	"os"
	"path/filepath"

	"github.com/Mellanox/rdmamap"
)

func main() {

	fmt.Println("\n[TEST 1] Calling rdmamap.GetRdmaDeviceList()")
	deviceList := rdmamap.GetRdmaDeviceList()
	if deviceList == nil || len(deviceList) == 0 {
		fmt.Println("  [FAIL] GetRdmaDeviceList() returned no devices.")
		fmt.Println("         This means it cannot read the directories in /sys/class/infiniband/")
	} else {
		fmt.Printf("  [SUCCESS] Found %d RDMA devices:\n", len(deviceList))
		for _, dev := range deviceList {
			fmt.Printf("    - %s\n", dev)

			// Test 3: For each found device, find its character devices
			fmt.Printf("      - Checking for character devices...\n")
			charDevs := rdmamap.GetRdmaCharDevices(dev)
			if len(charDevs) > 0 {
				fmt.Printf("        [SUCCESS] Found char devices: %v\n", charDevs)
			} else {
				fmt.Printf("        [WARN] Found no char devices for %s.\n", dev)
			}
		}
	}

	fmt.Println("\n[TEST 2] Calling rdmamap.GetRdmaDevicesForPcidev()")
	// We need to find the PCI address first. Let's try to find it for mlx5_0
	pciDir := "/sys/class/infiniband/mlx5_0/device"
	
	pciLinkTarget, err := os.Readlink(pciDir)
	if err != nil {
		fmt.Printf("  [INFO] Could not determine PCI address from %s. Skipping PCI test. Error: %v\n", pciDir, err)
	} else {
		pciAddr := filepath.Base(pciLinkTarget) // Extracts just the '0000:c4:00.0' part
		fmt.Printf("  [INFO] Found PCI address for mlx5_0: %s\n", pciAddr)
		
		pciDevs := rdmamap.GetRdmaDevicesForPcidev(pciAddr)
		if len(pciDevs) > 0 {
			fmt.Printf("  [SUCCESS] Found RDMA devices for PCI addr %s: %v\n", pciAddr, pciDevs)
		} else {
			fmt.Printf("  [FAIL] Found no RDMA devices for PCI addr %s.\n", pciAddr)
		}
	}
}	

@aojea does DRA/dranet rely on having a net device or can we use other means of discovery? Is there a reason that the rdmamap library function that specifically queries net devices was used?

I haven't looked at how the returned string is used yet in dranet - I suppose I might find the answer to that question.

[vsoch]

Lost my allocation 😭

The minimal changes are here: converged-computing/usernetes#1

I need to familiarize myself with the dranet code (will do reading over dinner) and then I'll test if I can change that discovery logic to also detect non-net devices. Taking a break now to work on some compatibility ML stuff.

[aojea]

@aojea does DRA/dranet rely on having a net device or can we use other means of discovery? Is there a reason that the rdmamap library function that specifically queries net devices was used?

We can use other means of discovery, @vsoch the discovery logic is pretty simple, is just here

dranet/pkg/inventory/db.go

Lines 146 to 181 in a15ec9b

	ifaces, err := nlHandle.LinkList()
	if err != nil {
	klog.Error(err, "unexpected error trying to get system interfaces")
	}
	for _, iface := range ifaces {
	klog.V(7).InfoS("Checking network interface", "name", iface.Attrs().Name)
	if gwInterfaces.Has(iface.Attrs().Name) {
	klog.V(4).Infof("iface %s is an uplink interface", iface.Attrs().Name)
	continue
	}
	if ignoredInterfaceNames.Has(iface.Attrs().Name) {
	klog.V(4).Infof("iface %s is in the list of ignored interfaces", iface.Attrs().Name)
	continue
	}
	// skip loopback interfaces
	if iface.Attrs().Flags&net.FlagLoopback != 0 {
	continue
	}
	// publish this network interface
	device, err := db.netdevToDRAdev(iface)
	if err != nil {
	klog.V(2).Infof("could not obtain attributes for iface %s : %v", iface.Attrs().Name, err)
	continue
	}
	devices = append(devices, *device)
	klog.V(4).Infof("Found following network interface %s", iface.Attrs().Name)
	}
	klog.V(4).Infof("Found %d devices", len(devices))
	if len(devices) > 0 {
	db.notifications <- devices
	}

[BenTheElder]

BTW you can also do: https://kind.sigs.k8s.io/docs/user/rootless/

With that guide, it should be possible to run the in repo e2e scripts against a rootless environment.