Critical Bug: Kubernetes prepends '/' to image names causing InvalidImageName errors in podman integration

[elasticdotventures]

🐛 Critical Bug Report: Image Name Resolution in Minikube Podman Integration

Summary

The minikube podman integration has a critical bug where Kubernetes prepends a forward slash (/) to image names during resolution, causing InvalidImageName errors and preventing pods from starting.

Environment

• Minikube: Latest with podman driver
• Podman: Running as container runtime
• OS: Linux 6.8.0-64-generic
• Kubernetes: minikube's embedded version
• Integration: Native podman integration via minikube podman-env

Expected Behavior

When deploying a Kubernetes pod with image app4dog/middleware:latest, Kubernetes should:

1. Resolve the image name correctly as app4dog/middleware:latest
2. Pull/use the image from minikube's podman image store
3. Start the pod successfully

Actual Behavior

Kubernetes incorrectly prepends a forward slash to the image name:

• Expected: app4dog/middleware:latest
• Actual: /app4dog/middleware:latest ❌

This causes the following error:

Failed to apply default image tag "/app4dog/middleware:latest": couldn't parse image name "/app4dog/middleware:latest": invalid reference format

Reproduction Steps

1. Setup minikube with podman:

   minikube start --driver=podman --container-runtime=cri-o

2. Build image in minikube's podman context:

   eval $(minikube podman-env)
   cd /path/to/app
   podman build -t app4dog/middleware:latest .

3. Verify image exists:

   DOCKER_HOST="tcp://192.168.58.2:2376" DOCKER_TLS_VERIFY="1" DOCKER_CERT_PATH="$HOME/.minikube/certs" podman images | grep app4dog
   # Shows: localhost/app4dog/middleware:latest (correctly built)

4. Deploy Kubernetes pod:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: middleware
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: middleware
     template:
       metadata:
         labels:
           app: middleware
       spec:
         containers:
         - name: middleware
           image: app4dog/middleware:latest
           imagePullPolicy: IfNotPresent

5. Observe failure:

   kubectl get pods
   # Shows: InvalidImageName status
   
   kubectl describe pod middleware-xxx
   # Shows: Failed to apply default image tag "/app4dog/middleware:latest": invalid reference format

Evidence

Pod Events showing the bug:

Events:
  Type     Reason         Age                From               Message
  ----     ------         ----               ----               -------
  Normal   Scheduled      65s                default-scheduler  Successfully assigned app4dog/middleware-xxx to minikube
  Warning  InspectFailed  12s (x6 over 65s)  kubelet            Failed to apply default image tag "/app4dog/middleware:latest": couldn't parse image name "/app4dog/middleware:latest": invalid reference format
  Warning  Failed         12s (x6 over 65s)  kubelet            Error: InvalidImageName

Root Cause Analysis

The issue appears to be in the image name resolution logic where Kubernetes (likely kubelet or the container runtime interface) is incorrectly prepending a / to image names that don't have a registry prefix.

Impact

• Critical: Prevents deployment of custom applications using podman integration
• Blocks adoption: Makes native podman integration unusable for development workflows
• Affects Helm/Tilt workflows: Breaks modern cloud-native development patterns

Testing Context

This bug was discovered while implementing a streamlined developer experience using Helm charts and Tilt with native podman integration. The podman integration works perfectly for image building and storage, but fails at the Kubernetes image name resolution phase.

Priority

High - Critical integration bug preventing podman driver from working with custom applications.