Skip to content

Conversation

nirs
Copy link
Contributor

@nirs nirs commented Jul 25, 2025

Replace 9p mounts with virtiofs for vfkit and krunkit. Testing shows
that virtiofs mount is 23 times faster with krunkit, and 8 times faster
with vfkit.

vfkit and krunkit support multiple virtiofs mounts but minikube
--mount-* flags are not ready for multiple mounts. We have the same
issue with KIC drivers, supporting multiple mounts but using only one.
We hope to improve this in the next release.

Example usage:

minikube start --driver krunkit --mount-string ~/models:/mnt/models

The arguments are parsed and validated when configuring the driver, so
invalid arguments fail quickly without starting the driver. The
validated mounts are stored in the machine config:

$ jq '.Driver.VirtiofsMounts' < ~/.minikube/machines/minikube/config.json
[
    {
    "HostPath": "/Users/nir/models",
    "GuestPath": "/mnt/models",
    "Tag": "f845b54d-00e3-493d-9541-3b37490b96db"
    }
]

Minikube generates a new random UUID for every virtiofs mount to
identify the file system inside the guest. In krunkit and vfkit, every
mount is add as:

--device virtio-fs,sharedDir=/host-path,mountTag=f845b54d-00e3-493d-9541-3b37490b96db

When the guest is started the shared directory is mounted via SSH using:

sudo mkdir -p /mnt/models
sudo mount -t virtiofs f845b54d-00e3-493d-9541-3b37490b96db /mnt/models

Example mount:

$ minikube ssh findmnt /mnt/models
TARGET      SOURCE                               FSTYPE   OPTIONS
/mnt/models f845b54d-00e3-493d-9541-3b37490b96db virtiofs rw,relatime

More work is needed to add VirtioFS to qemu. I'm starting with vfkit and
krunkit since they have identical interface (krunkit was designed as
drop-in replacement for vfkit).

To play with this PR, see medyagh/ai-playground-minikube#4 (comment)

Example chat with granite model

sh-5.2# llama-run -ngl 999 /mnt/models/granite-7b-lab-Q4_K_M.gguf -c 4098
> can you write a creative and fun haiku about minikube krunkit driver?

Krunkit's minikube dance,
In a microcosm, it thrives,
Boarding, dismounting,
A whimsical, tiny twirl.

Minikube's joyous spin,
A dance of virtual machines,
Krunkit's playful prance,
In a world so small, it finds its place.

A tiny, krunkit-powered waltz,
In the vast universe of virtual boxes,
Minikube's charm,
In every single box.

A minikube krunkit jive,
A dance of joy, a testament,
To the power of the small,
And the magic of Krunkit's ballet.

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 25, 2025
@k8s-ci-robot k8s-ci-robot requested review from medyagh and prezha July 25, 2025 23:57
@k8s-ci-robot
Copy link
Contributor

Hi @nirs. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 25, 2025
@medyagh
Copy link
Member

medyagh commented Jul 26, 2025

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 26, 2025
@nirs nirs force-pushed the drivers-virtiofs branch from df178d0 to c0b3c92 Compare July 26, 2025 00:19
@nirs
Copy link
Contributor Author

nirs commented Jul 26, 2025

@afbjorklund can you review?

@minikube-pr-bot

This comment has been minimized.

@nirs nirs force-pushed the drivers-virtiofs branch 2 times, most recently from b6650e0 to a324895 Compare July 26, 2025 18:20
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 26, 2025
@nirs nirs force-pushed the drivers-virtiofs branch from a324895 to 651bb9c Compare July 26, 2025 18:21
@nirs nirs changed the title drivers: Add VirtioFS support for vfkit and krunkit drivers: Add VirtioFS shares for vfkit and krunkit Jul 26, 2025
@nirs
Copy link
Contributor Author

nirs commented Jul 26, 2025

@medyagh @ComradeProgrammer can you approve the workflows?

@nirs nirs force-pushed the drivers-virtiofs branch from 651bb9c to d0f3378 Compare July 26, 2025 20:01
@nirs
Copy link
Contributor Author

nirs commented Jul 26, 2025

minikube start --virtiofs-share /Users/joe/models:models

This was a quick version to be able to play with virtuous. I think we need something more extensible that will work nicely when we add yaml support:

Flags

minikube start \
    --virtiofs-share path=/Users/Joe/models:mount=models \
    --virtiofs-share path=/Users/Joe/models:mount=models:writable

Minikube Yaml

virtiofs-shares:
- path: /Users/Joe/models
  mount: models
- path: /Users/Joe/project
  mount: project
  writable: true

Minikube JSON

{
  "virtiofs-shares": [
    {
      "path": "/Users/Joe/models",
      "mount": "models"
    },
    {
      "path": "/Users/Joe/project",
      "mount": "project",
      "writable": true
    }
  ]
}

@nirs nirs changed the title drivers: Add VirtioFS shares for vfkit and krunkit drivers: Add Virtiofs shares for vfkit and krunkit Jul 26, 2025
@nirs nirs force-pushed the drivers-virtiofs branch from d0f3378 to cbd8afb Compare July 28, 2025 22:49
@minikube-pr-bot

This comment has been minimized.

@k8s-ci-robot k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 16, 2025
@nirs
Copy link
Contributor Author

nirs commented Aug 16, 2025

Rebased on #21272, comments not addressed yet.

@minikube-pr-bot

This comment has been minimized.

@nirs nirs mentioned this pull request Aug 16, 2025
@nirs nirs force-pushed the drivers-virtiofs branch from b09d34e to ca9eb87 Compare August 16, 2025 18:35
@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 16, 2025
@nirs
Copy link
Contributor Author

nirs commented Aug 16, 2025

@medyagh changes in last version:

  • Split the virtiofs package to new commit for easier review and cleaner git history
  • Improve the comments for virtiofs.Mount mentioning source and target
  • Spit the docs changes to another docs: Update mount docs #21346 since the docs can be finished and improved after the release
  • Update commit message (--mount is not used now)

Tested locally with vfkit and krunkit:

  • using virtiofs mount
  • make integration for TestMountStart
  • make functional

Should be ready for merge.

@nirs nirs marked this pull request as ready for review August 16, 2025 18:43
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 16, 2025
@k8s-ci-robot k8s-ci-robot requested a review from medyagh August 16, 2025 18:43
@minikube-pr-bot

This comment has been minimized.

@nirs nirs marked this pull request as draft August 16, 2025 20:40
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 16, 2025
@nirs
Copy link
Contributor Author

nirs commented Aug 16, 2025

Marking as draft since it depends on #21272

@nirs
Copy link
Contributor Author

nirs commented Aug 17, 2025

/retest

nirs added 2 commits August 19, 2025 00:02
The package provides functions for parsing and validating mount string
and setting up the mount inside the guest.
Replace 9p mounts with virtiofs for vfkit and krunkit. Testing shows
that virtiofs mount is 23 times faster with krunkit, and 8 times faster
with vfkit.

vfkit and krunkit support multiple virtiofs mounts but minikube
--mount-* flags are not ready for multiple mounts. We have the same
issue with KIC drivers, supporting multiple mounts but using only one.
We hope to improve this in the next release.

Example usage:

    minikube start --mount-string ~/models:/mnt/models

The arguments are parsed and validated when configuring the driver, so
invalid arguments fail quickly without starting the driver. The
validated mounts are stored in the machine config:

    $ jq '.Driver.VirtiofsMounts' < ~/.minikube/machines/minikube/config.json
    [
      {
        "HostPath": "/Users/joe/models",
        "GuestPath": "/mnt/models",
        "Tag": "f845b54d-00e3-493d-9541-3b37490b96db"
      }
    ]

Minikube generates a new random UUID for every virtiofs mount to
identify the file system inside the guest. In krunkit and vfkit, every
mount is add as:

    --device virtio-fs,sharedDir=/host-path,mountTag=f845b54d-00e3-493d-9541-3b37490b96db

When the guest is started the shared directory is mounted via SSH using:

    sudo mkdir -p /mnt/models
    sudo mount -t virtiofs f845b54d-00e3-493d-9541-3b37490b96db /mnt/models

Example mount:

    $ minikube ssh findmnt /mnt/models
    TARGET      SOURCE                               FSTYPE   OPTIONS
    /mnt/models f845b54d-00e3-493d-9541-3b37490b96db virtiofs rw,relatime

More work is needed to add VirtioFS to qemu. I'm starting with vfkit and
krunkit since they have identical interface (krunkit was designed as
drop-in replacement for vfkit).
@nirs nirs force-pushed the drivers-virtiofs branch from ca9eb87 to 21dcbef Compare August 18, 2025 21:02
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 18, 2025
@nirs nirs marked this pull request as ready for review August 18, 2025 21:02
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 18, 2025
@medyagh medyagh changed the title drivers: Add Virtiofs mounts for vfkit and krunkit drivers: Add support for Virtiofs mounts for vfkit and krunkit Aug 18, 2025
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21149 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 50.9s    │ 50.3s                  │
│ enable ingress │ 17.1s    │ 16.2s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube ingress: 19.1s 15.0s 16.0s 19.5s 16.0s
Times for minikube (PR 21149) ingress: 15.6s 15.0s 16.0s 15.5s 19.1s

Times for minikube start: 48.9s 52.3s 54.6s 47.9s 51.0s
Times for minikube (PR 21149) start: 50.5s 51.8s 49.8s 49.3s 50.1s

docker driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21149 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 24.3s    │ 23.7s                  │
│ enable ingress │ 14.4s    │ 13.1s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube ingress: 13.8s 13.8s 13.3s 13.8s 17.3s
Times for minikube (PR 21149) ingress: 12.3s 13.8s 12.3s 13.8s 13.3s

Times for minikube start: 25.2s 26.9s 23.8s 22.6s 23.2s
Times for minikube (PR 21149) start: 23.5s 23.6s 24.6s 21.4s 25.6s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21149 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 22.7s    │ 21.9s                  │
│ enable ingress │ 32.9s    │ 23.2s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 21.1s 23.3s 24.6s 21.0s 23.6s
Times for minikube (PR 21149) start: 21.6s 21.9s 20.8s 21.0s 24.1s

Times for minikube ingress: 39.3s 22.8s 39.3s 23.8s 39.3s
Times for minikube (PR 21149) ingress: 23.8s 22.7s 23.8s 22.8s 22.8s

Copy link
Member

@medyagh medyagh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @nirs this will be an important part of the user experience for the AI develoeprs

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, nirs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 21, 2025
@medyagh medyagh merged commit 96229ad into kubernetes:master Aug 21, 2025
35 of 45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/mount cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants