[Feature]: improve GGUF loading from HuggingFace user experience

[ivanbaldo]

🚀 The feature, motivation and pitch

Something like vllm serve unsloth/Qwen3-4B-Instruct-2507-GGUF:Q4_K_XL should simply work, downloading the model from Huggingface automatically and caching it.

Or maybe specifying it as unsloth/Qwen3-4B-Instruct-2507-GGUF/Qwen3-4B-Instruct-2507-Q4_K_XL.gguf (although I think this is less preferred).

Alternatives

No response

Additional context

This is a followup to #20084 and #20793 which I personally think are incomplete solutions and not what users expect from other projects like Llama.cpp, Ollama, etc.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[yyzxw]

@ivanbaldo This means that the model name can end with just gguf, without having to start with http or https, right?

[simondanielsson]

@yyzxw If I am not mistaken, think it's two things:

1. Pure UX: enable <model>:<quant> syntax
2. Support downloading and caching the model directly from the hub, instead of having to wget and vllm serve
   

   vllm/vllm/transformers_utils/config.py

   Line 544 in 838d711

   " Loading GGUF from a remote repo directly is not yet "

---

On point 2: with the current version I cannot download and serve a GGUF model in one command: these all fail:

$ vllm serve unsloth/Qwen3-4B-Instruct-2507-GGUF/Qwen3-4B-Instruct-2507-Q4_K_M.gguf --hf-config-path unsloth/Qwen3-4B-Instruct-2507
...
(APIServer pid=6096)     raise OSError(
(APIServer pid=6096) OSError: Can't load the configuration of 'unsloth/Qwen3-4B-Instruct-2507-GGUF/Qwen3-4B-Instruct-2507-Q4_K_M.gguf'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'unsloth/Qwen3-4B-Instruct-2507-GGUF/Qwen3-4B-Instruct-2507-Q4_K_M.gguf' is the correct path to a directory containing a config.json file

$ vllm serve https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/resolve/main/Qwen3-4B-Instruct-2507-Q5_K_M.gguf --hf-config-path unsloth/Qwen3-4B-Instruct-2507
(APIServer pid=14238) OSError: It looks like the config file at '/var/tmp/tmpimtoj0jl' is not a valid JSON file.

$ vllm serve unsloth/Qwen3-4B-Instruct-2507-GGUF --hf-config-path unsloth/Qwen3-4B-Instruct-2507
(APIServer pid=15411) ValueError: Unrecognized model in unsloth/Qwen3-4B-Instruct-2507-GGUF. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: ...

However, as the vLLM docs suggest, running with a local model works:

$ vllm serve ~/.cache/huggingface/hub/models--unsloth--Qwen3-4B-Instruct-2507-GGUF/snapshots/a06e946bb6b655725eafa393f4a9745d460374c9/Qwen3-4B-Instruct-2507-Q4_K_M.gguf --hf-config-path unsloth/Qwen3-4B-Instruct-2507

I think we just need to combine the downloading and serving, and improve the UX.

Happy to take a stab at this., unless you are aware of any critical reason for or difficulty behind why this has not already been done 👍