[Bug]: "pooling_type='ALL' no longer supported for embeddings

[jupyterjazz]

Your current environment

The output of python collect_env.py

==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0
Clang version                : Could not collect
CMake version                : version 3.27.2
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.8.0+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.11.5 (main, Sep 11 2023, 13:54:46) [GCC 11.2.0] (64-bit runtime)
Python platform              : Linux-5.15.0-143-generic-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 11.5.119
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration : 
GPU 0: NVIDIA A100 80GB PCIe
GPU 1: NVIDIA A100 80GB PCIe
GPU 2: NVIDIA A100 80GB PCIe
GPU 3: NVIDIA A100 80GB PCIe
GPU 4: NVIDIA A100 80GB PCIe
GPU 5: NVIDIA A100 80GB PCIe
GPU 6: NVIDIA A100 80GB PCIe
GPU 7: NVIDIA A100 80GB PCIe

Nvidia driver version        : 535.161.07
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        48 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               128
On-line CPU(s) list:                  0-127
Vendor ID:                            AuthenticAMD
Model name:                           AMD EPYC 7543 32-Core Processor
CPU family:                           25
Model:                                1
Thread(s) per core:                   2
Core(s) per socket:                   32
Socket(s):                            2
Stepping:                             1
Frequency boost:                      enabled
CPU max MHz:                          2800.0000
CPU min MHz:                          1500.0000
BogoMIPS:                             5589.39
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm
Virtualization:                       AMD-V
L1d cache:                            2 MiB (64 instances)
L1i cache:                            2 MiB (64 instances)
L2 cache:                             32 MiB (64 instances)
L3 cache:                             512 MiB (16 instances)
NUMA node(s):                         2
NUMA node0 CPU(s):                    0-31,64-95
NUMA node1 CPU(s):                    32-63,96-127
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Mitigation; safe RET
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-nccl-cu12==2.27.3
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pyzmq==27.1.0
[pip3] torch==2.8.0
[pip3] torchaudio==2.8.0
[pip3] torchvision==0.23.0
[pip3] transformers==4.56.1
[pip3] triton==3.4.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-cublas-cu11        11.10.3.66               pypi_0    pypi
[conda] nvidia-cublas-cu12        12.6.4.1                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu11    11.7.101                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.6.80                  pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu11    11.7.99                  pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.6.77                  pypi_0    pypi
[conda] nvidia-cuda-runtime-cu11  11.7.99                  pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.6.77                  pypi_0    pypi
[conda] nvidia-cudnn-cu11         8.5.0.96                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.5.1.17                 pypi_0    pypi
[conda] nvidia-cufft-cu11         10.9.0.58                pypi_0    pypi
[conda] nvidia-cufft-cu12         11.3.0.4                 pypi_0    pypi
[conda] nvidia-cufile-cu12        1.11.1.6                 pypi_0    pypi
[conda] nvidia-curand-cu11        10.2.10.91               pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.7.77                pypi_0    pypi
[conda] nvidia-cusolver-cu11      11.4.0.1                 pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.7.1.2                 pypi_0    pypi
[conda] nvidia-cusparse-cu11      11.7.4.91                pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.5.4.2                 pypi_0    pypi
[conda] nvidia-cusparselt-cu12    0.6.3                    pypi_0    pypi
[conda] nvidia-nccl-cu11          2.14.3                   pypi_0    pypi
[conda] nvidia-nccl-cu12          2.26.2                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.6.85                  pypi_0    pypi
[conda] nvidia-nvtx-cu11          11.7.91                  pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.6.77                  pypi_0    pypi
[conda] pynvml                    11.5.3                   pypi_0    pypi
[conda] pytorch-lightning         2.4.0                    pypi_0    pypi
[conda] pytorch-metric-learning   2.2.0                    pypi_0    pypi
[conda] pyzmq                     26.2.1                   pypi_0    pypi
[conda] sentence-transformers     5.1.0                    pypi_0    pypi
[conda] torch                     2.7.0                    pypi_0    pypi
[conda] torch-vision              0.1.6.dev0               pypi_0    pypi
[conda] torchaudio                2.7.0                    pypi_0    pypi
[conda] torchmetrics              1.4.1                    pypi_0    pypi
[conda] torchvision               0.22.0                   pypi_0    pypi
[conda] transformers              4.52.0                   pypi_0    pypi
[conda] triton                    3.3.0                    pypi_0    pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.10.2
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV12    NODE    NODE    SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU1    NV12     X      NODE    NODE    SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU2    NODE    NODE     X      NODE    SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU3    NODE    NODE    NODE     X      SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU4    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE    32-63,96-127    1               N/A
GPU5    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE    32-63,96-127    1               N/A
GPU6    SYS     SYS     SYS     SYS     NODE    NODE     X      NODE    32-63,96-127    1               N/A
GPU7    SYS     SYS     SYS     SYS     NODE    NODE    NODE     X      32-63,96-127    1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
CUDA_VISIBLE_DEVICES=1
CUDA_VISIBLE_DEVICES=1
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

When attempting to get embeddings from jinaai/jina-embeddings-v4-vllm-text-matching (based on the Qwen2_5_VLForConditionalGeneration architecture), I encounter the following error:

(EngineCore_DP0 pid=1671528)   File "/home/admin/saba/vllm-bug/venv/lib/python3.11/site-packages/vllm/model_executor/model_loader/base_loader.py", line 45, in load_model
(EngineCore_DP0 pid=1671528)     model = initialize_model(vllm_config=vllm_config,
(EngineCore_DP0 pid=1671528)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1671528)   File "/home/admin/saba/vllm-bug/venv/lib/python3.11/site-packages/vllm/model_executor/model_loader/utils.py", line 64, in initialize_model
(EngineCore_DP0 pid=1671528)     return model_class(vllm_config=vllm_config, prefix=prefix)
(EngineCore_DP0 pid=1671528)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1671528)   File "/home/admin/saba/vllm-bug/venv/lib/python3.11/site-packages/vllm/model_executor/models/adapters.py", line 163, in __init__
(EngineCore_DP0 pid=1671528)     super().__init__(vllm_config=vllm_config, prefix=prefix, **kwargs)
(EngineCore_DP0 pid=1671528)   File "/home/admin/saba/vllm-bug/venv/lib/python3.11/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 943, in __init__
(EngineCore_DP0 pid=1671528)     self.language_model = init_vllm_registered_model(
(EngineCore_DP0 pid=1671528)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1671528)   File "/home/admin/saba/vllm-bug/venv/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 316, in init_vllm_registered_model
(EngineCore_DP0 pid=1671528)     return initialize_model(vllm_config=vllm_config, prefix=prefix)
(EngineCore_DP0 pid=1671528)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1671528)   File "/home/admin/saba/vllm-bug/venv/lib/python3.11/site-packages/vllm/model_executor/model_loader/utils.py", line 64, in initialize_model
(EngineCore_DP0 pid=1671528)     return model_class(vllm_config=vllm_config, prefix=prefix)
(EngineCore_DP0 pid=1671528)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1671528)   File "/home/admin/saba/vllm-bug/venv/lib/python3.11/site-packages/vllm/model_executor/models/adapters.py", line 196, in __init__
(EngineCore_DP0 pid=1671528)     self._init_pooler(vllm_config, prefix=prefix)
(EngineCore_DP0 pid=1671528)   File "/home/admin/saba/vllm-bug/venv/lib/python3.11/site-packages/vllm/model_executor/models/adapters.py", line 260, in _init_pooler
(EngineCore_DP0 pid=1671528)     self.pooler = DispatchPooler(
(EngineCore_DP0 pid=1671528)                   ^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1671528)   File "/home/admin/saba/vllm-bug/venv/lib/python3.11/site-packages/vllm/model_executor/layers/pooler.py", line 696, in __init__
(EngineCore_DP0 pid=1671528)     raise ValueError(
(EngineCore_DP0 pid=1671528) ValueError: pooler=SimplePooler(
(EngineCore_DP0 pid=1671528)   (pooling): AllPool()
(EngineCore_DP0 pid=1671528)   (head): EmbeddingPoolerHead(
(EngineCore_DP0 pid=1671528)     (activation): PoolerNormalize()
(EngineCore_DP0 pid=1671528)   )
(EngineCore_DP0 pid=1671528) ) does not support task='embed'. Supported tasks: {'encode'}

This happens when I override pooling_type to "ALL". This is necessary because the model has a custom pooling method, and I need all last hidden states to do pooling afterwards. (See: model usage example.)

This same code worked in previous versions of vLLM, so this seems to be a recent change.

Code to reproduce:

from vllm import LLM
from vllm.config import PoolerConfig

model = LLM(
    model="jinaai/jina-embeddings-v4-vllm-text-matching",
    task="embed",  # same error with "embedding" and "encode"
    override_pooler_config=PoolerConfig(pooling_type="ALL"),
)
texts = ["Test Test"]
outputs = model.encode(texts, pooling_task="embed")  # same error with model.embed(texts)

Is it possible to support "ALL" pooling_type for models with custom pooling methods? If not, is there a recommended workaround?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Shiv-Expert2503]

Hu @jupyterjazz

The problem is that a recent update to vLLM added a new, stricter rule. This rule now says: "If you're doing an 'embed' task, you are not allowed to use pooling_type='ALL'." This new check is what's causing the ValueError.

I can take this on.

[DarkLight1337]

Can you simply use task="encode" when initializing LLM and correspondingly pooling_task="encode" when calling the model?

[DarkLight1337]

cc @noooop

[noooop]

We are moving all pooling logic to the encode task API (online /pooling API).

You don’t need to set task="embed"; it should have been deprecated in the latest version.

try:
outputs = model.encode(texts, pooling_task="encode")

we have ensured that all pooling models can always use all pooling，You don’t need to manually set all pooling

[jupyterjazz]

Hi, thanks for the quick response.

we have ensured that all pooling models can always use all pooling

I guess the issue is that Qwen2_5_VLForConditionalGeneration is not a pooling model but we want to use it for embeddings. If I just load the model, the runner type "generate" gets chosen. And if I set runner="pooling" I get the same error I wrote above.

[DarkLight1337]

Can you try setting task="reward"?

[noooop]

Hi, thanks for the quick response.

we have ensured that all pooling models can always use all pooling

I guess the issue is that Qwen2_5_VLForConditionalGeneration is not a pooling model but we want to use it for embeddings. If I just load the model, the runner type "generate" gets chosen. And if I set runner="pooling" I get the same error I wrote above.

Mark

There are still a lot of issues with multimodal models turn into pooling models, and we’re working on fixing them.

[noooop]

According to VLLM's taxonomy,

• one request that only output one result is split into embed, classify, score...
• All other requests that output one or multiple outputs are stuffed into the encode API... This is an all-purpose API. In fact, all other APIs are completed by calling this API.

As vllm supports more and more models and usage scenarios. Many recent issues have forced us to rethink how to handle "output multiple outputs" usage scenarios.

First, we hope that pooling models can always use all pooling, and users don’t need to manually set using all pooling.

But now it is found that encode API (Ignore one request that only outputs one result for now, after all, it is an all-purpose API) has multiple usage scenarios

• embed for each token (multi-vector retrieval) <- Currently, this scenario is not well supported.
• classify for each token (TokenClassification & reward models) <- Currently, mainly supports this scenario
• This model is a pooling model, outputs something, but it is neither embed nor classify

Maybe we need to further split it up and design a suitable API for each scenario

Chaos, right? That's the situation.

Perhaps we should give up our hope that pooling models can always use all pooling. Users can only choose to use one pooling type. But this will prevent a lot of cool things from being done

There's a lot more to discuss here............................

---

ultimate question

what kind of API do you want to use?

from vllm import LLM
from vllm.config import PoolerConfig

model = LLM(
    model="jinaai/jina-embeddings-v4-vllm-text-matching",
    task="embed",  
    override_pooler_config=PoolerConfig(pooling_type="ALL"),
)
texts = ["Test Test"]
outputs = model.embed(texts) 

from vllm import LLM
from vllm.config import PoolerConfig

model = LLM(
    model="jinaai/jina-embeddings-v4-vllm-text-matching",
    runner="pooling"
)
texts = ["Test Test"]
outputs = model.encode(texts, pooling_task="encode") # <- Softmax is used by default here now, so we need to find a way to turn it off. 

[xsank]

@noooop I have a question, why use the task name encode instead of pooling?

[noooop]

@noooop I have a question, why use the task name encode instead of pooling?

In fact, @DarkLight1337 and I have discussed this many times, the most recent one being

#21720 (comment)

Because we are not clear about the usage scenarios of all pooling and encode api, we keep it unchanged. (So gradually all unclear usage scenarios use the encode API, I think it's time to refactor.

[noooop]

Another detail is that chunked prefill does not support all pooling (and mean pooling).

#20930 makes the LAST Pooling model use chunked prefill by default and disables the encode API.

(We auto disable chunked prefill if it uses all pooling (and mean pooling)).

from vllm import LLM
from vllm.config import PoolerConfig

model = LLM(
    model="jinaai/jina-embeddings-v4-vllm-text-matching",
    enable_chunked_prefill=False,  # <- In order to use the encode api
    runner="pooling"
)
texts = ["Test Test"]
outputs = model.encode(texts, pooling_task="encode") #<- Softmax is used by default here now, so we need to find a way to turn it off.

This caused us a lot of trouble PTAL #24853

Maybe we should find a way to support chunked prefill + all pooling (and mean pooling))

---

These will not bother ordinary users, but many of the people who use vllm are advanced users. LOL

---

cc @maxdebayser

Do you have time to implement chunked prefill + all pooling? Because one request becomes multiple part outputs, which is definitely not a simple modification.