[BUG] CharacterNormalizer make special token lower case #20236
Description
Describe the bug
After removal of SubwordTokenizer in 25.10, we switch to CharacterNormalizer + WordPieceVocabulary to do GPU tokenization. When do_lower_case=True is specified, even if special tokens are passed in, CharacterNormalizer will also lower case the special tokens, hence it cannot be recoginzed in subsequent tokenization steps as the special tokens are upper case in the vocab file (i.e. bert-uncased: https://huggingface.co/google-bert/bert-base-uncased/blob/main/vocab.txt)
Steps/Code to reproduce bug
from cudf.core.character_normalizer import CharacterNormalizer
import cudf
str_series = cudf.Series(['Hello World [SEP] Foo Bar'])
special = cudf.Series(["[BOS]", "[EOS]", "[SEP]", "[PAD]"])
normalizer = CharacterNormalizer(do_lower=True, special_tokens=special)
norm = normalizer.normalize(str_series.str)
print(norm[0])
Output is
hello world [sep] foo bar
Expected behavior
For special tokens, it should not be converted to lower case in the CharacterNormalizer output.
It should be
hello world [SEP] foo bar
Environment overview (please complete the following information)
cudf==25.10
Python:3.12.11
Driver Version: 535.104.05
CUDA Version: 12.2
It should be easily reproduced in other python/cuda version
Environment details
Click here to see environment details
**git*** Not inside a git repository ***OS Information*** DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.5 LTS" PRETTY_NAME="Ubuntu 22.04.5 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.5 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy Linux xxx 5.15.0-26-generic #26 SMP Fri Jan 17 02:37:14 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux ***GPU Information*** Sun Oct 12 02:25:56 2025 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA H100 80GB HBM3 Off | 00000000:5D:00.0 Off | 0 | | N/A 34C P0 118W / 700W | 1105MiB / 81559MiB | 0% Default | | | | Disabled | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| +---------------------------------------------------------------------------------------+ ***CPU*** Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 224 On-line CPU(s) list: 0-223 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Platinum 8480+ CPU family: 6 Model: 143 Thread(s) per core: 2 Core(s) per socket: 56 Socket(s): 2 Stepping: 8 Frequency boost: enabled CPU max MHz: 2001.0000 CPU min MHz: 800.0000 BogoMIPS: 4000.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities Virtualization: VT-x L1d cache: 5.3 MiB (112 instances) L1i cache: 3.5 MiB (112 instances) L2 cache: 224 MiB (112 instances) L3 cache: 210 MiB (2 instances) NUMA node(s): 4 NUMA node0 CPU(s): 0-27,112-139 NUMA node1 CPU(s): 28-55,140-167 NUMA node2 CPU(s): 56-83,168-195 NUMA node3 CPU(s): 84-111,196-223 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ***CMake*** /usr/bin/cmake cmake version 3.22.1 CMake suite maintained and supported by Kitware (kitware.com/cmake). ***g++*** /usr/bin/g++ g++ (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0 Copyright (C) 2021 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ***nvcc*** /usr/local/cuda/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2024 NVIDIA Corporation Built on Thu_Mar_28_02:18:24_PDT_2024 Cuda compilation tools, release 12.4, V12.4.131 Build cuda_12.4.r12.4/compiler.34097967_0 ***Python*** /opt/conda/bin/python Python 3.12.11 ***Environment Variables*** PATH : /apache/hadoop/bin:/apache/hbase/bin:/apache/pig/bin:/apache/hive/bin:/opt/conda/bin:/apache/hadoop/bin:/apache/spark/bin:/apache/hive/bin:/apache/hbase/bin:/apache/pig/bin:/usr/jdk/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/nvidia/bin:/opt/conda/bin:/opt/clients/latest/linux/amd64/ LD_LIBRARY_PATH : /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/nvidia/lib64 NUMBAPRO_NVVM : NUMBAPRO_LIBDEVICE : CONDA_PREFIX : PYTHON_PATH : ***conda packages*** /opt/conda/bin/conda # packages in environment at /opt/conda: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge absl-py 2.3.1 pypi_0 pypi accelerate 1.10.1 pypi_0 pypi aiofiles 24.1.0 pypi_0 pypi aiohappyeyeballs 2.6.1 pypi_0 pypi aiohttp 3.12.15 pypi_0 pypi aiosignal 1.4.0 pypi_0 pypi alembic 1.16.5 pypi_0 pypi annotated-types 0.7.0 pypi_0 pypi ansicolors 1.1.8 pypi_0 pypi antlr4-python3-runtime 4.8 pypi_0 pypi anyio 4.11.0 pypi_0 pypi appdirs 1.4.4 pypi_0 pypi archspec 0.2.5 pyhd8ed1ab_0 conda-forge argon2-cffi 25.1.0 pypi_0 pypi argon2-cffi-bindings 25.1.0 pypi_0 pypi arrow 1.3.0 pypi_0 pypi astor 0.8.1 pypi_0 pypi astroid 3.3.11 pypi_0 pypi asttokens 3.0.0 pypi_0 pypi async-lru 2.0.5 pypi_0 pypi attrs 25.3.0 pypi_0 pypi babel 2.17.0 pypi_0 pypi bcrypt 5.0.0 pypi_0 pypi beautifulsoup4 4.14.2 pypi_0 pypi benchmarker 0.1.0 pypi_0 pypi bidict 0.23.1 pypi_0 pypi bitsandbytes 0.48.1 pypi_0 pypi blake3 1.0.7 pypi_0 pypi bleach 6.2.0 pypi_0 pypi blinker 1.9.0 pypi_0 pypi blis 1.2.1 pypi_0 pypi boltons 25.0.0 pyhd8ed1ab_0 conda-forge boto3 1.40.45 pypi_0 pypi botocore 1.40.45 pypi_0 pypi brewer2mpl 1.4.1 pypi_0 pypi brotli-python 1.1.0 py312h2ec8cdc_3 conda-forge bzip2 1.0.8 h4bc722e_7 conda-forge c-ares 1.34.5 hb9d3cd8_0 conda-forge ca-certificates 2025.10.5 hbd8a1cb_0 conda-forge cachetools 6.2.0 pypi_0 pypi catalogue 2.0.10 pypi_0 pypi cbor2 5.7.0 pypi_0 pypi certifi 2025.10.5 pyhd8ed1ab_0 conda-forge cffi 2.0.0 pypi_0 pypi chardet 5.2.0 pypi_0 pypi charset-normalizer 3.4.2 pyhd8ed1ab_0 conda-forge click 8.3.0 pypi_0 pypi cloudpathlib 0.22.0 pypi_0 pypi cloudpickle 3.1.1 pypi_0 pypi colorama 0.4.6 pyhd8ed1ab_1 conda-forge coloredlogs 15.0.1 pypi_0 pypi colorlog 6.9.0 pypi_0 pypi comm 0.2.3 pypi_0 pypi compressed-tensors 0.10.2 pypi_0 pypi conda 25.7.0 py312h7900ff3_0 conda-forge conda-libmamba-solver 25.3.0 pyhd8ed1ab_0 conda-forge conda-package-handling 2.4.0 pyh7900ff3_2 conda-forge conda-package-streaming 0.12.0 pyhd8ed1ab_0 conda-forge confection 0.1.5 pypi_0 pypi configargparse 1.7.1 pypi_0 pypi contourpy 1.3.3 pypi_0 pypi cpp-expected 1.1.0 hff21bea_1 conda-forge cryptography 46.0.2 pypi_0 pypi cuda-bindings 12.9.2 pypi_0 pypi cuda-pathfinder 1.3.0 pypi_0 pypi cuda-python 12.9.0 pypi_0 pypi cupy-cuda12x 13.6.0 pypi_0 pypi cycler 0.12.1 pypi_0 pypi cymem 2.0.11 pypi_0 pypi dataproperty 1.1.0 pypi_0 pypi datasets 4.1.1 pypi_0 pypi debugpy 1.8.17 pypi_0 pypi decorator 5.2.1 pypi_0 pypi deepspeed 0.17.6 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi depyf 0.19.0 pypi_0 pypi dill 0.4.0 pypi_0 pypi diskcache 5.6.3 pypi_0 pypi distro 1.9.0 pyhd8ed1ab_1 conda-forge dnspython 2.8.0 pypi_0 pypi docopt 0.6.2 pypi_0 pypi einops 0.8.1 pypi_0 pypi ellement 0.2.36 pypi_0 pypi email-validator 2.3.0 pypi_0 pypi et-xmlfile 2.0.0 pypi_0 pypi evaluate 0.4.6 pypi_0 pypi executing 2.2.1 pypi_0 pypi fastapi 0.118.0 pypi_0 pypi fastapi-cli 0.0.13 pypi_0 pypi fastapi-cloud-cli 0.3.0 pypi_0 pypi fastjsonschema 2.21.2 pypi_0 pypi fastrlock 0.8.3 pypi_0 pypi ffmpy 0.6.1 pypi_0 pypi filelock 3.19.1 pypi_0 pypi flash-attn 2.7.4.post1 pypi_0 pypi flashinfer-python 0.2.11 pypi_0 pypi flask 3.1.2 pypi_0 pypi flask-cors 6.0.1 pypi_0 pypi flask-login 0.6.3 pypi_0 pypi flatbuffers 25.9.23 pypi_0 pypi fluent-logger 0.11.1 pypi_0 pypi fmt 11.1.4 h07f6e7f_1 conda-forge fonttools 4.60.1 pypi_0 pypi fqdn 1.5.1 pypi_0 pypi frozendict 2.4.6 py312h66e93f0_0 conda-forge frozenlist 1.7.0 pypi_0 pypi fsspec 2025.9.0 pypi_0 pypi future 1.0.0 pypi_0 pypi gevent 25.5.1 pypi_0 pypi geventhttpclient 2.3.4 pypi_0 pypi ggplot 0.11.5 pypi_0 pypi gguf 0.17.1 pypi_0 pypi gitdb 4.0.12 pypi_0 pypi gitpython 3.1.45 pypi_0 pypi gradio 5.49.0 pypi_0 pypi gradio-client 1.13.3 pypi_0 pypi greenlet 3.2.4 pypi_0 pypi groovy 0.1.2 pypi_0 pypi grpcio 1.67.1 pypi_0 pypi grpcio-health-checking 1.62.3 pypi_0 pypi guidance 0.3.0 pypi_0 pypi guidance-stitch 0.1.5 pypi_0 pypi gunicorn 23.0.0 pypi_0 pypi h11 0.16.0 pypi_0 pypi h2 4.2.0 pyhd8ed1ab_0 conda-forge hf-xet 1.1.10 pypi_0 pypi hjson 3.1.0 pypi_0 pypi hnswlib 0.8.0 pypi_0 pypi hpack 4.1.0 pyhd8ed1ab_0 conda-forge httpcore 1.0.9 pypi_0 pypi httptools 0.6.4 pypi_0 pypi httpx 0.28.1 pypi_0 pypi huggingface-hub 0.35.3 pypi_0 pypi humanfriendly 10.0 pypi_0 pypi hyperframe 6.1.0 pyhd8ed1ab_0 conda-forge icu 75.1 he02047a_0 conda-forge idna 3.10 pyhd8ed1ab_1 conda-forge immutabledict 4.2.1 pypi_0 pypi importlib-metadata 8.7.0 pypi_0 pypi iniconfig 2.1.0 pypi_0 pypi interegular 0.3.3 pypi_0 pypi invoke 2.2.0 pypi_0 pypi ipykernel 6.30.1 pypi_0 pypi ipython 9.6.0 pypi_0 pypi ipython-pygments-lexers 1.1.1 pypi_0 pypi ipywidgets 8.1.7 pypi_0 pypi isoduration 20.11.0 pypi_0 pypi isort 6.1.0 pypi_0 pypi itsdangerous 2.2.0 pypi_0 pypi jedi 0.19.2 pypi_0 pypi jinja2 3.1.6 pypi_0 pypi jiter 0.11.0 pypi_0 pypi jmespath 1.0.1 pypi_0 pypi joblib 1.5.2 pypi_0 pypi json5 0.12.1 pypi_0 pypi jsonpatch 1.33 pyhd8ed1ab_1 conda-forge jsonpointer 3.0.0 py312h7900ff3_1 conda-forge jsonschema 4.25.1 pypi_0 pypi jsonschema-specifications 2025.9.1 pypi_0 pypi jupyter-client 8.6.3 pypi_0 pypi jupyter-core 5.8.1 pypi_0 pypi jupyter-events 0.12.0 pypi_0 pypi jupyter-lsp 2.3.0 pypi_0 pypi jupyter-server 2.17.0 pypi_0 pypi jupyter-server-proxy 4.4.0 pypi_0 pypi jupyter-server-terminals 0.5.3 pypi_0 pypi jupyterlab 4.4.9 pypi_0 pypi jupyterlab-pygments 0.3.0 pypi_0 pypi jupyterlab-server 2.27.3 pypi_0 pypi jupyterlab-widgets 3.0.15 pypi_0 pypi kazoo 2.10.0 pypi_0 pypi keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.9 pypi_0 pypi krb5 1.21.3 h659f571_0 conda-forge langchain 0.3.27 pypi_0 pypi langchain-core 0.3.78 pypi_0 pypi langchain-text-splitters 0.3.11 pypi_0 pypi langcodes 3.5.0 pypi_0 pypi langdetect 1.0.9 pypi_0 pypi langsmith 0.4.32 pypi_0 pypi language-data 1.3.0 pypi_0 pypi lark 1.2.2 pypi_0 pypi ld_impl_linux-64 2.44 h1423503_1 conda-forge libarchive 3.7.7 h75ea233_4 conda-forge libcurl 8.14.1 h332b0f4_0 conda-forge libedit 3.1.20250104 pl5321h7949ede_0 conda-forge libev 4.33 hd590300_2 conda-forge libexpat 2.7.1 hecca717_0 conda-forge libffi 3.4.6 h2dba641_1 conda-forge libgcc 15.1.0 h767d61c_3 conda-forge libgcc-ng 15.1.0 h69a702a_3 conda-forge libgomp 15.1.0 h767d61c_3 conda-forge libiconv 1.18 h4ce23a2_1 conda-forge liblzma 5.8.1 hb9d3cd8_2 conda-forge libmamba 2.1.1 h430c389_0 conda-forge libmambapy 2.1.1 py312h07448e0_0 conda-forge libnghttp2 1.64.0 h161d5f1_0 conda-forge libnsl 2.0.1 hb9d3cd8_1 conda-forge libsolv 0.7.34 h9463b59_0 conda-forge libsqlite 3.50.3 hee844dc_1 conda-forge libssh2 1.11.1 hcf80075_0 conda-forge libstdcxx 15.1.0 h8f9b012_3 conda-forge libstdcxx-ng 15.1.0 h4852527_3 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libxcrypt 4.4.36 hd590300_1 conda-forge libxml2 2.13.8 h4bc477f_0 conda-forge libzlib 1.3.1 hb9d3cd8_2 conda-forge llguidance 0.7.30 pypi_0 pypi llvmlite 0.44.0 pypi_0 pypi lm-format-enforcer 0.10.12 pypi_0 pypi locust 2.41.3 pypi_0 pypi locust-cloud 1.27.2 pypi_0 pypi loguru 0.7.3 pypi_0 pypi loralib 0.1.2 pypi_0 pypi lxml 6.0.2 pypi_0 pypi lxml-html-clean 0.4.3 pypi_0 pypi lz4-c 1.10.0 h5888daf_1 conda-forge lzo 2.10 hd590300_1001 conda-forge mako 1.3.10 pypi_0 pypi mamba 2.1.1 had4a41a_0 conda-forge marisa-trie 1.3.1 pypi_0 pypi markdown 3.9 pypi_0 pypi markdown-it-py 4.0.0 pypi_0 pypi markupsafe 3.0.3 pypi_0 pypi matplotlib 3.10.6 pypi_0 pypi matplotlib-inline 0.1.7 pypi_0 pypi mbstrdecoder 1.1.4 pypi_0 pypi mccabe 0.7.0 pypi_0 pypi mdurl 0.1.2 pypi_0 pypi menuinst 2.3.1 py312h7900ff3_0 conda-forge mistral-common 1.8.5 pypi_0 pypi mistune 3.1.4 pypi_0 pypi ml-dtypes 0.5.3 pypi_0 pypi mosestokenizer 1.2.1 pypi_0 pypi mpmath 1.3.0 pypi_0 pypi msgpack 1.1.1 pypi_0 pypi msgspec 0.19.0 pypi_0 pypi multidict 6.6.4 pypi_0 pypi multiprocess 0.70.16 pypi_0 pypi murmurhash 1.0.13 pypi_0 pypi narwhals 2.6.0 pypi_0 pypi nbclient 0.10.2 pypi_0 pypi nbconvert 7.16.6 pypi_0 pypi nbformat 5.10.4 pypi_0 pypi ncurses 6.5 h2d0b736_3 conda-forge nest-asyncio 1.6.0 pypi_0 pypi networkx 3.5 pypi_0 pypi ninja 1.13.0 pypi_0 pypi nlohmann_json 3.11.3 he02047a_1 conda-forge nltk 3.9.2 pypi_0 pypi notebook 7.4.7 pypi_0 pypi notebook-shim 0.2.4 pypi_0 pypi numba 0.61.2 pypi_0 pypi numpy 2.2.6 pypi_0 pypi nvidia-cublas-cu12 12.6.4.1 pypi_0 pypi nvidia-cuda-cupti-cu12 12.6.80 pypi_0 pypi nvidia-cuda-nvrtc-cu12 12.6.77 pypi_0 pypi nvidia-cuda-runtime-cu12 12.6.77 pypi_0 pypi nvidia-cudnn-cu12 9.5.1.17 pypi_0 pypi nvidia-cudnn-frontend 1.14.1 pypi_0 pypi nvidia-cufft-cu12 11.3.0.4 pypi_0 pypi nvidia-cufile-cu12 1.11.1.6 pypi_0 pypi nvidia-curand-cu12 10.3.7.77 pypi_0 pypi nvidia-cusolver-cu12 11.7.1.2 pypi_0 pypi nvidia-cusparse-cu12 12.5.4.2 pypi_0 pypi nvidia-cusparselt-cu12 0.6.3 pypi_0 pypi nvidia-ml-py 13.580.82 pypi_0 pypi nvidia-nccl-cu12 2.26.2 pypi_0 pypi nvidia-nvjitlink-cu12 12.6.85 pypi_0 pypi nvidia-nvtx-cu12 12.6.77 pypi_0 pypi omegaconf 2.2.0 pypi_0 pypi onnx 1.19.0 pypi_0 pypi onnxruntime-gpu 1.23.0 pypi_0 pypi openai 2.1.0 pypi_0 pypi openai-harmony 0.0.4 pypi_0 pypi opencv-python-headless 4.12.0.88 pypi_0 pypi openfile 0.0.7 pypi_0 pypi openpyxl 3.1.5 pypi_0 pypi openssl 3.5.4 h26f9b46_0 conda-forge optimum 1.27.0 pypi_0 pypi optuna 4.5.0 pypi_0 pypi orjson 3.11.3 pypi_0 pypi outlines-core 0.2.10 pypi_0 pypi packaging 24.2 pypi_0 pypi pandas 2.3.3 pypi_0 pypi pandocfilters 1.5.1 pypi_0 pypi paramiko 4.0.0 pypi_0 pypi parso 0.8.5 pypi_0 pypi partial-json-parser 0.2.1.1.post6 pypi_0 pypi pathspec 0.12.1 pypi_0 pypi pathvalidate 3.3.1 pypi_0 pypi patsy 1.0.1 pypi_0 pypi peft 0.17.1 pypi_0 pypi pexpect 4.9.0 pypi_0 pypi pillow 11.3.0 pypi_0 pypi pip 25.2 pypi_0 pypi platformdirs 4.3.8 pyhe01879c_0 conda-forge plotly 6.3.1 pypi_0 pypi pluggy 1.6.0 pyhd8ed1ab_0 conda-forge portalocker 3.2.0 pypi_0 pypi preshed 3.0.10 pypi_0 pypi prometheus-client 0.23.1 pypi_0 pypi prometheus-fastapi-instrumentator 7.1.0 pypi_0 pypi prompt-toolkit 3.0.52 pypi_0 pypi propcache 0.4.0 pypi_0 pypi protobuf 5.29.5 pypi_0 pypi psutil 7.1.0 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi pure-eval 0.2.3 pypi_0 pypi py-cpuinfo 9.0.0 pypi_0 pypi py-grpc-prometheus 0.8.0 pypi_0 pypi py-spy 0.4.1 pypi_0 pypi py4j 0.10.9.9 pypi_0 pypi pyarrow 21.0.0 pypi_0 pypi pybase64 1.4.2 pypi_0 pypi pybind11-abi 4 hd8ed1ab_3 conda-forge pychomsky 0.3.4 pypi_0 pypi pycosat 0.6.6 py312h66e93f0_2 conda-forge pycountry 24.6.1 pypi_0 pypi pycparser 2.22 pyh29332c3_1 conda-forge pycryptodome 3.23.0 pypi_0 pypi pydantic 2.11.10 pypi_0 pypi pydantic-core 2.33.2 pypi_0 pypi pydantic-extra-types 2.10.5 pypi_0 pypi pydantic-settings 2.11.0 pypi_0 pypi pydub 0.25.1 pypi_0 pypi pyee 11.1.1 pypi_0 pypi pyflann 1.6.14 pypi_0 pypi pygments 2.19.2 pypi_0 pypi pyhive 0.7.0 pypi_0 pypi pylint 3.3.8 pypi_0 pypi pynacl 1.6.0 pypi_0 pypi pynvml 13.0.1 pypi_0 pypi pyodbc 5.2.0 pypi_0 pypi pyopenssl 25.3.0 pypi_0 pypi pyparsing 3.2.5 pypi_0 pypi pyppeteer 2.0.0 pypi_0 pypi pysocks 1.7.1 pyha55dd90_7 conda-forge pytablewriter 1.2.1 pypi_0 pypi pytest 8.4.2 pypi_0 pypi python 3.12.11 h9e4cc4f_0_cpython conda-forge python-dateutil 2.9.0.post0 pypi_0 pypi python-dotenv 1.1.1 pypi_0 pypi python-engineio 4.12.3 pypi_0 pypi python-json-logger 3.3.0 pypi_0 pypi python-multipart 0.0.20 pypi_0 pypi python-rapidjson 1.21 pypi_0 pypi python-socketio 5.13.0 pypi_0 pypi python_abi 3.12 8_cp312 conda-forge pytz 2025.2 pypi_0 pypi pyuip 0.1.11 pypi_0 pypi pyyaml 6.0.3 pypi_0 pypi pyzmq 27.1.0 pypi_0 pypi ray 2.49.2 pypi_0 pypi readline 8.2 h8c095d6_2 conda-forge referencing 0.36.2 pypi_0 pypi regex 2025.9.18 pypi_0 pypi reproc 14.2.5.post0 hb9d3cd8_0 conda-forge reproc-cpp 14.2.5.post0 h5888daf_0 conda-forge requests 2.32.4 pyhd8ed1ab_0 conda-forge requests-toolbelt 1.0.0 pypi_0 pypi rfc3339-validator 0.1.4 pypi_0 pypi rfc3986-validator 0.1.1 pypi_0 pypi rfc3987-syntax 1.1.0 pypi_0 pypi rich 14.1.0 pypi_0 pypi rich-toolkit 0.15.1 pypi_0 pypi rignore 0.7.0 pypi_0 pypi rouge-score 0.1.2 pypi_0 pypi rpds-py 0.27.1 pypi_0 pypi ruamel.yaml 0.18.14 py312h66e93f0_0 conda-forge ruamel.yaml.clib 0.2.8 py312h66e93f0_1 conda-forge ruff 0.13.3 pypi_0 pypi s3transfer 0.14.0 pypi_0 pypi sacrebleu 2.5.1 pypi_0 pypi safehttpx 0.1.6 pypi_0 pypi safetensors 0.6.2 pypi_0 pypi scikit-learn 1.7.2 pypi_0 pypi scipy 1.16.2 pypi_0 pypi seaborn 0.13.2 pypi_0 pypi semantic-version 2.10.0 pypi_0 pypi send2trash 1.8.3 pypi_0 pypi sentence-transformers 5.1.1 pypi_0 pypi sentencepiece 0.2.1 pypi_0 pypi sentry-sdk 2.39.0 pypi_0 pypi setproctitle 1.3.7 pypi_0 pypi setuptools 79.0.1 pypi_0 pypi setuptools-scm 9.2.0 pypi_0 pypi shellingham 1.5.4 pypi_0 pypi simdjson 3.12.3 h84d6215_0 conda-forge simpervisor 1.0.0 pypi_0 pypi simple-websocket 1.1.0 pypi_0 pypi six 1.17.0 pypi_0 pypi smart-open 7.3.1 pypi_0 pypi smmap 5.0.2 pypi_0 pypi sniffio 1.3.1 pypi_0 pypi soundfile 0.13.1 pypi_0 pypi soupsieve 2.8 pypi_0 pypi soxr 1.0.0 pypi_0 pypi spacy 3.8.7 pypi_0 pypi spacy-legacy 3.0.12 pypi_0 pypi spacy-loggers 1.0.5 pypi_0 pypi sqlalchemy 2.0.43 pypi_0 pypi sqlitedict 2.1.0 pypi_0 pypi srsly 2.5.1 pypi_0 pypi stack-data 0.6.3 pypi_0 pypi starlette 0.48.0 pypi_0 pypi statsmodels 0.14.5 pypi_0 pypi sympy 1.14.0 pypi_0 pypi tabledata 1.3.4 pypi_0 pypi tabulate 0.9.0 pypi_0 pypi tcolorpy 0.1.7 pypi_0 pypi tenacity 8.5.0 pypi_0 pypi tensorboard 2.20.0 pypi_0 pypi tensorboard-data-server 0.7.2 pypi_0 pypi tensorboardx 2.6.4 pypi_0 pypi terminado 0.18.1 pypi_0 pypi thinc 8.3.4 pypi_0 pypi threadpoolctl 3.6.0 pypi_0 pypi tiktoken 0.11.0 pypi_0 pypi tinycss2 1.4.0 pypi_0 pypi tk 8.6.13 noxft_hd72426e_102 conda-forge tokenizers 0.22.1 pypi_0 pypi tomlkit 0.13.3 pypi_0 pypi toolwrapper 2.1.0 pypi_0 pypi torch 2.7.1 pypi_0 pypi torch-hd 5.8.4 pypi_0 pypi torchaudio 2.7.1 pypi_0 pypi torchvision 0.22.1 pypi_0 pypi tornado 6.5.2 pypi_0 pypi tqdm 4.67.1 pyhd8ed1ab_1 conda-forge traitlets 5.14.3 pypi_0 pypi transformers 4.57.0 pypi_0 pypi triton 3.3.1 pypi_0 pypi tritonclient 2.60.0 pypi_0 pypi trl 0.23.1 pypi_0 pypi truststore 0.10.1 pyh29332c3_0 conda-forge twitter-common-dirutil 0.3.11 pypi_0 pypi twitter-common-lang 0.3.11 pypi_0 pypi twitter-common-log 0.3.11 pypi_0 pypi twitter-common-net 0.3.11 pypi_0 pypi twitter-common-options 0.3.11 pypi_0 pypi twitter-common-quantity 0.3.11 pypi_0 pypi typepy 1.3.4 pypi_0 pypi typer 0.19.2 pypi_0 pypi types-python-dateutil 2.9.0.20250822 pypi_0 pypi typing-extensions 4.15.0 pypi_0 pypi typing-inspection 0.4.2 pypi_0 pypi tzdata 2025.2 pypi_0 pypi uctools 1.3.0 pypi_0 pypi uip-toolkit 0.2.0 pypi_0 pypi unidecode 1.4.0 pypi_0 pypi uri-template 1.3.0 pypi_0 pypi urllib3 2.5.0 pyhd8ed1ab_0 conda-forge uvicorn 0.37.0 pypi_0 pypi uvloop 0.21.0 pypi_0 pypi vllm 0.10.1.1 pypi_0 pypi wandb 0.22.1 pypi_0 pypi wasabi 1.1.3 pypi_0 pypi watchfiles 1.1.0 pypi_0 pypi wcwidth 0.2.14 pypi_0 pypi weasel 0.4.1 pypi_0 pypi webcolors 24.11.1 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 1.8.0 pypi_0 pypi websockets 15.0.1 pypi_0 pypi werkzeug 3.1.3 pypi_0 pypi wheel 0.45.1 pyhd8ed1ab_1 conda-forge widgetsnbextension 4.0.14 pypi_0 pypi word2number 1.1 pypi_0 pypi wrapt 1.17.3 pypi_0 pypi wsproto 1.2.0 pypi_0 pypi xcmd 0.0.4 pypi_0 pypi xformers 0.0.31 pypi_0 pypi xgrammar 0.1.21 pypi_0 pypi xxhash 3.6.0 pypi_0 pypi yaml-cpp 0.8.0 h3f2d84a_0 conda-forge yarl 1.20.1 pypi_0 pypi zipp 3.23.0 pypi_0 pypi zk-shell 1.3.4 pypi_0 pypi zope-event 6.0 pypi_0 pypi zope-interface 8.0.1 pypi_0 pypi zstandard 0.23.0 py312h66e93f0_2 conda-forge zstd 1.5.7 hb8e6e7a_2 conda-forge
Additional context
Add any other context about the problem here.
Metadata
Metadata
Assignees
Type
Projects
Status
Status