model: add hunyuan dense

[stevenkuang-tencent]

Update:

• Support hunyuan_dense
• fix hunyuan_moe chat template

[CISC]

Just checking; is it using the same pre-tokenizer regex as the MoE?

[stevenkuang-tencent]

Just checking; is it using the same pre-tokenizer regex as the MoE?

There are two types of vocabulary in Hunyuan, regardless of whether it is moe or dense.

[CISC]

And are they all using the same regex, ie this one?

llama.cpp/src/llama-vocab.cpp

Lines 355 to 360 in 11dd5a4

	case LLAMA_VOCAB_PRE_TYPE_HUNYUAN:
	regex_exprs = {
	// original regex from tokenizer.json
	// "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
	"(?:'[sS]|'[tT]|'[rR][eE]|'[vV][eE]|'[mM]|'[lL][lL]|'[dD])|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
	};

[CISC]

It shouldn't be necessary to set this manually, a correctly configured model has this set in tokenizer_config.json and this will be picked up from there by gguf.SpecialVocab (called from _set_vocab_gpt2).

[CISC]

@stevenkuang-tencent gentle ping