is this project dead?

[devops724]

Hi , i long time awating for update to new cuda driver support, but last commit is around 2 months ago, is there any alternative allow me run on system with qwen 235B on 16G VRAM and 512G RAM in latest debian 13, cuda and nvidia driver?

[kirnat]

You can try ik_llama.cpp

[devops724]

You can try ik_llama.cpp

i can't find a way to compile this using cuda 12.8 in linux too,
rtx 50 series require minimum cuda 12.8

[Gadflyii]

You can try ik_llama.cpp

which does not use AMX in CPU+GPU hybrid unless something changed in the past few weeks. They are still using the upstream engine from lllama.cpp which is hard coded to use AVX512 when a GPU is present and will only use AMX in CPU only operations.

[Gadflyii]

You can try ik_llama.cpp

i can't find a way to compile this using cuda 12.8 in linux too, rtx 50 series require minimum cuda 12.8

I had no issues with Ubuntu 24.04 LTS, Cuda 12.8, and driver open 575-server

[kirnat]

True, ik doesn't have any specific AMX optimization, but going by the benchmark results from this repo (twhich may be outdated) I don't really see it outperforming ik in neither prefill nor generation. I wish I could be proven wrong since I could personally benefit from it.

[ikawrakow]

You can try ik_llama.cpp

which does not use AMX in CPU+GPU hybrid unless something changed in the past few weeks. They are still using the upstream engine from lllama.cpp which is hard coded to use AVX512 when a GPU is present and will only use AMX in CPU only operations.

ik_llama does not use the "upstream inference engine". If it did, what would be the point of it? Although it does not use AMX-specific optimisations, it is still significantly faster than the llama.cpp and KTransformers AMX implementations, at least as far as I can tell.

[Gadflyii]

Any plans to implement AMX then?

I am a big fan of IK_llama.cpp; but have not ran it in a long time.

I could try to do a run of both.

You can try ik_llama.cpp

which does not use AMX in CPU+GPU hybrid unless something changed in the past few weeks. They are still using the upstream engine from lllama.cpp which is hard coded to use AVX512 when a GPU is present and will only use AMX in CPU only operations.

ik_llama does not use the "upstream inference engine". If it did, what would be the point of it? Although it does not use AMX-specific optimisations, it is still significantly faster than the llama.cpp and KTransformers AMX implementations, at least as far as I can tell.

[Gadflyii]

I have a fork of llama.cpp they enables AMX in hybrid environments, grants about a 20-30% increase in the CPU inference portions of offloaded models if you want to try it.

True, ik doesn't have any specific AMX optimization, but going by the benchmark results from this repo (twhich may be outdated) I don't really see it outperforming ik in neither prefill nor generation. I wish I could be proven wrong since I could personally benefit from it.

[trilog-inc]

I pulled the AMXInt4 working fork of Ktransformers and saw some meaningful improvements. Would love to see these be implemented in ik_llama too

#1492 (comment)

I have a fork of llama.cpp they enables AMX in hybrid environments, grants about a 20-30% increase in the CPU inference portions of offloaded models if you want to try it.

True, ik doesn't have any specific AMX optimization, but going by the benchmark results from this repo (twhich may be outdated) I don't really see it outperforming ik in neither prefill nor generation. I wish I could be proven wrong since I could personally benefit from it.

[Gadflyii]

If you are running a hybrid CPU/GPU setup, try my llama.cpp fork, it works with amx int8/bf16 (I don’t have a 6th gen to try Int4). https://github.com/Gadflyii/llama.cpp Build as usual with all the AMX flags, and run llama-bench/cli/server with “—amx” to enable amx in a hybrid. Should see 30-40% uplift in the CPU offloaded layers / experts

…

On Fri, Sep 12, 2025 at 4:18 PM Myfrienderic ***@***.***> wrote: *trilog-inc* left a comment (kvcache-ai/ktransformers#1490) <#1490 (comment)> I pulled the AMXInt4 working fork of Ktransformers and saw some meaningful improvements. Would love to see these be implemented in ik_llama too #1492 (comment) <#1492 (comment)> I have a fork of llama.cpp they enables AMX in hybrid environments, grants about a 20-30% increase in the CPU inference portions of offloaded models if you want to try it. True, ik doesn't have any specific AMX optimization, but going by the benchmark results from this repo (twhich may be outdated) I don't really see it outperforming ik in neither prefill nor generation. I wish I could be proven wrong since I could personally benefit from it. — Reply to this email directly, view it on GitHub <#1490 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIJGCA3AX7FRKIN2ZHKQCBL3SM2BFAVCNFSM6AAAAACFIUX5C6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEOBWHA2DGNRYGY> . You are receiving this because you commented.Message ID: ***@***.***>

[trilog-inc]

Hi, I tried your fork and the uplift wasn't noticeable.

prompt eval time = 1016.33 ms / 16 tokens ( 63.52 ms per token, 15.74 tokens per second)
eval time = 149323.75 ms / 1162 tokens ( 128.51 ms per token, 7.78 tokens per second)

I used the following flags, as AMX specific flags aren't documented anywhere, but some posts mentioned the GGML_NATIVE flag being enough..

cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON

I also used an IQ4_XL quant. Would that have affected the results?

In any case, I think we should continue this conversation on that repo. Can you enable the discussions feature?

If you are running a hybrid CPU/GPU setup, try my llama.cpp fork, it works
with amx int8/bf16 (I don’t have a 6th gen to try Int4).

https://github.com/Gadflyii/llama.cpp

Build as usual with all the AMX flags, and run llama-bench/cli/server with
“—amx” to enable amx in a hybrid.

Should see 30-40% uplift in the CPU offloaded layers / experts
…

[Gadflyii]

Did you build with all the AMX flags? You need: -DGGML_NATIVE=ON -DGGML_CUDA=ON -DGGML_AMX_TILE=ON -DGGML_AMX_INT8=ON -DGGML_AMX_BF16=ON “—AmX” is only used as a flag at run as part of the command: Example: “ ./build/bin/llama-bench \ -m /AI/Qwen3_30B/Q4_0/Qwen3-30B-A3B-Thinking-2507-Q4_0.gguf \ -t 32 --amx -ngl 20 \ “

…

On Fri, Sep 12, 2025 at 10:56 PM Myfrienderic ***@***.***> wrote: *trilog-inc* left a comment (kvcache-ai/ktransformers#1490) <#1490 (comment)> Hi, I tried your fork and the uplift wasn't noticeable. prompt eval time = 1016.33 ms / 16 tokens ( 63.52 ms per token, 15.74 tokens per second) eval time = 149323.75 ms / 1162 tokens ( 128.51 ms per token, 7.78 tokens per second) I used the following flags, as AMX specific flags aren't documented anywhere, but some posts mentioned the GGML_NATIVE flag being enough.. cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON I also used an IQ4_XL quant. Would that have affected the results? In any case, I think we should continue this conversation on that repo. Can you enable the discussions feature? If you are running a hybrid CPU/GPU setup, try my llama.cpp fork, it works with amx int8/bf16 (I don’t have a 6th gen to try Int4). https://github.com/Gadflyii/llama.cpp Build as usual with all the AMX flags, and run llama-bench/cli/server with “—amx” to enable amx in a hybrid. Should see 30-40% uplift in the CPU offloaded layers / experts … <#m_-5299714883264582019_> — Reply to this email directly, view it on GitHub <#1490 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIJGCA4QHBLRYNG3BJGKDWD3SOIXRAVCNFSM6AAAAACFIUX5C6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEOBXGUYTANBRGU> . You are receiving this because you commented.Message ID: ***@***.***>

[Gadflyii]

I enabled discussions, I really appreciate you helping me test it. On Sat, Sep 13, 2025 at 1:17 AM ***@***.***> wrote:

…

Did you build with all the AMX flags? You need: -DGGML_NATIVE=ON -DGGML_CUDA=ON -DGGML_AMX_TILE=ON -DGGML_AMX_INT8=ON -DGGML_AMX_BF16=ON “—AmX” is only used as a flag at run as part of the command: Example: “ ./build/bin/llama-bench \ -m /AI/Qwen3_30B/Q4_0/Qwen3-30B-A3B-Thinking-2507-Q4_0.gguf \ -t 32 --amx -ngl 20 \ “ On Fri, Sep 12, 2025 at 10:56 PM Myfrienderic ***@***.***> wrote: > *trilog-inc* left a comment (kvcache-ai/ktransformers#1490) > <#1490 (comment)> > > Hi, I tried your fork and the uplift wasn't noticeable. > > prompt eval time = 1016.33 ms / 16 tokens ( 63.52 ms per token, 15.74 > tokens per second) > eval time = 149323.75 ms / 1162 tokens ( 128.51 ms per token, 7.78 tokens > per second) > > I used the following flags, as AMX specific flags aren't documented > anywhere, but some posts mentioned the GGML_NATIVE flag being enough.. > > cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON > > I also used an IQ4_XL quant. Would that have affected the results? > > In any case, I think we should continue this conversation on that repo. > Can you enable the discussions feature? > > If you are running a hybrid CPU/GPU setup, try my llama.cpp fork, it works > with amx int8/bf16 (I don’t have a 6th gen to try Int4). > > https://github.com/Gadflyii/llama.cpp > > Build as usual with all the AMX flags, and run llama-bench/cli/server with > “—amx” to enable amx in a hybrid. > > Should see 30-40% uplift in the CPU offloaded layers / experts > … <#m_-5881665051769918058_m_-5299714883264582019_> > > — > Reply to this email directly, view it on GitHub > <#1490 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AIJGCA4QHBLRYNG3BJGKDWD3SOIXRAVCNFSM6AAAAACFIUX5C6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEOBXGUYTANBRGU> > . > You are receiving this because you commented.Message ID: > ***@***.***> >

[magikRUKKOLA]

@devops724
Yes, this project is dead. The underlying fork of flalshinfer that ktransformers is using doens't work corrrectly. Even more so, if you will try to port the latest master of flashinfer to be usable with ktransformers you will hit the same issues. ktransformers is an example of a project that should be avoided at all costs. Speed is nothing if the output is a garbage.

[Gadflyii]

What doesn’t work correctly?

I don’t think it is fair to say the project is dead, they made a major commit just last week to add support for qwen-next.