Commit e0c3bd3

committed

turns out gemma v3 decided on qk norm over softcapping

Revert "start fashioning a version with attention softclamping a la gemma and get rid of maximum tracking" This reverts commit c56cc9c.

1 parent c56cc9c commit e0c3bd3Copy full SHA for e0c3bd3

2 files changed

+0

-2213

lines changed

native_sparse_attention_pytorch
- triton_native_sparse_attention_with_softclamp.py
test_triton_nsa_softclamp.py

2 files changed

+0

-2213

lines changed

Comments

(0)