Different results of SparseMax

[JiabinXue]

Hi, thank you very much for providing the open source code!
However, I meet a problem when I use this code.
The model is not converged when I replace the Softmax function in attention with the Sparsemax function in my model.
Next, I try to use the Sparsemax function which is implemented by https://github.com/msobroza/SparsemaxPytorch, the model is converged as soon as possible.
So, I want to know whether some errors are in this code.
Thank you!

[vene]

Hi,

Are the unit tests in my repository passing on your computer? Perhaps there is some version mismatch, otherwise I don't know.

I'm not familiar with @msobroza's implementation, I will try to look into it.

In any case: this repository is mostly intended for fusedmax & oscarmax. If all you need is sparsemax, my recommended implementation is the one in our fork of opennmt-py:

https://github.com/deep-spin/opennmt-sparse/blob/sp-research/onmt/modules/sparse_activations.py

This runs on the GPU and, in particular, the top-k implementation (which uses partial sorting) can scale up to d>50k without much slowdown from softmax without sacrificing exactness.

[JiabinXue]

Hi,
Thank you for your reply. I want to use the Fusedmax in my model.
However, I got the Attention like this
.

Then, when I use the Sparsemax, the attention is like this

Finally, I try to use the Sparsemax which is implemented by @msobroza, the attention is like this

I try to compare the result between these two implements, I found the output is just the same.
Are you meet the same problem?

Thank you!

[vene]

No, I have not had problems.

1. Are the unit tests passing?

2. Are you sure that the way you are using the functions applies normalization along the same dimension? SHould the rows or the columns sum to 1?

If 1 & 2 don't help, please provide minimal code to reproduce the issue, ideally with a small input/output of two stacked vectors so we can check if the dimension broadcasting is the issue.