Question about LongNet attention map overlap #108
Description
The attention maps of different Dilated Rates overlap with each other. In the figure, the same token is calculated only once, but in the code, the overlapped token is taken into account multiple times (because the output is a weighted sum of all Dilated Rates).
Would you have any comments on this? Is this expected behavior? @donglixp @shumingma