Attention and Decoder Module hidden state question

[pyoungkangkim]

Hi 312shan, great work!

I was analyzing your code and was curious about parts of the forward for the Decoder and Attention module.

. In the forward function of the Attention Module, you are doing
h = hidden.repeat(timestep, 1, 1).transpose(0, 1) # [32, 512]=>[32, 27, 512]
since you have passed the last layer of hidden state from Decoder's forward function as
attn_weights = self.attention(last_hidden[-1], encoder_outputs) # [32, 512][27, 32, 512]=>[32, 1, 27]

However, why is only the last of the hidden used? I'm assuming it's because of meeting matrix dimensionality requirements for the Attention Module, but lets say you specified multi-layers for the Decoder Module such as ,

decoder = Decoder(embed_size, hidden_size, en_size, n_layers=2, dropout=0.0)

Would this be losing half of information from the hidden state, that the Attention Module could have possibly used since you are only taking the last layer of the hidden state into account everytime?