Is the adaptive gradient clipping also applicable to layer normalization?

[shi27feng]

Hi,
My question is as the title.
Thanks

[vballoli]

I believe AGC is meant to replace any sort of normalization between two consequent layers (batch, layer, etc.).

[shi27feng]

Thank you for your prompt reply. Actually, we are thinking about removing the layer normalization inside the encoder module of the Transformer, we might need to try and bring some results back. If you have any suggestions, please let us know, thanks again.

[vballoli]

Tbh it's probably better to take an empirical direction and see how it performs with and without layernorm. It's probably not worth speculating since it's relatively new.