EnzymeCommissions always NaN using ESM fixed

[jasperhyp]

Hi, I used your EnzymeCommissions to benchmark ESM and it looks like the loss is still NaN at epoch 20. Shall I change the yaml (copied from GearNet)?

[jasperhyp]

The same thing happened for GO (BP, CC, MF).

[ChrisAllenMing]

Hi, I have checked the ESM on EC config in the GearNet repo. Everything looks fine to me. The most important points are lr_ratio: 0.1 and max_length=550 for reproducing the results. I am not sure whether some specific setting (like batch size, gpus) will lead to the training failure

[jasperhyp]

Interesting... I am using batch_size: 16 and one GPU. But that shouldn't affect losses. Otherwise, everything is the same.

[ChrisAllenMing]

Yeah, it's weird. I'm afraid I cannot tell why it is the case. BTW, for ESM on EC, I think batch_size: 16 may easily exceed the maximum GPU memory. I am curious about the max_length of truncation you use to let it runnable.

[jasperhyp]

Hmmm. I'll check the GO dataset to see if there's something weird I'm doing.
I used a 32GB GPU. I kept the max len line in the yaml file, which is 550.

[ChrisAllenMing]

Oh, I just find you fix the encoder, so you can have the batch size of 16. The truncation length and other configurations also look fine to me

[jasperhyp]

It's kinda weird but if I used

truncate_transform = transforms.TruncateProtein(max_length=550, random=False)
protein_view_transform = transforms.ProteinView(view="residue")
transform = transforms.Compose([truncate_transform, protein_view_transform])
dataset = datasets.EnzymeCommission(data_folder+"/protein-datasets", transform=transform)

it worked. I am guessing it's a dataset issue but I can't figure out why.

[Oxer11]

Hi! Have you tried this config in GearNet repo? This config should work well for EC. If not, you can raise an issue in the GearNet repo.