Some Questions about Pretraining

[U-rara]

I greatly admire the creative endeavor that went into the development of ProtST. I have some questions about its pretraining process. After reviewing the code and the paper, I discovered that the pretraining of ProtST does not involve validation. How should we determine which epoch's checkpoint to use for downstream tasks (Which epoch does the checkpoint mentioned in the Readme correspond to)? In our attempts to replicate ProtST, we found that for ESM2, the zero-shot performance tends to decrease after a larger number of epochs. Could this mean that the pretraining process is overfitting, and might this affect downstream tasks? We are keen to follow your innovative work and are very interested in these questions, looking forward to your response.