load_from_checkpoint has different validation results

[lanlanlan3]

when save ckpt, it has some metrics like val_acc=0.77.
but when load_from_checkpoint, trainer.validate(model), I got val_acc=0.02

code：
from main_pl import LitModel
ckpt_path = '...val_acc=0.77.ckpt'
model = LitModel.load_from_checkpoint(ckpt_path)
'''model.eval()'''
trainer = Trainer(gpus=-1)
'''or trainer = Trainer(gpus=-1, resume_from_checkpoint=ckpt_path)'''
trainer.validate(model) # val_acc=0.02

[rohitgr7]

try:

trainer.validate(model, ckpt_path=ckpt_path)

[lanlanlan3]

a similar problem:
#6678
bug it doesnot work for me

[lanlanlan3]

when I wrote the validation code after trainer.fit in the main_pl.py, I got similar val_acc.
but when I trained the model, and load ckpt in a ipython console, the results are very different.

##### in main_pl.py
if __name__ == '__main__':
    trainer = Trainer(max_epochs=1)
    trainer.fit(task)  # val_acc=0.76
    trainer.validate(task)  # val_acc=0.77
    
    ckpt_dir = 'lightning_logs/version_0/checkpoints'
    ckpt_name = os.listdir(ckpt_dir)[0]
    ckpt_path = os.path.join(ckpt_dir, ckpt_name)
    task = Task.load_from_checkpoint(ckpt_path)
    trainer.validate(task)  # val_acc=0.79

##### in  ipython console
from main_pl import *
ckpt_path = 'lightning_logs/version_0/checkpoints/epoch=0-step=102-train_loss=1.05857-train_acc=0.65137-val_loss=0.74743-val_acc=0.76663.ckpt'

#  pytorch lightning 
task = Task.load_from_checkpoint(ckpt_path)
trainer = pl.Trainer(gpus=-1)
trainer.validate(task)  # val_acc=0.036

# pytorch
device = torch.device('cuda')
model = task.model.to(device)
model.eval()
loss, acc = 0, 0
for batch in task.val_dataloader():
    batch = {k: v.to(device) for k, v in batch.items() if isinstance(v, torch.Tensor)}
    with torch.no_grad():
        outputs = model(**batch)
    loss += outputs['loss'].detach().cpu().item()
    acc + task.acc_fn(outputs['logits'], batch['labels'])
loss /= len(task.val_dataloader())
acc /= len(task.val_dataloader())  # 0.03

[ruiyan1995]

I meet the same issue, did you solve it? @lanlanlan3

[rohitgr7]

can anyone create a reproducible script using BoringModel to check this issue?

[strawhatboy]

same issue, how can I resolve it? totally weired,
tried to load the checkpoint in the training script, it's working well, but with another script with the same validation data, same validation dataloader, same code to load the checkpoint, result seems to be random. it looks like the model was initialized with random parameters.