{fit,validate,test,tune,predict} as subcommands of LightningCLI

[mauvilsa]

In the discussion of pull request #6862 a comment from @carmocca was:

Should the LightningCLI allow {fit,validate,test,tune,predict}? So by default you can do:

python simple_image_classifier.py fit --trainer.max_epochs=50 or python simple_image_classifier.py validate --trainer.limit_val_batches=30

Adding subcommands to the CLI is possible, wouldn't require too much effort and be mostly compatible with what is currently implemented. But there needs to be some discussion to understand if a change like this makes sense. I think that having subcommands make sense only if there are normal use cases in which each subcommand is executed independently in different command line calls. Below are some of my thoughts. I have not used lightning's other methods apart from fit so I might be misunderstanding some things.

1. tune. From what I understand tune is something to run before fit. If this is the case then I think it does not make sense to have it as a subcommand. It could be added as an init parameter.

2. test. In the pl_examples/basic_examples test is always used after fit and just doing a print of the result. In lightning-transformers there is a flag to run test after fit. I haven't seen an example in which test is run independently from fit. Though I do think it might make sense. It is possible to run fit and then in a later point in time want to run test for a particular checkpoint. Or having a pretrained model and want to run a test for it potentially on different test sets. If test is a subcommand some questions could be:

• What would it do? Just run test and no possibility for doing something additional?
• Would it log to a new experiment or somehow be able to log to the same place fit was logged?

3. validate. I guess this is similar to test though maybe less common to run independently from the command line.

4. predict. I think certainly makes sense to be a subcommand. But the purpose of running predict depends on the particular model and use case. People might want to implement code to further process the output of predict. This could be done by some method of LightningCLI.

[carmocca]

If this is the case then I think it does not make sense to have it as a subcommand.

I agree

It could be added as an init parameter.

I think it's fair to override the LightningCLI to run tune before fit

I haven't seen an example in which test is run independently from fit.

I personally have used this a lot. For example, you might want to save GPU costs so you train on a GPU cluster and then run test separately on CPU with a checkpoint on another machine.

It is possible to run fit and then in a later point in time want to run test for a particular checkpoint. Or having a pretrained model and want to run a test for it potentially on different test sets.

Exactly

Just run test and no possibility for doing something additional?

I think it could use the {before,after}_* hooks structure we have for fit to support "something additional"

Would it log to a new experiment or somehow be able to log to the same place fit was logged?

Isn't this choice a responsibility of the LightningModule/Trainer? What does the CLI have to do with it?

People might want to implement code to further process the output of predict. This could be done by some method of LightningCLI.

Yes, but it's expected that predict returns raw predictions (transformed by predict_step)

[mauvilsa]

I have looked further in particular at the methods test and predict. In the case of test it receives a parameter ckpt_path which can be used to specify which weights to test. But predict does not have the same parameter. In that case how do you specify the checkpoint to use for prediction? I also don't understand how the default ckpt_path='best' works. Is this the "best" for a single run of fit? If yes, then if fit was run in an independent execution, does this default work for running test in an instance of the trainer which did not run fit?

Since these are methods of Trainer it is required to instantiate the Trainer class before running them. Thus the trainer configuration is required. But in principle running a test or a prediction only requires the data and the model. What should be the trainer configuration in these cases? If people are required to give trainer configuration, what should it be? Which parameters of trainer actually influence anything in test and predict? Is there any place where it is explained why these methods are part of Trainer?

[carmocca]

But predict does not have the same parameter. In that case how do you specify the checkpoint to use for prediction?

This is a bug. We'll add it.

Is this the "best" for a single run of fit?

Yes

does this default work for running test in an instance of the trainer which did not run fit?

No, you would need to set your own ckpt path

What should be the trainer configuration in these cases? If people are required to give trainer configuration, what should it be?

It could be used to limit the number of batches, change the accelerator, the precision...
These are still valid configurations for test and predict.

Is there any place where it is explained why these methods are part of Trainer?

Note that Trainer is a legacy name as it does more things than just training (just kept for compatibility). You can think of it as an Engine or Runner instead.

[mauvilsa]

Note that Trainer is a legacy name as it does more things than just training (just kept for compatibility). You can think of it as an Engine or Runner instead.

I think it would make sense to rename the class. Keeping backwards compatibility could be as simple as Trainer = Engine.

[carmocca]

It's not so easy considering all the hooks that are passed the argument trainer and all the examples/docs/tutorials all around the web which talk about the trainer and use the trainer.

[carmocca]

Another point is that there should be a way to easily extend the support for new trainer functions.

For example, in Flash we have the flash.Trainer which is almost the same but includes a trainer.finetune() function. This calls fit internally but includes a fine-tuning callback.

https://github.com/PyTorchLightning/lightning-flash/blob/719cf5c778f3c20f32b380449a06b0ea3e87532b/flash/core/trainer.py#L54-L61

So the design should allow to:

class FlashCLI(LightningCLI):
    # adds support for trainer.finetune():
    # python train.py finetune ...
    ...

[carmocca]

If you are from the future, this is now done and merged to master!