Some thoughts about the evaluation / training API

[thomasaarholt]

Outline & Motivation

Hiya,

These are just my first impressions. They are probably wrong. Feel free to ignore.

I am a newcomer to Adalflow and trying to see if I can use it for my project. I thought I'd highlight a difficulty that I experience with the current API, as I am learning using the Question Answering tutorial.

The Component instance, typically parsed into an object called task_pipeline is itself a callable. Its return type (output = task_pipeline(...))
depends at runtime on whether the pipeline is in eval or train mode.

I rely heavily on the type checker (mypy or pyright) to tell me what to do with objects via autocompletion. Currently, the return type regardless of eval/train is listed as Parameter. When I print the type(output), I get either GeneratorOutput if eval or OutputParameter if training. The type checker is "static", it doesn't know if we are in train or eval mode, so it returns what it is able to, which is Union[Parameter, Unknown].

This makes it very difficult to understand I should "do" with my output from this point on.

Pitch

As a suggestion, instead of (or at least, in addition to) making the pipeline directly callable through __call__, you could use the task_pipeline.eval() and task_pipeline.train() methods to call the pipeline and return (with type hinting) the correct object. Or, if the toggled train/eval flag set by those methods is used for other stuff and you don't want to change them, then call them something like task_pipeline.eval_pipeline and task_pipeline.train_pipeline().

In pseudocode:

class Component(...):
    def train(self, **kwargs) -> GeneratorOutput:
        self._set_component_in_train()
        self(**kwargs)

    def eval(self, **kwargs) -> OutputParameter:
        self._set_component_in_eval()
        self(**kwargs)

or

class Component(...):
    def train_pipeline(self, **kwargs) -> GeneratorOutput:
        self.train()
        self(**kwargs)

    def eval_pipeline(self, **kwargs) -> OutputParameter:
        self.eval()
        self(**kwargs)

Both of these types of solutions now return the correct type, which makes it easier for the end user to a) know that they are indeed in train or eval mode, and b) use the type checker to auto-complete the correct properties or methods on those objects.

Again, just my two cents.

Additional context

No response

[liyin2015]

@thomasaarholt Thanks for the feedback.

Currently we use forward and call for train and eval. forward aligns with PyTorch's syntax. And call fits more for inference code(production serving code).

Right now we use bicall because sometimes its less code if we deal with both situations in the same method. If you want more clarity, feel free to use forward and call instead. Then each should only support one type.

We still need to think further to make these apis crystal clear, so I will keep this issue open until it reaches that stage.