Faster engine swap using vram (or ram?)

[SirVeggie]

The webui has a feature where you can load multiple models into vram for near instantaneous swap for hires or refiner use (or just normally swapping them around for any reason). I wonder if it would be possible to do something similar with the tensor engines? Something like having a static engine for both 512x512 and 1024x1024 and then swapping them during hires fix.

Currently I'm using a dynamic engine for this, but this kind of fast swapping would enable more flexibility like swapping to a different model entirely for hires fix (hopefully).

[chazzhou]

Engine swapping definitely needs improvement. I have a local proof of concept that supports swapping to an engine for the SDXL refiner model. However, the performance gain is killed by the longer overhead it takes. I will examine this when I have the opportunity.

@contentis, do you have any insights on this?