Support for keeping models cached in the memory!

[cihangoksu]

Hi,

I am trying to use Comfyscript and run multiple workflows with a manager class. Each workflow uses different flux models. Unfortunately every time I call these run_model1 or run_model2 functions, manager class "Request to load flux" models and that takes quite long and exhauistive. Is there a way to cache these models properly? Why setting them in the init function as follows does not work? Is it because I use "with Workflow()" for each method? I am using Real runtime "from comfy_script.runtime.real import *"

Thank you very much in advance!

Cheers,
Cihan

class COMFYUIMANAGER(object):
def init(self):
super(COMFYUIMANAGER, self).init()
self.model1 = UNETLoader("flux1-schnell.safetensors", "fp8_e4m3fn")
self.model2 = UnetLoaderGGUF("flux1-dev-Q8_0.gguf")
def run_model1(self, *):
with Workflow():
......some code here.............
model = self.model1
def run_model2(self, *):
with Workflow():
......some code here.............
model = self.model2

[Chaoses-Ib]

with Workflow() will indeed clean up the models. Though as I remember a model will only be unloaded if there are no references to it. (See #21 for details.)

Anyway, you can replace it with with torch.inference_mode() (and import torch first) to avoid any cleanup.

[cihangoksu]

Thanks a lot for your response! I replaced "with Workflow():" with "with torch.inference_mode()" and unfortunately manager still loads the models every time I call different methods. Here I share the full code. Can there be some culprit lines inside the methods?

comfyui_manager.txt

Thank you super much for your help :-)

[Chaoses-Ib]

Then it's probably because your VRAM is not enough to keep them.

[cihangoksu]

You are right, the models are super large. Thanks a lot again for your help! :-)