Skip to content
This repository was archived by the owner on Jun 2, 2025. It is now read-only.

Conversation

@facok
Copy link

@facok facok commented May 17, 2025

Addresses a performance issue where WAS_Load_Image_Batch was slow when dealing with directories containing a very large number of files, even after initial file listing cache was implemented. The bottleneck was identified as the cost of loading/deserializing a massive file path list from the database cache.

This commit refactors the caching mechanism within BatchImageLoader:

  • Introduces a module-level memory cache for faster access within a single session.
  • Changes the database cache storage format from a serialized object to a newline-separated text block (Batch Images Paths Cache Text category), which is generally more efficient for storing and retrieving large lists of strings.
  • Updates the loading and saving logic to handle this text block format.
  • Refines error handling during image loading.
  • Adjusts index updating in incremental mode to occur only after successful image loading.
  • Updates the IS_CHANGED method to correctly utilize the cached file paths based on the input index for 'single_image' mode.

Additionally, the call to update_history_images(new_paths) in load_batch_images has been commented out as it was suspected to be a significant bottleneck when processing the entire list of potentially millions of file paths on every node execution. If history tracking is required, update_history_images may need to be optimized to handle large lists efficiently or process only the single loaded file.

These changes aim to drastically reduce the time spent loading the file list cache, improving responsiveness when working with extensive image libraries.

…ctories

Addresses a performance issue where WAS_Load_Image_Batch was slow when dealing with directories containing a very large number of files, even after initial file listing cache was implemented. The bottleneck was identified as the cost of loading/deserializing a massive file path list from the database cache.

This commit refactors the caching mechanism within `BatchImageLoader`:
- Introduces a module-level memory cache for faster access within a single session.
- Changes the database cache storage format from a serialized object to a newline-separated text block (`Batch Images Paths Cache Text` category), which is generally more efficient for storing and retrieving large lists of strings.
- Updates the loading and saving logic to handle this text block format.
- Refines error handling during image loading.
- Adjusts index updating in incremental mode to occur only after successful image loading.
- Updates the `IS_CHANGED` method to correctly utilize the cached file paths based on the input index for 'single_image' mode.

Additionally, the call to `update_history_images(new_paths)` in `load_batch_images` has been commented out as it was suspected to be a significant bottleneck when processing the entire list of potentially millions of file paths on every node execution. If history tracking is required, `update_history_images` may need to be optimized to handle large lists efficiently or process only the single loaded file.

These changes aim to drastically reduce the time spent loading the file list cache, improving responsiveness when working with extensive image libraries.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant