GPU/TPU/memory not releasing #21690
Description
using any method or code to release memory used by keras is not working.
I tried this with tensorflow and jax .
This has been occurring since 2022 or earlier, and even in tf.keras
Impact of bug:
- for linux users: they need to kill process, to free the memory.
- for colab, we need to restart session.
- for windows users, they can neither kill process nor do anything, complete system restart is the only solution.
- For shared GPU in small companies or mid scale companies on a Linux Server, if someone uses a script to allocate GPU or memory, and is done with using it, the memory is not being released by the process, and other user can't even allocate GPU or TPU even when it is not being used by the process which allocated it first.
In this case one possible way is to find the process id which allocated GPU and then kill it through terminal. But this might kill wrong process by mistake, and cause lots of difficulties in collaboration.