Recursion Error when setting DataLoader Random Seed with Hydra configuration #8087
Description
Describe the bug
When creating a monai dataloader inside a hydra app, I get the following recursion error
RecursionError: maximum recursion depth exceeded in comparison
full_key: set_random_state
object_type=<DataclassName>
I have created a minimum working example below.
To Reproduce
Steps to reproduce the behavior:
- create the following script
monai_hydra.py
from monai.data import DataLoader
from hydra.core.config_store import ConfigStore
from dataclasses import dataclass, field
from monai.data.dataset import Dataset
import torch
from hydra import compose, initialize
import hydra
@dataclass
class LoaderConfig:
csv_path: str = './'
spatial_size: list = field(default_factory=lambda: [128,128,128])
class MyDataset(Dataset):
def __init__(self,csv_path, spatial_size):
self.csv_path = csv_path
self.spatial_size = spatial_size
def __len__(self):
return 10
def __getitem__(self, idx):
return torch.randn(self.spatial_size)
@hydra.main(config_name = 'loader_config')
def main(cfg) -> None:
print(cfg)
my_dataset = MyDataset(
csv_path = cfg.csv_path,
spatial_size = cfg.spatial_size
)
my_dataloader = DataLoader(
my_dataset,batch_size=1,num_workers=0,shuffle=False
)
if __name__=='__main__':
cs = ConfigStore()
cs.store(name='loader_config',node=LoaderConfig)
with initialize(version_base=None):
cfg = compose(config_name='loader_config',overrides=[])
main(cfg)- run the following in the terminal
python monai_hydra.py
Note that changing the above to have the following in the excutable path makes the above run fine.
if __name__=='__main__':
# cs = ConfigStore()
# cs.store(name='loader_config',node=LoaderConfig)
# with initialize(version_base=None):
# cfg = compose(config_name='loader_config',overrides=[])
cfg = LoaderConfig()
main(cfg)Expected behavior
The dataloader should initialise without any issues.
Environment
Ensuring you use the relevant python executable, please paste the output of:
python -c "import monai; monai.config.print_debug_info()"
/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/utils/tf32.py:66: UserWarning: torch.backends.cuda.matmul.allow_tf32 = True by default.
This value defaults to True when PyTorch version in [1.7, 1.11] and may affect precision.
See https://docs.monai.io/en/latest/precision_accelerating.html#precision-and-accelerating
warnings.warn(
================================
Printing MONAI config...
================================
MONAI version: 1.3.1
Numpy version: 1.24.3
Pytorch version: 1.11.0
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 96bfda00c6bd290297f5e3514ea227c6be4d08b4
MONAI __file__: /home/<username>/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/__init__.py
Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
ITK version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: NOT INSTALLED or UNKNOWN VERSION.
scikit-image version: 0.20.0
scipy version: 1.9.1
Pillow version: 9.4.0
Tensorboard version: 2.13.0
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.12.0
tqdm version: 4.65.0
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: 5.9.8
pandas version: 1.5.3
einops version: 0.3.0
transformers version: 4.19.2
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: NOT INSTALLED or UNKNOWN VERSION.
clearml version: NOT INSTALLED or UNKNOWN VERSION.
For details about installing the optional dependencies, please visit:
https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 22.04.4 LTS
Platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.10
Processor: x86_64
Machine: x86_64
Python version: 3.8.5
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 6
Num logical CPUs: 12
Num usable CPUs: 12
CPU usage (%): [8.0, 7.5, 9.3, 6.8, 6.2, 77.2, 8.6, 8.0, 8.1, 9.2, 9.9, 28.8]
CPU freq. (MHz): 3657
Load avg. in last 1, 5, 15 mins (%): [5.4, 5.6, 4.4]
Disk usage (%): 79.3
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 15.3
Available memory (GB): 6.9
Used memory (GB): 8.0
================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 11.3
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: None
cuDNN version: 8200
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'compute_37']
GPU 0 Name: NVIDIA GeForce GTX 1650 with Max-Q Design
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 16
GPU 0 Total memory (GB): 3.8
GPU 0 CUDA capability (maj.min): 7.5
Also using hydra-core 1.3.2
Full trace
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/hydra/main.py", line 83, in decorated_main
return task_function(cfg_passthrough)
File "monai_hydra.py", line 35, in main
my_dataloader = DataLoader(
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/dataloader.py", line 87, in __init__
set_rnd(dataset, int(_seed))
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
seed = set_rnd(obj.__dict__[key], seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
seed = set_rnd(obj.__dict__[key], seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
_seed = set_rnd(item, seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
seed = set_rnd(obj.__dict__[key], seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
seed = set_rnd(obj.__dict__[key], seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
_seed = set_rnd(item, seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
seed = set_rnd(obj.__dict__[key], seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
seed = set_rnd(obj.__dict__[key], seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
_seed = set_rnd(item, seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
_seed = set_rnd(item, seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
seed = set_rnd(obj.__dict__[key], seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
seed = set_rnd(obj.__dict__[key], seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
_seed = set_rnd(item, seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
seed = set_rnd(obj.__dict__[key], seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
seed = set_rnd(obj.__dict__[key], seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
_seed = set_rnd(item, seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
seed = set_rnd(obj.__dict__[key], seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
seed = set_rnd(obj.__dict__[key], seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
_seed = set_rnd(item, seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
seed = set_rnd(obj.__dict__[key], seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
seed = set_rnd(obj.__dict__[key], seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
_seed = set_rnd(item, seed=seed)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 359, in __getattr__
self._format_and_raise(key=key, value=None, cause=e)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
format_and_raise(
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
_raise(ex, cause)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/_utils.py", line 797, in _raise
raise ex.with_traceback(sys.exc_info()[2]) # set env var OC_CAUSE=1 for full trace
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
return self._get_impl(
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 442, in _get_impl
node = self._get_child(
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 73, in _get_child
child = self._get_node(
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 475, in _get_node
self._validate_get(key)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 164, in _validate_get
self._format_and_raise(
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
format_and_raise(
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/_utils.py", line 843, in format_and_raise
object_type = OmegaConf.get_type(node)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 650, in get_type
return OmegaConf._get_obj_type(c)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 905, in _get_obj_type
if is_structured_config(c):
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/_utils.py", line 448, in is_structured_config
return is_attr_class(obj) or is_dataclass(obj)
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/_utils.py", line 442, in is_attr_class
if attr is None or isinstance(obj, Node):
File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/abc.py", line 98, in __instancecheck__
return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison
full_key: set_random_state
object_type=LoaderConfig