Skip to content
This repository was archived by the owner on Sep 11, 2023. It is now read-only.

Commit 12ff283

Browse files
authored
Updating docs for set_fsspec_for_multiprocessing
1 parent a7d1199 commit 12ff283

File tree

1 file changed

+9
-3
lines changed

1 file changed

+9
-3
lines changed

nowcasting_dataset/utils.py

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,9 +22,15 @@ def set_fsspec_for_multiprocess() -> None:
2222
"""
2323
Clear reference to the loop and thread.
2424
25-
This is necessary otherwise
26-
gcsfs hangs in the ML training loop. Only required for fsspec >= 0.9.0
27-
See https://github.com/dask/gcsfs/issues/379#issuecomment-839929801
25+
This is a nasty hack that was suggested but NOT recommended by the lead fsspec developer!
26+
27+
This appears necessary otherwise gcsfs hangs when used after forking multiple worker processes.
28+
Only required for fsspec >= 0.9.0
29+
30+
See:
31+
- https://github.com/fsspec/gcsfs/issues/379#issuecomment-839929801
32+
- https://github.com/fsspec/filesystem_spec/pull/963#issuecomment-1131709948
33+
2834
TODO: Try deleting this two lines to make sure this is still relevant.
2935
"""
3036
fsspec.asyn.iothread[0] = None

0 commit comments

Comments
 (0)