-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Description
The default spawn method is fork which apparently duplicates all the memory of the main process into the child. If gpu-tracker does that by default on linux machines, that creates a problem for jobs ran on those machines. Instead, we can have the tracker process use its own context that uses the 'spawn' method instead of 'fork' like so:
import multiprocessing as mp
ctx = mp.get_context('spawn')
class _TrackingProcess(ctx.Process)
If this works as expected, we should test out how memory usage differs when using the 'spawn' method versus 'fork'.
- Create a script that first allocates a bunch of memory.
- After all that memory is allocated, create a Tracker object and start tracking.
- See how memory differs using the 'spawn' method versus the 'fork' method.
We probably want to do the above test using the global setting withmp.set_start_method
to see if there truly is an issue before we bother with usingget_context
and inheriting fromctx.Process
.
Metadata
Metadata
Assignees
Labels
No labels