Skip to content

spawn method context #57

@erikhuck

Description

@erikhuck

The default spawn method is fork which apparently duplicates all the memory of the main process into the child. If gpu-tracker does that by default on linux machines, that creates a problem for jobs ran on those machines. Instead, we can have the tracker process use its own context that uses the 'spawn' method instead of 'fork' like so:

import multiprocessing as mp

ctx = mp.get_context('spawn')
class _TrackingProcess(ctx.Process)

If this works as expected, we should test out how memory usage differs when using the 'spawn' method versus 'fork'.

  • Create a script that first allocates a bunch of memory.
  • After all that memory is allocated, create a Tracker object and start tracking.
  • See how memory differs using the 'spawn' method versus the 'fork' method.
    We probably want to do the above test using the global setting with mp.set_start_method to see if there truly is an issue before we bother with using get_context and inheriting from ctx.Process.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions