Skip to content

discopop_explorer: add checkpointing #739

@lukasrothenberger

Description

@lukasrothenberger

The discopop_explorer regenerates all data and data structures upon every execution, even if the input data has not changed.
To speed up repeated execution (which might happen during development), checkpointing should be added.

For this, i propose the following steps:

If no prior data exists:

  • calculate a hash over the current input files and store them
  • create the data structures
  • store the data structures after creation alongside the calculated hash value in the form of JSON dumps

If prior data exists:

  • calculate hashes over the current input files
  • if the calculated hash is equivalent to the stored hash:
    • load the dumped data structures and skip the corresponding processing steps
    • continue with processings steps, where the data structures can not be recovered
  • if the calculated hash is not equivalent to the stored hash:
    • clear the old stored data
    • proceed as if no prior data existed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions