-
Notifications
You must be signed in to change notification settings - Fork 22
Open
Labels
enhancementNew feature or requestNew feature or requestgood first issueGood for newcomersGood for newcomers
Description
The discopop_explorer regenerates all data and data structures upon every execution, even if the input data has not changed.
To speed up repeated execution (which might happen during development), checkpointing should be added.
For this, i propose the following steps:
If no prior data exists:
- calculate a hash over the current input files and store them
- create the data structures
- store the data structures after creation alongside the calculated hash value in the form of JSON dumps
If prior data exists:
- calculate hashes over the current input files
- if the calculated hash is equivalent to the stored hash:
- load the dumped data structures and skip the corresponding processing steps
- continue with processings steps, where the data structures can not be recovered
- if the calculated hash is not equivalent to the stored hash:
- clear the old stored data
- proceed as if no prior data existed.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestgood first issueGood for newcomersGood for newcomers