-
Notifications
You must be signed in to change notification settings - Fork 8
Open
Milestone
Description
It would be nice to have an epi archive backed by disk / many files + we talked about the current exploration framework based on the targets package + annoyances going through epix_slide when also looping across things other than forecast_date (+ annoyances with epix_slide when the slide function outputs errors/etc.) I think this points to:
- epi_archive should be something more general that can have various backends. I sketched some backends we might imagine here.
- Maybe we want just a smarter/faster
epix_as_ofrather thanepix_slide. (epix_slideisn't smart at all yet, but it'd be easier than making epix_as_of smarter. Makingepix_as_ofsmarter probably involves caching the last query & result or some sort of partition of the data, whileepix_slidewould get to order things as it likes, partition once, etc.) - We often want a way to let slide computations output errors/warnings instead/alongside "real" results, rather than have
epix_slidecompletely give up or spit out all the warnings without associatedforecast_as_ofs at the end. I thinkpurrrmight have some useful helper functions here (purrr::safely?). - Would we want a
derived_epi_archivethat does some of this targets/targets-like stuff, or based on some (hypothetical?) library that does parallelism with only one pool of workers & adjusts for BLAS/gurobi/etc. using parallelism to prevent CPU/swap thrashing / OOM killing (investigatecoro?)? It might still have some of the awkwardness ofepix_slide, but might make some things look nicer; e.g., metaforecaster's forecasts would be aderived_epi_archivemaybe atop amerge_epi_archiveatop (a) thederived_epi_archiveholding the base forecaster's forecasts and (b) the response data'sram_epi_archive.
Alternatively, if we don't have derived or cached archived concepts, can we integrate our slide computations with a framework that does? E.g., targets.
Metadata
Metadata
Assignees
Labels
No labels