Add RFC for dynamic catalog modification #47
Conversation
infvg
force-pushed
the
dynamic-catalog
branch
from
86219e7 to
94cb499
Compare
| 1. Add an option of having catalogs stored remotely as opposed to locally. | ||
| 2. Add a way to interact with the external store via Presto | ||
|
|
||
| ### Dynamic Catalog Store |
There was a problem hiding this comment.
Some high level feedback -
-
Let's call out what also happens at worker startup - With the dynamic catalog store, I think it makes sense to not have any sort of local disk based catalog info for any node. Instead, at startup the worker always gets (via a pull or a push by the RM) the most up-to-date catalog info over the wire
-
We should flesh out how the coordinator decides to atomically switch to new (set of catalog configs). IMO this should be done via the heartbeat mechanism + consensus (say 2/3rd of the workers ack that they have new configs). However, I am not familiar with the heartbeat mechanism - specifically if we can make any latency and atomicity guarantees on it
-
Even with the atomic config switchover, queries that are already in-progress can fail if their configs are over-written. To address this, we need a way to support multiple config-sets (at least two - the current and previous ones). We can associate each query with a
catalog_config_idto address this
3 participants
No description provided.