1515
1616## Purpose
1717
18- This plugin purpose is to detect missing gaps (historica and recent) in the intended stored block sequence, then
18+ This plugins purpose is to detect missing gaps (historical and recent) in the intended stored block sequence, then
1919autonomously and asynchronously fetch the missing blocks from another source and store them.
2020
2121## Goals
2222
23231 . Detect gaps on start-up and while running.
24- 2 . Fetch missing blocks from another BN, must be configurable.
25- 3 . Asynchronously recover (fetch and store) the missing blocks, without blocking the live ingestion path (new incoming blocks) .
24+ 2 . Fetching missing blocks from another BN, must be configurable.
25+ 3 . Asynchronously recover (fetch and store) the missing blocks, without blocking the live ingestion of incoming blocks.
26264 . Instrumentation, logging, metrics and error handling for the backfill process.
2727
2828## Terms
@@ -73,16 +73,17 @@ The plugin will autonomously detect gaps in the block range and fetch missing bl
7373
74741 . At start-up a loop is defined that runs every ` backfill.scanInterval `
75752 . At every interval the plugin detects missing gaps in the intended block range against the actual stored blocks using
76- the ` HistoricalBlockFacility ` . It will use a gRPC client to request the serverStatus from configured target block
77- nodes and determine the available range of recent blocks available.
78- 3 . If gaps are found, it initiates the backfill process.
79- 4 . The plugin uses a gRPC client to connect to other Block Nodes to fetch the missing blocks.
80- 5 . Once the blocks are fetched, the plugin creates a ` BlockNotification ` of type ` BackfilledBlockNotification ` and sends
76+ the ` HistoricalBlockFacility ` .
77+ 3 . If greedy backfill logic is enabled it will use a gRPC client to request the serverStatus from configured target
78+ block nodes and determine the available range of recent blocks available to consider in addition.
79+ 4 . If gaps are found, it initiates the backfill process.
80+ 5 . The plugin uses a gRPC client to connect to other Block Nodes to fetch the missing blocks.
81+ 6 . Once the blocks are fetched, the plugin creates a ` BlockNotification ` of type ` BackfilledBlockNotification ` and sends
8182 it to the ` MessagingFacility ` .
82- 6 . The ` VerificationPlugin ` will then process the ` BackfilledBlockNotification ` and if the block is valid, it will
83+ 7 . The ` VerificationPlugin ` will then process the ` BackfilledBlockNotification ` and if the block is valid, it will
8384 create a ` VerificationNotification ` and send it to the ` MessagingFacility ` for further processing.
84- 7 . The PersistencePlugin will then store the block in the local storage.
85- 8 . The ` BackfillPlugin ` will receive the ` PersistenceNotification ` and update its internal state accordingly, marking
85+ 8 . The PersistencePlugin will then store the block in the local storage.
86+ 9 . The ` BackfillPlugin ` will receive the ` PersistenceNotification ` and update its internal state accordingly, marking
8687 the backfill process as complete for that block.
8788
8889## Diagram
@@ -141,8 +142,7 @@ received and periodically to ensure block node does not fall too far behind.
141142 usually this would be done by the ` PublisherPlugin ` or any other plugin that knows the latest block or wants to
142143 ensure the Block Node is up-to-date.
1431442 . BackfillPlugin will handle the ` NewestBlockKnownToNetwork ` message and will check if there are any gaps in the block
144- range available in the local storage. It will also use a gRPC client to request the serverStatus from configured
145- peer block nodes and determine the available range of recent blocks available.
145+ range available in the local storage.
1461463 . If gaps are found, it will initiate the backfill process as described in the Autonomous Backfill section.
1471474 . The process will be the same as the Autonomous Backfill, but it will be triggered by the ` NewestBlockKnownToNetwork `
148148 message instead of the periodic scan.
@@ -162,8 +162,6 @@ sequenceDiagram
162162 %% Dispatch to backfill
163163 MessagingFacility->>BackfillPlugin: NewestBlockKnownToNetwork(latestBlock)
164164 BackfillPlugin->>HistoricalBlockFacility: detectMissingGaps(…, latestBlock)
165- BackfillPlugin->>+GrpcClient: getServerStatusFromPeerBlockNodes()
166- BackfillPlugin-->>BackfillPlugin: detectMissingRecentGaps(min(firstBlocks), max(lastBlocks))
167165
168166 alt Gaps found
169167 BackfillPlugin->>GrpcClient: fetchMissingBlocks(gapRange, batchSize)
0 commit comments