Skip to content

Commit 3d92e48

Browse files
committed
Update design doc
Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
1 parent ec0f974 commit 3d92e48

File tree

1 file changed

+13
-15
lines changed

1 file changed

+13
-15
lines changed

docs/design/backfill-plugin.md

Lines changed: 13 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -15,14 +15,14 @@
1515

1616
## Purpose
1717

18-
This plugin purpose is to detect missing gaps (historica and recent) in the intended stored block sequence, then
18+
This plugins purpose is to detect missing gaps (historical and recent) in the intended stored block sequence, then
1919
autonomously and asynchronously fetch the missing blocks from another source and store them.
2020

2121
## Goals
2222

2323
1. Detect gaps on start-up and while running.
24-
2. Fetch missing blocks from another BN, must be configurable.
25-
3. Asynchronously recover (fetch and store) the missing blocks, without blocking the live ingestion path (new incoming blocks).
24+
2. Fetching missing blocks from another BN, must be configurable.
25+
3. Asynchronously recover (fetch and store) the missing blocks, without blocking the live ingestion of incoming blocks.
2626
4. Instrumentation, logging, metrics and error handling for the backfill process.
2727

2828
## Terms
@@ -73,16 +73,17 @@ The plugin will autonomously detect gaps in the block range and fetch missing bl
7373

7474
1. At start-up a loop is defined that runs every `backfill.scanInterval`
7575
2. At every interval the plugin detects missing gaps in the intended block range against the actual stored blocks using
76-
the `HistoricalBlockFacility`. It will use a gRPC client to request the serverStatus from configured target block
77-
nodes and determine the available range of recent blocks available.
78-
3. If gaps are found, it initiates the backfill process.
79-
4. The plugin uses a gRPC client to connect to other Block Nodes to fetch the missing blocks.
80-
5. Once the blocks are fetched, the plugin creates a `BlockNotification` of type `BackfilledBlockNotification` and sends
76+
the `HistoricalBlockFacility`.
77+
3. If greedy backfill logic is enabled it will use a gRPC client to request the serverStatus from configured target
78+
block nodes and determine the available range of recent blocks available to consider in addition.
79+
4. If gaps are found, it initiates the backfill process.
80+
5. The plugin uses a gRPC client to connect to other Block Nodes to fetch the missing blocks.
81+
6. Once the blocks are fetched, the plugin creates a `BlockNotification` of type `BackfilledBlockNotification` and sends
8182
it to the `MessagingFacility`.
82-
6. The `VerificationPlugin` will then process the `BackfilledBlockNotification` and if the block is valid, it will
83+
7. The `VerificationPlugin` will then process the `BackfilledBlockNotification` and if the block is valid, it will
8384
create a `VerificationNotification` and send it to the `MessagingFacility` for further processing.
84-
7. The PersistencePlugin will then store the block in the local storage.
85-
8. The `BackfillPlugin` will receive the `PersistenceNotification` and update its internal state accordingly, marking
85+
8. The PersistencePlugin will then store the block in the local storage.
86+
9. The `BackfillPlugin` will receive the `PersistenceNotification` and update its internal state accordingly, marking
8687
the backfill process as complete for that block.
8788

8889
## Diagram
@@ -141,8 +142,7 @@ received and periodically to ensure block node does not fall too far behind.
141142
usually this would be done by the `PublisherPlugin` or any other plugin that knows the latest block or wants to
142143
ensure the Block Node is up-to-date.
143144
2. BackfillPlugin will handle the `NewestBlockKnownToNetwork` message and will check if there are any gaps in the block
144-
range available in the local storage. It will also use a gRPC client to request the serverStatus from configured
145-
peer block nodes and determine the available range of recent blocks available.
145+
range available in the local storage.
146146
3. If gaps are found, it will initiate the backfill process as described in the Autonomous Backfill section.
147147
4. The process will be the same as the Autonomous Backfill, but it will be triggered by the `NewestBlockKnownToNetwork`
148148
message instead of the periodic scan.
@@ -162,8 +162,6 @@ sequenceDiagram
162162
%% Dispatch to backfill
163163
MessagingFacility->>BackfillPlugin: NewestBlockKnownToNetwork(latestBlock)
164164
BackfillPlugin->>HistoricalBlockFacility: detectMissingGaps(…, latestBlock)
165-
BackfillPlugin->>+GrpcClient: getServerStatusFromPeerBlockNodes()
166-
BackfillPlugin-->>BackfillPlugin: detectMissingRecentGaps(min(firstBlocks), max(lastBlocks))
167165
168166
alt Gaps found
169167
BackfillPlugin->>GrpcClient: fetchMissingBlocks(gapRange, batchSize)

0 commit comments

Comments
 (0)