-
Notifications
You must be signed in to change notification settings - Fork 76
Add async backing technical reference #1179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
charlesHetterich
wants to merge
17
commits into
staging/product-ia
Choose a base branch
from
charles/async-backing-technincal-ref
base: staging/product-ia
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
8a1b7e3
first pass async backing technical reference
charlesHetterich 30e9e3d
split "async backing" into "async backing" & "inclusion pipeline" sec…
charlesHetterich f21afb1
renamed parachain language to rollup
charlesHetterich 5bc9f0f
committing all difs generated by workflow script on commits
charlesHetterich e7e40dc
replaced all references of *rollup* --> *parachain/parablock*
charlesHetterich 9ddd44f
added description to inclusion-pipeline.md
charlesHetterich 353d84a
fixed typos pointed out by copilot and updated diagrams
charlesHetterich 3f7c4ad
committing generated content
charlesHetterich 9c21f6b
added links to definitions for , , and
charlesHetterich a8561ce
removed unnecessary from internal link
charlesHetterich 214af0d
improve `Generation` and `Backing` definitions within the Inclusion P…
charlesHetterich 864b3bf
adding generated files
charlesHetterich 426bf7a
added intro to inclusion pipeline page
charlesHetterich 1ee9875
add ai generated pages
charlesHetterich 1aed941
improved backing definition in the inclusion pipeline reference
charlesHetterich 86f6a2f
improved sync vs async section of async backing
charlesHetterich 317b284
fixed mermaid chart
charlesHetterich File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,198 @@ | ||
| TODO | ||
| --- | ||
| title: Asynchronous Backing | ||
| description: Understand how asynchronous backing pipelines parachain block production, the protocol changes it introduces on the Relay Chain, and how parachains participate safely and efficiently. | ||
| categories: Polkadot Protocol | ||
| --- | ||
|
|
||
| # Asynchronous Backing | ||
|
|
||
| ## Introduction | ||
|
|
||
| Parachain blocks secured by the Polkadot relay chain are processed through a mulit-step pipeline called the [Inclusion Pipeline](#inclusion-pipeline). | ||
|
|
||
| Asynchronous backing (often shortened to **_Async Backing_**) decouples the steps of this pipeline which enables parallel processing. Specifically, it allows for the generation and backing steps to kick off for a given parablock *before its parent block completes the pipeline*. | ||
charlesHetterich marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| The introduction of this technique improves throughput of the Polkadot network and lays the foundation for many blocks of the same parachain to be processed in parallel via [Elastic Scaling](/reference/parachains/consensus/elastic-scaling){target=\_blank}. | ||
|
|
||
| ## How Async Backing Works | ||
|
|
||
| Before diving into *asynchronous* backing, it is helpful to first review each component of the pipeline. | ||
|
|
||
| ### Inclusion Pipeline | ||
|
|
||
| ```mermaid | ||
| %%{init: {"flowchart": {"nodeSpacing": 40, "rankSpacing": 60}}}%% | ||
| flowchart LR | ||
| %% Keep the pipeline on one row (container is hidden) | ||
| subgraph Row[" "] | ||
| direction LR | ||
| G["Generation"] --> B["Backing"] --> I["Inclusion"] | ||
| end | ||
| style Row fill:none,stroke:none | ||
|
|
||
| %% Context: plain text (no box) pointing to both G and B | ||
| C["Context"]:::nobox | ||
| C -.-> G | ||
| C -.-> B | ||
|
|
||
| classDef nobox fill:none,stroke:none,color:inherit; | ||
| ``` | ||
charlesHetterich marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| **Context**: Context of state is provided as input in order for collators and validators to build a parablock during the generation and backing stages, respectively. This context is provided by two sources: | ||
|
|
||
| * **Relay Parent**: The relay chain block which a given parablock is anchored to. Note that the relay parent of a parablock and the relay block including that parablock are always different. This context source lives on the relay chain. | ||
|
|
||
charlesHetterich marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| * **Unincluded Segments**: Chains of candidate parablocks that have yet to be included in the relay chain, i.e. they can contain parablocks at any stage pre-inclusion. The core functionality that asynchronous backing brings is the ability to build on these unincluded segments of block ancestors rather than building only on ancestors included in the relay chain state. This context source lives on the collators. | ||
|
|
||
| **Generation**: Collators *execute* their blockchain's core functionality to generate a new block, producing a [candidate receipt](), which is passed to validators selected for backing. | ||
|
|
||
| **Backing**: A subset of active validators verify if the parablock follows the state transition rules of the parachain and sign *Proof of Validity* (PoV) statements that can have a positive or negative outcome. With enough positive statements, the block is backed and included in the relay chain, but is still pending approval. | ||
|
|
||
| **Inclusion**: Validators gossip [erasure code chunks]() and put the parablock through the final [approval process]() before a parablock is considered *included* in the relay chain. | ||
|
|
||
| ### Sync VS. Async | ||
|
|
||
| In the synchronous scenario, both the collators and validators draw context from the Relay Parent of the prior parablock, which lives on the relay chain. This makes the Backing and Generation steps tightly coupled to the prior parablock completing the inclusion pipeline. As a result, one parablock can be processed every other relay blocks. | ||
|
|
||
| ```mermaid | ||
| --- | ||
| displayMode: compact | ||
| config: | ||
| themeCSS: " | ||
| #item1 { fill: #4f2eb7; stroke: #000; } \n | ||
| #item2 { fill: #ffc62a; stroke: #000; } \n | ||
| #item3 { fill: #4faff9; stroke: #000; } \n | ||
| #r { fill: #eb4172; stroke: #000; } \n | ||
| #p1padTop { fill:none;stroke:none;opacity:0; } \n | ||
| text.taskText[id^=p1padTop] { fill: none !important; color: #000 !important; } \n | ||
|
|
||
|
|
||
| /* Default inside task text: white */ \n | ||
| .taskText { fill: #fff !important; color: #fff !important; font-weight: 700; font-size: 18px; } \n | ||
|
|
||
| /* Outside labels and section titles: black */ \n | ||
| .taskTextOutside, .sectionTitle { fill: #000 !important; color: #000 !important; font-weight: 700; font-size: 18px; } \n | ||
|
|
||
| /* Inside text for #r items: black */ \n | ||
| text.taskText[id^=r] { fill: #fff !important; color: #000 !important; } \n | ||
| " | ||
| themeVariables: | ||
| sectionBkgColor: '#fff' | ||
| gantt: | ||
| numberSectionStyles: 1 | ||
| barHeight: 70 | ||
| gridLineStartPadding: 100 | ||
| --- | ||
| %%{init: {"gantt": {"barHeight": 70 }}}%% | ||
| gantt | ||
| dateFormat YYYY | ||
| axisFormat %y | ||
| %% this next line doesn't recognise 'decade' or 'year', but will silently ignore | ||
| tickInterval '10year' | ||
|
|
||
| R1 : r, 1905, 1907 | ||
| R2 : r, 1911, 1913 | ||
| R3 : r, 1917, 1919 | ||
| R4 : r, 1923, 1925 | ||
|
|
||
| SPACING : p1padTop, 1905, 1907 | ||
| SPACING : p1padTop, 1911, 1913 | ||
| SPACING : p1padTop, 1917, 1919 | ||
| SPACING : p1padTop, 1923, 1925 | ||
|
|
||
| section P1 | ||
| X : item1, 1900, 1901 | ||
| Backing : item2, 1901, 1906 | ||
| Inclusion : item3, 1906, 1912 | ||
|
|
||
| section P2 | ||
| X : item1, 1912, 1913 | ||
| Backing : item2, 1913, 1918 | ||
| Inclusion : item3, 1918, 1924 | ||
| ``` | ||
|
|
||
| In the asynchronous scenario, where both the collators and validators have access to Unincluded Segments as an additional context source, the Backing and Generation steps are no longer coupled to the prior block completing the full inclusion pipeline. Instead, the prior parablock only needs to complete the generation step and be added to the Unincluded Segments before the next parablock can begin the Backing and Generation steps. | ||
|
|
||
| This results in one parablock being processed *every* relay block, and allows for more time to execute during the Generation step (0.5s --> 2s). | ||
|
|
||
| ```mermaid | ||
| --- | ||
| displayMode: compact | ||
| config: | ||
| themeCSS: " | ||
| #item1 { fill: #4f2eb7; stroke: #000; } \n | ||
| #item2 { fill: #ffc62a; stroke: #000; } \n | ||
| #item3 { fill: #4faff9; stroke: #000; } \n | ||
| #r { fill: #eb4172; stroke: #000; } \n | ||
| #p1padTop { fill:none;stroke:none;opacity:0; } \n | ||
| text.taskText[id^=p1padTop] { fill: none !important; color: #000 !important; } \n | ||
|
|
||
|
|
||
| /* Default inside task text: white */ \n | ||
| .taskText { fill: #fff !important; color: #fff !important; font-weight: 700; font-size: 18px; } \n | ||
|
|
||
| /* Outside labels and section titles: black */ \n | ||
| .taskTextOutside, .sectionTitle { fill: #000 !important; color: #000 !important; font-weight: 700; font-size: 18px; } \n | ||
|
|
||
| /* Inside text for #r items: black */ \n | ||
| text.taskText[id^=r] { fill: #fff !important; } \n | ||
| " | ||
| themeVariables: | ||
| sectionBkgColor: '#fff' | ||
| gantt: | ||
| numberSectionStyles: 1 | ||
| barHeight: 70 | ||
| gridLineStartPadding: 100 | ||
| --- | ||
| %%{init: {"gantt": {"barHeight": 70 }}}%% | ||
| gantt | ||
| dateFormat YYYY | ||
| axisFormat %y | ||
| %% this next line doesn't recognise 'decade' or 'year', but will silently ignore | ||
| tickInterval '10year' | ||
|
|
||
| R1 : r, 1905, 1907 | ||
| R2 : r, 1911, 1913 | ||
| R3 : r, 1917, 1919 | ||
| R4 : r, 1923, 1925 | ||
| R5 : r, 1929, 1931 | ||
|
|
||
| SPACING : p1padTop, 1905, 1907 | ||
| SPACING : p1padTop, 1911, 1913 | ||
| SPACING : p1padTop, 1917, 1919 | ||
| SPACING : p1padTop, 1923, 1925 | ||
| SPACING : p1padTop, 1929, 1931 | ||
|
|
||
| section P1 | ||
| X : item1, 1900, 1902 | ||
| Backing : item2, 1902, 1912 | ||
| Inclusion : item3, 1912, 1918 | ||
|
|
||
| section P2 | ||
| X : item1, 1906, 1908 | ||
| Backing : item2, 1908, 1918 | ||
| Inclusion : item3, 1918, 1924 | ||
|
|
||
| section P3 | ||
| X : item1, 1912, 1914 | ||
| Backing : item2, 1914, 1924 | ||
| Inclusion : item3, 1924, 1930 | ||
|
|
||
| section P4 | ||
| X : item1, 1918, 1920 | ||
| Backing : item2, 1920, 1930 | ||
| ``` | ||
|
|
||
| ### Compute Advantage | ||
| Below is a table showing the main advantages of asynchronous over synchronous backing. | ||
|
|
||
| | | Sync Backing | Async Backing | Async Backing Advantage | | ||
| | ------------------------------------ | ------------ | ------------ | ----------------------------------------- | | ||
| | **Parablocks included every** | 12 seconds | 6 seconds | **2x** more parablocks included | | ||
| | **Parablock maximum execution time** | 0.5 seconds | 2 seconds | **4x** more execution time in a parablock | | ||
| | **Total Computer Gain (per core)** | | | **8x Compute Throughput** | | ||
|
|
||
| ## Configurations | ||
| The following configurations can be set by onchain governance: | ||
|
|
||
| * [`max_candidate_depth`](https://github.com/paritytech/polkadot-sdk/blob/f204e3264f945c33b4cea18a49f7232c180b07c5/polkadot/primitives/src/vstaging/mod.rs#L49): the number of parachain blocks a collator can produce that are not yet included in the relay chain. | ||
| * [`allowed_ancestry_len`](https://github.com/paritytech/polkadot-sdk/blob/f204e3264f945c33b4cea18a49f7232c180b07c5/polkadot/primitives/src/vstaging/mod.rs#L54): the oldest relay chain parent a parachain block can be built on top of. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.