cmd/dbcmd: add inspect trie tool to analysis trie storage #28892

fynnss · 2024-01-29T07:46:35Z

This pr adds a tool names inpsect-trie, aimed to analyze the mpt and its node storage more efficiently.

Example

./geth db inspect-trie --datadir server/data-seed/ latest 4000

Result

MPT shape
Account Trie
Top N Storage Trie

+-------+-------+--------------+-------------+--------------+
|   -   | LEVEL | SHORTNODECNT | FULLNODECNT | VALUENODECNT |
+-------+-------+--------------+-------------+--------------+
|   -   |   0   |      0       |      1      |      0       |
|   -   |   1   |      0       |     16      |      0       |
|   -   |   2   |      76      |     32      |      74      |
|   -   |   3   |      66      |      1      |      66      |
|   -   |   4   |      2       |      0      |      2       |
| Total |  144  |      50      |     142     |
+-------+-------+--------------+-------------+--------------+
AccountTrie
+-------+-------+--------------+-------------+--------------+
|   -   | LEVEL | SHORTNODECNT | FULLNODECNT | VALUENODECNT |
+-------+-------+--------------+-------------+--------------+
|   -   |   0   |      0       |      1      |      0       |
|   -   |   1   |      0       |     16      |      0       |
|   -   |   2   |     108      |     84      |     104      |
|   -   |   3   |     195      |      5      |     195      |
|   -   |   4   |      10      |      0      |      10      |
| Total |  313  |     106      |     309     |
+-------+-------+--------------+-------------+--------------+
ContractTrie-0xc874e65ccffb133d9db4ff637e62532ef6ecef3223845d02f522c55786782911
+-------+-------+--------------+-------------+--------------+
|   -   | LEVEL | SHORTNODECNT | FULLNODECNT | VALUENODECNT |
+-------+-------+--------------+-------------+--------------+
|   -   |   0   |      0       |      1      |      0       |
|   -   |   1   |      0       |     16      |      0       |
|   -   |   2   |      57      |     14      |      56      |
|   -   |   3   |      33      |      0      |      33      |
| Total |  90   |      31      |     89      |
+-------+-------+--------------+-------------+--------------+
ContractTrie-0x1d7dcb6a0ce5227c5379fc5b0e004561d7833b063355f69bfea3178f08fbaab4
+-------+-------+--------------+-------------+--------------+
|   -   | LEVEL | SHORTNODECNT | FULLNODECNT | VALUENODECNT |
+-------+-------+--------------+-------------+--------------+
|   -   |   0   |      0       |      1      |      0       |
|   -   |   1   |      5       |      8      |      5       |
|   -   |   2   |      16      |      1      |      16      |
|   -   |   3   |      2       |      0      |      2       |
| Total |  23   |      10      |     23      |
+-------+-------+--------------+-------------+--------------+
ContractTrie-0xaa8a4783ebbb3bec45d3e804b3c59bfd486edfa39cbeda1d42bf86c08a0ebc0f
+-------+-------+--------------+-------------+--------------+
|   -   | LEVEL | SHORTNODECNT | FULLNODECNT | VALUENODECNT |
+-------+-------+--------------+-------------+--------------+
|   -   |   0   |      0       |      1      |      0       |
|   -   |   1   |      9       |      3      |      9       |
|   -   |   2   |      7       |      1      |      7       |
|   -   |   3   |      2       |      0      |      2       |
| Total |  18   |      5       |     18      |
+-------+-------+--------------+-------------+--------------+
ContractTrie-0x9d2804d0562391d7cfcfaf0013f0352e176a94403a58577ebf82168a21514441
+-------+-------+--------------+-------------+--------------+
|   -   | LEVEL | SHORTNODECNT | FULLNODECNT | VALUENODECNT |
+-------+-------+--------------+-------------+--------------+
|   -   |   0   |      0       |      1      |      0       |
|   -   |   1   |      6       |      4      |      6       |
|   -   |   2   |      8       |      0      |      8       |
| Total |  14   |      5       |     14      |
+-------+-------+--------------+-------------+--------------+
ContractTrie-0x17e3eb95d0e6e92b42c0b3e95c6e75080c9fcd83e706344712e9587375de96e1
+-------+-------+--------------+-------------+--------------+
|   -   | LEVEL | SHORTNODECNT | FULLNODECNT | VALUENODECNT |
+-------+-------+--------------+-------------+--------------+
|   -   |   0   |      0       |      1      |      0       |
|   -   |   1   |      5       |      3      |      5       |
|   -   |   2   |      7       |      0      |      7       |
| Total |  12   |      4       |     12      |
+-------+-------+--------------+-------------+--------------+
ContractTrie-0xc017ca90c8aa37693c38f80436bb15bde46d7b30a503aa808cb7814127468a44
Contract Trie, total trie num: 142, ShortNodeCnt: 620, FullNodeCnt: 204, ValueNodeCnt: 615

MariusVanDerWijden · 2024-01-30T08:09:07Z

This PR on mainnet:

+-------+-------+--------------+-------------+--------------+
|   -   | LEVEL | SHORTNODECNT | FULLNODECNT | VALUENODECNT |
+-------+-------+--------------+-------------+--------------+
|   -   |   0   |      0       |      1      |      0       |
|   -   |   1   |      0       |     16      |      0       |
|   -   |   2   |      0       |     256     |      0       |
|   -   |   3   |      0       |    4096     |      0       |
|   -   |   4   |      0       |    65536    |      0       |
|   -   |   5   |      0       |   1048576   |      0       |
|   -   |   6   |     438      |  16776930   |     263      |
|   -   |   7   |  100144570   |  56122080   |   97505275   |
|   -   |   8   |  124204988   |   3221357   |  124000341   |
|   -   |   9   |   6480231    |    10439    |   6479228    |
|   -   |  10   |    20884     |      2      |    20883     |
|   -   |  11   |      4       |      0      |      4       |
| Total |   -   |  230851115   |  77249289   |  228005994   |
+-------+-------+--------------+-------------+--------------+
AccountTrie
EOA accounts num:  190618297
+-------+-------+--------------+-------------+--------------+
|   -   | LEVEL | SHORTNODECNT | FULLNODECNT | VALUENODECNT |
+-------+-------+--------------+-------------+--------------+
|   -   |   0   |      0       |      1      |      0       |
|   -   |   1   |      0       |     16      |      0       |
|   -   |   2   |      0       |     256     |      0       |
|   -   |   3   |      0       |    4096     |      0       |
|   -   |   4   |      0       |    65536    |      0       |
|   -   |   5   |      0       |   1048576   |      0       |
|   -   |   6   |   2632166    |  13611933   |   2388687    |
|   -   |   7   |   40897826   |   4112954   |   40656890   |
|   -   |   8   |   8404095    |    48282    |   8401135    |
|   -   |   9   |    96671     |     48      |    96668     |
|   -   |  10   |      96      |      0      |      96      |
| Total |   -   |   52030854   |  18891698   |   51543476   |
+-------+-------+--------------+-------------+--------------+
ContractTrie-0x159e4890cde46d5cadbfcd08a052f495ae791244728e06d910cc743ed05eae12
+-------+-------+--------------+-------------+--------------+
|   -   | LEVEL | SHORTNODECNT | FULLNODECNT | VALUENODECNT |
+-------+-------+--------------+-------------+--------------+
|   -   |   0   |      0       |      1      |      0       |
|   -   |   1   |      0       |     16      |      0       |
|   -   |   2   |      0       |     256     |      0       |
|   -   |   3   |      0       |    4096     |      0       |
|   -   |   4   |      0       |    65536    |      0       |
|   -   |   5   |      0       |   1048576   |      0       |
|   -   |   6   |   6267714    |   5853598   |   6029411    |
|   -   |   7   |   13620075   |   519213    |   13588531   |
|   -   |   8   |   1047203    |    2323     |   1047042    |
|   -   |   9   |     4648     |      0      |     4648     |
| Total |   -   |   20939640   |   7493615   |   20669632   |
+-------+-------+--------------+-------------+--------------+
ContractTrie-0xfe1c2c3bf003e59420de2a964984544a947ac6de636a2dedb89b689ab278b65e
+-------+-------+--------------+-------------+--------------+
|   -   | LEVEL | SHORTNODECNT | FULLNODECNT | VALUENODECNT |
+-------+-------+--------------+-------------+--------------+
|   -   |   0   |      0       |      1      |      0       |
|   -   |   1   |      0       |     16      |      0       |
|   -   |   2   |      0       |     256     |      0       |
|   -   |   3   |      0       |    4096     |      0       |
|   -   |   4   |      0       |    65536    |      0       |
|   -   |   5   |      5       |   1048574   |      2       |
|   -   |   6   |   6360087    |   4255479   |   6169042    |
|   -   |   7   |   9567427    |   284426    |   9549861    |
|   -   |   8   |    572674    |    1007     |    572609    |
|   -   |   9   |     2016     |      0      |     2016     |
| Total |   -   |   16502209   |   5659391   |   16293530   |
+-------+-------+--------------+-------------+--------------+
ContractTrie-0xfbd1592b616670c3c966b8ed9942aecdb2b25ab10837f7d1c8d1e66c10f243a8
+-------+-------+--------------+-------------+--------------+
|   -   | LEVEL | SHORTNODECNT | FULLNODECNT | VALUENODECNT |
+-------+-------+--------------+-------------+--------------+
|   -   |   0   |      0       |      1      |      0       |
|   -   |   1   |      0       |     16      |      0       |
|   -   |   2   |      0       |     256     |      0       |
|   -   |   3   |      0       |    4096     |      0       |
|   -   |   4   |      0       |    65536    |      0       |
|   -   |   5   |      10      |   1048568   |      8       |
|   -   |   6   |   6345222    |   4047200   |   6161871    |
|   -   |   7   |   9060682    |   259799    |   9044727    |
|   -   |   8   |    522892    |     917     |    522841    |
|   -   |   9   |     1834     |      0      |     1834     |
| Total |   -   |   15930640   |   5426389   |   15731281   |
+-------+-------+--------------+-------------+--------------+
ContractTrie-0x4e6f19bdf5c8f4aa5e46a382f33fbed981bd7d93c25885c5ecdabfdc102f8b67
+-------+-------+--------------+-------------+--------------+
|   -   | LEVEL | SHORTNODECNT | FULLNODECNT | VALUENODECNT |
+-------+-------+--------------+-------------+--------------+
|   -   |   0   |      0       |      1      |      0       |
|   -   |   1   |      0       |     16      |      0       |
|   -   |   2   |      0       |     256     |      0       |
|   -   |   3   |      0       |    4096     |      0       |
|   -   |   4   |      0       |    65536    |      0       |
|   -   |   5   |      53      |   1048541   |      31      |
|   -   |   6   |   6191387    |   3228255   |   6037188    |
|   -   |   7   |   7104178    |   173532    |   7093403    |
|   -   |   8   |    348985    |     493     |    348948    |
|   -   |   9   |     986      |      0      |     986      |
| Total |   -   |   13645589   |   4520726   |   13480556   |
+-------+-------+--------------+-------------+--------------+
ContractTrie-0x9f13f88230a70de90ed5fa41ba35a5fb78bc55d11cc9406f17d314fb67047ac7

fynnss · 2024-01-30T09:35:39Z

Nice job! @MariusVanDerWijden

Currently, only the hash of the contract address can be printed. The contract address can be obtained through the preimage.

cmd/geth/dbcmd.go

trie/inspect_trie.go

trie/trie.go

trie/inspect_trie.go

MariusVanDerWijden · 2025-07-22T08:47:16Z

Still getting killed with OOM for me...

Inspected 3834181111 trie nodes in 10h8m32.000775427s
Inspected 3834883531 trie nodes in 10h8m40.000154241s
+-------+-------+--------------+-------------+--------------+
|   -   | LEVEL | SHORTNODECNT | FULLNODECNT | VALUENODECNT |
+-------+-------+--------------+-------------+--------------+
|   -   |   0   |      0       |      1      |      0       |
|   -   |   1   |      0       |     16      |      0       |
|   -   |   2   |      0       |     256     |      0       |
|   -   |   3   |      0       |    4096     |      0       |
|   -   |   4   |      0       |    65536    |      0       |
|   -   |   5   |      0       |   1048576   |      0       |
|   -   |   6   |      8       |  16777213   |      3       |
|   -   |   7   |  101606337   |  82780951   |   98088636   |
|   -   |   8   |  189771140   |   6555308   |  189363030   |
|   -   |   9   |   13212109   |    27657    |   13209879   |
|   -   |  10   |    55340     |     10      |    55337     |
|   -   |  11   |      20      |      0      |      20      |
| Total |   -   |  304644954   |  107259620  |  300716905   |
+-------+-------+--------------+-------------+--------------+
AccountTrie
``

Co-authored-by: Fynn <zcheng1004@gmail.com> Co-authored-by: lightclient <lightclient@protonmail.com> Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>

gballet

I left a few comments. Regarding verkle/binary, I can add support for it later.

gballet · 2025-10-27T13:40:29Z

trie/inspect.go

+// triestat tracks the type and count of trie nodes at each level in the trie.
+//
+// Note: theoretically it is possible to have up to 64 trie level. Since it is
+// unlikely to encounter such a large trie, the stats are capped at 16 levels to
+// avoid substantial unneeded allocation.
+type triestat struct {
+	level [16]stat
+}


there is a similar structure in core/stateless/stats, it should be merged.

Let me look into this.

gballet · 2025-10-27T13:46:14Z

core/rawdb/database.go

 	}

-	table := newTableWriter(os.Stdout)
+	table := tablewriter.NewWriter(os.Stdout)


I don't see the point of moving this to its own package, the tree writer is only used in trie, which is the one importing rawdb.

I moved it to it's own package because it isn't exported from rawdb. It felt wrong to expose it publicly in that package.

holiman reviewed Jan 31, 2024

View reviewed changes

fynnss force-pushed the inspect-trie branch 2 times, most recently from 3bdeec1 to 1581066 Compare January 31, 2024 11:48

fynnss requested a review from holiman January 31, 2024 13:00

holiman suggested changes Feb 6, 2024

View reviewed changes

fynnss force-pushed the inspect-trie branch from c7bb1af to 67a49a1 Compare February 7, 2024 01:52

fynnss force-pushed the inspect-trie branch from 67a49a1 to 36b7785 Compare February 18, 2024 07:13

fynnss requested a review from holiman February 18, 2024 07:14

MariusVanDerWijden force-pushed the inspect-trie branch from 36b7785 to 1649310 Compare June 27, 2025 08:37

MariusVanDerWijden requested a review from rjl493456442 as a code owner June 27, 2025 08:37

lightclient self-assigned this Sep 24, 2025

trie: add inspector

f651600

Co-authored-by: Fynn <zcheng1004@gmail.com> Co-authored-by: lightclient <lightclient@protonmail.com> Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>

lightclient force-pushed the inspect-trie branch from b66c822 to f651600 Compare September 24, 2025 01:55

lightclient requested a review from gballet as a code owner September 24, 2025 01:55

core/rawdb: move tablewriter re-export to internal package

2f238ab

gballet self-assigned this Sep 29, 2025

gballet added this to the 1.16.6 milestone Oct 16, 2025

lightclient and others added 4 commits October 20, 2025 19:01

trie: refactor inspect

e2308aa

cmd/geth: fix lint

ec5831d

cmd/geth: fix lint

7a9c1dc

trie,cmd: improve results display

a22690d

gballet requested changes Oct 27, 2025

View reviewed changes

fjl modified the milestones: 1.16.6, 1.16.8 Nov 4, 2025

cmd/dbcmd: add inspect trie tool to analysis trie storage #28892

Are you sure you want to change the base?

cmd/dbcmd: add inspect trie tool to analysis trie storage #28892

Conversation

fynnss commented Jan 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example

Result

Uh oh!

MariusVanDerWijden commented Jan 30, 2024

Uh oh!

fynnss commented Jan 30, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MariusVanDerWijden commented Jul 22, 2025

Uh oh!

gballet left a comment

Choose a reason for hiding this comment

Uh oh!

gballet Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

lightclient Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

gballet Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

lightclient Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

fynnss commented Jan 29, 2024 •

edited

Loading