-
Notifications
You must be signed in to change notification settings - Fork 14
chore: BlockAccessors Hardening & Improved Error Handling
#1833
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: BlockAccessors Hardening & Improved Error Handling
#1833
Conversation
5e07b85 to
3617276
Compare
3617276 to
e75166d
Compare
...block-access/src/main/java/org/hiero/block/node/access/service/BlockAccessServicePlugin.java
Show resolved
Hide resolved
...s.historic/src/main/java/org/hiero/block/node/blocks/files/historic/FilesHistoricConfig.java
Show resolved
Hide resolved
e75166d to
db09b26
Compare
|
We could also add tests for:
|
db09b26 to
1e574a7
Compare
mustafauzunn
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to add the new paths to block-node/app/build.gradle.kts to not break the local runs
Rebase and removal of link-based changes is complete.
ata-nas
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jsync-swirlds thanks for the help here while I am on PTO. I leave some cleanup suggestions.
...storic/src/main/java/org/hiero/block/node/blocks/files/historic/BlockFileHistoricPlugin.java
Outdated
Show resolved
Hide resolved
block-node/base/src/main/java/org/hiero/block/node/base/tar/TaredBlockIterator.java
Outdated
Show resolved
Hide resolved
...storic/src/main/java/org/hiero/block/node/blocks/files/historic/BlockFileHistoricPlugin.java
Show resolved
Hide resolved
...iles.historic/src/main/java/org/hiero/block/node/blocks/files/historic/ZipBlockAccessor.java
Outdated
Show resolved
Hide resolved
...files.historic/src/main/java/org/hiero/block/node/blocks/files/historic/ZipBlockArchive.java
Outdated
Show resolved
Hide resolved
...ic/src/test/java/org/hiero/block/node/blocks/files/historic/BlockFileHistoricPluginTest.java
Outdated
Show resolved
Hide resolved
...ic/src/test/java/org/hiero/block/node/blocks/files/historic/BlockFileHistoricPluginTest.java
Outdated
Show resolved
Hide resolved
....historic/src/test/java/org/hiero/block/node/blocks/files/historic/ZipBlockAccessorTest.java
Show resolved
Hide resolved
fe24e5c to
1f955ea
Compare
mustafauzunn
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Timeout for e2e tests.
and also
java.lang.ClassCastException: class sun.nio.fs.UnixPath cannot be cast to class com.google.common.jimfs.JimfsPath (sun.nio.fs.UnixPath is in module java.base of loader 'bootstrap'; com.google.common.jimfs.JimfsPath is in module com.google.common.jimfs@1.3.1 of loader 'app')
at org.hiero.block.node.blocks.files.historic@0.24.0-SNAPSHOT/org.hiero.block.node.blocks.files.historic.ZipBlockArchiveTest$FunctionalTests.testMinStoredSingleZipFile(ZipBlockArchiveTest.java:168)
for ZipBlockArchiveTest tests
Yes, I am working on this error. It's quite strange as it only happens from command line; in IDE it does not happen (so it cannot be debugged). The above error was, indeed, related to erasure and JimFS not fully capturing the FileSystem structures. I switched back to using hard links, as a result. |
1f955ea to
2865e7a
Compare
2865e7a to
e8a8774
Compare
...iles.historic/src/main/java/org/hiero/block/node/blocks/files/historic/ZipBlockAccessor.java
Show resolved
Hide resolved
...recent/src/test/java/org/hiero/block/node/blocks/files/recent/BlockFileRecentPluginTest.java
Show resolved
Hide resolved
block-node/spi/src/main/java/org/hiero/block/node/spi/historicalblocks/BlockAccessor.java
Show resolved
Hide resolved
...es.recent/src/main/java/org/hiero/block/node/blocks/files/recent/BlockFileBlockAccessor.java
Outdated
Show resolved
Hide resolved
...iles.historic/src/main/java/org/hiero/block/node/blocks/files/historic/ZipBlockAccessor.java
Show resolved
Hide resolved
...storic/src/main/java/org/hiero/block/node/blocks/files/historic/BlockFileHistoricPlugin.java
Show resolved
Hide resolved
|
@jsync-swirlds an idea for unit tests to be added to accessors if we do not have such: create a block and persist it, issue an accessor - it will create hard link, delete the actual block file, make sure the accessor can still read the data. Good to have for all (production) accessor types we have. This should be very easy to do, as we have tons of unit tests, simple copy of the closes test we have and a slight modification to it should do the trick. |
I would like to add these in a follow-up PR; this one is already getting bloated and taking too long. |
aa81ffa to
feca78b
Compare
Nana-EC
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice changes. A lot to introduce.
I did 1 good pass but would like to circle back again on a few key areas based on responses
block-node/base/src/main/java/org/hiero/block/node/base/tar/TaredBlockIterator.java
Show resolved
Hide resolved
block-node/app/src/testFixtures/java/org/hiero/block/node/app/fixtures/blocks/BlockUtils.java
Show resolved
Hide resolved
...ic/src/test/java/org/hiero/block/node/blocks/files/historic/BlockFileHistoricPluginTest.java
Show resolved
Hide resolved
...s.historic/src/test/java/org/hiero/block/node/blocks/files/historic/ZipBlockArchiveTest.java
Show resolved
Hide resolved
block-node/spi/src/main/java/org/hiero/block/node/spi/historicalblocks/BlockAccessorBatch.java
Show resolved
Hide resolved
* BlockAccessors are now Autocloseable
* BlockAccessors now create hard links for the duration of their
lifespan
* This allows us to ensure data is accessible for the duration of the
accessor's life
* Tests are updated and migrated
* Deleted redundant configuration tests
* Added tests:
* Tests for subsequent reads to ensure that closing an accessor does
not affect the data or the ability for a new accessor to access it
* Added a recursive deletion visitor
* Introduced a BlockAccessorBatch
* Added reuse of `ZipFileSystem` instances to cut down on
repeated reread of large zip files and dictionaries.
* Adjusted block accessor close semantics to avoid double-close.
* Adjusted BlockFile to use a visitor for some file tree walks.
Signed-off-by: Atanas Atanasov <a.v.atanasov98@gmail.com>
Signed-off-by: Joseph S. <121976561+jsync-swirlds@users.noreply.github.com>
feca78b to
e205eca
Compare
Nana-EC
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unblocking per comments.
There's a lot of follow ups to improve confidence but the threshold for now seems good.
AlfredoG87
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lot of changes and improvements, LGTM 👍
lifespan
accessor's life
not affect the data or the ability for a new accessor to access it
Reviewer Notes
Related Issue(s)
Resolves #1273