Skip to content

Conversation

@ata-nas
Copy link
Contributor

@ata-nas ata-nas commented Nov 5, 2025

  • BlockAccessors are now Autocloseable
  • BlockAccessors now create hard links for the duration of their
    lifespan
    • This allows us to ensure data is accessible for the duration of the
      accessor's life
  • Tests are updated and migrated
  • Deleted redundant configuration tests
  • Added tests:
    • Tests for subsequent reads to ensure that closing an accessor does
      not affect the data or the ability for a new accessor to access it

Reviewer Notes

  • migrating accessors to an autocloseable variant
  • improving the ability to retrieve data by having hard links

Related Issue(s)

Resolves #1273

@ata-nas ata-nas added this to the 0.23.0 milestone Nov 5, 2025
@ata-nas ata-nas self-assigned this Nov 5, 2025
@ata-nas ata-nas added the pull request label to get past the "label required" check when no label is needed or appropriate. label Nov 5, 2025
@ata-nas ata-nas linked an issue Nov 5, 2025 that may be closed by this pull request
@ata-nas ata-nas force-pushed the 1273-blockaccessors-hardening-improved-error-handling branch 12 times, most recently from 5e07b85 to 3617276 Compare November 6, 2025 15:12
@ata-nas ata-nas marked this pull request as ready for review November 6, 2025 15:12
@ata-nas ata-nas requested review from a team as code owners November 6, 2025 15:12
@ata-nas ata-nas force-pushed the 1273-blockaccessors-hardening-improved-error-handling branch from 3617276 to e75166d Compare November 6, 2025 15:13
@ata-nas ata-nas force-pushed the 1273-blockaccessors-hardening-improved-error-handling branch from e75166d to db09b26 Compare November 6, 2025 15:31
@ata-nas
Copy link
Contributor Author

ata-nas commented Nov 6, 2025

We could also add tests for:

  1. Clearing the links root path on startup for recents
  2. Clearing the links root path on startup for historic

@ata-nas ata-nas force-pushed the 1273-blockaccessors-hardening-improved-error-handling branch from db09b26 to 1e574a7 Compare November 6, 2025 15:37
Copy link
Contributor

@mustafauzunn mustafauzunn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to add the new paths to block-node/app/build.gradle.kts to not break the local runs

@jsync-swirlds jsync-swirlds dismissed their stale review November 14, 2025 23:02

Rebase and removal of link-based changes is complete.

Copy link
Contributor Author

@ata-nas ata-nas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsync-swirlds thanks for the help here while I am on PTO. I leave some cleanup suggestions.

@jsync-swirlds jsync-swirlds force-pushed the 1273-blockaccessors-hardening-improved-error-handling branch 2 times, most recently from fe24e5c to 1f955ea Compare November 18, 2025 03:27
Copy link
Contributor

@mustafauzunn mustafauzunn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Timeout for e2e tests.
and also

java.lang.ClassCastException: class sun.nio.fs.UnixPath cannot be cast to class com.google.common.jimfs.JimfsPath (sun.nio.fs.UnixPath is in module java.base of loader 'bootstrap'; com.google.common.jimfs.JimfsPath is in module com.google.common.jimfs@1.3.1 of loader 'app')
        at org.hiero.block.node.blocks.files.historic@0.24.0-SNAPSHOT/org.hiero.block.node.blocks.files.historic.ZipBlockArchiveTest$FunctionalTests.testMinStoredSingleZipFile(ZipBlockArchiveTest.java:168)

for ZipBlockArchiveTest tests

@jsync-swirlds
Copy link
Contributor

jsync-swirlds commented Nov 18, 2025

Timeout for e2e tests. and also

java.lang.ClassCastException: class sun.nio.fs.UnixPath cannot be cast to class com.google.common.jimfs.JimfsPath (sun.nio.fs.UnixPath is in module java.base of loader 'bootstrap'; com.google.common.jimfs.JimfsPath is in module com.google.common.jimfs@1.3.1 of loader 'app')
        at org.hiero.block.node.blocks.files.historic@0.24.0-SNAPSHOT/org.hiero.block.node.blocks.files.historic.ZipBlockArchiveTest$FunctionalTests.testMinStoredSingleZipFile(ZipBlockArchiveTest.java:168)

for ZipBlockArchiveTest tests

Yes, I am working on this error. It's quite strange as it only happens from command line; in IDE it does not happen (so it cannot be debugged).
It appears, so far, to be related to containers and Erasure.

The above error was, indeed, related to erasure and JimFS not fully capturing the FileSystem structures.
Fixing that, however, uncovered an inability to hold files open with JimFS. This led to enough complexity that it became clear the "open zip files" approach could not work in the short term.

I switched back to using hard links, as a result.

@jsync-swirlds jsync-swirlds force-pushed the 1273-blockaccessors-hardening-improved-error-handling branch from 1f955ea to 2865e7a Compare November 21, 2025 02:28
@jsync-swirlds jsync-swirlds force-pushed the 1273-blockaccessors-hardening-improved-error-handling branch from 2865e7a to e8a8774 Compare November 21, 2025 22:50
@ata-nas
Copy link
Contributor Author

ata-nas commented Nov 24, 2025

@jsync-swirlds an idea for unit tests to be added to accessors if we do not have such: create a block and persist it, issue an accessor - it will create hard link, delete the actual block file, make sure the accessor can still read the data. Good to have for all (production) accessor types we have. This should be very easy to do, as we have tons of unit tests, simple copy of the closes test we have and a slight modification to it should do the trick.

@jsync-swirlds
Copy link
Contributor

@jsync-swirlds an idea for unit tests to be added to accessors if we do not have such: create a block and persist it, issue an accessor - it will create hard link, delete the actual block file, make sure the accessor can still read the data. Good to have for all (production) accessor types we have. This should be very easy to do, as we have tons of unit tests, simple copy of the closes test we have and a slight modification to it should do the trick.

I would like to add these in a follow-up PR; this one is already getting bloated and taking too long.

@jsync-swirlds jsync-swirlds force-pushed the 1273-blockaccessors-hardening-improved-error-handling branch 2 times, most recently from aa81ffa to feca78b Compare November 25, 2025 01:58
Copy link
Contributor

@Nana-EC Nana-EC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice changes. A lot to introduce.
I did 1 good pass but would like to circle back again on a few key areas based on responses

* BlockAccessors are now Autocloseable
* BlockAccessors now create hard links for the duration of their
  lifespan
  * This allows us to ensure data is accessible for the duration of the
    accessor's life
* Tests are updated and migrated
* Deleted redundant configuration tests
* Added tests:
  * Tests for subsequent reads to ensure that closing an accessor does
    not affect the data or the ability for a new accessor to access it
* Added a recursive deletion visitor
* Introduced a BlockAccessorBatch
* Added reuse of `ZipFileSystem` instances to cut down on
  repeated reread of large zip files and dictionaries.
* Adjusted block accessor close semantics to avoid double-close.
* Adjusted BlockFile to use a visitor for some file tree walks.

Signed-off-by: Atanas Atanasov <a.v.atanasov98@gmail.com>
Signed-off-by: Joseph S. <121976561+jsync-swirlds@users.noreply.github.com>
@jsync-swirlds jsync-swirlds force-pushed the 1273-blockaccessors-hardening-improved-error-handling branch from feca78b to e205eca Compare November 25, 2025 23:06
Copy link
Contributor

@Nana-EC Nana-EC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unblocking per comments.
There's a lot of follow ups to improve confidence but the threshold for now seems good.

Copy link
Contributor

@AlfredoG87 AlfredoG87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lot of changes and improvements, LGTM 👍

@jsync-swirlds jsync-swirlds merged commit 2c8ddf1 into main Nov 26, 2025
11 of 14 checks passed
@jsync-swirlds jsync-swirlds deleted the 1273-blockaccessors-hardening-improved-error-handling branch November 26, 2025 00:23
@codecov
Copy link

codecov bot commented Nov 26, 2025

Codecov Report

❌ Patch coverage is 66.34921% with 106 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...blocks/files/historic/BlockFileHistoricPlugin.java 65.16% 24 Missing and 7 partials ⚠️
...k/node/blocks/files/historic/ZipBlockAccessor.java 55.26% 13 Missing and 4 partials ⚠️
...g/hiero/block/node/archive/s3/S3ArchivePlugin.java 65.95% 12 Missing and 4 partials ⚠️
...ode/blocks/files/recent/BlockFileRecentPlugin.java 51.72% 13 Missing and 1 partial ⚠️
...tream/subscriber/BlockStreamSubscriberSession.java 46.66% 6 Missing and 2 partials ⚠️
...de/blocks/files/recent/BlockFileBlockAccessor.java 71.42% 6 Missing ⚠️
.../node/access/service/BlockAccessServicePlugin.java 44.44% 3 Missing and 2 partials ⚠️
...main/java/org/hiero/block/node/base/BlockFile.java 87.09% 1 Missing and 3 partials ⚠️
...ro/block/node/blocks/files/historic/BlockPath.java 33.33% 2 Missing ⚠️
...ck/node/blocks/files/historic/ZipBlockArchive.java 66.66% 2 Missing ⚠️
... and 1 more

❌ Your patch status has failed because the patch coverage (66.34%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

@@             Coverage Diff              @@
##               main    #1833      +/-   ##
============================================
- Coverage     80.72%   79.55%   -1.18%     
- Complexity     1178     1195      +17     
============================================
  Files           127      128       +1     
  Lines          5557     5684     +127     
  Branches        591      610      +19     
============================================
+ Hits           4486     4522      +36     
- Misses          800      885      +85     
- Partials        271      277       +6     
Files with missing lines Coverage Δ Complexity Δ
...locks/files/historic/BlockStagingFileAccessor.java 44.73% <100.00%> (-5.27%) 6.00 <2.00> (ø)
...ode/blocks/files/historic/FilesHistoricConfig.java 100.00% <ø> (ø) 1.00 <0.00> (ø)
...ck/node/blocks/files/recent/FilesRecentConfig.java 100.00% <ø> (ø) 1.00 <0.00> (ø)
...block/node/spi/historicalblocks/BlockAccessor.java 62.50% <ø> (ø) 2.00 <0.00> (ø)
.../node/spi/historicalblocks/BlockAccessorBatch.java 100.00% <100.00%> (ø) 9.00 <9.00> (?)
.../hiero/block/node/base/tar/TaredBlockIterator.java 83.67% <90.00%> (-0.54%) 16.00 <0.00> (ø)
...ro/block/node/blocks/files/historic/BlockPath.java 92.68% <33.33%> (-4.69%) 10.00 <1.00> (+1.00) ⬇️
...ck/node/blocks/files/historic/ZipBlockArchive.java 67.37% <66.66%> (-0.49%) 24.00 <2.00> (ø)
...main/java/org/hiero/block/node/base/BlockFile.java 90.10% <87.09%> (-3.14%) 24.00 <6.00> (-1.00)
.../node/access/service/BlockAccessServicePlugin.java 82.00% <44.44%> (-4.96%) 10.00 <0.00> (ø)
... and 6 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pull request label to get past the "label required" check when no label is needed or appropriate.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BlockAccessors Hardening & Improved Error Handling

7 participants