Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
27a81c8
Adding new tools for record file processing
jasperpotts Oct 8, 2025
d3c900a
Adding new tools for record file processing. Cleaning up readme files…
jasperpotts Oct 9, 2025
6e644bd
Added download command
jasperpotts Oct 9, 2025
0150e0e
progress
jasperpotts Oct 9, 2025
dbb6507
progress
jasperpotts Oct 9, 2025
3a37046
progress
jasperpotts Oct 9, 2025
c433401
progress and spotify
jasperpotts Oct 9, 2025
519968d
progress and spotify
jasperpotts Oct 10, 2025
544fc53
progress and spotify
jasperpotts Oct 13, 2025
3a25e32
some refactoring
jasperpotts Oct 14, 2025
1d65e0b
fixing issues
jasperpotts Oct 14, 2025
75231d9
added test record files
jasperpotts Oct 14, 2025
c92d22b
added test and working on validation
jasperpotts Oct 14, 2025
7b9bac8
working on validation
jasperpotts Oct 15, 2025
ab11412
working on validation
jasperpotts Oct 15, 2025
00f31e6
working on validation
jasperpotts Oct 15, 2025
668e2ad
working on validation
jasperpotts Oct 15, 2025
019bd53
working on validation
jasperpotts Oct 15, 2025
ddf7a69
working on validation
jasperpotts Oct 15, 2025
d736e95
working on validation
jasperpotts Oct 15, 2025
75303ba
working on validation
jasperpotts Oct 16, 2025
d07b98c
working on validation
jasperpotts Oct 16, 2025
5b677b2
working on validation
jasperpotts Oct 16, 2025
3f154bc
Added warnings file
jasperpotts Oct 16, 2025
1ae3b63
Progress
jasperpotts Oct 17, 2025
42aa9c3
Added ls day listing command and fixed issue in Day download
jasperpotts Oct 17, 2025
e8ef5ff
Added ls record file listing command and fixed issue in Day download
jasperpotts Oct 21, 2025
cbc82f6
Apply Spotify
jasperpotts Oct 21, 2025
0946a95
fix bug
jasperpotts Oct 21, 2025
ad8750d
faster day reading
jasperpotts Oct 21, 2025
8dc84e0
Much faster reading of tar.zstd 40mb/s to 600mb/s
jasperpotts Oct 21, 2025
d5dc56f
Clean up of commands structure
jasperpotts Oct 22, 2025
b82d0ee
Added resume-ability to validate command
jasperpotts Oct 22, 2025
4d6c0e3
Refactoring of signature handling and adding wrapping
jasperpotts Oct 22, 2025
278f1b4
Added wrap command
jasperpotts Oct 23, 2025
0d29be2
Fix mirror node commands
jasperpotts Oct 23, 2025
e63b35b
progress on downloader v2
jasperpotts Oct 23, 2025
50b7567
progress on downloader v2
jasperpotts Oct 23, 2025
6575e5d
progress on downloader v2
jasperpotts Oct 24, 2025
d7c2782
progress on downloader v2
jasperpotts Oct 24, 2025
e77b849
progress on downloader v2
jasperpotts Oct 24, 2025
7d61a07
progress on downloader v2
jasperpotts Oct 24, 2025
56a9159
Added address book saving, fixed issues in validate command
jasperpotts Oct 24, 2025
1504f4d
Fixing docs
jasperpotts Oct 24, 2025
f0155e2
Fixing docs
jasperpotts Oct 25, 2025
f242f7e
Fixing docs
jasperpotts Oct 25, 2025
69206e4
Fix handling on sidecar files
jasperpotts Nov 4, 2025
928a03d
Fix issues with corrupt gzip files
jasperpotts Nov 5, 2025
8b2daa9
Change thread settings for DownloadDaysV2
jasperpotts Nov 6, 2025
7376e3a
Added retry for bad user
jasperpotts Nov 6, 2025
1fa4326
Added clean command
jasperpotts Nov 6, 2025
9a386ce
more fully featured clean command
jasperpotts Nov 7, 2025
bf8fbed
Fix CleanDayOfBadRecordSets to handle partially complete files
jasperpotts Nov 7, 2025
e5eb8cd
Added extended LS command to list record file hashes
jasperpotts Nov 10, 2025
cdca636
Box fix in AddressBookRegistry and tuned Validate to use less RAM
jasperpotts Nov 11, 2025
18b62ab
Fixes and improvements to CLI commands
jasperpotts Nov 11, 2025
538c946
Added simple main for testing to validate command
jasperpotts Nov 11, 2025
aa6d4ce
Refactor validate command to work with dates rather than day indexes
jasperpotts Nov 11, 2025
e2bd177
Added start at day to validate command
jasperpotts Nov 12, 2025
d51a24c
Added support to choose to use Zstd jni lib. Fix for ignoring hidden …
jasperpotts Nov 12, 2025
52786c1
Added speed printing in form or 300x realtime to validate command
jasperpotts Nov 13, 2025
ec8e3ae
Applied Spotless
jasperpotts Nov 13, 2025
b1b3833
Added MD5 retry to downloadsV2
jasperpotts Nov 13, 2025
f591dd3
Added number of sidecar files to record file info. Fixed tool.sh run …
jasperpotts Nov 13, 2025
05409da
Added skip for MD5 failure on signature files in DownloadDays2 Also f…
jasperpotts Nov 14, 2025
9d3a028
Progress on wrapping
jasperpotts Nov 14, 2025
e596c92
Spotless apply
jasperpotts Nov 14, 2025
6c91849
Added mirror node update data files command
jasperpotts Nov 14, 2025
fb06ab2
Fix issues with 0x hex prefixes in days json. Fix bug with handling o…
jasperpotts Nov 14, 2025
ee53bb0
Work on wrap command
jasperpotts Nov 15, 2025
2fe080d
Spotless apply
jasperpotts Nov 15, 2025
315bca6
Added more printing to LS command for debugging stream issues. Added …
jasperpotts Nov 17, 2025
5552c75
Steps on wrapped block work
jasperpotts Nov 18, 2025
e3fa929
Added BlockArchiveType
jasperpotts Nov 18, 2025
039c987
Applied spotless
jasperpotts Nov 18, 2025
17d2af5
Added BlockReader
jasperpotts Nov 18, 2025
6c369b1
Added BlockReader and tests
jasperpotts Nov 18, 2025
481daa6
Added progress to ToWrappedBlocksCommand
jasperpotts Nov 18, 2025
6a33763
Added "days updateDayListings" new command to update the "listingsByD…
jasperpotts Nov 19, 2025
1c7208d
Fix UpdateBlockData to not download a day that has not finished yet, …
jasperpotts Nov 19, 2025
8fc0544
refactor and cleanup of packages and commandline tools
jasperpotts Nov 19, 2025
9f7635c
cleanup of metadata, added metadata commands
jasperpotts Nov 20, 2025
fde9fba
Added block header to wrapped blocks
jasperpotts Nov 20, 2025
95a5927
Step on the way to wrapped blocks, more refactor coming
jasperpotts Nov 20, 2025
ddfc8cd
Step on the way to wrapped blocks, more refactor coming
jasperpotts Nov 21, 2025
65d915d
Still needs lots of testing and there are a bunch of TODOs but it is …
jasperpotts Nov 21, 2025
591a727
implemented writing out all the files
jasperpotts Nov 21, 2025
4d5ac2f
progress and adding tests
jasperpotts Nov 21, 2025
7943749
added v5 and v6 tests and implemented hash computation
jasperpotts Nov 21, 2025
9b8e612
Applied spotless
jasperpotts Nov 21, 2025
8c2e129
Added test for ParsedRecordBlock
jasperpotts Nov 21, 2025
a7a0beb
improved test for ParsedRecordBlock
jasperpotts Nov 21, 2025
6c3c246
Added tests
jasperpotts Nov 21, 2025
543797d
work on wrap command
jasperpotts Nov 22, 2025
9b885ea
work on wrap command
jasperpotts Nov 22, 2025
d957062
Helped performance when execution is created
jasperpotts Nov 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .editorconfig
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ root = true

[*]
indent_style = space
indent_size = 2
indent_size = 4
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
Expand Down
90 changes: 90 additions & 0 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Project-specific Copilot instructions

## How to build and run
- when working with gradle module `:tools-and-tests:tools` it is actually `:tools`. Default full build task is
`shadowJar` which builds all modules and creates an "all-in-one" jar `tools-and-tests/tools/build/libs/tools-0.21.0-SNAPSHOT-all.jar`.
- to run the block CLI tool `:tools` after building, use: `java -jar tools-and-tests/tools/build/libs/tools-0.21.0-SNAPSHOT-all.jar [args]`

## Coding style and conventions
This provides explicit, actionable guidance for generating Java source that matches this repository's coding style and formatting rules.

- File header and license:
- Every Java source file must start with the SPDX license header exactly as a single-line comment:
// SPDX-License-Identifier: Apache-2.0
- The SPDX header must be the first non-empty line in the file, followed by the `package` statement.

- Package and imports:
- Always include a `package` declaration after the SPDX header.
- Do not use wildcard imports (`import java.util.*`).
- Preferred import ordering: java.*, javax.*, org.*, com.*, then project packages (alphabetical within each group).
- Keep a single blank line between the package statement and imports, and a single blank line between imports and type declaration.

- Javadoc:
- Provide Javadoc for all public classes, interfaces, enums, methods, and fields.
- For short descriptions prefer a single-line Javadoc, e.g.:
/** The Constants class defines the constants for the block simulator. */
- For longer descriptions, use a standard block Javadoc and avoid blank lines without <p> tags if a paragraph split is required, e.g.:
/**
* Builds X and returns Y.
*
* <p>This method does ...
*
* @param input the input value
* @return the result
*/
- Include `@param`, `@return`, and `@throws` tags where applicable for public APIs.
- Close all HTML tags and use `<ul>`, `<li>` for lists (no Markdown lists inside Javadoc).

- Naming, modifiers, and fields:
- Use long descriptive variable names rather than single-letter names (except loop indices i, j, k).
- Prefer explicit types for public method signatures and fields; prefer `final` for fields that do not change.
- Static constants: `public static final` with UPPER_SNAKE_CASE naming.
- For utility classes, add a private constructor to prevent instantiation.

- Formatting and style:
- Indentation: 4 spaces.
- Brace style: opening brace on same line as declaration (K&R).
- Line length: aim for <= 120 characters.
- Use single blank line to separate logical blocks inside methods.
- Use descriptive boolean variable names (e.g., `isEnabled`, `colorfulLogFormatterEnabled`).

- Java version and modern APIs:
- Target Java 21 compatibility.
- Prefer modern JDK APIs where appropriate: use `switch` expressions for multi-branch selection, `Objects.requireNonNullElse`, java.time classes, and `HexFormat` for hex conversions.
- Avoid adding third-party dependencies for simple tasks that can be done with the JDK.
- Prefer `record` for simple immutable data carriers where it makes sense and public API stability allows it.

- Logging:
- Follow existing project pattern of `java.util.logging`.
- For obtaining a logger: `private static final Logger LOGGER = Logger.getLogger(MyClass.class.getName());`

- Tests:
- Place tests under the appropriate `src/test/java` module and match package naming of the production code.
- Tests should be self-contained, use expressive variable names, and include descriptive test method names.

- Avoid:
- Avoid using `var` in public API signatures. Use `var` only in short local contexts where the inferred type is obvious.
- Avoid wildcard imports.
- Avoid unnecessary suppression of warnings; if suppression is used, scope it narrowly and justify in a short comment.

- Examples and small templates
- File header template:
// SPDX-License-Identifier: Apache-2.0
package org.hiero.block.example;
/**
* Brief single-line class description.
*/
public final class ExampleClass {
/** Explanation of constant. */
public static final String EXAMPLE_CONSTANT = "value";
/** Constructor prevents instantiation. */
private ExampleClass() {}
}

- Spotless and formatting note for Copilot:
- The repository uses an organization-level Spotless plugin wrapper (`org.hiero.gradle.check.spotless`). Generated files should conform to the above formatting so they pass Spotless checks.
- If you are adding or editing Gradle files, prefer the repository's existing Gradle plugin usage over adding new formatting plugins unless approved.

- When in doubt:
- Match nearby files in the same package/module: copy class-level javadoc style, ordering, and formatting from local examples.

18 changes: 18 additions & 0 deletions BUILDING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Build & run (development)

Prerequisites
- Use the Gradle wrapper that's committed to the repo. From project root use `./gradlew` (Unix) or `gradlew.bat` (Windows).

Common tasks

Root build (all modules)
- Build: `./gradlew build`
- Run tests: `./gradlew test`

The project is made up of several apps/modules. They each have their own README and BUILDING instructions:
- `/block-node`: The Block Node Server implementation.
- `/common`: Common library used by both the Block Node and the Simulator.
- `/tools-and-tests/protobuf-protoc`: The HAPI API protobuf compiled with the Google protoc compiler.
- `/tools-and-tests/simulator`: A simulator for the Block Node, which can be used to test the Block Node in a local environment.
- [/tools-and-tests/tools](tools-and-tests/tools/BUILDING.md): A set of command line tools for working with Block Stream files.
- `/tools-and-tests/suites`: A set of e2e tests that can be used to verify the correctness of the Block Node.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ The Block Node will also expose additional targeted value adding APIs to the Hie
- [`simulator`](docs/simulator/README.md): A simulator for the Block Node, which can be used to test the Block Node in a local environment.
- `common`: Module responsible for holding common literals, utilities and types used by the other modules.
- `suites`: A set of e2e tests that can be used to verify the correctness of the Block Node.
- `tools`: A set of command line tools for working with Block Stream files.
- [tools](tools-and-tests/tools/README.md): A set of command line tools for working with Block Stream files.

## Getting Started

Expand Down
2 changes: 1 addition & 1 deletion docs/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ This document will outline the structure and content of Block Node documentation

- [Hiero Block Node](block-node/README.md): Provides an introduction to Hiero Block Node, its purpose and how it can be used.
- [Simulator](simulator/README.md): Offers instructions on how the Simulator can be utilized.
- [Tools](tools/README.md): Provides information on the command line tools available.
- [Tools](../tools-and-tests/tools/README.md): Provides information on the command line tools available.

## Concepts

Expand Down
179 changes: 0 additions & 179 deletions docs/tools/README.md

This file was deleted.

53 changes: 0 additions & 53 deletions docs/tools/quickstart.md

This file was deleted.

Loading
Loading