graph/db: helper tests and some benchmarks for SQL #10109
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @ellemouton, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request significantly expands the testing and performance analysis capabilities within the graph/db package. It introduces robust helper tests for populating graph databases across various backend types, including a dedicated test for the KVDB to SQL migration process. Crucially, it adds a suite of benchmarks to measure key graph operations like cache loading and data iteration, providing valuable insights into performance characteristics. Additionally, the existing SQL migration logic has been improved with better progress reporting.
Highlights
- New Testing Utilities: Introduced helper tests (
TestPopulateDBs) to populate local graph databases from a source graph, supporting various backend types (bbolt, sqlite, postgres for both KVDB and native SQL). This includes asyncGraphutility for data replication. - SQL Migration Helper: Added
TestPopulateViaMigrationto specifically test and facilitate the migration of graph data from KVDB-backed databases to native SQL databases, leveraging the existing migration logic. - Performance Benchmarks: Implemented several benchmark tests (
BenchmarkCacheLoading,BenchmarkForEachNode,BenchmarkForEachChannelAndPolicy) to measure the performance of graph cache loading and graph traversal operations across different database backends. - Migration Progress Logging: Enhanced the SQL migration functions (
migrateNodes,migrateChannelsAndPolicies,migratePruneLog,migrateClosedSCIDIndex,migrateZombieIndex) by adding rate-limited progress logging, providing real-time feedback during large data migrations. - Logging Infrastructure Update: Updated the
btclogimport tobtclog/v2and adjusted thegraphdb.Subsystemlogger registration fromAddV1SubLoggertoAddSubLoggerfor consistency.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request introduces helper tests and benchmarks for SQL, including helper tests to populate a local graph from a source graph, another helper to do the same via the Migrate SQL function, and benchmark tests to test cache loading and describe graph calls against various graph DB backend types. Several comments were made to address line length issues.
ellemouton
force-pushed
the
graphPerf1
branch
2 times, most recently
from
ca813c0 to
0bbe463
Compare
ellemouton
force-pushed
the
graphPerf1
branch
2 times, most recently
from
b841b78 to
a8695e1
Compare
This is a prep commit that just adds all the boilerplate connection logic for connecting to various graph DBs. These will be used in upcoming commits which will add tests and benchmarks.
This commit adds a helper test that is similar to TestPopulateDBs except for the fact that it uses the MigrateGraphToSQL function directly to migrate a local kvdb-sql graph to a native SQL one.
Add a benchmark test to test graph cache loading against various local graph DBs.
This commit adds benchmark tests for the ForEachNode and ForEachChannel DB calls which are called by DescribeGraph.
|
cc @guggero for override merge 🙏 the test fail is invoice registry related |
This PR adds:
These benchmarks will be useful for upcoming PRs which will start introducing some performance improvements. The benckmarks will let us ensure that we are in fact introducing valuable improvements