Add support for exporting data in the Data Loader via Consensus Commit

[inv-jishnu]

Description

This PR adds support for transactional mode in data export within the Dataloader core, along with corresponding updates to the CLI.

Previously, the export functionality only supported storage mode because there was no transactional scanner API that allowed streaming data without loading everything into memory. With the introduction of a suitable transactional scanner, we can now support export in transaction mode as well.

Related issues and/or PRs

NA

Changes made

On core

• Added changes to support transaction for export in TableMetadataService, ExportManager, and ScalarDBDao
• Added/updated unit tests.
  On CLI
• Added a parameter to input the Scalardb mode.
• Added checks to use the transaction based on input for metadata retrieval and export.

Checklist

The following is a best-effort checklist. If any items in this checklist are not applicable to this PR or are dependent on other, unmerged PRs, please still mark the checkboxes after you have read and understood each item.

• I have commented my code, particularly in hard-to-understand areas.
• I have updated the documentation to reflect the changes.
• I have considered whether similar issues could occur in other products, components, or modules if this PR is for bug fixes.
• Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
• Tests (unit, integration, etc.) have been added for the changes.
• My changes generate no new warnings.
• Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

NA

Release notes

Added transaction mode in data export on data loader core and CLI

[inv-jishnu]

@brfrn169 san
I have made changes as suggested.
Please take a look at this again when you get a chance.
Thank you.

[brfrn169]

Is it best that I add this to the 3.16.1 project when available instead? (and remove it from 3.16 for now)?

@ypeckstadt Basically, we don’t include new features in patch versions.

As discussed with @feeblefakie, we’ve decided not to release 3.16 this week but to release it next week instead. So I think we can include this PR in that release. Thanks.

[thongdk8]

FYI, as discussed in another place Slack conv. With the default config of transaction manager consensus-commit when exporting data in transaction mode, the OMM will occur if the export data exceeds the available memory. So we are expecting to run exporting in transaction mode with scalar.db.transaction_manager=single-crud-operation, with single-crud-operation transaction manager the memory usage will be safe as exporting in storage mode.

[inv-jishnu]

@brfrn169 san, @komamitsu san,
I have made updates based on feedback.
Please take a look at this PR when you get a chance.
Thank you

[komamitsu]

Don't need to consider #2740 (comment) ?

[inv-jishnu]

@komamitsu san, @brfrn169 san (cc @ypeckstadt )

Based on comment from @thongdk8 , for export operations in transaction mode, is it acceptable to explicitly set scalar.db.transaction_manager to single-crud-operation, even if the user hasn’t specified this value in the database configuration file?

[komamitsu]

@inv-jishnu I think it's a reasonable and safe option. But it depends on the requirement (e.g., users can use consensus-commit to export snapshot data even it has the risk of OOM). Probably, you can talk with @ypeckstadt to clarify the requirement.

[brfrn169]

Based on comment from @thongdk8 , for export operations in transaction mode, is it acceptable to explicitly set scalar.db.transaction_manager to single-crud-operation, even if the user hasn’t specified this value in the database configuration file?

@inv-jishnu Let me discuss it with @feeblefakie.

[ypeckstadt]

Based on our discussion, this code and PR is necessary and will represent the consensus-commit approach for the export. (which needs to be supported). At the same time, the single CRUD operation will be handled via another PR. That PR will basically remove all usage of DistributedStorage and replace it with usage of the Single CRUD interface.

[ypeckstadt]

After discussion in the team., it's clear that this PR is not required anymore and we the export will just be done using the DistributedTransactionManager instead.

@inv-jishnu FYI