Add handlers for ExportToFs / ImportFromFs #28126

stanislav-shchetinin · 2025-11-01T09:53:25Z

Changelog entry

...

Changelog category

Not for changelog (changelog entry is not required)

Description for reviewers

Merge after #28072

github-actions · 2025-11-01T09:56:20Z

🟢 2025-11-01 09:56:19 UTC The validation of the Pull Request description is successful.

ydb/core/grpc_services/rpc_import_base.h

CyberROFL · 2025-11-01T12:38:36Z

ydb/core/grpc_services/rpc_export.cpp

Как будто можно сделать общую функцию для s3 и fs.

Отдельным pr-ом сделаю рефакторинг

CyberROFL · 2025-11-01T12:39:26Z

ydb/core/grpc_services/rpc_export.cpp

А почему шифрование не поддерживается как для s3?

Отдельным pr-ом сделаю добавление в .proto и в хэндлеры шифрование

Copilot

Pull Request Overview

This PR adds file system (FS) backup support for YDB, enabling export to and import from local or network file systems (like NFS). The implementation follows the existing patterns for S3-based backup operations.

Key changes:

Added ExportToFs and ImportFromFs RPC methods with validation for base paths and compression settings
Created comprehensive validation tests for FS backup parameters
Refactored the YDB_SDK_CLIENT macro into a shared header for code reuse

Reviewed Changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
ydb/services/ydb/ydb_import.cpp	Registers ImportFromFs RPC endpoint
ydb/services/ydb/ydb_export.cpp	Registers ExportToFs RPC endpoint
ydb/services/ydb/backup_ut/ya.make	Adds new validation test file to build
ydb/services/ydb/backup_ut/s3_backup_test_base.h	Extracts YDB_SDK_CLIENT macro to shared header
ydb/services/ydb/backup_ut/fs_backup_validation_ut.cpp	Comprehensive validation tests for FS backup parameters
ydb/services/ydb/backup_ut/backup_test_base.h	Shared YDB_SDK_CLIENT macro definition
ydb/public/sdk/cpp/src/client/operation/operation.cpp	Adds FS operation templates
ydb/public/sdk/cpp/src/client/import/import.cpp	Implements ImportFromFs client method
ydb/public/sdk/cpp/src/client/export/export.cpp	Implements ExportToFs client method
ydb/public/sdk/cpp/include/ydb-cpp-sdk/client/import/import.h	Declares ImportFromFs settings and response types
ydb/public/sdk/cpp/include/ydb-cpp-sdk/client/export/export.h	Declares ExportToFs settings and response types
ydb/public/api/protos/ydb_import.proto	Documents absolute path requirement for base_path
ydb/public/api/protos/ydb_export.proto	Documents absolute path requirement for base_path
ydb/core/tx/schemeshard/schemeshard_audit_log.cpp	Adds audit logging for FS export/import
ydb/core/protos/import.proto	Adds ImportFromFsSettings to proto definitions
ydb/core/protos/export.proto	Adds ExportToFsSettings to proto definitions
ydb/core/grpc_services/service_import.h	Declares DoImportFromFsRequest handler
ydb/core/grpc_services/service_export.h	Declares DoExportToFsRequest handler
ydb/core/grpc_services/rpc_import_base.h	Adds FS import kind handling to operation conversion
ydb/core/grpc_services/rpc_import.cpp	Implements ImportFromFs RPC handler with validation
ydb/core/grpc_services/rpc_export_base.h	Adds FS export kind handling to operation conversion
ydb/core/grpc_services/rpc_export.cpp	Implements ExportToFs RPC handler with validation
ydb/core/cms/console/console_configs_subscriber.cpp	Clears and updates config when changed

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ydb/services/ydb/backup_ut/fs_backup_validation_ut.cpp

Copilot · 2025-11-06T10:06:19Z

ydb/public/sdk/cpp/include/ydb-cpp-sdk/client/export/export.h

+    struct TItem {
+        std::string Src;


The TItem struct in TExportToFsSettings lacks documentation. Add comments explaining what Src and Dst represent (e.g., source path in database and destination path on filesystem).

Suggested change

struct TItem {

std::string Src;

struct TItem {

/// Source path in the database to export from.

std::string Src;

/// Destination path on the filesystem to export to.

Copilot · 2025-11-06T10:06:19Z

ydb/public/sdk/cpp/include/ydb-cpp-sdk/client/import/import.h

+
+    struct TItem {
+        // Source path.
+        // Path to the exported table/directory in FS (relative to base_path)


The comment says 'exported table/directory' but this is for import settings where the correct terminology would be 'the exported data' or 'backup location'. Consider rephrasing to 'Path to the backup data in FS (relative to base_path)'.

Suggested change

// Path to the exported table/directory in FS (relative to base_path)

// Path to the backup data in FS (relative to base_path)

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

github-actions · 2025-11-07T08:34:41Z

⚪ 2025-11-07 08:34:41 UTC Pre-commit check linux-x86_64-release-asan for 380022c has started.
⚪ 2025-11-07 08:34:57 UTC Artifacts will be uploaded here
⚪ 2025-11-07 08:37:14 UTC ya make is running...
🟡 2025-11-07 11:18:51 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
18893	18514	0	135	221	23

🟢 2025-11-07 11:19:04 UTC Build successful.
🔴 2025-11-07 11:19:31 UTC ydbd size 3.8 GiB changed* by +3.0 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash	main: `0e1c687`	merge: `380022c`	diff	diff %
ydbd size	4 072 619 024 Bytes	4 075 750 696 Bytes	+3.0 MiB	+0.077%
ydbd stripped size	1 511 727 848 Bytes	1 513 091 656 Bytes	+1.3 MiB	+0.090%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-11-07T08:34:47Z

⚪ 2025-11-07 08:34:46 UTC Pre-commit check linux-x86_64-relwithdebinfo for 380022c has started.
⚪ 2025-11-07 08:35:03 UTC Artifacts will be uploaded here
⚪ 2025-11-07 08:37:27 UTC ya make is running...
🟡 2025-11-07 11:15:55 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
42399	39539	0	5	2816	39

⚪ 2025-11-07 11:16:19 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-11-07 11:27:50 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
113 (only retried tests)	100	0	0	0	13

🟢 2025-11-07 11:27:59 UTC Build successful.
🟡 2025-11-07 11:28:19 UTC ydbd size 2.3 GiB changed* by +1.5 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `196ff72`	merge: `380022c`	diff	diff %
ydbd size	2 431 555 072 Bytes	2 433 108 520 Bytes	+1.5 MiB	+0.064%
ydbd stripped size	516 486 256 Bytes	516 757 552 Bytes	+264.9 KiB	+0.053%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

pnv1

SDK part LGTM

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

github-actions bot added the not-for-changelog label Nov 1, 2025

stanislav-shchetinin linked an issue Nov 1, 2025 that may be closed by this pull request

Add a GRPC handler for ExportToNFS / ImportFromNFS #27945

Closed

stanislav-shchetinin requested a review from CyberROFL November 1, 2025 10:55

CyberROFL requested changes Nov 1, 2025

View reviewed changes

stanislav-shchetinin added 4 commits November 5, 2025 16:14

done

2b41972

fix build

afb16c0

done tests

30e35e4

common part in tests

505f73f

stanislav-shchetinin force-pushed the nfs-handlers branch from fe2b782 to 505f73f Compare November 5, 2025 13:14

This comment was marked as outdated.

Sign in to view

small fix

f167571

This comment was marked as outdated.

Sign in to view

fix tests

ca2e316

This comment was marked as outdated.

Sign in to view

stanislav-shchetinin marked this pull request as ready for review November 6, 2025 10:02

stanislav-shchetinin requested review from a team as code owners November 6, 2025 10:02

Copilot AI review requested due to automatic review settings November 6, 2025 10:02

stanislav-shchetinin requested a review from a team as a code owner November 6, 2025 10:02

stanislav-shchetinin requested a review from vporyadke November 6, 2025 10:02

Copilot AI reviewed Nov 6, 2025

View reviewed changes

CyberROFL self-requested a review November 6, 2025 10:11

nfs -> fs in ut comments

562e40e

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

This comment was marked as outdated.

Sign in to view

remove

abc0354

CyberROFL approved these changes Nov 7, 2025

View reviewed changes

pnv1 approved these changes Nov 11, 2025

View reviewed changes

stanislav-shchetinin merged commit ed7c647 into ydb-platform:main Nov 11, 2025
14 checks passed

qyryq pushed a commit to qyryq/ydb that referenced this pull request Nov 24, 2025

Add handlers for ExportToFs / ImportFromFs (ydb-platform#28126)

297f69f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

	// Path to the exported table/directory in FS (relative to base_path)
	// Path to the backup data in FS (relative to base_path)

Add handlers for ExportToFs / ImportFromFs #28126

Add handlers for ExportToFs / ImportFromFs #28126

Uh oh!

Conversation

stanislav-shchetinin commented Nov 1, 2025

Changelog entry

Changelog category

Description for reviewers

Uh oh!

github-actions bot commented Nov 1, 2025

Uh oh!

Uh oh!

CyberROFL Nov 1, 2025

Choose a reason for hiding this comment

Uh oh!

stanislav-shchetinin Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

CyberROFL Nov 1, 2025

Choose a reason for hiding this comment

Uh oh!

stanislav-shchetinin Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

github-actions bot commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pnv1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Nov 7, 2025 •

edited

Loading

github-actions bot commented Nov 7, 2025 •

edited

Loading