Feat: add kmesh-daemon upgrades traffic without disruption #1503

072020127 · 2025-09-11T09:14:55Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Implemented logic for storing and reading persistent MapSpecs
Added retrieval of the current MapSpec
Implemented comparison logic for Map structure
Added unit tests and e2e test programs

Which issue(s) this PR fixes:

Fixes #1409

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

gemini-code-assist

Summary of Changes

Hello @072020127, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant feature enabling seamless, traffic-undisrupted upgrades for the kmesh-daemon. It achieves this by persisting BPF map specifications, comparing them during daemon restarts, and intelligently migrating map data to new versions. This ensures that ongoing network traffic is not interrupted when the daemon is updated.

Highlights

Persistent BPF MapSpecs: Implemented logic for storing and reading persistent BPF MapSpecs, which are crucial for maintaining state across kmesh-daemon upgrades.
MapSpec Comparison Logic: Added retrieval of the current MapSpec and implemented robust comparison logic for BPF Map structures to detect changes between versions.
Seamless Daemon Upgrades: Introduced an 'Update' mode for kmesh-daemon, allowing BPF programs and maps to be updated without traffic disruption by migrating existing map data.
Comprehensive Testing: Included new unit tests for MapSpec loading and comparison, as well as an end-to-end test to validate the traffic-undisrupted upgrade process.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

BPF maps persist, Upgrades flow, no traffic lost, Code ensures smooth ride.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a significant feature for non-disruptive upgrades of the kmesh-daemon. The core logic involves persisting BPF map specifications (MapSpecs), comparing them across versions, and migrating maps when their structure changes. The implementation adds new files for handling BPF updates and loading MapSpecs, along with extensive unit and e2e tests.

My review focuses on the new update logic. I've found a few critical issues related to error handling that could lead to panics, and a bug where changes in struct bitfield sizes are not detected, potentially causing data corruption. I've also identified areas with unhandled errors and code duplication that should be addressed to improve robustness and maintainability. The suggestions provided aim to fix these issues and refactor the code for better clarity.

gemini-code-assist · 2025-09-11T09:17:51Z

pkg/bpf/restart/bpf_update.go

+		if map_old.Offset != uint32(map_new.Offset) {
+            log.Info(map_old.BitfieldSize,  uint32(map_new.BitfieldSize))
+			diff.OffsetChanged = true
+			break
+		}


The BitfieldSize of a struct member is not being compared. This is a critical bug, as a change in bitfield size alters the struct layout and should trigger a map migration to avoid data corruption. The existing log.Info call should be replaced with a proper comparison.

if map_old.Offset != uint32(map_new.Offset) || map_old.BitfieldSize != uint32(map_new.BitfieldSize) { diff.OffsetChanged = true break }

gemini-code-assist · 2025-09-11T09:17:51Z

pkg/bpf/bpf.go

+	mapspec, _ := restart.LoadCompileTimeSpecs(config)
+	restart.SnapshotSpecsByPkg(mapspec)


The error returned by restart.LoadCompileTimeSpecs is ignored. This could lead to restart.SnapshotSpecsByPkg being called with a nil mapspec, which might cause a panic. The error should be checked and handled to prevent this.

Suggested change

mapspec, _ := restart.LoadCompileTimeSpecs(config)

restart.SnapshotSpecsByPkg(mapspec)

mapspec, err := restart.LoadCompileTimeSpecs(config)

if err != nil {

log.Errorf("failed to load compile time specs: %v", err)

return nil

}

restart.SnapshotSpecsByPkg(mapspec)

gemini-code-assist · 2025-09-11T09:17:51Z

pkg/bpf/restart/bpf_update.go

+	specsByPkg, err := LoadCompileTimeSpecs(config)
+	if specsByPkg == nil {
+		log.Errorf("load oldSpecsByPkg failed")
+        return nil
+	}


The error returned by LoadCompileTimeSpecs is not handled. If an error occurs, specsByPkg could be nil and the function might proceed, potentially causing a panic later. It's crucial to check and handle this error. Also, the log message on line 68 refers to oldSpecsByPkg which seems to be a typo.

Suggested change

specsByPkg, err := LoadCompileTimeSpecs(config)

if specsByPkg == nil {

log.Errorf("load oldSpecsByPkg failed")

return nil

}

specsByPkg, err := LoadCompileTimeSpecs(config)

if err != nil {

log.Errorf("load compile-time map specs failed: %v", err)

return nil

}

gemini-code-assist · 2025-09-11T09:17:51Z

pkg/bpf/restart/bpf_update.go

+func migrateMap(
+    oldMapSpec *PersistedMapSpec,
+    newMapSpec *ebpf.MapSpec,
+    pkgName, mapName, pinPath string,
+) (*ebpf.Map, error)  {
+	if oldMapSpec == nil {
+		return createEmptyMap(newMapSpec, pinPath, mapName, nil)
+	}
+    if oldMapSpec.Type != newMapSpec.Type.String() ||
+       oldMapSpec.KeySize != newMapSpec.KeySize ||
+       oldMapSpec.ValueSize != newMapSpec.ValueSize || 
+       oldMapSpec.MaxEntries != newMapSpec.MaxEntries {
+       oldMap, err := ebpf.LoadPinnedMap(pinPath, &ebpf.LoadPinOptions{})
+        if err == nil {
+            return createEmptyMap(newMapSpec, pinPath, mapName, oldMap)
+        } else {
+            return createEmptyMap(newMapSpec, pinPath, mapName, nil)
+        }
+	}
+
+	oldKeyStruct := oldMapSpec.KeyInfo
+    newKeyStruct, okNewKey := newMapSpec.Key.(*btf.Struct)
+	if okNewKey {
+        diff := diffStructInfoAgainstBTF(oldKeyStruct, newKeyStruct, make(map[string]bool))
+        if diff.Added || diff.Removed || diff.TypeChanged ||
+            diff.OffsetChanged || diff.NestedChanged {
+			oldMap, err := ebpf.LoadPinnedMap(pinPath, &ebpf.LoadPinOptions{})
+            if err == nil {
+                return createEmptyMap(newMapSpec, pinPath, mapName, oldMap)
+            } else {
+                return createEmptyMap(newMapSpec, pinPath, mapName, nil)
+            }
+		}
+    } else { 
+        if newMapSpec.Key == nil {
+            if len(oldKeyStruct.Members) != 0 || oldKeyStruct.Name != "" {
+                oldMap, err := ebpf.LoadPinnedMap(pinPath, &ebpf.LoadPinOptions{})
+                if err == nil {
+                    return createEmptyMap(newMapSpec, pinPath, mapName, oldMap)
+                } else {
+                    return createEmptyMap(newMapSpec, pinPath, mapName, nil)
+                }
+            }
+        } else {
+            newKeyTypeName := newMapSpec.Key.TypeName()
+            if len(oldKeyStruct.Members) != 0 || newKeyTypeName != oldKeyStruct.Name  {
+                oldMap, err := ebpf.LoadPinnedMap(pinPath, &ebpf.LoadPinOptions{})
+                if err == nil {
+                    return createEmptyMap(newMapSpec, pinPath, mapName, oldMap)
+                } else {
+                    return createEmptyMap(newMapSpec, pinPath, mapName, nil)
+                }
+            }
+        }
+    }
+
+    oldValueStruct := oldMapSpec.ValueInfo
+    newValueStruct, okNewValue := newMapSpec.Value.(*btf.Struct)
+    if okNewValue {
+        diff := diffStructInfoAgainstBTF(oldValueStruct, newValueStruct, make(map[string]bool))
+        if diff.Added || diff.Removed || diff.TypeChanged ||
+            diff.OffsetChanged || diff.NestedChanged {
+            oldMap, err := ebpf.LoadPinnedMap(pinPath, &ebpf.LoadPinOptions{})
+            if err == nil {
+                return createEmptyMap(newMapSpec, pinPath, mapName, oldMap)
+            } else {
+                log.Info(diff.Added ,diff.Removed,diff.TypeChanged ,
+            diff.OffsetChanged,diff.NestedChanged)
+                return createEmptyMap(newMapSpec, pinPath, mapName, nil)
+            }
+        }
+    } else {
+        if newMapSpec.Value == nil {
+            if len(oldValueStruct.Members) != 0 || oldValueStruct.Name != "" {
+                oldMap, err := ebpf.LoadPinnedMap(pinPath, &ebpf.LoadPinOptions{})
+                if err == nil {
+                    return createEmptyMap(newMapSpec, pinPath, mapName, oldMap)
+                } else {
+                    return createEmptyMap(newMapSpec, pinPath, mapName, nil)
+                }
+            }
+        } else {
+            newValTypeName := newMapSpec.Value.TypeName()
+            if len(oldValueStruct.Members) != 0 || newValTypeName != oldValueStruct.Name  {
+                oldMap, err := ebpf.LoadPinnedMap(pinPath, &ebpf.LoadPinOptions{})
+                if err == nil {
+                    return createEmptyMap(newMapSpec, pinPath, mapName, oldMap)
+                } else {
+                    return createEmptyMap(newMapSpec, pinPath, mapName, nil)
+                }
+            }
+        }
+    }
+    log.Info("pass all check")
+	return nil, nil
+}


The migrateMap function has high complexity and significant code duplication. The logic to decide whether to recreate a map is repeated in multiple branches. This makes the code hard to read and maintain. Consider refactoring this function to extract the map recreation logic into a helper function and simplify the conditional checks.

codecov · 2025-09-16T03:25:36Z

Codecov Report

❌ Patch coverage is 0.43573% with 457 lines in your changes missing coverage. Please review.
✅ Project coverage is 36.84%. Comparing base (5951594) to head (e907308).
⚠️ Report is 28 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/bpf/restart/bpf_update.go	0.00%	325 Missing ⚠️
pkg/bpf/restart/new_version_mapspec_loader.go	0.00%	116 Missing ⚠️
pkg/bpf/bpf.go	15.38%	9 Missing and 2 partials ⚠️
pkg/bpf/ads/sock_connection.go	0.00%	1 Missing ⚠️
pkg/bpf/ads/sock_ops.go	0.00%	1 Missing ⚠️
pkg/bpf/workload/skb.go	0.00%	1 Missing ⚠️
pkg/bpf/workload/sock_connection.go	0.00%	1 Missing ⚠️
pkg/bpf/workload/sock_ops.go	0.00%	1 Missing ⚠️

❌ Your patch check has failed because the patch coverage (0.43%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Files with missing lines	Coverage Δ
pkg/bpf/ads/sock_connection.go	`0.00% <0.00%> (ø)`
pkg/bpf/ads/sock_ops.go	`0.00% <0.00%> (ø)`
pkg/bpf/workload/skb.go	`0.00% <0.00%> (ø)`
pkg/bpf/workload/sock_connection.go	`0.00% <0.00%> (ø)`
pkg/bpf/workload/sock_ops.go	`0.00% <0.00%> (ø)`
pkg/bpf/bpf.go	`39.63% <15.38%> (-0.79%)`	⬇️
pkg/bpf/restart/new_version_mapspec_loader.go	`0.00% <0.00%> (ø)`
pkg/bpf/restart/bpf_update.go	`0.00% <0.00%> (ø)`

... and 2 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3d9cfcd...e907308. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

hzxuzhonghu

I am concerned this is a very big change, could introduce regression. Can you please add a feature flag, we can turn on until we have enough tests and documents

hzxuzhonghu · 2025-09-16T08:11:53Z

pkg/bpf/restart/new_version_mapspec_loader.go

+
+// LoadCompileTimeSpecs loads all compile-time MapSpecs from bpf2go-generated packages.
+// Returns a nested map: packageName to mapName to *ebpf.MapSpec
+func LoadCompileTimeSpecs(config *options.BpfConfig) (map[string]map[string]*ebpf.MapSpec, error) {


what do you mean by CompileTime

My intention is to describe that these specs are fixed when the binary is compiled (via bpf2go), as opposed to the specs loaded from the persisted on-disk snapshot. Maybe LoadEmbeddedSpecs will be clearer to reflect its role?

hzxuzhonghu · 2025-09-16T08:14:43Z

pkg/bpf/restart/persisted_mapspec_loader.go

+		} else {
+			specs["KmeshCgroupSockWorkload"] = coll.Maps
+		}
+		if coll, err := dualengine.LoadKmeshCgroupSockWorkloadCompat(); err != nil {


we should only call either LoadKmeshCgroupSockWorkload or LoadKmeshCgroupSockWorkloadCompat

please check

if helper.KernelVersionLowerThan5_13() { spec, err = bpf2go.LoadKmeshSockopsCompat() } else { spec, err = bpf2go.LoadKmeshSockops() }

hzxuzhonghu · 2025-09-16T08:17:51Z

pkg/bpf/restart/new_version_mapspec_loader.go

+
+// LoadCompileTimeSpecs loads all compile-time MapSpecs from bpf2go-generated packages.
+// Returns a nested map: packageName to mapName to *ebpf.MapSpec
+func LoadCompileTimeSpecs(config *options.BpfConfig) (map[string]map[string]*ebpf.MapSpec, error) {


I am wondering how we can not miss one map, even after we add a new map later.

I wrote a go file for gen-bpf_stpecs.go in the/back directory, which can automatically generate LoadCompileTimeSpecs by analyzing the bpf2Go.go file. If there are new maps or progs in the future, it can be easily kept synchronized.

hzxuzhonghu · 2025-09-16T08:19:22Z

pkg/bpf/restart/new_version_mapspec_loader_enhanced.go

+func LoadCompileTimeSpecs(config *options.BpfConfig) (map[string]map[string]*ebpf.MapSpec, error) {
+	specs := make(map[string]map[string]*ebpf.MapSpec)
+
+	if config.KernelNativeEnabled() {


With the tag //go:build enhanced, it is for kernel native mode, so we donot need the below else branch

hzxuzhonghu · 2025-09-16T08:19:58Z

pkg/bpf/bpf.go

 		return nil
 	}

+	mapspec, err := restart.LoadCompileTimeSpecs(config)


Suggested change

mapspec, err := restart.LoadCompileTimeSpecs(config)

mapSpecs, err := restart.LoadCompileTimeSpecs(config)

hzxuzhonghu · 2025-09-16T08:39:58Z

pkg/bpf/restart/bpf_update.go

+// It will migrate any BPF maps whose on‑disk pin already exists but whose
+// compiled MapSpec has changed
+func UpdateMapHandler(versionMap *ebpf.Map, kmBpfPath string, config *options.BpfConfig) *ebpf.Map {
+	persistedSpecs, err := LoadPersistedSnapshot()


persistedSpecs can be nil, which cause panic later, please check all

hzxuzhonghu · 2025-09-16T08:43:51Z

pkg/bpf/restart/bpf_update.go

+					log.Warnf("failed to remove old map pinpath: %v (continuing)", err)
+				}
+			case hasNew && !hasOld:
+				if _, err := migrateMap(nil, newSpec, pkgName, mapName, pinPath); err != nil {


Would prefer directly use call the map create function

hzxuzhonghu · 2025-09-16T08:45:34Z

pkg/bpf/restart/bpf_update.go

+	return versionMap
+}
+
+func updateVersionInfo(versionMap *ebpf.Map) {


we can share with that one storeVersionInfo in pkg/bpf/bpf.go

I need to import package restart in pkg/bpf/bpf.go. If i call storeVersionInfo in pkg/bpf/bpf.go, this will result in recursive import.

hzxuzhonghu · 2025-09-16T08:51:18Z

test/e2e/upgarde_test.go

+}
+
+// patchKmesh applies a strategic merge patch to the Kmesh DaemonSet and waits for rollout completion.
+func patchKmesh_upgarde(t framework.TestContext, patchData string) {


typo upgrade

hzxuzhonghu · 2025-09-16T08:52:30Z

test/e2e/upgarde_test.go

+func upgradeKmesh(t framework.TestContext) {
+	newImage := os.Getenv("KMESH_UPGRADE_IMAGE")
+	if newImage == "" {
+		newImage = "localhost:5000/kmesh"


isnot this same image with old daemonset?

I attached a new image which is build in run_test.sh set_daemonupgarde_testcase_image() to KMESH_UPGRADE_IMAGE. But E2e test istio 1.23~1.25 report no space left in device. I think the reason is there are two images existing ((localhost:5000/kmesh) and localhost:5000/kmesh/test-upgrade-map-change)

kmesh-bot · 2025-09-21T09:51:39Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kevin-wangzefeng for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LiZhenCheng9527 · 2025-10-15T02:17:51Z

We update configuration of e2e to fix this errors. Please rebase and then resubmit the code.

Signed-off-by: 072020127 <mhy200253@gmail.com>

kmesh-bot added the kind/feature label Sep 11, 2025

kmesh-bot requested review from Okabe-Rintarou-0 and YaoZengzeng September 11, 2025 09:14

kmesh-bot added the size/XXL label Sep 11, 2025

gemini-code-assist bot reviewed Sep 11, 2025

View reviewed changes

072020127 force-pushed the main branch from 9d71daf to 7ea199f Compare September 15, 2025 03:42

kmesh-bot added the do-not-merge/contains-merge-commits label Sep 15, 2025

072020127 force-pushed the main branch from 3e5f5f3 to 59ecb57 Compare September 15, 2025 07:03

kmesh-bot added do-not-merge/contains-merge-commits and removed do-not-merge/contains-merge-commits labels Sep 15, 2025

072020127 force-pushed the main branch from df31519 to 843f6bb Compare September 16, 2025 07:20

kmesh-bot removed the do-not-merge/contains-merge-commits label Sep 16, 2025

hzxuzhonghu reviewed Sep 16, 2025

View reviewed changes

072020127 force-pushed the main branch from 843f6bb to d39b94d Compare September 21, 2025 09:51

kmesh-bot added the do-not-merge/contains-merge-commits label Sep 21, 2025

072020127 force-pushed the main branch 4 times, most recently from a21a29c to aee4427 Compare September 21, 2025 11:08

kmesh-bot added do-not-merge/contains-merge-commits and removed do-not-merge/contains-merge-commits labels Sep 21, 2025

072020127 force-pushed the main branch from 8e70a2d to f222714 Compare September 26, 2025 02:24

kmesh-bot removed the do-not-merge/contains-merge-commits label Sep 26, 2025

072020127 force-pushed the main branch 3 times, most recently from ba67a19 to 5ffbc87 Compare September 29, 2025 02:25

kmesh-bot added the do-not-merge/contains-merge-commits label Oct 15, 2025

072020127 added 19 commits October 15, 2025 16:29

add kmesh-daemon upgrades traffic without disruption

2bb08bb

Signed-off-by: 072020127 <mhy200253@gmail.com>

Modify the e2e image acquisition logic and migrateMap

3213645

Signed-off-by: 072020127 <mhy200253@gmail.com>

Modify the e2e image acquisition logic and migrateMap

4db6ffc

Signed-off-by: 072020127 <mhy200253@gmail.com>

make gen check

ccf1bdf

Signed-off-by: 072020127 <mhy200253@gmail.com>

add copy right

ed284d8

Signed-off-by: 072020127 <mhy200253@gmail.com>

golangci-lint

4d85d39

Signed-off-by: 072020127 <mhy200253@gmail.com>

modify bpf_test.go

1c367ff

Signed-off-by: 072020127 <mhy200253@gmail.com>

deal with ut

c992fb6

Signed-off-by: 072020127 <mhy200253@gmail.com>

pkg/bpf/bpf_test.go

76e18a6

Signed-off-by: 072020127 <mhy200253@gmail.com>

add auto generate map spec loader and fix bug

b66113b

Signed-off-by: 072020127 <mhy200253@gmail.com>

add auto generate map spec loader and fix bug

ff3e93f

Signed-off-by: 072020127 <mhy200253@gmail.com>

add copy right

29002a0

Signed-off-by: 072020127 <mhy200253@gmail.com>

fix Makefile

7a5f6f4

Signed-off-by: 072020127 <mhy200253@gmail.com>

fix os import

8766f62

Signed-off-by: 072020127 <mhy200253@gmail.com>

fix upgrade test

06c9f0b

Signed-off-by: 072020127 <mhy200253@gmail.com>

e2e test add new map

35e3c9f

Signed-off-by: 072020127 <mhy200253@gmail.com>

add new test case

ff7f984

Signed-off-by: 072020127 <mhy200253@gmail.com>

fix run test

73d0a48

Signed-off-by: 072020127 <mhy200253@gmail.com>

Re-trigger CI

715adb6

Signed-off-by: 072020127 <mhy200253@gmail.com>

072020127 force-pushed the main branch from 5a80c93 to 715adb6 Compare October 15, 2025 08:30

kmesh-bot removed the do-not-merge/contains-merge-commits label Oct 15, 2025

072020127 added 2 commits October 15, 2025 16:45

rename mapspce loader files

8ce1737

Signed-off-by: 072020127 <mhy200253@gmail.com>

add flag skip build update image

e907308

Signed-off-by: 072020127 <mhy200253@gmail.com>

		mapspec, _ := restart.LoadCompileTimeSpecs(config)
		restart.SnapshotSpecsByPkg(mapspec)

	mapspec, err := restart.LoadCompileTimeSpecs(config)
	mapSpecs, err := restart.LoadCompileTimeSpecs(config)

Feat: add kmesh-daemon upgrades traffic without disruption #1503

Are you sure you want to change the base?

Feat: add kmesh-daemon upgrades traffic without disruption #1503

Conversation

072020127 commented Sep 11, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hzxuzhonghu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kmesh-bot commented Sep 21, 2025

Uh oh!

LiZhenCheng9527 commented Oct 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Sep 16, 2025 •

edited

Loading