fix: Issue with etcd key disappearing and unable to auto-re-register #4960
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
这种极端情况确实存在，需要重新注册服务 |
zcong1993
reviewed
Jun 18, 2025
core/discov/publisher.go
Outdated
| @@ -125,6 +125,8 @@ func (p *Publisher) keepAliveAsync(cli internal.EtcdClient) error { | |||
| } | |||
|
|
|||
| threading.GoSafe(func() { | |||
| ticker := time.NewTicker(5 * time.Second) | |||
core/discov/publisher.go
Outdated
| @@ -135,6 +137,16 @@ func (p *Publisher) keepAliveAsync(cli internal.EtcdClient) error { | |||
| } | |||
| return | |||
| } | |||
| case <-ticker.C: | |||
| resp, err := cli.Get(cli.Ctx(), p.fullKey) | |||
There was a problem hiding this comment.
这部分可以复用 doKeepAlive 的逻辑，感觉可以写成：
resp, err := cli.Get(cli.Ctx(), p.fullKey)
if err != nil {
logc.Errorf(cli.Ctx(), "etcd publisher check register key error: %s", err.Error())
continue
}
if len(resp.Kvs) == 0 {
logc.Errorf(cli.Ctx(), "etcd publisher key %s not found, will re-register", p.fullKey)
if err := p.doKeepAlive(); err != nil {
logc.Errorf(cli.Ctx(), "etcd publisher KeepAlive: %s", err.Error())
}
return
}
|
这两天太忙了，周六或者周日我再完善下。 |
kevwan
changed the title
|
优化了下，从定时器检查改为watch这个key，如果收到delete事件就会重新注册。 |
kevwan
force-pushed
the
fix-etcd-register
branch
from
f979851 to
04fe8dd
Compare
|
Thanks for your contribution! Will be merged in v1.9.0, which will be released in August. |
kevwan
force-pushed
the
fix-etcd-register
branch
2 times, most recently
from
1411476 to
7dc1613
Compare
kevwan
force-pushed
the
fix-etcd-register
branch
from
7dc1613 to
e833031
Compare
kevwan
force-pushed
the
fix-etcd-register
branch
from
e833031 to
585d307
Compare
kevwan
approved these changes
Aug 1, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
There is an issue in production where when go-zero uses etcd, under certain conditions, the registered key disappears, causing frontend services to be unable to find backend services.
Reproduction conditions
Then I simulated it locally by deleting the registered key in etcd. It won't regenerate, even if the service process still exists. So I used a simple strategy for re-registration: if getting this key doesn't exist, then put it. After using the new strategy, deleting this key will regenerate it.
(Since the time spent handling this was relatively short, only 30 minutes, the PR may not be very complete. Welcome to discuss with the author)