-
Notifications
You must be signed in to change notification settings - Fork 395
feat(agent): support multiple flow log senders #10177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
svc-design
wants to merge
27
commits into
deepflowio:main
Choose a base branch
from
svc-design:codex/analyze-deepoflow-agent-net-flow-log
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
feat(agent): support multiple flow log senders #10177
svc-design
wants to merge
27
commits into
deepflowio:main
from
svc-design:codex/analyze-deepoflow-agent-net-flow-log
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…et-flow-log Support multiple flow log senders
…or-high-throughput Add MPMC queue implementation 引入了基于 crossbeam-channel 的 全新 MPMC(多生产者多消费者)队列实现,用于替代或补充现有的 ring-buffer 实现。 🔧 主要改动 在 agent/crates/public/src/queue/ 下新增: mpmc_queue.rs:实现基于 crossbeam_channel 的 Sender、Receiver 和 StatsHandle,支持 MPMC 模型; 修改 queue/mod.rs: 引入并导出 mpmc_queue,增加 bounded_mpmc() 构造函数
…ub/workflows Replace custom Cirun runners with default GitHub runners
…-atomicusize Fix AtomicUsize import
…public-module Fix AtomicUsize import
…ersion Update arm64 runner for verify agent
…-deepflow-agent Fix clone derivation for BoxedTaggedFlow
…rror Fix rustfmt issues in agent
…ation-for-performance Add dynamic queue with expansion
…ent-of-thread-and-queue-settings Add auto tuning for sender configuration
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is for:
Feature:
背景问题
在专属采集器或高速数据面(如 2 Mpps、L4 Flow 限速 4 万条/s)下,现有链路中仅包含单个发送队列和单个
UniformSenderThread
,很容易出现写入过快、队列被写满、日志覆盖丢弃的问题,最终影响日志完整性与可观测性。本次改动实现
✅ 主要改动
在 agent/crates/public/src/queue/ 下新增:mpmc_queue.rs:实现基于 crossbeam_channel 的 Sender、Receiver 和 StatsHandle,支持 MPMC 模型;
修改 queue/mod.rs: 引入并导出 mpmc_queue,增加 bounded_mpmc() 构造函数
✅ 新增功能:
引入可配置的 日志发送并发度参数,包括:
l4_flow_senders
l7_flow_senders
metric_senders
pcap_senders
等每类日志类型按配置值创建多个:
DebugSender
队列(queue::bounded_with_debug
)UniformSenderThread
实例所有 sender 命名后缀追加编号(如
3-flowlog-to-collector-sender-0
~-3
)✅ 日志发送改为并发分发:
负载均衡策略:目前支持:
发送线程存入
Vec<UniformSenderThread<_>>
,在Trident::start()
中统一启动,优雅退出时统一停止✅ 配置示例(YAML):