Q&A configuration size & complexity

[janweinkauff]

Question

I’m currently working on a project that aims to route telemetry data from a large number of tenants to a high number of distinct sinks, ranging from thousands to tens of thousands and beyond.

We’ve begun benchmarking Vector’s behavior with large configurations. Initial results using single configuration files with 10,000 components show startup times (for both vector and vector validate) between 20 seconds (for ~100 components) and 2800 seconds (for ~10,000 components).

Now we’re trying to better understand the sensible limits for configuration files and how to scale Vector effectively in such scenarios. It seems we’ll need to find a strategy for sharding configurations across many Vector instances.

Some questions we’d love feedback on:

• Does it make sense to create configuration files with more than 100, 1,000 or 10,000 components?
• What are your experiences or best practices for splitting large configurations into smaller, manageable pieces?
• Have you conducted any benchmarks to test the impact of configuration file size and complexity on Vector’s performance?

Vector Config

No response

Vector Logs

No response

[pront]

Hi @janweinkauff,

Does it make sense to create configuration files with more than 100, 1,000 or 10,000 components?

Technically, there is nothing preventing you to create such large pipelines. Performance needs to examined though.

Can you help me understand why you need so many components?

Maybe we can leverage templating here. It also sounds like a sharding strategy can help here.

What are your experiences or best practices for splitting large configurations into smaller, manageable pieces?

We allow spreading configuration over multiple files in a folder. For your case, I would implement a pre-config layer to manage them though.

Have you conducted any benchmarks to test the impact of configuration file size and complexity on Vector’s performance?

No, we usually focus on much smaller pipelines and stress test by pushing through as many events as possible.