How to handle connection storms/churns during topology changes

[ryanrazsa]

Hi there.

I am looking for some advice on how to handle connection storms that are happening during topology changes. Currently when we do some topology changes on redis, it seems like the rueidis client starts hammering redis with connection requests. This is causing the nodes to get overwhelmed and not come back up.

Is there any tweaks on the rueidis client that can be done to mitigate this kind of problem?

Currently using v1.0.60

The current client options we are using

rueidis.ClientOption{
	ConnWriteTimeout: 10 * time.Second, // explicitly set to the rueidis default; otherwise, it would be computed from Dialer.KeepAlive - e.g 60s * 10
	Dialer: net.Dialer{
		KeepAlive: time.Second * 60, 
	}, // To decrease the pings
	DisableCache:      true, // client cache is not enabled on our redis instance
	PipelineMultiplex: 4,
	MaxFlushDelay:     20 * time.Microsecond,
	AlwaysPipelining:  true,
	DisableRetry:      true,
},

When this is happening there are consistent spikes in the metrics
rueidis_dial_attempt
rueidis_dial_success
rueidis_dial_conns

[rueian]

Hi @ryanrazsa, PipelineMultiplex=4 is probably too much. It will use 16 connections to each of your cluster nodes for doing auto pipelining. Generally, PipelineMultiplex=0 is enough in the cluster mode.

[ryanrazsa]

Thanks for the reply. We have tried lower pipeline multiplex, but it gives us higher latency as the multiplex goes down. We are doing 16k to 20k read tps per instance(1.3M to 2.5M for the cluster). Is there any way to get rueidis to slowly reestablish connections with jitter and backoff during these times?

[rueian]

One way I can think of is having a customized DialCtxFn function doing the rate limiting or jittering.

[ryanrazsa]

Thank you I will give this a try in the coming weeks. I am not too familiar with this option so I'll poke around