keepalive not working

[nomercy9]

Hi, I am currently facing an issue that after configuring the keepalive, I still cannot see any keepalive being sent in TCPDUMP.

06:12:49.118951 enP57554s1 Out IP [client vm ip].[port] > [server lb ip].https: Flags [P.], seq 14565:19180, ack 7298, win 501, options [nop,nop,TS val 2833495177 ecr 3777546711], length 4615
06:12:49.120686 enP57554s1 In  IP [server lb ip].https > [client vm ip].[port]: Flags [.], ack 19180, win 486, options [nop,nop,TS val 3777560471 ecr 2833495177], length 0
06:12:49.120686 eth0  In  IP [server lb ip].https > [client vm ip].[port]: Flags [.], ack 19180, win 486, options [nop,nop,TS val 3777560471 ecr 2833495177], length 0
06:12:49.120698 azv80b6b3c7132 Out IP [server lb ip].https > [app container ip].[port]: Flags [.], ack 19180, win 486, options [nop,nop,TS val 3777560471 ecr 2833495177], length 0
06:12:49.789801 enP57554s1 In  IP [server lb ip].https > [client vm ip].[port]: Flags [P.], seq 7298:7648, ack 19180, win 501, options [nop,nop,TS val 3777561140 ecr 2833495177], length 350
06:12:49.789801 eth0  In  IP [server lb ip].https > [client vm ip].[port]: Flags [P.], seq 7298:7648, ack 19180, win 501, options [nop,nop,TS val 3777561140 ecr 2833495177], length 350
06:12:49.789858 azv80b6b3c7132 Out IP [server lb ip].https > [app container ip].[port]: Flags [P.], seq 7298:7648, ack 19180, win 501, options [nop,nop,TS val 3777561140 ecr 2833495177], length 350
06:12:49.789916 azv80b6b3c7132 In  IP [app container ip].[port] > [server lb ip].https: Flags [.], ack 7648, win 501, options [nop,nop,TS val 2833495848 ecr 3777561140], length 0
06:12:49.789943 eth0  Out IP [client vm ip].[port] > [server lb ip].https: Flags [.], ack 7648, win 501, options [nop,nop,TS val 2833495848 ecr 3777561140], length 0
06:12:49.789947 enP57554s1 Out IP [client vm ip].[port] > [server lb ip].https: Flags [.], ack 7648, win 501, options [nop,nop,TS val 2833495848 ecr 3777561140], length 0
06:17:26.197582 enP57554s1 In  IP [server lb ip].https > [client vm ip].[port]: Flags [R.], seq 1638214070, ack 19180, win 0, length 0
06:17:26.197582 eth0  In  IP [server lb ip].https > [client vm ip].[port]: Flags [R.], seq 1638214070, ack 19180, win 0, length 0

public static void main(String[] args) {
		ConnectionProvider provider =
				ConnectionProvider.builder("custom")
				                  .maxConnections(50)
				                  .maxIdleTime(Duration.ofSeconds(20))           
				                  .maxLifeTime(Duration.ofSeconds(60))           
				                  .pendingAcquireTimeout(Duration.ofSeconds(60)) 
				                  .evictInBackground(Duration.ofSeconds(120))    
				                  .build();

		HttpClient client = HttpClient.create(provider)
                      .option(ChannelOption.SO_KEEPALIVE, true)            
				          .option(EpollChannelOption.TCP_KEEPIDLE, 100)        
				          .option(EpollChannelOption.TCP_KEEPINTVL, 10)        
				          .option(EpollChannelOption.TCP_KEEPCNT, 2);

Expected Behavior

After 100s of idle, there should be keepalive sent to the server. and after 2 Keepalive, client should sent Fin to the server to close the connection.

Actual Behavior

No Keepalive nor FIN captured in TCPDUMP but only server RST the connection after idle timeout.

Possible Solution

• Reactor version(s) used:
  Springboot 3.4
• JVM version (java -version): 21
• OS and version (eg. uname -a):

[violetagg]

@nomercy9 Please provide uname -a

[nomercy9]

@nomercy9 Please provide uname -a

Linux [xxx] 5.15.0-1092-azure #101-Ubuntu SMP Fri Jun 27 23:06:02 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

@violetagg

[violetagg]

@nomercy9 I cannot reproduce this, I see that pings are sent. Can you enable the reactor.netty logs and verify that you run with Epoll. The thread name should contain epoll and in the logs you should have Default Epoll support : true

[nomercy9]

@violetagg

{"@timestamp":"2025-09-29T08:35:56.48267736Z","timestamp":"2025-09-29 15:35:56,482","level":"DEBUG","message":"[http] resources will use the default LoopResources: DefaultLoopResources {prefix=reactor-http, daemon=true, selectCount=4, workerCount=4} NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"reactor.netty.tcp.TcpResources"}
{"@timestamp":"2025-09-29T08:35:56.482901563Z","timestamp":"2025-09-29 15:35:56,482","level":"DEBUG","message":"[http] resources will use the default ConnectionProvider: reactor.netty.resources.DefaultPooledConnectionProvider@8f65a66 NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"reactor.netty.tcp.TcpResources"}
{"@timestamp":"2025-09-29T08:35:56.483027364Z","timestamp":"2025-09-29 15:35:56,483","level":"DEBUG","message":"Successfully started bean 'reactorResourceFactory' NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"org.springframework.context.support.DefaultLifecycleProcessor"}
{"@timestamp":"2025-09-29T08:35:56.483138765Z","timestamp":"2025-09-29 15:35:56,483","level":"DEBUG","message":"Starting beans in phase 1 NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"org.springframework.context.support.DefaultLifecycleProcessor"}
{"@timestamp":"2025-09-29T08:35:57.776177965Z","timestamp":"2025-09-29 15:35:57,776","level":"DEBUG","message":"-Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024 NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.util.internal.InternalThreadLocalMap"}
{"@timestamp":"2025-09-29T08:35:57.776341267Z","timestamp":"2025-09-29 15:35:57,776","level":"DEBUG","message":"-Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096 NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.util.internal.InternalThreadLocalMap"}
{"@timestamp":"2025-09-29T08:35:58.087395475Z","timestamp":"2025-09-29 15:35:58,087","level":"DEBUG","message":"-Dio.netty.leakDetection.level: simple NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.util.ResourceLeakDetector"}
{"@timestamp":"2025-09-29T08:35:58.087580878Z","timestamp":"2025-09-29 15:35:58,087","level":"DEBUG","message":"-Dio.netty.leakDetection.targetRecords: 4 NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.util.ResourceLeakDetector"}
{"@timestamp":"2025-09-29T08:35:58.284636863Z","timestamp":"2025-09-29 15:35:58,284","level":"DEBUG","message":"brotli4j not in the classpath; Brotli support will be unavailable. NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.handler.codec.compression.Brotli"}
{"@timestamp":"2025-09-29T08:35:58.289745323Z","timestamp":"2025-09-29 15:35:58,289","level":"DEBUG","message":"Default io_uring support : false NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"reactor.netty.resources.DefaultLoopIOUring"}
{"@timestamp":"2025-09-29T08:35:58.382534799Z","timestamp":"2025-09-29 15:35:58,382","level":"DEBUG","message":"-Dio.netty.native.workdir: /tmp (io.netty.tmpdir) NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.util.internal.NativeLibraryLoader"}
{"@timestamp":"2025-09-29T08:35:58.382692901Z","timestamp":"2025-09-29 15:35:58,382","level":"DEBUG","message":"-Dio.netty.native.deleteLibAfterLoading: true NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.util.internal.NativeLibraryLoader"}
{"@timestamp":"2025-09-29T08:35:58.382758302Z","timestamp":"2025-09-29 15:35:58,382","level":"DEBUG","message":"-Dio.netty.native.tryPatchShadedId: true NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.util.internal.NativeLibraryLoader"}
{"@timestamp":"2025-09-29T08:35:58.382856503Z","timestamp":"2025-09-29 15:35:58,382","level":"DEBUG","message":"-Dio.netty.native.detectNativeLibraryDuplicates: true NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.util.internal.NativeLibraryLoader"}
{"@timestamp":"2025-09-29T08:35:58.477299598Z","timestamp":"2025-09-29 15:35:58,477","level":"DEBUG","message":"Successfully loaded the library /tmp/libnetty_transport_native_epoll_x86_642540664934307383265.so NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.util.internal.NativeLibraryLoader"}
{"@timestamp":"2025-09-29T08:35:58.483587371Z","timestamp":"2025-09-29 15:35:58,483","level":"DEBUG","message":"-Djava.net.preferIPv4Stack: true NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.util.NetUtil"}
{"@timestamp":"2025-09-29T08:35:58.483704873Z","timestamp":"2025-09-29 15:35:58,483","level":"DEBUG","message":"-Djava.net.preferIPv6Addresses: false NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.util.NetUtil"}
{"@timestamp":"2025-09-29T08:35:58.485683596Z","timestamp":"2025-09-29 15:35:58,485","level":"DEBUG","message":"Loopback interface: lo (lo, 127.0.0.1) NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.util.NetUtilInitializations"}
{"@timestamp":"2025-09-29T08:35:58.48693481Z","timestamp":"2025-09-29 15:35:58,486","level":"DEBUG","message":"/proc/sys/net/core/somaxconn: 4096 NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.util.NetUtil"}
{"@timestamp":"2025-09-29T08:35:58.487277514Z","timestamp":"2025-09-29 15:35:58,487","level":"DEBUG","message":"Default Epoll support : true NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"reactor.netty.resources.DefaultLoopEpoll"}
{"@timestamp":"2025-09-29T08:35:58.572141499Z","timestamp":"2025-09-29 15:35:58,572","level":"DEBUG","message":"-Dio.netty.eventLoopThreads: 2 NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.channel.MultithreadEventLoopGroup"}
{"@timestamp":"2025-09-29T08:35:58.66880832Z","timestamp":"2025-09-29 15:35:58,668","level":"DEBUG","message":"-Dio.netty.globalEventExecutor.quietPeriodSeconds: 1 NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.util.concurrent.GlobalEventExecutor"}
{"@timestamp":"2025-09-29T08:35:58.776794273Z","timestamp":"2025-09-29 15:35:58,776","level":"DEBUG","message":"org.jctools-core.MpscChunkedArrayQueue: available NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.util.internal.PlatformDependent"}
{"@timestamp":"2025-09-29T08:35:58.884488922Z","timestamp":"2025-09-29 15:35:58,884","level":"DEBUG","message":"Default ResolvedAddressTypes: IPV4_ONLY NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.resolver.dns.DnsNameResolver"}
{"@timestamp":"2025-09-29T08:35:58.884632624Z","timestamp":"2025-09-29 15:35:58,884","level":"DEBUG","message":"Localhost address: localhost/127.0.0.1 NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.resolver.dns.DnsNameResolver"}
{"@timestamp":"2025-09-29T08:35:58.884692524Z","timestamp":"2025-09-29 15:35:58,884","level":"DEBUG","message":"Windows hostname: null NR-LINKING||[Redacted]|||[Redacted]|","stackTrace":"","logger":"io.netty.resolver.dns.DnsNameResolver"}

Here is the log I captured and I found Default Epoll support : true

[violetagg]

@nomercy9 That's good. So I tried the setup below and I cannot reproduce the issue, so please try to provide steps that we need to follow in order to reproduce the issue.

Reactor Netty version 1.2.10
Netty version 4.1.127.Final

Java version

openjdk version "21.0.8" 2025-07-15 LTS
OpenJDK Runtime Environment Corretto-21.0.8.9.1 (build 21.0.8+9-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.8.9.1 (build 21.0.8+9-LTS, mixed mode, sharing)

Docker

Ubuntu 24.04

Linux 1c5035c8a0f1 6.10.14-linuxkit #1 SMP Thu Aug 14 19:26:13 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

public class Application {

	public static void main(String[] args) {
		ConnectionProvider provider =
				ConnectionProvider.builder("custom")
						.maxConnections(50)
						.maxIdleTime(Duration.ofSeconds(20))
						.maxLifeTime(Duration.ofSeconds(60))
						.pendingAcquireTimeout(Duration.ofSeconds(60))
						.evictInBackground(Duration.ofSeconds(120))
						.build();

		HttpClient client = HttpClient.create(provider)
				.option(ChannelOption.SO_KEEPALIVE, true)
				.option(EpollChannelOption.TCP_KEEPIDLE, 30)
				.option(EpollChannelOption.TCP_KEEPINTVL, 10)
				.option(EpollChannelOption.TCP_KEEPCNT, 2);

		client.get()
				.uri("https://httpbin.org/get")
				.responseContent()
				.aggregate()
				.asString()
				.block();
	}

}

[nomercy9]

03:33:51.330525 eth0  In  IP [server lb ip].https > [vm ip].[port]: Flags [P.], seq 8113:8463, ack 11423, win 501, options [nop,nop,TS val 1817234340 ecr 3727635777], length 350
03:33:51.330581 azv1d28d00f305 Out IP [server lb ip].https > [container ip].[port]: Flags [P.], seq 8113:8463, ack 11423, win 501, options [nop,nop,TS val 1817234340 ecr 3727635777], length 350
03:33:51.374901 azv1d28d00f305 In  IP [container ip].[port] > [server lb ip].https: Flags [.], ack 8463, win 501, options [nop,nop,TS val 3727637864 ecr 1817234340], length 0
03:33:51.374948 eth0  Out IP [vm ip].[port] > [server lb ip].https: Flags [.], ack 8463, win 501, options [nop,nop,TS val 3727637864 ecr 1817234340], length 0
03:33:51.374949 enP57554s1 Out IP [vm ip].[port] > [server lb ip].https: Flags [.], ack 8463, win 501, options [nop,nop,TS val 3727637864 ecr 1817234340], length 0
03:39:02.984962 eth0  In  IP [server lb ip].https > [vm ip].[port]: Flags [R.], seq 2706896608, ack 11423, win 0, length 0

Hi @violetagg , I have tried the configuration but I still get the same issue.

{"@timestamp":"2025-09-30T03:33:42.077279619Z","timestamp":"2025-09-30 10:33:42,077","level":"DEBUG","message":"Creating a new [custom] client pool [PoolFactory{evictionInterval=PT2M, leasingStrategy=fifo, maxConnections=500, maxIdleTime=20000, maxLifeTime=60000, metricsEnabled=false, pendingAcquireMaxCount=1000, pendingAcquireTimeout=60000}] for [domain/<unresolved>:80] NR-LINKING|NjM3NjA3MnxBUE18QVBQTElDQVRJT058OTU0MjYwNjI0|[Redacted]|ef696d359a104c69909115a942fb2343|7c1246f0d7f55b24|[Redacted]","stackTrace":"","logger":"reactor.netty.resources.PooledConnectionProvider"}


{"@timestamp":"2025-09-30T03:33:42.871304795Z","timestamp":"2025-09-30 10:33:42,871","level":"DEBUG","message":"[f4a0431e] Created a new pooled channel, now: 0 active connections, 0 inactive connections 0 pending acquire requests. NR-LINKING|NjM3NjA3MnxBUE18QVBQTElDQVRJT058OTU0MjYwNjI0|[Redacted]|||[Redacted]|","stackTrace":"","logger":"reactor.netty.resources.PooledConnectionProvider"}


Executing the call...


{"@timestamp":"2025-09-30T03:35:42.193159085Z","timestamp":"2025-09-30 10:35:42,193","level":"DEBUG","message":"[f4a0431e, L:/[Redacted]:57596 ! R:[Redacted]] onStateChange(PooledConnection{channel=[id: 0xf4a0431e, L:/[Redacted]:57596 ! R:[Redacted]]}, [disconnecting]) NR-LINKING|NjM3NjA3MnxBUE18QVBQTElDQVRJT058OTU0MjYwNjI0|[Redacted]|||[Redacted]|","stackTrace":"","logger":"reactor.netty.resources.DefaultPooledConnectionProvider"}

Even after the channel cleaned and disconnected, there is no Fin sent by the app

[violetagg]

@nomercy9 Please provide steps that we need to follow in order to reproduce the issue.