LEAK: ByteBuf.release() was not called before it's garbage-collected #580
Description
Bug Report
Hello we are using Spring Boot 3.0.2, Spring Data R2DBC and r2dbc-postgresql. Under normal conditions everything works fine. Though when stress testing our service we observe Netty leaks which force our service to restart. I have attached an example stacktrace below.
Stacktrace:
2023-01-27T11:57:32.841Z ERROR 1 --- [or-http-epoll-1] io.netty.util.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. bookstore-springboot
27/01/2023, 13:57:32 557a70965771472db1e734e2cf132246 Recent access records: bookstore-springboot
27/01/2023, 13:57:32 557a70965771472db1e734e2cf132246 Created at: bookstore-springboot
27/01/2023, 13:57:32 557a70965771472db1e734e2cf132246 io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:403) bookstore-springboot
27/01/2023, 13:57:32 557a70965771472db1e734e2cf132246 io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) bookstore-springboot
27/01/2023, 13:57:32 557a70965771472db1e734e2cf132246 io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) bookstore-springboot
27/01/2023, 13:57:32 557a70965771472db1e734e2cf132246 io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53) bookstore-springboot
27/01/2023, 13:57:32 557a70965771472db1e734e2cf132246 io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120) bookstore-springboot
27/01/2023, 13:57:32 557a70965771472db1e734e2cf132246 io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75) bookstore-springboot
27/01/2023, 13:57:32 557a70965771472db1e734e2cf132246 io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785) bookstore-springboot
27/01/2023, 13:57:32 557a70965771472db1e734e2cf132246 io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) bookstore-springboot
27/01/2023, 13:57:32 557a70965771472db1e734e2cf132246 io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) bookstore-springboot
27/01/2023, 13:57:32 557a70965771472db1e734e2cf132246 io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) bookstore-springboot
27/01/2023, 13:57:32 557a70965771472db1e734e2cf132246 io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) bookstore-springboot
27/01/2023, 13:57:32 557a70965771472db1e734e2cf132246 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) bookstore-springboot
27/01/2023, 13:57:32 557a70965771472db1e734e2cf132246 java.base/java.lang.Thread.run(Thread.java:833) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 2023-01-27T11:57:30.835Z ERROR 1 --- [tor-tcp-epoll-1] io.netty.util.ResourceLeakDetector : LEAK: DataRow.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 Recent access records: bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 Created at: bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.r2dbc.postgresql.message.backend.DataRow.<init>(DataRow.java:37) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.r2dbc.postgresql.message.backend.DataRow.decode(DataRow.java:141) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.r2dbc.postgresql.message.backend.BackendMessageDecoder.decodeBody(BackendMessageDecoder.java:65) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.r2dbc.postgresql.message.backend.BackendMessageDecoder.decode(BackendMessageDecoder.java:39) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:208) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:294) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:403) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:411) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:333) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:454) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) bookstore-springboot
27/01/2023, 13:57:30 557a70965771472db1e734e2cf132246 java.base/java.lang.Thread.run(Thread.java:833)
Our code is very simple we are just using Spring Data Repositories:
interface BooksRepository : CoroutineCrudRepository<BookEntity, UUID> {
fun findBy(pageable: Pageable): Flow<BookEntity>
}
and then just using the above repository like this:
// Method in BooksDataStore
fun getBooks(limit: Int): Flow<Book> {
val pageRequest = PageRequest.ofSize(limit)
return booksRepository.findBy(pageRequest).map { it.toBook() }
}
@GetMapping
fun getBooks(
@RequestParam("limit", required = false, defaultValue = "1000") limit: Int,
): Flow<Book> {
return booksDataStore.getBooks(limit)
}
Is there anything we did configure wrongly or we have to watchout?