"Connection reset" error in TechEmpower benchmarks

[p8]

The Falcon Rack implementation in the TechEmpower bench raises the following errors:

    with unhandled exception.","event":{"type":"failure","root":"/rack","class":"Errno::ECONNRESET","message":"Connection reset by peer","backtrace":[
    /usr/local/lib/ruby/3.5.0+0/socket.rb:457:in 'BasicSocket#__read_nonblock'
    /usr/local/lib/ruby/3.5.0+0/socket.rb:457:in 'BasicSocket#read_nonblock'
    /usr/local/bundle/gems/io-stream-0.10.0/lib/io/stream/buffered.rb:138:in 'IO::Stream::Buffered#sysread'
    /usr/local/bundle/gems/io-stream-0.10.0/lib/io/stream/readable.rb:347:in 'IO::Stream::Readable#fill_read_buffer'
    /usr/local/bundle/gems/io-stream-0.10.0/lib/io/stream/readable.rb:268:in 'IO::Stream::Readable#gets'
    /usr/local/bundle/gems/protocol-http1-0.35.1/lib/protocol/http1/connection.rb:356:in 'Protocol::HTTP1::Connection#read_line?'
    /usr/local/bundle/gems/protocol-http1-0.35.1/lib/protocol/http1/connection.rb:378:in 'Protocol::HTTP1::Connection#read_request_line'
    /usr/local/bundle/gems/protocol-http1-0.35.1/lib/protocol/http1/connection.rb:399:in 'Protocol::HTTP1::Connection#read_request'
    /usr/local/bundle/gems/async-http-0.91.0/lib/async/http/protocol/http1/request.rb:26:in 'Async::HTTP::Protocol::HTTP1::Request.read'
    /usr/local/bundle/gems/async-http-0.91.0/lib/async/http/protocol/http1/server.rb:49:in 'Async::HTTP::Protocol::HTTP1::Server#next_request'
    /usr/local/bundle/gems/async-http-0.91.0/lib/async/http/protocol/http1/server.rb:66:in 'Async::HTTP::Protocol::HTTP1::Server#each'
    /usr/local/bundle/gems/async-http-0.91.0/lib/async/http/server.rb:49:in 'Async::HTTP::Server#accept'
    /usr/local/bundle/gems/falcon-0.52.3/lib/falcon/server.rb:57:in 'Falcon::Server#accept'
    /usr/local/bundle/gems/io-endpoint-0.15.2/lib/io/endpoint/wrapper.rb:216:in 'block (2 levels) in IO::Endpoint::Wrapper#accept'
    /usr/local/bundle/gems/async-2.32.0/lib/async/task.rb:207:in 'block in Async::Task#run'
    /usr/local/bundle/gems/async-2.32.0/lib/async/task.rb:452:in 'block in Async::Task#schedule'

The log can be found here (it's 56 MB):
https://tfb-status.techempower.com/unzip/results.2025-09-18-05-29-59-113.zip/results/20250910230442/rack-falcon/run/rack-falcon.log

The tests can be run with the following command:

./tfb --test rack-falcon

If I change the configuration to use non-persistent HTTP, the errors no longer occur but things get 10 times slower.

rack hostname do
   endpoint Async::HTTP::Endpoint.parse('http://0.0.0.0:8080').with(
     protocol: Async::HTTP::Protocol::HTTP1.new(persistent: false)
   )
 end

[ioquatix]

Thanks, I'll investigate.

[ioquatix]

Could you try testing with this branch?

socketry/protocol-http1#49

[p8]

@ioquatix Yes, that fixes the issue.

[ioquatix]

It's likely the reason for this issue is that the benchmark is not correctly closing the connections to the server.

[p8]

Yes, it's probably the same issue as: #205

[ioquatix]

On the one hand, we could just simply ignore badly behaving clients. But on the other hand, maybe it would be better to know about it and fix those clients. I don't have a strong opinion about where I land on this, but actively ignoring errors is not something I'm completely comfortable with, because in the past it's created situations that are extremely hard to debug (silently dropping errors).

[travisbell]

I recall seeing some related errors like I reported on #216. I stopped seeing Errno::EPIPE errors after a different patch (#289), but do continue to see Errno::ECONNRESET randomly in development:

Async::Task: Reading HTTP/1.1 requests for Async::HTTP::Protocol::HTTP1::Server. [oid=0x4088] [ec=0x4080] [pid=59948] [2025-10-02 09:00:03 -0600]
               | Task may have ended with unhandled exception.
               |   Errno::ECONNRESET: Connection reset by peer
               |   → <internal:io> 63:in 'IO#read_nonblock'
               |     /Users/travisbell/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/io-stream-0.10.0/lib/io/stream/buffered.rb:138 in 'IO::Stream::Buffered#sysread'
               |     /Users/travisbell/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/io-stream-0.10.0/lib/io/stream/readable.rb:347 in 'IO::Stream::Readable#fill_read_buffer'
               |     /Users/travisbell/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/io-stream-0.10.0/lib/io/stream/readable.rb:268 in 'IO::Stream::Readable#gets'
               |     /Users/travisbell/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/protocol-http1-0.35.1/lib/protocol/http1/connection.rb:356 in 'Protocol::HTTP1::Connection#read_line?'
               |     /Users/travisbell/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/protocol-http1-0.35.1/lib/protocol/http1/connection.rb:378 in 'Protocol::HTTP1::Connection#read_request_line'
               |     /Users/travisbell/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/protocol-http1-0.35.1/lib/protocol/http1/connection.rb:399 in 'Protocol::HTTP1::Connection#read_request'
               |     /Users/travisbell/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/async-http-0.91.0/lib/async/http/protocol/http1/request.rb:26 in 'Async::HTTP::Protocol::HTTP1::Request.read'
               |     /Users/travisbell/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/async-http-0.91.0/lib/async/http/protocol/http1/server.rb:49 in 'Async::HTTP::Protocol::HTTP1::Server#next_request'
               |     /Users/travisbell/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/async-http-0.91.0/lib/async/http/protocol/http1/server.rb:66 in 'Async::HTTP::Protocol::HTTP1::Server#each'
               |     /Users/travisbell/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/async-http-0.91.0/lib/async/http/server.rb:49 in 'Async::HTTP::Server#accept'
               |     /Users/travisbell/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/falcon-0.52.3/lib/falcon/server.rb:57 in 'Falcon::Server#accept'
               |     /Users/travisbell/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/wrapper.rb:216 in 'block (2 levels) in IO::Endpoint::Wrapper#accept'
               |     /Users/travisbell/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/async-2.33.0/lib/async/task.rb:207 in 'block in Async::Task#run'
               |     /Users/travisbell/.rbenv/versions/3.4.5/lib/ruby/gems/3.4.0/gems/async-2.33.0/lib/async/task.rb:452 in 'block in Async::Task#schedule'

Like I mentioned in the other issue, I don't usually see these in production but there's load balancers in between the client and the server so they're probably clients that are doing "the right thing". I only usually see these while working locally, and usually even then, specifically with Firefox for some reason.

However, on the other hand, I don't see these errors while running other servers like Puma. What does that mean? Puma is handling this client behaviour differently, or are they just ignoring this type of bad client like you're suggesting? 🤷🏼

[ioquatix]

After careful consideration, I'm ignoring Errno::ECONNRESET only in read_line?.

[p8]

Thanks @ioquatix !