The benchmark result from 2025-02-10 is definitely incorrect

[KostyaTretyak]

The first inaccuracy, which is immediately noticeable, is the abnormally strong difference between benchmarks of the same frameworks, but on node.js and bun.js. For example, ditsmod on node.js shows 3.5 K per second, and on bun.js - 99 K. In fact, ditsmod v3 on bun.js works 1.5 - 2 times faster, but definitely not 30 times faster.

Even if we take the results for node.js alone, the benchmarks are still clearly incorrect. For example, nestjs-express shows even better results than bare express, which is impossible if both these frameworks are running on the same version of node.js. Correct results should show that express is always 15-20% faster than nestjs-express.

In addition, express should show more than half the performance of ditsmod or fastify.

[waghanza]

Could you check the implementation then ?

Maybe a cluster thing then

[cyrusmsk]

Many frameworks changed significantly
For example in Go - only one showing numbers close to previous results
Every other solution dropped to around 4k-10k from 100k-500k
Same for C++ frameworks - significant drop

[KostyaTretyak]

@waghanza, I can't tell you what's wrong with the workflow, but something is definitely not going as it should.

This is easy to check:

cd javascript/ditsmod
npm install
npm run build
NODE_APP=dist/main.js node cluster.mjs

From second terminal run 3-5 times this command:

wrk -H 'Connection: close' -d 5s -c 8 --timeout 8 -t 4 http://0.0.0.0:3000

The outputs:

Running 5s test @ http://0.0.0.0:3000
  4 threads and 8 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.94ms  386.05us   5.53ms   73.05%
    Req/Sec     1.87k   108.61     2.05k    80.50%
  37315 requests in 5.00s, 3.35MB read
Requests/sec:   7457.23
Transfer/sec:    684.55KB

Stop ditsmod in first terminal and then run this commands:

cd ../ditsmod-bun
bun install
bun run build
bun run start

From second terminal run 3-5 times this command:

wrk -H 'Connection: close' -d 5s -c 8 --timeout 8 -t 4 http://0.0.0.0:3000

The last command outputs:

Running 5s test @ http://0.0.0.0:3000
  4 threads and 8 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   414.17us  473.23us  10.70ms   91.65%
    Req/Sec     3.70k   550.42     6.43k    93.07%
  74308 requests in 5.10s, 5.31MB read
  Socket errors: connect 0, read 74302, write 0, timeout 0
Requests/sec:  14568.88
Transfer/sec:      1.04MB

So, ditsmod runs x2 as fast on Bun.js as it does on Node.js, but definitely not x30 as shown in the benchmark results from 2025-02-10.

[waghanza]

Could you make the same for all tree endpoints?

Maybe results are totally wrong for one of them

[KostyaTretyak]

It doesn't make sense for everyone to do this, but for example for Koa:

• Node.js - 7190 (3254 in the last benchmarks on the website)
• Bun.js - 12219 (93368 in the last benchmarks on the website)

[akyoto]

The numbers are definitely wrong.
Take a look at Go and compare it with the 2024-12 results.

I have no clue why my own library is the only one showing the same results.

I'm just guessing, but maybe the machine got occupied with other tasks during the benchmark and a re-run could fix it?

[KostyaTretyak]

No, the current results should be different from 2024-12 because the HTTP keep alive feature is not currently used. But still, the benchmarks are definitely incorrect.

[waghanza]

The whole command is

wrk -H 'Connection: close' --connections 64 --threads 8 --duration 15 --timeout 1 --script /home/waghanza/web-frameworks/pipeline.lua http://172.17.0.2:3000/

@KostyaTretyak

The full log is here =>https://gist.github.com/waghanza/92366d18948972463a918f3e3ece2be7

Seems that there is a lot of socket errors in favor of bun

[KostyaTretyak]

@waghanza, regarding bun errors, I don't know if the benchmark results can be trusted. Other frameworks also throw these errors when running on bun.

But these logs do not contain logs from the framework. Why? Are they specifically disabled?

These logs should look like:

info [AppModule]: PreRouterExtension: setted route GET "/".

[Kaliumhexacyanoferrat]

So are we sure that we do not have server implementations that just ignore the connection close instruction as it is not common anymore? 🤔

To be honest it now feels more like a TCP handshake benchmark than a web server benchmark (or maybe more a HTTP/1.0 one). As long as I understood the new changes correctly.

[KostyaTretyak]

server implementations that just ignore the connection close instruction as it is not common anymore

The server receives not just the instruction, but also the HTTP version. If it does not follow the instruction to close the connection, then in the same way it can ignore any instructions at its discretion.

Can compliance with these instructions be easily verified?

To be honest it now feels more like a TCP handshake benchmark than a web server benchmark

Single Page Applications (which are very popular now) often use two web servers: one for static files, one for dynamic requests. Benchmarks are usually run for frameworks with dynamic requests, and probably in 90% of cases, such a feature as "keep alive" is not used (in particular because of use a single GraphQL query).

So, in the real world, dynamic queries are handled by closing the connection after a few requests (usually 1-3 requests). In my opinion, in real-world conditions, dynamic queries never get thousands of requests on a single connection, as has been the case in benchmarks so far.

[Kaliumhexacyanoferrat]

Can compliance with these instructions be easily verified?

You could do curl -v http://server http://server with the Connection: close header which issues two requests and states whether the connection has been re-used or not.

If you look at the results for Java it is pretty obvious that the first two frameworks are not HTTP compliant. See RFC 9112 section 9.6.

In my opinion, in real-world conditions, dynamic queries never get thousands of requests on a single connection, as has been the case in benchmarks so far.

I agree, both ends of this range feel kind of synthetic.

[cyrusmsk]

Many of these things were described just in previous Issue: #8116

Andrea even prepared a simple validation toolkit
And yes - many solutions from this repo don’t give a %%%% about http standard :)

[akyoto]

But why would somebody send "Connection: close" and immediately follow up with another request on the same connection?

I believe that's what's happening here in the benchmarks and this combination doesn't seem to make sense.

The RFC also states this:

Connection: close as a request header field indicates that this is the **last** request
that the client will send on this connection

A client that sends a "close" connection option MUST NOT send further requests on that connection

^ The benchmark seems to violate this rule.

[KostyaTretyak]

The benchmark seems to violate this rule

These rules should work for real-world applications. But in the real world, no web server receives thousands of requests per second on a single connection. Closing the connection after each request brings the benchmarks much closer to real-world usage than allowing thousands of requests to be made on a single connection.