Create new types of tests

[mmaryo]

Hello,

Currently, the web-framework benchmark tests an empty endpoint.

However, web frameworks are about more than just request delay; they are mainly about processing complex tasks like JSON serialization, object composition, DB requests, etc.

Have you ever considered enriching the benchmark with new types of tests? It could provide a more comprehensive evaluation of web frameworks.

Imagine the following endpoints:
/ - the current behavior, we check the response time
/external-service - we call external services (a function that does a sleep)
/serialization - we send and receive a large JSON body in POST request with few query parameters
/http-client - we test the standard HTTP client and load a large file hosted in the server itself
/web - we do complete web scenarios with JSON in requests, DB requests, data transformations, and large JSON in response

We also deprecated the endpoints :
/user/{id}
/user
Because they do not bring value compared to the / endpoint

The current benchmark does not need to be updated. But each new piece of code that is not a framework update must implement that logic.

[waghanza]

Hey,

Thanks for this proposal.

Seems to go in a direction that I'm confortable with

I suggest slight enhancement.

• /user/0 -> instead of /
  • the idea is to test routing condition
  • and the response
• the database used for the /web endpoint should be in-memory and local and the same across all framework (standard like sqlite)
  • better to avoid network latency and else that could mess the results
  • better to avoid persistency that could mess the reuslts

[cyrusmsk]

Can you make please a message with description of new endpoints and the plan when they will be enabled in testing?

[waghanza]

Not sure to get you @cyrusmsk

[cyrusmsk]

As I understood you want to introduce new endpoints for testing? With sleep with json maybe with SQLite.
I’m wondering about more detailed description and the timeline of incorporation of these new tests

[waghanza]

The idea is to make new tests, so as this project could be meaningul. But we plan to do this incrementally, no timeline defined at this time

[cyrusmsk]

The idea is to make new tests, so as this project could be meaningul. But we plan to do this incrementally, no timeline defined at this time

In that case maybe new filter field will be helpful. Something like "Old tests" - "New tests". So in the result will be possible to see different implementations without any breaking changes.

[waghanza]

You mean add this on UI

If you make somes PRs why not ... Just a joke

I'm more on the line of having a new UI when all tests are ready, and in the meantime we can just make the backend code step by step

[waghanza]

We are going to write new test (add some endpoints)

• /, check the routing part of a framework
• /external-service, check how frameworks call external services (database, authentication api ...)
• /serialization, do some computation with few query parameters analysis
• /http-client, how the framework loads a big file
• /web - all scenarios with a real db (and json response)

I plan to use a real sqlite database

• convient for me to create a database as a file
• without the burden of setting up a database
• no noise about network latency during database communication
• but their is IO operation (due to file database)
• however all frameworks will be implemented like that, not sure if this is so terrible do use a local database

any thought @whiplash @cyrusmsk @ahopkins @appleboy @dalisoft @sy-records @System-Glitch @pimbrouwers @panesofglass @0xTim @leocavalcante @lukaszlenart ?

[waghanza]

cc @mario-huang

[System-Glitch]

Many frameworks don't have anything related to the features you want to test. It would essentially be testing the standard library. I'm not sure those would be very valuable. I thinking mainly of:

• how the framework loads a big file
• do some computation with few query parameters analysis

[dalisoft]

@waghanza

I think /external-service is not necessary as this may different at different cases. And maybe even out-of-scope for this project. Database performance is not web-server task (i could be wrong, fix me if it is)
But these kind of tests are interesting for me:

• /static - Static endpoint like {URL}/health
• /foo/:bar - Dynamic endpoint to parse parameter
• /foo/query?bar=1 - Static endpoint with query
• /upload/json - JSON deserialize/parse performance of server or it's middleware
• /serialization - JSON serialize performance of server or it's middleware
• /upload/BIG_FILE.mp4 - File upload performance

At best, if those tests could be run at same LAN/network but over 10GbE network not at same machine as running server

Here another benchmark alternative to TechEmpower and this repo - https://github.com/SaltyAom/bun-http-framework-benchmark

[waghanza]

so for you @System-Glitch

• /serialization, do some computation with few query parameters analysis
• /http-client, how the framework loads a big file
  are not necessary since not in all frameworks

I'll argue that this is what we have mainly in server-side API, thus necessary to implement.
Also, if certain frameworks could not have those feature (which I doubt, since we could implement it it any language), we should just drop them of

---

for you @dalisoft /external-service is not necessary, but I can argue that some API could use any external service (the first one is a distant db)

[cyrusmsk]

Do all frameworks will require to implement all tests? or they can skip some of them to be presented in the table?
Will the frontend allow to see separate results of the tests or only combined result of all tests?

[waghanza]

Do all frameworks will require to implement all tests ?

Yes, the new ones

Will the frontend allow to see separate results of the tests or only combined result of all tests?

Not thought about it. But could be intersting to have an customer choice to pick 1 endpoint or all

[cyrusmsk]

@dalisoft @System-Glitch I think the project should decide on the scope..
I don't have heavy knowledge in this, but what I found when I've made some simple research it is possible to say that there are web-servers and web-frameworks. And web-frameworks could be micro and full-stack.
It seems there is no established terminology - like when we can say it is just a web-server or it is already a micro web-framework or when we can say that the solution is not micro framework anymore but already full stack (sometimes general purpose) web-framework.

The name of the repo is "web-frameworks" so probably it should be more than web-server functionality, but maybe not necessary "full stack" frameworks. Especially most of the presented solutions in the repo are web-servers/micro frameworks.

So there could be 2 types of tests for micro and full stack frameworks and 2 tables.

@waghanza also recently I watched the video (https://www.youtube.com/watch?v=qB422eeOFr8) and we can leverage on the ideas from this test. One of the nice idea is the size of the headers in request. Sending small request for Hello world - not really relevant to real use cases. Maybe only in some micro-service architecture (but probably they will use something like gPRC in that case).