-
Notifications
You must be signed in to change notification settings - Fork 677
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Practical proposal for a minimal I/O benchmark #6317
Comments
Any news about improvements? @waghanza |
No, but it should be the next move for this project. Not sure about an incremental update (hard to deal with with front end ui). 'll work on something that reflect that reflect real world workload + easy to implement |
Maybe start with another branch that will work independently? |
But not sure about having a dynamic feature list. In terms of maintainability, in the long term, it is not so easy, and it could complexity results |
are you aware of the Techempower framework benchmarks? I wasn't sure if this was still happening, since they haven't posted a benchmark in over a year - but it looks like they just finished a new benchmark run last week: TechEmpower/FrameworkBenchmarks#8501 it's a very comprehensive benchmark, across many different platforms and frameworks - and it looks like there's quite a lot of new entries this year, including https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/JavaScript They're likely already doing most of what you had planned, for most of the same frameworks, so with new results coming up, I would need to ask if perhaps we're reinventing the wheel here? |
I haven't check recently, but in the past Techempower had a lot of cheats and hacks in the implementations, that has nothing relevant to "production ready" solutions. |
I'll say that techempower and this project do not have the same goal. This project is more for developers that are willing to test their frameworks, and this one is more to have some clues about what stack to choose when you start a project. For exemple, in TFB some frameworks are checked is some outdated version of language ... here we pledge to use the same version language (and a LTS one), currently java 17 for exemple, unless the framework behind is not ready for it. |
Huh, you're right - I never realized, but just look at some of the dockerfiles in the Javascript folder... everything from node 14 to 20. How is that a fair comparison of anything. 🤔 |
Yes. Some implementations also have hacks.. in routing. So users need to careful to find the proper numbers in that benchmark |
so what is you plan for the next steps? |
To discuss a typical workload scenario and to start implementing it. It will be too complicated to have multiple scenarios. I hope I'll start a discussion on Friday |
Which typical scenarios are existed in that case? |
Currently, this is a benchmark of raw HTTP round-trip overhead and basic routing.
This is meaningful, but doesn't tell you much about real-world scenarios, where actual body content needs to be sent: there is no real-world application in which every route returns an empty body.
I've read the other issues regarding this issue, including #110, #3803 and #1728.
Other proposals have suggested making database connections, which doesn't strictly say much about web frameworks, and starts to move into the territory of full-stack frameworks, which most of these frameworks aren't. You would need to pick database integrations for each of the implementations, which moves away from testing raw framework overhead. It's also considerably more work to implement. Things like JSON encoding gets you into similar situations: now you're benchmarking the standard language run-time (or a library) rather than the framework.
I'd like to propose something simpler:
10kb.txt
,1mb.txt
,100mb.txt
.files/10kb.txt
etc. and return the file.10kb.txt
should be cached in-memory: this will indicate raw output performance. (REST APIs, cached static assets, etc.)This should be fairly easy to add to almost every implementation, and will make the results a lot more realistic. People looking to use a framework for a REST API might be more interested in the
10kb.txt
results, while someone looking to serve images or videos might look at the1mb.txt
or100mb.txt
results, and so on.Of course, this won't tell you anything about JSON, XML or GraphQL parsing or encoding overhead - but this is currently a benchmark of raw server overhead, and this proposal keeps it that way. It's still useful information for someone building from scratch, or someone choosing a web server for building a library - it can inform you about baseline server performance, all application logic aside.
And it's probably a more reasonable and realistic next step, as there are a lot of benchmarks in many different tech stacks that need to be updated.
I would suggesting adding something like
features: [ "io" ]
toconfig.yaml
to indicate this benchmark feature is implemented, so that benchmarks can progressively move forward - and eventually adding an option to the website to only display frameworks that implement this feature.The text was updated successfully, but these errors were encountered: