Introduction
Processing large CSV files efficiently is a common requirement in many applications, from data analysis to ETL (Extract, Tr...
For further actions, you may consider blocking this person and/or reporting abuse
I couldn't help but see abnormalities with results. And no justice for nodejs there.
I cloned your repository and run tests by writing my javascript version running with both nodejs and bunjs.
Results good.
Processor: Ryzen 7 5800H
RAM: 16GB DDR4
Nodejs and Bunjs:
The same golang code for reference:
EDIT:
Added a PR github.com/rocklinda/csv-parsing-b...
Used node 22
why the second loop in Go, because you could have looked for the topProduct in the first loop.
(then you don't need a map, saving memory and such...)
I think this might be faster :-)
You are right I just realized now. Thank you for your sharp eye.
Can you check if the outcome is a bit more in favour of Go? (and let us know)
I made some changes you can see in my article above. Changing from two loops to one loop doesn't have any difference, I still need a map for storing variables and getting a product with the highest price. But, I think if in real-world cases which lot of logic will make a difference. Do you have any thoughts about this? Let me know...
Linda, please have a look at this (to understand more about what I meant)
goplay.space/#Ny0OT89_zNP
I can't test it, since I don't have the file.
Trick is: no map, no loop for lookup. This must be slightly better.
I think you misunderstood the case, topProduct is the sum of all the sales not the product with the highest price. There's no way I can get the sum value of each product with an integer variable. I still need a map to collect all the sums and get the highest value of total sales. My approach is the same as storing values in an array in PHP/NodeJS or a list in Python.
You're missing the case of duplicate products.
If a product appears more than once in the list, you need to add the values.
Using Polars with Python is more flexible and powerful for handling large datasets and performing complex data manipulation. While Golang is powerful, it isn't designed for working with datasets like data frames. However, Golang could dominate if a framework similar to Polars is developed for it
Thank you for your insight, in this case, I just want to try plain Python. Next time I will try Polars Python with a larger dataset.
I check GitHub. Why nestjs run web server? Other tests direct run. It's not fair. Maybe run js only nodejs or bun.
I thought about it as well, you are right. I will fix it later.
just an fyi: if you're benchmarking a php script, hrtime stopwatch is a better option.
php.net/manual/en/class.hrtime-sto...
In Go, you are converting the quantity to an int to then convert it to a float directly after. Converting to float immediately should noticably impact performance.
You could probably refactor the PHP code to use generators for better speed. 1 million rows is enough data to see an improvement. It's a tipping point, if it isn't enough data it's actually slower - go figure
I've never used generators before. Yeah, I will refactor this later to see the difference.
JIT enabled?
Why use a library for Javascript if you won't use a library for python?
The initial idea is that I really want to know if the NestJS framework is fast enough or not because I used this framework at my previous workplace. However, the nodeJS is already updated in GitHub with help from @amankrokx you can take a look.
I think the huge execution time difference is not because of using nestjs but rather the streaming way of getting CSV file and parsing it. The function call with rows can also contribute to it.
If we read the CSV all at once and then process it, I beleive it will run way within 6 seconds.
Then again, I haven't really used nestjs so maybe some other factors might be involved.
But overall, processing is processing, as long as compiler/interpreter produces similar machine code, it should perform similarly.
So compiled languages are faster than interpreted languages?
I can't say that is correct or incorrect because if we want to know about this we need to compare between all compile languages and interpreter. In this experiment I only used Go.
I would like to see Golang compared to other native code. All the other languages you tested are interpreted.
Yeah, that's a good idea. The reason why I use those languages is because of their popularity in BE technology, I don't consider between interpreter and compile.