Recently I released a small package for Go.
While building this, I came across various problems and best practices. So I just wanted to share my learnings.
This is my first priority when I build something for public. This makes your library widely compatible and reusable.
In this library, I made the decoder compatible with io.Reader. Now, I don't have to care about the source. It can be an HTTP request, a file, etc. Anything that implements io.Reader, is supported. I recommend you to read this excellent article from Mat Ryer
In simple words, avoid using external libraries for some simple tasks. Try to use the standard library as much as you can.
In the context of this library, I intentionally avoided some functionality, like validating resulting JSON, so that the importing library can reuse its existing dependencies. Stick to the original problem instead of providing some fancy functionality.
Writing unit tests is fun and I would say it makes your program more robust and easy to maintain. Whenever you change some logic, tests make sure your program is working as intended or not. If tests fail, then you know what to do!
This library provides 100% test coverage and this helped me allot.
You can test your program with the following command.
go test -cover
Don't just get to the conclusion without benchmarking your programs. Go tooling makes it super easy to benchmark your programs. You get the following information from benchmarks for a single function.
- Number of operations performed
- Time required for each operation in nanoseconds
- Data processed per second
- Memory required per operation
- Allocations made per operation
You can benchmark your program with the following command.
go test -bench=. -benchmem
Have a look at benchmarks from this package. As you can see, using io.Reader, memory is being reused resulting in almost 500% performance increase in data processing than existing available library jsonc
goos: windows goarch: amd64 pkg: github.com/akshaybharambe14/go-jsonc/benchmarks BenchmarkOwnSmallJSONBytes-4 256599 4952 ns/op 353.00 MB/s 0 B/op 0 allocs/op BenchmarkOwnSmallJSONBytesReader-4 206823 5832 ns/op 299.70 MB/s 6224 B/op 5 allocs/op BenchmarkJSONCSmallJSONBytes-4 171474 6925 ns/op 252.41 MB/s 1792 B/op 1 allocs/op BenchmarkOwnBigJSONBytes-4 33517 35921 ns/op 462.26 MB/s 0 B/op 0 allocs/op BenchmarkOwnBigJSONBytesReader-4 105244 11292 ns/op 1470.45 MB/s 6224 B/op 5 allocs/op BenchmarkJSONCBigJSONBytes-4 19599 61422 ns/op 270.34 MB/s 18432 B/op 1 allocs/op PASS ok github.com/akshaybharambe14/go-jsonc/benchmarks 26.250s
If you think your program is not fast enough, then you can profile your programs. This really helps in finding the culprits.
In my case, I identified that my decode function was spending almost 40% time in just checking the byte can be added to resulting JSON or not. Results, I ended up gaining a 40% performance increase.
You don't need any separate tools to profile your programs. Benchmarks can be used for this also. You can generate profiles and read them with pprof.
go test -bench=. -benchmem -memprofile memprofile.out -cpuprofile profile.out
I was happy with the above tweaks, but still, I was trying to avoid those 5 allocations per operations in the case of io.Reader as input (See benchmarks). But seriously stop being greedy and check it in the memory profile if those allocations are avoidable or not.
In my case, internal functions were making allocations to the heap and I can't control them. So I stopped there.
Don't stop here and work on the feedback provided by your colleagues, friends, and community. File issues and work on them in your free time.
That's it for now. It was a nice experience and I am looking forward to contributing to some opensource go projects.
Give it a try. Your feedback is highly appreciated. I will surely work on your inputs, that's what I just learned.