loading...
Cover image for What I learnt while working my first open source package for Go

What I learnt while working my first open source package for Go

akshaybharambe14 profile image Akshay Bharambe ・3 min read

Recently I released a small package for Go.

GitHub logo akshaybharambe14 / go-jsonc

go-jsonc provides a way to work with commented json by converting it to plain json.

While building this, I came across various problems and best practices. So I just wanted to share my learnings.

Make it compatible with the standard library

This is my first priority when I build something for public. This makes your library widely compatible and reusable.

In this library, I made the decoder compatible with io.Reader. Now, I don't have to care about the source. It can be an HTTP request, a file, etc. Anything that implements io.Reader, is supported. I recommend you to read this excellent article from Mat Ryer

Reduce your API surface

In simple words, avoid using external libraries for some simple tasks. Try to use the standard library as much as you can.

In the context of this library, I intentionally avoided some functionality, like validating resulting JSON, so that the importing library can reuse its existing dependencies. Stick to the original problem instead of providing some fancy functionality.

Unit Tests are must

Writing unit tests is fun and I would say it makes your program more robust and easy to maintain. Whenever you change some logic, tests make sure your program is working as intended or not. If tests fail, then you know what to do!

This library provides 100% test coverage and this helped me allot.

You can test your program with the following command.

go test -cover

Benchmark your program

Don't just get to the conclusion without benchmarking your programs. Go tooling makes it super easy to benchmark your programs. You get the following information from benchmarks for a single function.

  • Number of operations performed
  • Time required for each operation in nanoseconds
  • Data processed per second
  • Memory required per operation
  • Allocations made per operation

You can benchmark your program with the following command.

go test -bench=. -benchmem

Have a look at benchmarks from this package. As you can see, using io.Reader, memory is being reused resulting in almost 500% performance increase in data processing than existing available library jsonc

goos: windows
goarch: amd64
pkg: github.com/akshaybharambe14/go-jsonc/benchmarks
BenchmarkOwnSmallJSONBytes-4              256599              4952 ns/op         353.00 MB/s           0 B/op          0 allocs/op
BenchmarkOwnSmallJSONBytesReader-4        206823              5832 ns/op         299.70 MB/s        6224 B/op          5 allocs/op
BenchmarkJSONCSmallJSONBytes-4            171474              6925 ns/op         252.41 MB/s        1792 B/op          1 allocs/op
BenchmarkOwnBigJSONBytes-4                 33517             35921 ns/op         462.26 MB/s           0 B/op          0 allocs/op
BenchmarkOwnBigJSONBytesReader-4          105244             11292 ns/op        1470.45 MB/s        6224 B/op          5 allocs/op
BenchmarkJSONCBigJSONBytes-4               19599             61422 ns/op         270.34 MB/s       18432 B/op          1 allocs/op
PASS
ok      github.com/akshaybharambe14/go-jsonc/benchmarks 26.250s

CPU profiling and memory profiling

If you think your program is not fast enough, then you can profile your programs. This really helps in finding the culprits.

In my case, I identified that my decode function was spending almost 40% time in just checking the byte can be added to resulting JSON or not. Results, I ended up gaining a 40% performance increase.

You don't need any separate tools to profile your programs. Benchmarks can be used for this also. You can generate profiles and read them with pprof.

go test -bench=. -benchmem -memprofile memprofile.out -cpuprofile profile.out

You can't avoid all allocations

I was happy with the above tweaks, but still, I was trying to avoid those 5 allocations per operations in the case of io.Reader as input (See benchmarks). But seriously stop being greedy and check it in the memory profile if those allocations are avoidable or not.

In my case, internal functions were making allocations to the heap and I can't control them. So I stopped there.

Work on feedbacks

Don't stop here and work on the feedback provided by your colleagues, friends, and community. File issues and work on them in your free time.

That's it for now. It was a nice experience and I am looking forward to contributing to some opensource go projects.

GitHub logo akshaybharambe14 / go-jsonc

go-jsonc provides a way to work with commented json by converting it to plain json.

Give it a try. Your feedback is highly appreciated. I will surely work on your inputs, that's what I just learned.

Happy coding!

Posted on Feb 17 by:

akshaybharambe14 profile

Akshay Bharambe

@akshaybharambe14

A software developer by profession. I Have 2+ years of experience in writing code and building systems. Pretty good in Golang. My free time includes working on pet projects and making myself better.

Discussion

markdown guide
 

I've used golang in two or three college projects and so far so good. Though, I felt overwhelmed sometimes while looking at the stdlib because I felt like there was too many interfaces and such and many ways to do one thing. What do you recommend looking at to get a good grasp on the stdlib? Thanks!

 

I felt like there was too many interfaces and such and many ways to do one thing. This is a little contradictory statement as go encourages us to have only one possible way to do something.

The interfaces in the standard libraries are essential for compatibility. I recommend you go through the documentation of the respective interface. Go has well-written documentation for the standard library. Let me if I missed something.