Go producer-consumer problem

In the producer-consumer problem, a producer repeatedly generates data and writes it into a buffer. The consumer repeatedly reads the data in the buffer, removing it from the buffer and using the data in some way.

In this particular problem, the producer reads in tweets from a stream, and a consumer is processing the data to find out whether someone has tweeted about Golang or not. The task is to implement a producer and consumer that run concurrently to increase the throughput.

Solution

Key takeaway

The use buffered channel

You can opt to implement this using an unbuffered channel. When no goroutine reads the channel, the execution will pause. So this will decrease throughput. The length of the buffer depends on your need. For this problem, I just use ten. The buffered channel is declared at line 34.

Clean up your goroutines

When you launch a goroutine, make sure that it will eventually exit. Go runtime can not detect that a goroutine will never be used again. If a goroutine does not exit, the scheduler will still periodically give it time to do nothing, which slows down your program. This is called a goroutine leak.

In our solution, we clean up our goroutine on line 12. Once the stream is closed, we also close the link channel.

Range and close

You can read from a channel using a for-loop construct. It will keep reading from the channel until the channel is closed.

So in our solution, line 20 will keep reading the channel link. Until that channel is closed (at line 12).

Wait for the process to finish

In our solution, line 43, the execution will pause until a value is written to the done channel. And we write to this channel at the end of our consumer function (line 27) to indicate that we are done. If you don't wait, the program will exit immediately.

Wrap up

So in this article, we again see Go concurrency's building block, namely goroutines and channels. The following article will examine a more traditional means of writing concurrent code through memory access synchronization and the primitives that follow that technique.