What is the io.Reader in Go?

If you write Go programs, you'll come across the io.Reader interface quite a bit. It's used all over the place to represent reading data from lots of different sources, but its type definition is actually pretty humble.

type Reader interface {
    Read(p []byte) (n int, err error)
}

Just one method! Which means if you make a type with a method called Read, and that method takes in a byte slice and returns an int plus any error that occurs, your type is an io.Reader. But what is the idea of this interface, and how do we use it in our code?

Taking a look at the function signature

Reader is a little interface with a strong abstraction! What is that abstraction exactly? The idea with the Read method is that it represents reading bytes of data from some source, so that we can then use those bytes in our code. That source could be files, cameras, network connections, or just a plain old string. For example, if we're reading data from files, the io.Reader we would use is a *os.File.

Before we try this out, let's take a look at the parts of the function signature for Reader.Read:

Read(p []byte) (n int, err error)

• p []byte is a byte slice we pass into the Read method. The Reader copies the data it reads from its data source (like a file) over to that byte slice.
• The returned n int tells you how many bytes we read in this call to Read.
• The returned err error is any error that might happen reading the data, like reaching the end of a file.

Let's give this a try with a file and see how we use it. Copy this text and save it to a file in a new directory called sloth.txt:

I'm a sloth and hibiscus flowers are tasty!

Now, let's write some Go code to read it using a File's Read method. Copy this code and save it to a file main.go in the same directory as sloth.txt:

package main

import (
    "log"
    "os"
)

func main() {
    file, err := os.OpenFile("sloth.txt")
    if err != nil {
        log.Fatalf("error opening sloth.txt: %v", err)
    }
    defer file.Close()

    // Make a byte slice that's big enough to store a few words of the message
    // we're reading
    bytesRead := make([]byte, 33)

    // Now read some data, passing in our byte slice
    n, err := file.Read(bytesRead)
    if err != nil {
        log.Fatalf("error reading from sloth.txt: %v", err)
    }

    // Take a look at the bytes we copied into the byte slice
    log.Printf("We read \"%s\" into bytesRead (%d bytes)",
        string(bytesRead), n)
}

Run this with go run main.go and the output you should be getting is:

We read "I'm a sloth and hibiscus flowers " (33 bytes) into bytesRead

So what happens when we call os.File.Read is, Go reads data from the file, copying it to bytesRead, and since no error happened, the error returned is nil. We're passing our bytesRead slice into the Read function like we pass a cup into a soda fountain!

So that's how we get our bytes, but why are we even returning how many bytes we read?

To see why, let's see what happens if we do a second call to Read. We want to see this sloth's opinion on hibiscus flowers! Add this to the end of the main function:

    n, err = file.Read(bytesRead)
    if err != nil {
        log.Fatalf("error reading from sloth.txt: %v", err)
    }

    log.Printf("We read \"%s\" into bytesRead (%d bytes)",
        string(bytesRead), n)

Now the output to go run main.go should be:

2020/03/01 13:38:50 We read "I'm a sloth and hibiscus flowers " into bytesRead (33 bytes)
2020/03/01 13:38:50 We read "are tasty!
 and hibiscus flowers " into bytesRead (11 bytes)

As expected we read in that our sloth indeed loves hibiscus flowers, but why does the third line of our output say and hibiscus flowers?

The reason is that while our Reader copied the 11 remaining bytes from the file to the beginning our bytesRead slice, it didn't touch the rest of the slice. The rest of the slice contains the bytes we read on the previous call to Read!

So what the returned integer n does for us, is it tells us which bytes in the byte slice contain the content we read. That way, code working with those bytes will know which bytes in the slice it should be paying attention to, and which bytes it shouldn't pay attention to.

Handling errors with io.Reader

Let's call file.Read one more time:

    n, err = file.Read(bytesRead)
    if err == io.EOF {
        log.Printf("End of file reached; we got %d bytes on the last read", n)
    } else {
        log.Fatalf("unexpected error reading from the file: %v\n", err)
    }

Run this with go run main.go, and the final output will be something like:

2020/03/01 13:44:35 We read "I'm a sloth and hibiscus flowers " into bytesRead (33 bytes)
2020/03/01 13:44:35 We read "are tasty!
 and hibiscus flowers " into bytesRead (11 bytes)
2020/03/01 13:44:35 End of file reached; we got 0 bytes on the last read

The error we got is io.EOF, which stands for "end of file", and that error is one we generally expect to get. When you encounter EOF, typically what you do is just have your code work with any bytes that were read in, and then stop reading data from that io.Reader since there isn't any data left.

However, EOF isn't the only kind of error that io.Reader.Read can return. If we get an error besides EOF, typically that means either there's a bug in our code, or something unexpected happened. For example, try copying this code to close-early.go that reads from a file we already closed:

package main

import (
    "log"
    "os"
)

func main() {
    file, err := os.OpenFile("sloth.txt")
    if err != nil {
        log.Fatalf("error opening sloth.txt: %v", err)
    }
    file.Close()

    bytesRead := make([]byte, 0, 33)
    if _, err := file.Read(bytesRead); err != nil {
        log.Fatalf("error reading from sloth.txt: %#v", err)
    }
}

If you run that with go run close-early.go, you should get:

2020/03/01 13:45:45 error reading from sloth.txt: &os.PathError{Op:"read", Path:"sloth.txt",
  Err:(*errors.errorString)(0xc000010110)}
exit status 1

An os.PathError because we're trying to read from a file we already closed!

So in general, if you're running io.Reader.Read, there are three cases to work with:

• ✅ If we get a nil error, we have our code work with the bytes we copied from the Reader to bytesRead[:n], and can keep reading more data.
• 🛑 If we get io.EOF, we should stop reading data, and if any bytes were copied to bytesRead, then we have your code work with those last n bytes in bytesRead[:n]
• ⚠️ If we get a different kind of error, it's probably unexpected. If any bytes were read in, then have your code work with them, then handle the error with whatever error handling logic you see fit.

Notice that even if we got a non-nil error, we might have still successfully read in some bytes our code we can work with, so the code working with io.Reader typically should still be processing those bytes.

Building on top of io.Reader

While io.Reader.Read gives us a generalized way to read from a lot of data sources, in a Go project, often you won't be calling Read directly; instead we have code that builds on top of that method. Some of those are:

• If you want to read all the bytes from a source into one long byte slice, you can pass your Reader into ioutil.ReadAll!
• If you want to read a file line-by-line, you can have a bufio.Scanner read in a line of data with its Scan method, and the Scanner will handle looking for newlines so your code doesn't have to think about that for you!
• Working with image data? The image package will help you decode your image into a fancy image.Image interface!
• Deserializing data from JSON for a web app? json.Decoder.Decode's got the JSON logic covered!

Why use this code instead of just Read? Because the io.Reader is a low-level interface; its job is to read data from a source, and copy it over to a byte slice, but it's not concerned with what bytes you read in. Actually working with those bytes is handled by the code that calls Read.

To see this in action, let's try deserializing some JSON! Save this Go code to sloth-facts.go:

package main

import (
    "encoding/json"
    "fmt"
    "strings"
)

type AnimalFacts struct {
    CommonName     string   `json:"commonName"`
    ScientificName string   `json:"scientificName"`
    HeightInInches int      `json:"heightInInches"`
    FavoriteFoods  []string `json:"favoriteFoods"`
    CanSwim        bool     `json:"canSwim"`
}

func main() {
    jsonData := `
{
    "commonName":     "brown-throated three-toed sloth",
    "scientificName": "Bradypus variegatus",
    "heightInInches": 31,
    "favoriteFoods":  ["Cecropia leaves", "Hibiscus flowers"],
    "canSwim":        true
}`

    // an io.Reader that isn't a file!
    rdr := strings.NewReader(jsonData)

    var sloth AnimalFacts
    if err := json.NewDecoder(rdr).Decode(&sloth); err != nil {
        fmt.Printf("error deserializing JSON: %v", err)
        return
    }

    fmt.Printf("Hi! I'm a %s (%s)! I'm about %d\" tall, and love eating %v!\n",
        sloth.CommonName, sloth.ScientificName, sloth.HeightInInches,
        sloth.FavoriteFoods)
    if sloth.CanSwim {
        fmt.Println("By the way, I can swim!")
    }
}

Run this with go run sloth-facts.go and you should get this output:

Hi! I'm a brown-throated three-toed sloth (Bradypus variegatus)! I'm about 31"
tall, and love eating [Cecropia leaves Hibiscus flowers]!
By the way, I can swim

What's happening here?

1. We take our JSON data from a string, and make an io.Reader with it. But instead of using a file as our io.Reader, we're making a strings.Reader, which lets us read data from a plain old string!
2. json.NewDecoder takes in an io.Reader, so we pass our strings.Reader into that to set up a json.Decoder, a struct that reads JSON data from our Reader, into Go variables.
3. We pass a pointer to our AnimalFacts struct into json.Decoder.Decode, and our JSON data gets read into that struct!
4. We print our data out in a cute sloth facts message, ready for a zookeeper to put on the sign at the sloth exhibit!

If we had read our JSON data with plain calls to io.Reader.Read, we would have to keep track of curly braces and commas, which field of the JSON we were currently on, and more. But with json.Decoder.Decode, the Go Team has already handled thinking that through for us, building JSON decoding logic on top of Read!

As you can see, io.Reader is a really versatile interface! And that's why in Go code, you'll see io.Readers everywhere, and you can use code that works with the Read method to work with any data format, an awesome Go interface abstraction!