DEV Community

Steve Layton
Steve Layton

Posted on • Originally published at shindakun.dev on

Golang Data Connector Part 2

#go

Intro

Golang Data Connector Part 2

Last time we started our small data collector. The idea is we want to move data from a Ghost blog over to something else. This time we're going to extend our starting point to actually make use of the data that comes back from the API call. That means we need to take a closer look at the incoming data... and with that in mind let's get started.

Adding Struct-ure

Currently, we are receiving our data and converting it from []byte to string so we can print it to the screen. That's cool and all but not very useful. Instead, we need to see if we can parse the data and make it usable for the next stage of this project.

To do this we'll need to convert the JSON into a struct. OK, so how do we do that? There are a couple of different ways but the easiest I've found is to use the site JSON to Go. This site converts a JSON object into what is a matching Golang struct.

Golang Data Connector Part 2

We can simply paste the raw output from the call to https://demo.ghost.io/ghost/api/v4/content/posts/?key=22444f78447824223cefc48062 into the form. It will spit out the exact struct we need:

type AutoGenerated struct {
    Posts []struct {
        ID string `json:"id"`
        UUID string `json:"uuid"`
        Title string `json:"title"`
        Slug string `json:"slug"`
        HTML string `json:"html"`
        CommentID string `json:"comment_id"`
        FeatureImage string `json:"feature_image"`
        Featured bool `json:"featured"`
        Visibility string `json:"visibility"`
        CreatedAt time.Time `json:"created_at"`
        UpdatedAt time.Time `json:"updated_at"`
        PublishedAt time.Time `json:"published_at"`
        CustomExcerpt string `json:"custom_excerpt"`
        CodeinjectionHead interface{} `json:"codeinjection_head"`
        CodeinjectionFoot interface{} `json:"codeinjection_foot"`
        CustomTemplate interface{} `json:"custom_template"`
        CanonicalURL interface{} `json:"canonical_url"`
        EmailRecipientFilter string `json:"email_recipient_filter"`
        URL string `json:"url"`
        Excerpt string `json:"excerpt"`
        ReadingTime int `json:"reading_time"`
        Access bool `json:"access"`
        OgImage interface{} `json:"og_image"`
        OgTitle interface{} `json:"og_title"`
        OgDescription interface{} `json:"og_description"`
        TwitterImage interface{} `json:"twitter_image"`
        TwitterTitle interface{} `json:"twitter_title"`
        TwitterDescription interface{} `json:"twitter_description"`
        MetaTitle interface{} `json:"meta_title"`
        MetaDescription interface{} `json:"meta_description"`
        EmailSubject interface{} `json:"email_subject"`
        Frontmatter interface{} `json:"frontmatter"`
        Plaintext string `json:"plaintext,omitempty"`
    } `json:"posts"`
    Meta struct {
        Pagination struct {
            Page int `json:"page"`
            Limit int `json:"limit"`
            Pages int `json:"pages"`
            Total int `json:"total"`
            Next interface{} `json:"next"`
            Prev interface{} `json:"prev"`
        } `json:"pagination"`
    } `json:"meta"`
}
Enter fullscreen mode Exit fullscreen mode

Let's start by updating AutoGenerated to say Ghost. From there we can add the entire struct to the top of our previous code - right above main(). That is if we want to just added. In our imagined receiving system the entire struct isn't necessary so I've left out a bit. I've also updated several of the interface{} entries to be strings since that's what we'll actually be getting if the value is set.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "time"
)

type Ghost struct {
    Posts []struct {
        ID string `json:"id"`
        UUID string `json:"uuid"`
        Title string `json:"title"`
        Slug string `json:"slug"`
        HTML string `json:"html"`
        FeatureImage string `json:"feature_image"`
        Featured bool `json:"featured"`
        Visibility string `json:"visibility"`
        CreatedAt time.Time `json:"created_at"`
        UpdatedAt time.Time `json:"updated_at"`
        PublishedAt time.Time `json:"published_at"`
        CustomExcerpt string `json:"custom_excerpt"`
        URL string `json:"url"`
        Excerpt string `json:"excerpt"`
        ReadingTime int `json:"reading_time"`
        Access bool `json:"access"`
        OgImage string `json:"og_image"`
        OgTitle string `json:"og_title"`
        OgDescription string `json:"og_description"`
        TwitterImage string `json:"twitter_image"`
        TwitterTitle string `json:"twitter_title"`
        TwitterDescription string `json:"twitter_description"`
        MetaTitle string `json:"meta_title"`
        MetaDescription string `json:"meta_description"`
        Plaintext string `json:"plaintext,omitempty"`
    } `json:"posts"`
    Meta struct {
        Pagination struct {
            Page int `json:"page"`
            Limit int `json:"limit"`
            Pages int `json:"pages"`
            Total int `json:"total"`
            Next interface{} `json:"next"`
            Prev interface{} `json:"prev"`
        } `json:"pagination"`
    } `json:"meta"`
}

func main() {

    ...snip...
Enter fullscreen mode Exit fullscreen mode
Enjoy this post?
How about buying me a coffee?

More Printing

Alright, we're slowly getting somewhere! Now, we are going to replace our fmt.Println() with code to actually use our struct. We'll read the body in using the same manner as last time.

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }
Enter fullscreen mode Exit fullscreen mode

But this time instead of printing we'll create a variable p of the type Ghost. Using p we then json.Unmarshal our received body.

    var p Ghost
    err = json.Unmarshal(body, &p)
    if err != nil {
        log.Fatal(err)
    }
Enter fullscreen mode Exit fullscreen mode

From here we now have access to range over the entirety of p.Posts and once again we print them out to the screen.

    for i := range p.Posts {
        fmt.Printf("%v", p.Posts[i])
    }
Enter fullscreen mode Exit fullscreen mode

Next Time

That's all for this quick post, you can see the full code listing below. Next time we'll be taking one more step forward and actually use the http package to send our posts to another service.

Code Listing 

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "time"
)

type Ghost struct {
    Posts []struct {
        ID string `json:"id"`
        UUID string `json:"uuid"`
        Title string `json:"title"`
        Slug string `json:"slug"`
        HTML string `json:"html"`
        FeatureImage string `json:"feature_image"`
        Featured bool `json:"featured"`
        Visibility string `json:"visibility"`
        CreatedAt time.Time `json:"created_at"`
        UpdatedAt time.Time `json:"updated_at"`
        PublishedAt time.Time `json:"published_at"`
        CustomExcerpt string `json:"custom_excerpt"`
        URL string `json:"url"`
        Excerpt string `json:"excerpt"`
        ReadingTime int `json:"reading_time"`
        Access bool `json:"access"`
        OgImage string `json:"og_image"`
        OgTitle string `json:"og_title"`
        OgDescription string `json:"og_description"`
        TwitterImage string `json:"twitter_image"`
        TwitterTitle string `json:"twitter_title"`
        TwitterDescription string `json:"twitter_description"`
        MetaTitle string `json:"meta_title"`
        MetaDescription string `json:"meta_description"`
        Plaintext string `json:"plaintext,omitempty"`
    } `json:"posts"`
    Meta struct {
        Pagination struct {
            Page int `json:"page"`
            Limit int `json:"limit"`
            Pages int `json:"pages"`
            Total int `json:"total"`
            Next interface{} `json:"next"`
            Prev interface{} `json:"prev"`
        } `json:"pagination"`
    } `json:"meta"`
}

func main() {
    req, err := http.NewRequest(http.MethodGet, "https://demo.ghost.io/ghost/api/v4/content/posts/?key=22444f78447824223cefc48062", nil)
    if err != nil {
        log.Fatal(err)
    }

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        log.Fatal(err)
    }

    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }

    var p Ghost
    err = json.Unmarshal(body, &p)
    if err != nil {
        log.Fatal(err)
    }

    for i := range p.Posts {
        fmt.Printf("%v", p.Posts[i])
    }
}
Enter fullscreen mode Exit fullscreen mode

Top comments (0)

Read next

func25 profile image

Go Sync Mutex: Normal and Starvation Mode

Phuong Le -

mfbmina profile image

Circuit Breaker in Go apps

Matheus Mina -

mr_destructive profile image

Safely using Maps in Golang: Differences in declaration and initialization

Meet Rajesh Gor -

nerdherd profile image

Golang Struct Field Scopes

Nerdherd -