loading...
Cover image for Attempting to Learn Go - Building Dev Log Part 03

Attempting to Learn Go - Building Dev Log Part 03

shindakun profile image Steve Layton ・6 min read

Attempting to Learn Go (23 Part Series)

1) Attempting to Learn Go - Building a Downloader Part 01 2) Attempting to Learn Go - Building a Downloader Part 02 3 ... 21 3) Attempting to Learn Go - Building a Downloader Part 03 4) Attempting to Learn Go - Building a Downloader Part 04 5) Attempting to Learn Go - Building a Downloader Part 05 6) Attempting to Learn Go - Consuming a REST API 7) Attempting to Learn Go - Continuing REST Adventures 8) Attempting to Learn Go - Now Sending REST Requests 9) Attempting to Learn Go - REST API and A Bit On Templates 10) Attempting to Learn Go - Sending Email Via API Again 11) Attempting to Learn Go - Let's Get Modular! 12) Attempting to Learn Go - Let's Get Modular - Again! 13) Attempting to Learn Go - Building Dev Log Part 01 14) Attempting to Learn Go - Building Dev Log Part 02 15) Attempting to Learn Go - Building Dev Log Part 03 16) Attempting to Learn Go - Building Dev Log Part 04 17) Attempting to Learn Go - Building Dev Log Part 05 18) Attempting to Learn Go - Listing Files By Extension 19) Attempting to Learn Go - Sorting and Moving Files by Extension 20) Attempting to Learn Go - Issuer 01 21) Attempting to Learn Go - Issuer 02 22) Attempting to Learn Go - Issuer 03 - Cloud Function Go! 23) Attempting to Learn Go - Issuer 04 - Addendum

Building Out A Dev Blog

Here we go again! If you've been following these posts, welcome back, and if not welcome aboard! This time we'll be continuing our work to extend the basic code we put together to create a static site generator. We actually jump quite a bit forward this week. We've now are at a point where we can convert an entire directory of well-formatted markdown files into HTML ready to host. Anyway, let's keep on with our attempts to learn Go...


Diving In

I'm still toying around with a different way to show our updated code but, for now, I think the code base is small enough I can just start from the top and work my way down. I'm going to gloss over the bits that haven't changed since last week so, if you haven't read that I suggest you do!

Now then, let's jump down to our first real changes.

package main

import (
  "bufio"
  "bytes"
  "fmt"
  "html/template"
  "io/ioutil"
  "os"
  "regexp"
  "strings"

  "github.com/microcosm-cc/bluemonday"
  "github.com/russross/blackfriday"
  yaml "gopkg.in/yaml.v2"
)

const delim = "---"

type post struct {
  Title       string
  Published   bool
  Description string
  Tags        []string
  CoverImage  string
  Series      string
  PostBody    template.HTML
}

var templ = `<!DOCTYPE html>
<html lang="en">
  <head>
    <title>{{.Title}}</title>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="HandheldFriendly" content="True">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="referrer" content="no-referrer-when-downgrade" />
    <meta name="description" content="{{.Description}}" />
  </head>
  <body>
    <div class="post">
      <h1>{{.Title}}</h1>
      {{.PostBody}}
    </div>
  </body>
  </html>
  `

Naming Stuff Is Hard

Here we get to the first (small) change, the function is the same as last week but I've renamed it from loadFile() to getContents(). It seems to more accurately capture what it does.

func getContents(f *string) ([]byte, error) {
  b, err := ioutil.ReadFile(*f)
  if err != nil {
    return nil, err
  }
  return b, nil
}

One thing you should notice throughout this section is I've tried to make sure we're are breaking out code into standalone functions. This should keep our main() nice and make it so we can write tests. Below we have parseFM() which wraps up our un-marshaling and returning the first stage of our front matter. It may be more correctly called parseYAML() but, we'll leave it as is for now. isNil() has not changed and still is not really used, I will see to that before the next post, possibly.

func parseFM(b *[]byte) (map[string]interface{}, error) {
  m := make(map[string]interface{})
  err := yaml.Unmarshal(*b, &m)
  if err != nil {
    msg := fmt.Sprintf("error: %v\ninput:\n%s", err, b)
    return nil, fmt.Errorf(msg)
  }
  return m, nil
}

func isNil(i interface{}) bool {
  if i != nil {
    return false
  }
  return true
}

As before we need to split up the data we loaded from the markdown file. A call to splitData() will take care of it. I've fed in various bits of malformed data and the if statement seemed to work as expected each time. I also ran my current set of markdown dev.to posts through and each was converted without a problem. They can all be found over at dev.shindakun.net.

func splitData(f *[]byte) ([][]byte, error) {
  b := bytes.Split(*f, []byte(delim))
  if len(b) < 3 || len(b[0]) != 0 {
    return nil, fmt.Errorf("Front matter is damaged")
  }
  return b, nil
}

Once we've split the data and parsed out the front matter we call makePost() to build the post struct and the template. I'm passing back the *post because we're currently executing the template just before we write the file to disk, not in this function itself. It will also be handy when I build out something to create an index.html. Again, we're not really using isNil() as intended mostly because I don't like the if/else jumble that it will create. We'll get that taken care of one of these days.

// makePost creates the post struct, returns that and the template HTML
func makePost(fm map[string]interface{}, contents []byte,
  s [][]byte) (*template.Template, *post) {
  p := &post{}

  if isNil(fm["title"]) {
    panic("isNil tripped at title")
  } else {
    p.Title = fm["title"].(string)
  }
  p.Published = fm["published"].(bool)
  p.Description = fm["description"].(string)

  // TODO: Strip space after comma prior to parse?
  tmp := fm["tags"].(string)
  p.Tags = strings.Split(tmp, ", ")

  p.CoverImage = fm["cover_image"].(string)
  p.Series = fm["series"].(string)

  pBody := contents[len(s[1])+(len(delim)*2):]

  out := blackfriday.Run(pBody)

  bm := bluemonday.UGCPolicy()
  bm.AllowAttrs("class").Matching(regexp.MustCompile("^language-[a-zA-Z0-9]+$")).OnElements("code")
  p.PostBody = template.HTML(bm.SanitizeBytes(out))

  tm := template.Must(template.New("msg").Parse(templ))
  return tm, p
}

There you go, we're down to our main() function! Right of the bat, you'll see that we're now using ioutil.ReadDir() to get a list of the contents of the current directory.

func main() {
  d, err := ioutil.ReadDir(".")
  if err != nil {
    panic(err)
  }

We then range through the list of the current directory and immediately check to see if the current file ends with .md. If it does the address of our string gets passed into getContents() and if all goes well we'll get the contents back.

  for _, f := range d {
    if t := f.Name(); strings.HasSuffix(t, ".md") {
      contents, err := getContents(&t)
      if err != nil {
        panic(err)
      }

We'll pass a pointer to our contents into splitData() and then pass the second chunk of s, that is s[1], into parseFM(). This gives us the base building blocks for our post. Both splitData() and parseFM() could probably be moved into makePost(), then we could pass the contents directly into it and do all the work there. That's something to consider for when refactoring.

      s, err := splitData(&contents)
      if err != nil {
        panic(err)
      }

      fm, err := parseFM(&s[1])
      if err != nil {
        msg := fmt.Sprintf("error: %v\ninput:\n%s", err, s[1])
        panic(msg)
      }

      tm, p := makePost(fm, contents, s)

In this current version of the code, we're going to take the current filename and lop off the extension. We'll then create a new file with the same name as the original and append .html. Again, this is all currently happening in the same directory and we're being really sloppy and not checking for existing files or anything. os.Create() uses 0666 (or equivalent permissions in Windows) for the file permissions so in theory we should be able to write and overwrite with no issues. This is another case where we might not be doing what may be considered "correct" - instead, we are just going through an iterative process. This lets everyone see how the code has changed, warts and all.

File created, o, we then use bufio.NewWriter() to create a "Writer" with a buffer. We can then execute our template and write directly into the file which is closed after we flush the buffer.

      fin := strings.TrimSuffix(t, ".md")
      o, err := os.Create(fin + ".html")
      if err != nil {
        panic(err)
      }
      defer o.Close()

      buf := bufio.NewWriter(o)

      err = tm.Execute(buf, p)
      if err != nil {
        panic(err)
      }
      buf.Flush()
    }
  }
}

If we have any files left we jump back up to the top of our loop and do the whole things again!


Next Time

We have just about everything in place! But first, we need to create and write out index.html. I think I'm going implement that over the next day or so and put up a quick follow up post. After that, I'll need to decide if we want to do one more post in the "dev site" series and extend the code to use external templates. It shouldn't be that much more work, I suppose I'll wait and see how busy the next week looks on Monday. See you next time!


You can find the code for this and most of the other Attempting to Learn Go posts in the repo on GitHub.



Attempting to Learn Go (23 Part Series)

1) Attempting to Learn Go - Building a Downloader Part 01 2) Attempting to Learn Go - Building a Downloader Part 02 3 ... 21 3) Attempting to Learn Go - Building a Downloader Part 03 4) Attempting to Learn Go - Building a Downloader Part 04 5) Attempting to Learn Go - Building a Downloader Part 05 6) Attempting to Learn Go - Consuming a REST API 7) Attempting to Learn Go - Continuing REST Adventures 8) Attempting to Learn Go - Now Sending REST Requests 9) Attempting to Learn Go - REST API and A Bit On Templates 10) Attempting to Learn Go - Sending Email Via API Again 11) Attempting to Learn Go - Let's Get Modular! 12) Attempting to Learn Go - Let's Get Modular - Again! 13) Attempting to Learn Go - Building Dev Log Part 01 14) Attempting to Learn Go - Building Dev Log Part 02 15) Attempting to Learn Go - Building Dev Log Part 03 16) Attempting to Learn Go - Building Dev Log Part 04 17) Attempting to Learn Go - Building Dev Log Part 05 18) Attempting to Learn Go - Listing Files By Extension 19) Attempting to Learn Go - Sorting and Moving Files by Extension 20) Attempting to Learn Go - Issuer 01 21) Attempting to Learn Go - Issuer 02 22) Attempting to Learn Go - Issuer 03 - Cloud Function Go! 23) Attempting to Learn Go - Issuer 04 - Addendum

Posted on by:

shindakun profile

Steve Layton

@shindakun

I've been known to write some code from time to time.

Discussion

markdown guide
 

About this snippet:

func getContents(f *string) ([]byte, error) {
  b, err := ioutil.ReadFile(*f)
  if err != nil {
    return nil, err
  }
  return b, nil
}

You're taking in *string when string is better for what you want to do.

But more importantly, you are not checking your pointer:

if f == nil {
    return nil, errors.New("contextual error about f being nil")
}

I'll make the assumption that you are doing it this way because you are already checking before the function, but I would say that this would be better done inside so you never have to think about it again.

On another note: I don't think the function expresses what it's doing correctly. Consider the following alternative signatures:

  • func getFileContents(filename string) ([]byte, error) {}
  • func contentsOf(f *os.File) ([]byte, error) {}
  • func contentsOf(r io.Reader) ([]byte, error) { return ioutil.ReadAll(r) }

And in that last case, if you're operating on readers, why not just... not use a function at all!

As an *os.File is a Reader, you would eliminate the boilerplate code, and be able to leverage one of the most powerful interfaces in the standard library.

 

Thanks for the comments! I really should be passing in just the string as you've pointed out, there is no reason to use the pointer. My thinking in making this a function on its own was to be able to write tests for it later, which I suppose means I had better be checking for that pointer. I'm not sure just wrapping reading a file into a function is worth it just to be able to write a test down the road though so it might be worth doing it away with it entirely.

Definitely, something for me to consider. Thanks again!

 

Worth considering as well that using a Reader rather than a concrete file here will make testing this easier later anyway, since you'll be able to use any Reader. not just a file, you can mock your input using buffers or any old strings.NewReader("thing")!

Thanks for taking the time to respond.

That is a fantastic point! I think using a Reader is going to be the way I go.

 

Hey, nice post series. I always learn something new from it.

What would you think of renaming some functions to make them more unambiguous? I know, Go is known for making names as short as possible, but ... πŸ€”

For example

func getContents(f *string) ([]byte, error) {}

The code within explains itself, as it is short enough, but if you read only the function signature, it is not quit clear. Does it get the content of a string and returns it as a byte slice? What, if the function gets longer? I would have to skim a lot more code to understand it.

How about

func getFileContent(f string)

or

func getContent(file string)

And why do you use a reference to the Λ‹fileΛ‹ string instead of a simple value?

Another one would be

func parseFM()

Abbreviations are hard to follow, it not within the closed context of a function. Maybe

func parseFrontMatter()

would make it instantly clear?!

Looking forward to your next post.

 

Thanks for the comment! It would definitely make it clearer. I always go back and forth on names, I've been thinking about learning more toward descriptive functions like getFileContent(f string) especially for stuff I'm going to be posting. Now, is probably a good time to try and commit to that. I can't remember why I wrote it that way and didn't just send in a string. It's not like we're memory constrained it doesn't need to be that way and the passed in string would just be garbage collected eventually.

 

Hi, good job :)

Try avoiding things like f *[]byte as slices and maps are already pointers by themselves. Also, it's a good practice to check whether entries exist in the map before converting them to types:

p.Published = fm["published"].(bool)

Could be:

pubIntf, ok := fm["published"]
if ok {
   published, ok := pubIntf.(bool)
   if ok {
     p.Published = published
   }
}

While it does seem redundant in personal/toy projects, I have seen some applications panic and crash due to such simple assumptions when someone later modifies the input map and misses some values :)

P.S. It would be a lot more robust if you parsed that yaml into a struct!

 

Thanks for the comment! I've been putting off dealing with checking on the front matter map for two posts. πŸ˜… isNil() basically only exists for checking the map but I keep putting off implementing the full thing because I was working with "known good" files. But I kind of like the pattern in your example I will probably be using that. I think at some point it might go into a struct if I keep extending the code.