loading...
Cover image for Attempting to Learn Go - Building Dev Log Part 05

Attempting to Learn Go - Building Dev Log Part 05

shindakun profile image Steve Layton ・7 min read

Attempting to Learn Go (23 Part Series)

1) Attempting to Learn Go - Building a Downloader Part 01 2) Attempting to Learn Go - Building a Downloader Part 02 3 ... 21 3) Attempting to Learn Go - Building a Downloader Part 03 4) Attempting to Learn Go - Building a Downloader Part 04 5) Attempting to Learn Go - Building a Downloader Part 05 6) Attempting to Learn Go - Consuming a REST API 7) Attempting to Learn Go - Continuing REST Adventures 8) Attempting to Learn Go - Now Sending REST Requests 9) Attempting to Learn Go - REST API and A Bit On Templates 10) Attempting to Learn Go - Sending Email Via API Again 11) Attempting to Learn Go - Let's Get Modular! 12) Attempting to Learn Go - Let's Get Modular - Again! 13) Attempting to Learn Go - Building Dev Log Part 01 14) Attempting to Learn Go - Building Dev Log Part 02 15) Attempting to Learn Go - Building Dev Log Part 03 16) Attempting to Learn Go - Building Dev Log Part 04 17) Attempting to Learn Go - Building Dev Log Part 05 18) Attempting to Learn Go - Listing Files By Extension 19) Attempting to Learn Go - Sorting and Moving Files by Extension 20) Attempting to Learn Go - Issuer 01 21) Attempting to Learn Go - Issuer 02 22) Attempting to Learn Go - Issuer 03 - Cloud Function Go! 23) Attempting to Learn Go - Issuer 04 - Addendum

Building Out A Dev Blog

Welcome back to what might be the final trip into our static site generator. For now at least! This week we are going to add a very basic command line function. This will allow us to target a source directory and a destination directory. We'll also look at a couple of things that came up in the comments. Let's get right into it!


Nested If/Else

@ladydascalie commented on the makePost() function, point out the wordy if/else statements. By moving the else up and out of the if makes this much easier to read. This is one instance were being verbose to a fault makes the code harder to follow.

Original Snippet

 descIntf, ok := fm["description"]
 if ok {
 description, ok := descIntf.(string)
 if ok {
 post.Description = description
    } else {
 post.Description = ""
    }
  } else {
 post.Description = ""
  }

Refactored Snippet

descIntf, ok := fm["description"]
post.Description = ""
if ok {
 if desc, ok := descIntf.(string); ok {
 post.Description = desc
    }
}

I moved the post.Description-esc portions over the assignment of the interface. Either way works in the end. I also began pulling out many of the panic()'s that littered the code. Many were there because I wasn't quite sure how I wanted to handle these cases and decided that failing fast was a good first step. I'll comment a bit more on them as we go through the code.


Constants

@dirkolbrich also left a comment on the use of my constant delimiter. When I first started I was thinking that we would accept different delimiters. We won't be adding that we'll stay with the three dashes, like Jekyll. But, by passing it into splitData() the function will be easier to test and comprehend. I should also pass it into makePost() but the signature is very long already so, for now, I didn't. It was also pointed out that I had not used all upper case on the constant name, const delimiter = "---". For many languages, you would use DELIMITER to show in the code that it is a constant. I hadn't noticed it at the time and neither had go lint. Interesting. Go has some excellent resources so I checked Effective Go and the Go Wiki. And as it turns out constants are not all caps, Go prescribes the use of mixed case. This makes a certain amount of sense. A constant/variable (or function, or struct, etc.) starting with a capital letter gets exported. We don't want the delimiter exported so lower case it is. There are exceptions to this rule in the Go code base but, I'm not going to argue one way or the other. Maybe when I've got a better grasp of the language though...


Code Walkthrough

Once again we'll start from the top and I'll try to call out what's changed. Near the top not much, we've added flag and log. flag so we can use command line options and log for formatting errors.

package main

import (
  "bufio"
  "bytes"
  "flag"
  "fmt"
  "html/template"
  "io"
  "io/ioutil"
  "log"
  "os"
  "regexp"
  "strings"

  "github.com/microcosm-cc/bluemonday"
  "github.com/russross/blackfriday"
  yaml "gopkg.in/yaml.v2"
)

const delimiter = "---"

type post struct {
  Title       string
  Published   bool
  Description string
  Tags        []string
  CoverImage  string
  Series      string
  PostBody    template.HTML
}

type index struct {
  Pages []Page
}

type Page struct {
  FileName string
  Title    string
}

var indexTempl = `<!DOCTYPE html>
<html lang="en">
  <head>
    <title>shindakun's dev site</title>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="HandheldFriendly" content="True">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="referrer" content="no-referrer-when-downgrade" />
    <meta name="description" content="shindakun's dev site" />
  </head>
  <body>
    <div class="index">
    {{ range $key, $value := .Pages }}
      <a href="/{{ $value.FileName }}">{{ $value.Title }}</a>
    {{ end }}
    </div>
  </body>
  </html>
`

var postTempl = `<!DOCTYPE html>
<html lang="en">
  <head>
    <title>{{.Title}}</title>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="HandheldFriendly" content="True">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="referrer" content="no-referrer-when-downgrade" />
    <meta name="description" content="{{.Description}}" />
  </head>
  <body>
    <div class="post">
      <h1>{{.Title}}</h1>
      {{.PostBody}}
    </div>
  </body>
  </html>
`

I could get rid of getContentsOf() and use ioutil.ReadAll() in main(). I've left it though for now since I keep telling myself I'm going to write tests and it'd be an easy function to test. I had it returning depending on the error state in an if. There was no need for that as ioutil.ReadAll() matches the return already so it can return.

func getContentsOf(r io.Reader) ([]byte, error) {
  return ioutil.ReadAll(r)
}

func parseFrontMatter(b []byte) (map[string]interface{}, error) {
  fm := make(map[string]interface{})
  err := yaml.Unmarshal(b, &fm)
  if err != nil {
    msg := fmt.Sprintf("error: %v\ninput:\n%s", err, b)
    return nil, fmt.Errorf(msg)
  }
  return fm, nil
}

func splitData(fm []byte, delimiter string) ([][]byte, error) {
  b := bytes.Split(fm, []byte(delimiter))
  if len(b) < 3 || len(b[0]) != 0 {
    return nil, fmt.Errorf("Front matter is damaged")
  }
  return b, nil
}

makePost() is where the largest set of changes occurred. As mentioned in the opening those changes are a direct response to a very good comment on the last article. I was thinking about splitting this function up since as it is we're returning the template and the post{}. It may be clearer, in the long run, to populate the post struct first. Then feed that into a different function to populate the template. That's some easy refactoring I can tackle if we do have one more post in this part of the "ATLG" series.

// makePost creates the post struct, returns that and the template HTML
func makePost(fm map[string]interface{}, contents []byte, s [][]byte) (*template.Template, *post, bool) {
  post := &post{}

  post.Published = false
  pubIntf, ok := fm["published"]
  if ok {
    if published, ok := pubIntf.(bool); ok {
      post.Published = published
    }
  }

  if !post.Published {
    return nil, nil, true
  }

  post.Title = ""
  titleIntf, ok := fm["title"]
  if ok {
    if title, ok := titleIntf.(string); ok {
      post.Title = title
    }
  }

  if post.Title == "" {
    return nil, nil, true
  }

  post.Description = ""
  descIntf, ok := fm["description"]
  if ok {
    if description, ok := descIntf.(string); ok {
      post.Description = description
    }
  }

  post.Tags = []string{}
  tagsIntf, ok := fm["tags"]
  if ok {
    if tags, ok := tagsIntf.(string); ok {
      post.Tags = strings.Split(tags, ", ")
    }
  }

  post.CoverImage = ""
  covIntf, ok := fm["cover_image"]
  if ok {
    if coverImage, ok := covIntf.(string); ok {
      post.CoverImage = coverImage
    }
  }

  post.Series = ""
  seriesIntf, ok := fm["series"]
  if ok {
    if series, ok := seriesIntf.(string); ok {
      post.Series = series
    }
  }

  pBody := contents[len(s[1])+(len(delimiter)*2):]

  bf := blackfriday.Run(pBody)

  bm := bluemonday.UGCPolicy()
  bm.AllowAttrs("class").Matching(regexp.MustCompile("^language-[a-zA-Z0-9]+$")).OnElements("code")
  post.PostBody = template.HTML(bm.SanitizeBytes(bf))

  tm := template.Must(template.New("post").Parse(postTempl))
  return tm, post, false
}

writeIndex() has changed a bit. Notice that we now take in a destination directory string and return an error. This is much nicer than just panicking on error. I have a bit more digging to do in working with my own error structs and better ways to surface and handle errors.

func writeIndex(idx index, destination string) error {
  indexFile, err := os.Create(destination + "/" + "index.html")
  if err != nil {
    return err
  }
  defer indexFile.Close()

  buffer := bufio.NewWriter(indexFile)
  tm := template.Must(template.New("index").Parse(indexTempl))
  err = tm.Execute(buffer, idx)
  if err != nil {
    return err
  }
  buffer.Flush()
  return nil
}

func main() {
  var idx index

Here we have the beginning part of our command line option code. Note that I'm using the standard library flag package and not something like Cobra. This keeps it very simple for now. Requiring only an option for a source directory and a destination directory. The source directory would be the location of all our markdown files. The destination is where we are going to save our generated HTML. Note, that I am still using panic() if we can't create the directory or if it already exists. I have decided it's better to fail out if the directory is already there for now. I have also been trying out log.Panicf() though in this iteration it only appears once. Later we could add switches for overwriting existing files, or deleting everything first, etc.

  destination := flag.String("destination", "", "destination folder")
  source := flag.String("source", "", "source directory")
  flag.Parse()

  if _, err := os.Stat(*destination); os.IsNotExist(err) {
    err := os.Mkdir(*destination, 0777)
    if err != nil {
      panic(err)
    }
  } else {
    log.Panicf("error: destination '%s' already exists", *destination)
  }

  _, err := ioutil.ReadDir(*destination)
  if err != nil {
    panic(err)
  }

  srcDir, err := ioutil.ReadDir(*source)
  if err != nil {
    panic(err)
  }

We've also swapped out some of the panic()'s and continue's in our main loop. We don't want the process to fail out so logging any issues and continuing on is the way to go. continue exits the loop and "continues" on from the top.

  for _, file := range srcDir {
    if fileName := file.Name(); strings.HasSuffix(fileName, ".md") {

      openedFile, err := os.Open(fileName)
      if err != nil {
        log.Println(fileName, err)
        continue
      }

      contents, err := getContentsOf(openedFile)
      if err != nil {
        openedFile.Close()
        log.Println(fileName, err)
        continue
      }
      openedFile.Close()

      s, err := splitData(contents, delimiter)
      if err != nil {
        log.Println(fileName, err)
        continue
      }

      fm, err := parseFrontMatter(s[1])
      if err != nil {
        msg := fmt.Sprintf("%v\n", err)
        log.Println(fileName, msg)
        continue
      }

Yes, I did leave a couple of panic()'s in place. If we can't create the output file or execute the template we come to a halt. I am assuming if one has a problem then it's better to not try and continue.

      template, post, skip := makePost(fm, contents, s)
      if !skip {
        trimmedName := strings.TrimSuffix(fileName, ".md")
        outputFile, err := os.Create(*destination + "/" + trimmedName + ".html")
        if err != nil {
          panic(err)
        }
        buffer := bufio.NewWriter(outputFile)

        err = template.Execute(buffer, post)
        if err != nil {
          panic(err)
        }
        buffer.Flush()
        outputFile.Close()

        indexLinks := Page{
          FileName: trimmedName + ".html",
          Title:    post.Title,
        }
        idx.Pages = append(idx.Pages, indexLinks)
      }
    }
  }

I couldn't decide whether I wanted to error out if something went wrong with the index. For now, I've left it as a logline.

  if len(idx.Pages) > 0 {
    err := writeIndex(idx, *destination)
    if err != nil {
      log.Println(err)
    }
  }
}

Next Time

I haven't quite decided on what we are going to look at next time. I can continue building this out or we can try something new. I was thinking about writing a Go backend for a very old project I put together way back when. It was a PHP/HTML5 music player. I had ported part of it over to Glitch, which you can see part of at https://carnation-motion.glitch.me/. It may take a second to start up if the VM has shut down. I don't think any songs will play on that version but the basic UI is in place. We'll see where the mood takes me over the next week.


You can find the code for this and most of the other Attempting to Learn Go posts in the repo on GitHub.



Attempting to Learn Go (23 Part Series)

1) Attempting to Learn Go - Building a Downloader Part 01 2) Attempting to Learn Go - Building a Downloader Part 02 3 ... 21 3) Attempting to Learn Go - Building a Downloader Part 03 4) Attempting to Learn Go - Building a Downloader Part 04 5) Attempting to Learn Go - Building a Downloader Part 05 6) Attempting to Learn Go - Consuming a REST API 7) Attempting to Learn Go - Continuing REST Adventures 8) Attempting to Learn Go - Now Sending REST Requests 9) Attempting to Learn Go - REST API and A Bit On Templates 10) Attempting to Learn Go - Sending Email Via API Again 11) Attempting to Learn Go - Let's Get Modular! 12) Attempting to Learn Go - Let's Get Modular - Again! 13) Attempting to Learn Go - Building Dev Log Part 01 14) Attempting to Learn Go - Building Dev Log Part 02 15) Attempting to Learn Go - Building Dev Log Part 03 16) Attempting to Learn Go - Building Dev Log Part 04 17) Attempting to Learn Go - Building Dev Log Part 05 18) Attempting to Learn Go - Listing Files By Extension 19) Attempting to Learn Go - Sorting and Moving Files by Extension 20) Attempting to Learn Go - Issuer 01 21) Attempting to Learn Go - Issuer 02 22) Attempting to Learn Go - Issuer 03 - Cloud Function Go! 23) Attempting to Learn Go - Issuer 04 - Addendum

Posted on by:

shindakun profile

Steve Layton

@shindakun

I've been known to write some code from time to time.

Discussion

markdown guide
 

Well this has been a very interesting series thus far.

I could recommend a few things which were good learning exercises for me early on:

  • Write a program to sort files within a folder by their extension

    • Later make it sort them in logical folders ex: .txt in Documents, .jpg in Images etc...
  • Write a batch picture downloader. Easily done against an api such as Reddit, for example, to collect all images in a page and save them somewhere.

    • Bonus points if you can get them to sort using the previous program!
 

Well this has been a very interesting series thus far.

Thanks! I certainly hope so!

I really like both of your ideas, I think I may have to see about implementing the first one at some point in the next couple of days. Would be good practice.