nao1215/csv - Read csv with validation in golang

What is nao1215/csv package

The nao1215/csv package is a library that performs validation when loading CSV files. Validation is executed based on the rules specified in the struct tags.

The nao1215/csv package was developed with inspiration from go-playground/validator and shogo82148/go-header-csv. I would like to express my gratitude here.

How to use

Please attach the "validate:" tag to your structure and write the validation rules after it. It's crucial that the "order of columns" matches the "order of field definitions" in the structure. The csv package does not automatically adjust the order.

When using csv.Decode, please pass a pointer to a slice of structures tagged with struct tags. The csv package will perform validation based on the struct tags and save the read results to the slice of structures if there are no errors. If there are errors, it will return them as []error.

package main

import (

    "bytes"

    "fmt"

<span class="s">"github.com/nao1215/csv"</span>



    

    




)

func main() {

    input := id,name,age

1,Gina,23

a,Yulia,25

3,Den1s,30



    buf := bytes.NewBufferString(input)

    c, err := csv.NewCSV(buf)

    if err != nil {

        panic(err)

    }

<span class="k">type</span> <span class="n">person</span> <span class="k">struct</span> <span class="p">{</span>
    <span class="n">ID</span>   <span class="kt">int</span>    <span class="s">`validate:"numeric"`</span>
    <span class="n">Name</span> <span class="kt">string</span> <span class="s">`validate:"alpha"`</span>
    <span class="n">Age</span>  <span class="kt">int</span>    <span class="s">`validate:"gt=24"`</span>
<span class="p">}</span>
<span class="n">people</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="n">person</span><span class="p">,</span> <span class="m">0</span><span class="p">)</span>

<span class="n">errs</span> <span class="o">:=</span> <span class="n">c</span><span class="o">.</span><span class="n">Decode</span><span class="p">(</span><span class="o">&amp;</span><span class="n">people</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">errs</span><span class="p">)</span> <span class="o">!=</span> <span class="m">0</span> <span class="p">{</span>
    <span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">errs</span> <span class="p">{</span>
        <span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">err</span><span class="o">.</span><span class="n">Error</span><span class="p">())</span>
    <span class="p">}</span>
<span class="p">}</span>

<span class="c">// Output:</span>
<span class="c">// line:2 column age: target is not greater than the threshold value: threshold=24.000000, value=23.000000</span>
<span class="c">// line:3 column id: target is not a numeric character: value=a</span>
<span class="c">// line:4 column name: target is not an alphabetic character: value=Den1s</span>



    

    




}

Struct tags

You set the validation rules following the "validate:" tag according to the rules in the table below. If you need to set multiple rules, please enumerate them separated by commas.

Validation rule without arguments

Validation rule with numeric argument

Why I wrote nao1215/csv

It's the beginning of a long tale of memories.

Have you ever had the task of finding 10 errors within a CSV containing tens of thousands of rows and over 300 columns? I've experienced that several times.

I developed nao1215/sqly(article is here), which allows executing SQL on CSV files, because I found correcting CSVs to be challenging. However, sqly alone was insufficient.

In Japan, it's common to manage master data using Excel. However, it's not practical for apps to directly read data from Excel. Therefore, it's sometimes necessary to convert Excel to CSV and create apps to import the data into multiple database tables. However, there are cases where incorrect data is present in the CSV (it's strange to have incorrect data in master data, isn't it?), which prevents it from being imported into the database.

I despised this task.

I just wanted to make it even a bit easier. I had someone else implement a validation process to check the CSV columns and identify where errors were (I didn't do it myself). However, I believed I could write the validation process more generically.

Over a year has passed since I started contemplating this. To bid farewell to this challenge, I secluded myself in my room for a day and created nao1215/csv. It took over a year to accomplish what could have been done in just one day.

Next work

I plan to increase the variety of validations and implement faster validation processes. As a side note, I am not currently working with CSVs. I spent a day to resolve lingering regrets from past work. Therefore, the development pace of nao1215/csv may seem slow.

If everyone could gift GitHub Stars, I believe the development pace would be faster.

Thank you for reading this article.