Let's start the project
cd $GOPATH/src/github.com/drpaneas
mkdir checkfile; cd checkfile
git init -q
go mod init
touch main.go
code . # for vscode
Coding Task
Write a function that checks if a specific directory exists on disk. For example:
if folderExists(directory) {
// do something
}
Bonus: Feel free to add fileExists(filename)
as homework.
Acceptance Criteria
- Use https://github.com/spf13/afero for filesystem abstraction
- Make sure it works cross-platform
- For logging use https://github.com/Sirupsen/logrus
- Put the function into
pkg/utils
package - Write a unit test
- Make sure there are testable examples
Hints:
- Writing a unit test for filesystem checking is not trivial. You should NOT create a real file on the system, because then your test will depend on the I/O of the file system itself. The last resort is mocking the filesystem. There are quite a few powerful libraries like spf13/afero for this purpose (mocking of a filesystem). These packages will create temporary files in the background and clean up afterward.
- As an exercise, try to write the test first. Let it fail. Then try to fix it, by writing the actual implementation of the function.
Extra Help
- https://golangcode.com/check-if-a-file-exists/
- https://www.youtube.com/watch?v=ifBUfIb7kdo
- https://github.com/spf13/afero#using-afero-for-testing
- An example using Afero for testing: http://krsacme.com/golang-fs-tests/
- For logging https://linuxhint.com/golang-logrus-package/
NOTICE:
If you are unable to mock this test, in worst case scenario, create a folder pkg/utils/testdata
and put actual files you can use for your test.
Let's code
// main.go
package main
import (
"os"
log "github.com/sirupsen/logrus"
)
// exists reports whether the named file or directory exists
func exists(path string, isDir bool) bool {
if path == "" {
log.Debug("Path is empty")
return false
}
info, err := os.Stat(path)
if err != nil {
if os.IsNotExist(err) { // look for the specific error type
return false
}
}
return isDir == info.IsDir()
}
// FolderExists reports whether the provided directory exists.
func FolderExists(path string) bool {
return exists(path, true)
}
// FileExists reports whether the provided directory exists.
func FileExists(path string) bool {
return exists(path, false)
}
func main() {
// Just for demo purposes
if FolderExists("logs") {
log.Info("Folder exists")
} else {
log.Error("Folder does not exist")
}
if FileExists("helloworld.txt") {
log.Info("File exists")
} else {
log.Error("File does not exist")
}
}
One way to test this is by actually using the program itself.
Running it as it is, it's expected to fail:
ERRO[0000] Folder does not exist
ERRO[0000] Folder does not exist
If we create the two necessary things we look for, then it's expected to work:
# Create the file and directory
touch helloworld.txt
mkdir logs
# Run it again
go run main.go
INFO[0000] Folder exists
INFO[0000] File exists
Normally you would create separate package and not use main
.
So let's do it!
mkdir -p pkg/utils/
touch pkg/utils/filesystem.go
The code will look like this:
// main.go
// -------
package main
import (
"github.com/drpaneas/checkfile/pkg/utils"
log "github.com/sirupsen/logrus"
)
func main() {
if utils.FolderExists("logs") {
log.Info("Folder exists")
} else {
log.Error("Folder does not exist")
}
if utils.FileExists("helloworld.txt") {
log.Info("File exists")
} else {
log.Error("File does not exist")
}
}
// pkg/utils/filesystem.go
// -----------------------
package utils
import (
"os"
log "github.com/sirupsen/logrus"
)
// exists reports whether the named file or directory exists
func exists(path string, isDir bool) bool {
if path == "" {
log.Debug("Path is empty")
return false
}
info, err := os.Stat(path)
if err != nil {
if os.IsNotExist(err) { // look for the specific error type
return false
}
}
return isDir == info.IsDir()
}
// FolderExists reports whether the provided directory exists.
func FolderExists(path string) bool {
return exists(path, true)
}
// FileExists reports whether the provided directory exists.
func FileExists(path string) bool {
return exists(path, false)
}
Just to be on the safe side we didn't forget anything major, let's run it again to make sure everything is alright.
# Run it again
go run main.go
INFO[0000] Folder exists
INFO[0000] File exists
Test with testdata
Ok,we need to write tests for pkg/utils/filesystem.go
.
One way of doing this is by accessing a real file system to do testing, then we're talking about what you'd call system or integration testing. That's fine in itself: for instance, you could run such tests in a throw-away container as part of a CI pipeline.
But with this approach you can't sensibly do what is called unit-testing.
To proceed with that, we are going to use the generated boilerplate code that vscode
happily offers (give you have the Go extensions installed).
Place the mouse cursor over the FolderExists()
function and press the keys: CTRL+SHIFT+P
.
From the drop-down menu, type: Go: test
and select the one for the function.
This will create the filesystem_test.go
file and populate it with a table test.
Add your test logic there:
// pkg/utils/filesystem_test.go
// ----------------------------
package utils
import "testing"
// Add those variables as well
var (
existingFolder = "./testdata/a-folder-that-exists"
nonExistingFolder = "./testdata/a-folder-that-does-not-exists"
)
func TestFolderExists(t *testing.T) {
type args struct {
path string
}
tests := []struct {
name string
args args
want bool
}{
// ----- test logic start ----- //
{
"Returns true when given folder exists",
args{path: existingFolder},
true,
},
{
"Returns false when given folder does not exist",
args{path: nonExistingFolder},
false,
},
// ----- test logic end ----- //
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := FolderExists(tt.args.path); got != tt.want {
t.Errorf("FolderExists() = %v, want %v", got, tt.want)
}
})
}
}
The tests is relying upon real data on the actual hard drive.
Run tests and you will see they will fail:
go test ./... -v
? github.com/drpaneas/checkfile [no test files]
=== RUN TestFolderExists
=== RUN TestFolderExists/Returns_true_when_given_folder_exists
filesystem_test.go:33: FolderExists() = false, want true
=== RUN TestFolderExists/Returns_false_when_given_folder_does_not_exist
--- FAIL: TestFolderExists (0.00s)
--- FAIL: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
--- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
FAIL
FAIL github.com/drpaneas/checkfile/pkg/utils 0.098s
FAIL
Create the necessary testdata and try again:
mkdir -p pkg/utils/testdata/a-folder-that-exists
$ go test ./... -v? github.com/drpaneas/checkfile [no test files]
=== RUN TestFolderExists
=== RUN TestFolderExists/Returns_true_when_given_folder_exists
=== RUN TestFolderExists/Returns_false_when_given_folder_does_not_exist
--- PASS: TestFolderExists (0.00s)
--- PASS: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
--- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
PASS
ok github.com/drpaneas/checkfile/pkg/utils (cached)
Now, see the coverage. Go into the package and run cover
.
cover
PASS
coverage: 77.8% of statements
ok github.com/drpaneas/checkfile/pkg/utils 0.293s
PS: You need to have this snippet into your ~/.bashrc
:
# Go coverage trick
cover () {
t="/tmp/go-cover.$$.tmp"
go test -coverprofile=$t $@ && go tool cover -html=$t && unlink $t
}
Let's enhance the testing coverage, by adding:
{
"Returns false when provided path is empty",
args{path: ""},
false,
},
Run the tests again and the coverage:
go test ./...
$ cover ./pkg/utils
ok github.com/drpaneas/checkfile/pkg/utils 0.232s coverage: 88.9% of statements
Add the tests for the other function as well:
The variables first:
var (
existingFile = "./testdata/a-file-that-exists"
nonExistingFile = "./testdata/a-file-that-doesn-not-exist"
)
The test code:
{
"Returns true when given file exists",
args{path: existingFile},
true,
},
{
"Returns false when given file does not exist",
args{path: nonExistingFile},
false,
},
{
"Returns false when provided path is empty",
args{path: ""},
false,
},
And create the appropriate file for testing:
touch pkg/utils/testdata/a-file-that-exists
Run the tests (they should all pass):
$ go test -v ./...? github.com/drpaneas/checkfile [no test files]
=== RUN TestFolderExists
=== RUN TestFolderExists/Returns_true_when_given_folder_exists
=== RUN TestFolderExists/Returns_false_when_given_folder_does_not_exist
=== RUN TestFolderExists/Returns_false_when_provided_path_is_empty
--- PASS: TestFolderExists (0.00s)
--- PASS: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
--- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
--- PASS: TestFolderExists/Returns_false_when_provided_path_is_empty (0.00s)
=== RUN TestFileExists
=== RUN TestFileExists/Returns_true_when_given_file_exists
=== RUN TestFileExists/Returns_false_when_given_file_does_not_exist
=== RUN TestFileExists/Returns_false_when_provided_path_is_empty
--- PASS: TestFileExists (0.00s)
--- PASS: TestFileExists/Returns_true_when_given_file_exists (0.00s)
--- PASS: TestFileExists/Returns_false_when_given_file_does_not_exist (0.00s)
--- PASS: TestFileExists/Returns_false_when_provided_path_is_empty (0.00s)
PASS
ok github.com/drpaneas/checkfile/pkg/utils 0.301s
Check the testing coverage:
cover ./pkg/utils
ok github.com/drpaneas/checkfile/pkg/utils 0.268s coverage: 100.0% of statements
Although we already hit the 100% testing coverage, I would like to add another test-case.
Add these two code block respectively:
// TestFolderExists()
// ...
{
"Returns false when provided path is not a directory",
args{path: existingFile},
false,
},
// TestFileExists
// ...
{
"Returns false when provided path is not a file",
args{path: existingFolder},
false,
},
... and that's it!
Run again the tests to verify everything works and we are good to go.
go test ./...
? github.com/drpaneas/checkfile [no test files]
ok github.com/drpaneas/checkfile/pkg/utils 0.332s
Test without testdata
Ok now let's do the same without the actual testdata.
To unit-test your function you need to replace something it uses with something "virtualised".
Exactly how to do that is an open question.
Right now, a work is being done on bringing filesystem virtualisation right into the Go standard library, but while it's not there yet, there's a 3rd-party package which has been doing that for ages, — for instance, github.com/spf13/afero.
First let's remove the actual real data, and see the tests fail:
rm -r ./pkg/utils/testdata
So now the tests are failing obviously:
go test ./...
? github.com/drpaneas/checkfile [no test files]
--- FAIL: TestFolderExists (0.00s)
--- FAIL: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
filesystem_test.go:47: FolderExists() = false, want true
--- FAIL: TestFileExists (0.00s)
--- FAIL: TestFileExists/Returns_true_when_given_file_exists (0.00s)
filesystem_test.go:86: FileExists() = false, want true
FAIL
FAIL github.com/drpaneas/checkfile/pkg/utils 0.404s
FAIL
To do that we will use Afero.
Writing a unit test for filesystem checking is not trivial. You should NOT create a real file on the system, because then your test will depend on the I/O of the file system itself.
Let's modify our code to use Afero for both implementation and test.
So for the filesystem.go
just add:
// Making the global, sharing the memory filesystem
var (
FS afero.Fs = afero.NewMemMapFs()
AFS *afero.Afero = &afero.Afero{Fs: FS}
)
and replace the os.Stat
to AFS.Stat
. That's all.
For the filesystem_test.go
just add:
func init() {
// create test files and directories
AFS.MkdirAll(existingFolder, 0755)
afero.WriteFile(AFS, existingFile, []byte("test file"), 0755)
}
Run the tests and they should pass:
go test ./... -v
? github.com/drpaneas/checkfile [no test files]
=== RUN TestFolderExists
=== RUN TestFolderExists/Returns_true_when_given_folder_exists
=== RUN TestFolderExists/Returns_false_when_given_folder_does_not_exist
=== RUN TestFolderExists/Returns_false_when_provided_path_is_empty
=== RUN TestFolderExists/Returns_false_when_provided_path_is_not_a_directory
--- PASS: TestFolderExists (0.00s)
--- PASS: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
--- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
--- PASS: TestFolderExists/Returns_false_when_provided_path_is_empty (0.00s)
--- PASS: TestFolderExists/Returns_false_when_provided_path_is_not_a_directory (0.00s)
=== RUN TestFileExists
=== RUN TestFileExists/Returns_true_when_given_file_exists
=== RUN TestFileExists/Returns_false_when_given_file_does_not_exist
=== RUN TestFileExists/Returns_false_when_provided_path_is_empty
=== RUN TestFileExists/Returns_false_when_provided_path_is_not_a_file
--- PASS: TestFileExists (0.00s)
--- PASS: TestFileExists/Returns_true_when_given_file_exists (0.00s)
--- PASS: TestFileExists/Returns_false_when_given_file_does_not_exist (0.00s)
--- PASS: TestFileExists/Returns_false_when_provided_path_is_empty (0.00s)
--- PASS: TestFileExists/Returns_false_when_provided_path_is_not_a_file (0.00s)
PASS
ok github.com/drpaneas/checkfile/pkg/utils 0.349s
Notice we have no testdata any more.
Basically, Afero
allows you to not use os
directly but write all your code in a way so that instead of the os
package it calls methods on an instance of a type implementing particular interface: in the production code that object is a thin shim for the os
package, and in the testing code it's replaced by whatever you wish; afero
has a readily-available in-memory FS backend to do this.
To sum up, although it's a good idea to do this, there might be cases where Afero fails to offer you this isolation.
For example if you try to do anything on top of the basic filesystem usage, then Afero might not be enough.
For example, if you try to extract, zip or compress a fail, you will see that those functions are not compatible with Afero, thus you will have either to send a PR to afero or suffer by creating lot's of extra code for achieving that.
Top comments (0)