I have been working on my own Web Desktop OS called "ArozOS" for nearly 4+ years now. I have been setting up servers to store my data and provide different kind of services, all using my trusty Web Desktop OS for simple server management. However, separating the resources into so many servers are sometime a bit difficult to manage.
In this post, I will show you how I design a new file system for ArozOS and make it link up to all my servers. Creating a truly private cloud experience that pretty much make my life much easier in finding files, backup or even music collections in my sea of hard disks.
If this is your first time reading my post, I would recommend you to take a look at my old post over here so you can have a glance on how my web desktop OS looks like in general.
Writing Local Program is Simple
Writing local programs that use local disk as storage is pretty straight forward. You can use io/ioutil or the os package to access anything you need locally. This is how the original ArozOS File System was designed. For every file you saw on the web desktop interface, it correspond to a local file in your server's hard disk.
However, I always want to bridge the file systems between servers together using some kind of protocol. Some of the ArozOS users already bridging their own servers using SMB / Samba, WebDAV (with davfs2 on Linux) or SFTP mounting. These solution are quick and dirty way to get things done, however this causes some issues as this created platform dependencies on the arozos system.
I didn't plan to solve this (As 1st rule of programming: If it works, don't touch it) until a feature request pop up on our Github repo asking for a new function to be implemented: Allows users to mount a webdav folder into arozos as vroot.
And of course we can use rsync or davfs2 to get the job done. However, as I am kind of a person who don't like hacky stuffs run on a 24/7 server, I started to think of a proper solution to this issue.
Virtual File System Layer
During the old beta development of the web desktop system, we did encounter a lot of security concerning path escape issues. That is why we developed the Virtual File System Layer on the ArozOS 1.0. This VFS layer provide an virtualized path for each file access and make the underlying infrastructure modification much easier.
Thus, in this update, I designed something that slip between the old VFS layer and Physical File System layer provided by the OS and call it the "File System Abstraction" layer.
File System Abstraction Driver
To allow ArozOS to support different kind of file system clients, a driver needed to be developed for each of the file system types. For the most basic one, it is the local disk driver. It basically re-map the function from os and ioutil into a special interface structure (which I will show you later) and allow the interface to be used to access local file system.
Other network file system or protocol can also be used this way. By developing a driver and adapt to the arozos file system interface, now any network file system protocol can be used as a local virtual drive in your ArozOS system, making them transparent for the WebApp that lies on top of the Virtual File System layer.
Replacing os, ioutil and more
As local file system is no longer the only file system type that can be used in ArozOS, we need to develop a special interface that allows file operations transparency between the Virtual File System and the target file system. Hence, we will need to define our own file system access interface for all the legacy virtual file system functions. This is where we referenced Afero, a VFS implementation that use for unit testing.
Afero provide a really similar file system methods to the os package but lacking a lot of features regarding file operation itself.
That is why we decided to combine the os.File functions with the afero file system functions. And here are what we came up with.
File Abstraction Interface
The file abstraction interface is pretty much the same as os.File Method but in interface{}. This allow any struct that provide the same functions be used as File in the arozos system.
type File interface {
Chdir() error
Chmod(mode fs.FileMode) error
Chown(uid, gid int) error
Close() error
Name() string
Read(b []byte) (n int, err error)
ReadAt(b []byte, off int64) (n int, err error)
Readdirnames(n int) (names []string, err error)
ReadFrom(r io.Reader) (n int64, err error)
Readdir(n int) ([]fs.FileInfo, error)
Seek(offset int64, whence int) (ret int64, err error)
Stat() (fs.FileInfo, error)
Sync() error
Truncate(size int64) error
Write(b []byte) (n int, err error)
WriteAt(b []byte, off int64) (n int, err error)
WriteString(s string) (n int, err error)
}
File System Abstraction Interface
The File System Abstraction Interface is the most important part of the design. This allow us to perform file operation with ease between two file system types. For example, with these functions, we can do direct data streaming between SMB server and WebDAV server using one of the ArozOS server as middle-man. Or we can easily convert a file that only serve via SMB into a HTTP Stream.
type FileSystemAbstraction interface {
//Fundamental Functions
Chmod(string, os.FileMode) error
Chown(string, int, int) error
Chtimes(string, time.Time, time.Time) error
Create(string) (arozfs.File, error)
Mkdir(string, os.FileMode) error
MkdirAll(string, os.FileMode) error
Name() string
Open(string) (arozfs.File, error)
OpenFile(string, int, os.FileMode) (arozfs.File, error)
Remove(string) error
RemoveAll(string) error
Rename(string, string) error
Stat(string) (os.FileInfo, error)
Close() error
VirtualPathToRealPath(string, string) (string, error)
RealPathToVirtualPath(string, string) (string, error)
FileExists(string) bool
IsDir(string) bool
Glob(string) ([]string, error)
GetFileSize(string) int64
GetModTime(string) (int64, error)
WriteFile(string, []byte, os.FileMode) error
ReadFile(string) ([]byte, error)
ReadDir(string) ([]fs.DirEntry, error)
WriteStream(string, io.Reader, os.FileMode) error
ReadStream(string) (io.ReadCloser, error)
Walk(string, filepath.WalkFunc) error
}
You might notice the first section of the interface is pretty close to what Afero offers. For the second half, it is a mix of different functions that commonly available in utilities packages. For example, you will see Glob and Walk from filepath, or ReadDir from ioutil.
Streaming between File System Abstractions
The most important function that make the abstraction interface flexible is the WriteStream and ReadStream function.
The WriteStream and ReadStream function allow us to provide and write a byte stream into the target virtual driver, no matter the target virtual drive is a local or remote FS.
With these two functions, we can do something like doing real-time streaming between two remote FS using different protocol
or converting a remote FS protocol that is not supported by your device into something that support by basically every devices
I got myself a universal gateway to all my files!
Currently I am still rewriting majority of my system to support this kind of virtual file system architecture. I guess it might take another 4 - 6 months to complete. But after the hard works, I think I will be getting one of my best written piece of software - A universal gateway to all my files from all my servers, accessible with just a single browser tab! 😀
Top comments (0)