SeaweedFS vs JuiceFS

SeaweedFS is an efficient distributed file system that can read and write small data blocks quickly, and is prototyped after Facebook's Haystack. In this article, we will compare the differences in design and features between SeaweedFS and JuiceFS to help readers make a better choice for themselves.

SeaweedFS System Architecture

SeaweedFS consists of 3 parts, a Volume Server to store files in the underlying layer, a Master Server to manage clusters, and an optional Filer component to provide more features upwards.

Volume Server and Master Server

In terms of system operation, Volume Server and Master Server work together as file storage. Volume Server focuses on writing and reading data, and Master Server provides management service for clusters and Volumes.

When reading and writing data, the implementation of SeaweedFS is similar to Haystack in that a user-created Volume is a large disk file (Superblock in the figure below). In this Volume, all files written by the user (Needle in the figure below) are merged into the large disk file.

Before starting to write data, the caller needs to make a write request to SeaweedFS (Master Server), which then returns a File ID (consisting of a Volume ID and offset) based on the current amount of data, along with basic metadata information such as file length and Chunk. When the write is complete, the caller needs to associate the file with the returned File ID in an external system (e.g. MySQL). When reading the data, the contents of the file can be obtained very efficiently because the data can be directly referenced by File ID.

Filer

On top of the above-mentioned underlying storage unit, SeaweedFS provides a component called Filer, which connects to the Volume Server and Master Server. Thus it can provide rich functions and features (such as POSIX support, WebDAV, S3 interface, etc.). Like JuiceFS, Filer also needs to use an external database to store metadata.

For the sake of illustration, all the SeaweedFS mentioned below include Filer components.

JuiceFS Architecture

JuiceFS uses a separate storage architecture for data and metadata, where the file data itself is splitted and stored in an object store (e.g. Amazon S3), while the metadata is stored in a database of the user's choice (e.g. Redis, MySQL). By sharing the same Metadata Engine and object storage, JuiceFS achieves a distributed file system with strong consistency guarantee, and also has many features such as POSIX full compatibility and high performance.

Comparison in Metadata

Both SeaweedFS and JuiceFS support external databases to store metadata . In terms of the supported databases, SeaweedFS supports up to 24 databases. Due to high requirements for database transactions (see below), JuiceFS supports 3 types of transactional databases, 10 in total.

Atomic Operations

To ensure the atomicity of all metadata operations, JuiceFS needs to use a transactional database at the implementation level. SeaweedFS, on the other hand, has weaker requirements for database transaction capabilities. And it only enables transactions for some of the databases (SQL, ArangoDB, and TiKV) when performing rename operations. Also, since the original directory or file is not locked when copying metadata during the rename operation, data corruption may occur.

Change Log (changelog)

SeaweedFS generates a changelog for all metadata operations, which can be used for data replication (see below), operation auditing, etc., while JuiceFS does not implement this feature yet.

Comparison in Storage

As mentioned above, the data storage of SeaweedFS is implemented by Volume Server + Master Server, which supports features such as merged storage and correction code for small data blocks. The data storage of JuiceFS is based on Object Storage Service, and the relevant features are provided by the user-selected Object Storage.

File Splitting

When storing data, SeaweedFS and JuiceFS both split files into smaller chunks and then persist them in the underlying data system. SeaweedFS splits files into 8MB chunks, and for very large files (over 8GB), it also saves the chunk indexes to the underlying data system. JuiceFS, on the other hand, splits into 64MB Chunks and then into 4MB Objects, optimizing the performance of random writes, sequential reads, repeated writes, etc. by using the internal concept of a Slice. (See An introduction to the workflow of processing read and write for details)

Tiered Storage

For newly created Volumes, SeaweedFS stores the data locally, while for older Volumes, SeaweedFS supports uploading them to the cloud to achieve tiered storage. In this regard, JuiceFS relies on external services.

Data Compression

JuiceFS supports data compression using LZ4 or ZStandard, while SeaweedFS chooses compression algorithm based on the information such as extension and file type of the written files.

Storage Encryption

JuiceFS supports encryption in transit and encryption at rest. When encryption at rest is enabled, the user needs to provide a self-managed key, and all written data will be encrypted based on this key. (See "Data Encryption" for details.)

SeaweedFS also supports in-transit encryption and encryption at rest. When data encryption is enabled, all data written to Volume Server is encrypted with a random key, and the information about the corresponding random key is managed by the Filer, which maintains the metadata.

Access Protocols

POSIX Compatibility

JuiceFS is fully POSIX compatible, while SeaweedFS is only partially POSIX compatible (see Issue 1558 and Wiki), which is still being improved.

S3 Protocol

JuiceFS implements S3 gateway through MinIO S3 gateway. It provides a S3-compatible RESTful API for files in JuiceFS, and enables the management with tools such as s3cmd, AWS CLI, MinIO Client (mc), etc. if mounting is not convenient.

SeaweedFS currently supports about 20 S3 APIs, covering common read, write, check, and delete requests, with extension of functions for some specific requests (e.g. Read), as detailed in Amazon-S3-API.

WebDAV Protocol

Both JuiceFS and SeaweedFS support the WebDAV protocol.

HDFS Compatibility

JuiceFS is fully compatible with the HDFS API, not only with Hadoop 2.x and Hadoop 3.x, but also with various components of the Hadoop ecosystem. SeaweedFS provides basic compatibility with the HDFS API, but some operations (such as truncate, concat, checksum, and extended attributes) are not yet supported.

CSI Drivers

Both JuiceFS and SeaweedFS provide a Kubernetes CSI Driver to help users use the corresponding file system in the Kubernetes ecosystem.

Extended Features

Client-side caching

JuiceFS has a variety of client-side caching strategies, covering everything from metadata to data caching, and allows users to customize according to their scenarios (details), while SeaweedFS does not have client-side caching capabilities.

Cluster Data Replication

For data replication between multiple clusters, SeaweedFS supports two asynchronous replication modes, Active-Active and Active-Passive, both of which achieve consistency between different clusters through log replaying. For each changelog, there is a signature message to ensure that the same modification is not applied multiple times. In Active-Active mode, where the number of cluster nodes exceeds 2, some operations of SeaweedFS will be limited, such as renaming directories.

JuiceFS does not natively support data synchronization between clusters and relies on the data replication capabilities of the metadata engine and the object store.

On-cloud Data Caching

SeaweedFS can be used as a cache for the object stored on the cloud and supports manual cache warm-up . Modifications to cached data are synchronized asynchronously to the object store. JuiceFS needs to store files to the object store by chunks and does not yet support cache acceleration for data already in the object store.

Trash

By default, JuiceFS enables the Trash feature, which automatically moves user-deleted files to the .trash directory under the JuiceFS root directory for a specified period of time before the data is actually cleaned up. SeaweedFS does not support this feature currently.

Operations and Maintenance Tools

JuiceFS provides two subcommands, juciefs stats and juicefs profile, allowing users to view real-time or playback performance metrics from a certain time period. JuiceFS has also developed an external metrics API that allows users to easily monitor using Prometheus and Grafana.

SeaweedFS supports both Push and Pull approaches to work with Prometheus and Grafana, and provides an interactive weed shell tool for users to perform a range of operations and maintenance tasks (e.g., checking the current cluster status, listing files, etc.).

Other comparisons

• In terms of release date, SeaweedFS was released in April 2015 and has accumulated 16.4K stars so far, while JuiceFS was released in January 2021 and has accumulated 7.3K stars.
• In terms of the project, both JuiceFS and SeaweedFS adopt the commercial-friendly Apache License 2.0. SeaweedFS is mainly maintained by Chris Lu personally, while JuiceFS is mainly maintained by Juicedata, inc.
• Both JuiceFS and SeaweedFS are written in Go language.

Comparison Table

From [JuiceFS/Juicedata]~