What is Bottleπ€?
π¨π»βπ»: Bottle is a lightweight kv storage engine based on LSM-TREE.
Project Home pages: https://bottle.ibyte.me/
Project Introduction
The first thing to note is that Bottle is a KV embedded storage engine, not a KV database.
Other
I am a novice, self-taught go language programming, and then realized this project through the bitcask paper, I hope you can give me a star, thank you, my English is not very good.
The function implementation of this project is completely based on the
bitcask
paper.
Interested friends can go and see this course. If you think it is good, you can give me a small star β Thank you.
Install Moduleπ
You just need to install the Bottle module in your project to use:
go get -u github.com/auula/bottle
Bottle Source codeπ²
.
βββ LICENSE
βββ README.md
βββ bottle-logo.svg
βββ bottle.go # Main File
βββ bottle_test.go
βββ encoding.go # Data encoding module
βββ encrypted.go # Data encrypted module
βββ encrypted_test.go
βββ example # Exmaple code
βΒ Β βββ config.yaml
βΒ Β βββ demo_exchange.go
βΒ Β βββ demo_put_get.go
βββ go.mod
βββ go.sum
βββ gopher-bottle.svg
βββ hashed.go # Hashed function
βββ item.go # Data log Entry
βββ option.go # Bottle Configure
1 directory, 17 files
Data Directory π
Since the bottle design is based on a single-process program, each storage instance corresponds to a data
directory, data
is the log merge structure data directory, and index
is the index
data version.
Log consolving structural data The current version is merged every time the data is started. The default is that all data files under the DATA data folder occupies the total and more than 1GB
of more than 1GB
will trigger a merge, and the data that is not used after the merger is discarded.
Of course, if the dirty data merge requirements are not reached, the data file is archived in the size of the configured, each data has a version number, and is set to read-only mount, the process work directory structure is as follows:
./testdata
βββ data
β βββ 1.data
βββ index
βββ 1646378326.index
βββ 1646378328.index
2 directories, 3 files
When the storage engine starts working, the folder and files can only be operated by this process to ensure data security.
Basic API β
Below is a usage example of a basic data manipulation I wrote:
package main
import (
"fmt"
"github.com/auula/bottle"
)
func init() {
// Open a storage instance by default configuration
err := bottle.Open(bottle.DefaultOption)
// And handle possible errors
if err != nil {
panic(err)
}
}
// Userinfo test data structure
type Userinfo struct {
Name string
Age uint8
Skill []string
}
func main() {
// PUT Data
bottle.Put([]byte("foo"), []byte("66.6"))
// If it is converted to string, it is a string
fmt.Println(bottle.Get([]byte("foo")).String())
// If there is no default, it is 0
fmt.Println(bottle.Get([]byte("foo")).Int())
// If it is not success, it is false.
fmt.Println(bottle.Get([]byte("foo")).Bool())
// If it is not success, it is 0.0
fmt.Println(bottle.Get([]byte("foo")).Float())
user := Userinfo{
Name: "Leon Ding",
Age: 22,
Skill: []string{"Java", "Go", "Rust"},
}
var u Userinfo
// Save the data object by BSON, and set the timeout for 5 seconds
// TTL timeout can not set demand
bottle.Put([]byte("user"), bottle.Bson(&user), bottle.TTL(5))
// Analysis of structures by unwrap
bottle.Get([]byte("user")).Unwrap(&u)
// Print value
fmt.Println(u)
// Remove key 'foo'
bottle.Remove([]byte("foo"))
// Close handleable error that may happen
if err := bottle.Close(); err != nil {
fmt.Println(err)
}
}
The following is a separate variable assignment and catch handling errors:
data := bottle.Get([]byte("user"))
if data.IsError() {
fmt.Println(data.Err)
} else {
fmt.Println(data.Value)
}
Encryptorπ
The data encryptor
records the value of the data, that is, the block encryption at the field level, instead of encrypting the entire file once, which will cause performance consumption, so the block data segment encryption method is adopted.
The following example uses the bottle.SetEncryptor(Encryptor,[]byte)
function to set the data encryptor and configure the 16-bit data encryption key.
func init() {
bottle.SetEncryptor(bottle.AES(), []byte("1234567890123456"))
}
You can also customize the interface to implement the data encryptor:
// SourceData for encryption and decryption
type SourceData struct {
Data []byte
Secret []byte
}
// Encryptor used for data encryption and decryption operation
type Encryptor interface {
Encode(sd *SourceData) error
Decode(sd *SourceData) error
}
The following code is the implementation code of the built-in AES
encryptor, just implement the bottle.Encryptor
interface, and the data source is the bottle.SourceData
structure field:
// AESEncryptor Implement the Encryptor interface
type AESEncryptor struct{}
// Encode source data encode
func (AESEncryptor) Encode(sd *SourceData) error {
sd.Data = aesEncrypt(sd.Data, sd.Secret)
return nil
}
// Decode source data decode
func (AESEncryptor) Decode(sd *SourceData) error {
sd.Data = aesDecrypt(sd.Data, sd.Secret)
return nil
}
The specific encryptor implementation code can be viewed in encrypted.go
Hashed π§°
If you need to customize the salad function, you can implement the bottle.Hashed interface:
type Hashed interface {
Sum64([]byte) uint64
}
Then complete your hash function configuration by built-in bottle.SetHashFunc(hash Hashed)
settings.
Index Size π
The index pre-set size will greatly affect your program acception and read data. If you can expect the index size required to run during initialization, and in initialization, you can decreaseThe performance issues that run data migration and expansion brought by the program during operation.
func init() {
// setting indexes size
bottle.SetIndexSize(1000)
}
Configuration Information
You can also use the default configuration, you can initialize your storage engine with the structure of the built-in bottle.Option
, and the configuration instance is as follows:
func init() {
// Custom configuration information
option := bottle.Option{
// Working directory
Directory: "./data",
// Algorithm opens encryption
Enable: true,
// Customize the secret, you can use a built-in secret
Secret: bottle.Secret,
// Custom data size, storage unit is KB
DataFileMaxSize: 1048576,
}
// Custom configuration information
bottle.Open(option)
}
Of course, you can also use the built-in bottle.Load(path string)
function to load the configuration file to start Bottle, the configuration file format is YAML
, the configurable items are as follows:
# Bottle config options
Enable: TRUE
Secret: "1234567890123456"
Directory: "./testdata"
DataFileMaxSize: 536870912
It is important to note that the key implemented by the built-in encryptor must be 16
bits, if you are a custom implementation encrypler to set your custom encryprators through bottle.SetEncryptor(Encryptor,[]byte)
, then this secretThe number of digits will not be restricted.
Vision π
If you have any questions about this project, you can contact me from the GitHub home page, you can also initiate a pull request
, after your code is merged, you will appear in the following list~
Top comments (0)