If you're interested in the final result, you can find it here: https://sapphire-db.com/

Intro

Around one year ago I started developing a new project. The project initially consisted of an asp.net core application as a backend server and an angular client application as a UI. One of the key requirements was the data synchronization in realtime.

I already knew and liked firebase and had the wish to be able to use it in the project. Sadly that was not possible because another requirement was that the application should have a asp.net core backend and can be self-hosted.

I made some research and sadly did not find any other solution that would make the development easier with the required tech stack.

The idea

After some days of frustration I made the decision to start developing my own realtime database. After some thinking I wrote down my most important requirements and features the database should have.

• Extend an existing asp.net core application that uses entity framework

The main idea was, that the realtime database should extend an existing application. The API should be as easy as possible and I wanted to make as few changes to an existing application as possible. That would make the migration of other existing applications much easier.

• Work with every database supported by ef core

The second idea was to build the realtime database on top of entity framework core. This would have the advantage that it could support any database, supported by entity framework core.

• No change in the usage of ef core on server side

I also did not want to make big changes in the existing code on server side. This means changes in the API for querying and managing data on server side directly, for example if you have a procedure that has no realtion to interaction with the client application. The realtime database should extend the existing code and manage synchronization in background automatically.

• Easy to use CRUD-operations and API

The most important feature for me was, that the API should be as easy as possible. I wanted to query data from the server on client side with a single line of code, like this:

this.db.collection('data').values();

• No SignalR

I already had some bad experiences using SignalR and decided that the project should not depend on it.

• Actions (Alternative to methods in controllers)

Because I'm already going to implement a connection and a new API on client side I thought, it would also be very helpful, to have something like custom actions you can define and easily call through the already existing connection.

Development

With the main features in mind I started creating first drafts and making tests.

I dont want to go to much into details but want to give a short overview of the main technical points related to the synchronization of data.

If you are interested I will write more articles explaining more of the concepts and technology used.

You can also check out the sources to learn more:

Notification about changes

The first thing I wanted to achieve was an easy way to get notified when something in the database changes.

I thought about triggers on database level but they would not be flexible enough and would not work with some database systems like the InMemory-database.

The solution I came up with, was to create a custom DbContext that hooks into the SaveChanges-method and triggers a notifier. The cool thing about that is, that EF Core already comes with a ChangeTracker that provides you all changes that affect the next change operation. The code I came up with looks like this:

public override int SaveChanges(bool acceptAllChangesOnSuccess)
{
    List<ChangeResponse> changes = GetChanges();
    int result = base.SaveChanges(acceptAllChangesOnSuccess);
    notifier.HandleChanges(changes, GetType());
    return result;
}

private List<ChangeResponse> GetChanges()
{
    return ChangeTracker.Entries()
        .Where(e => e.State == EntityState.Added || e.State == EntityState.Deleted || e.State == EntityState.Modified)
        .Select(e => new ChangeResponse(e, this)).ToList();
}

With this small piece of code I was already able to achieve one of the key requirements. You dont have to change anything in the code you are already using, except from changing all DbContexts to the new DbContext with the custom SaveChanges method.

Communication

The next challenge to solve was the communcation between client and server.

The final communication itself consists of two simple components. A command and a response. Every action on server side has a command. The client sends a command with a unique id to the server. The server will handle this command and on completion it will respond with a response containing the same unique id as the command. The client then can handle the response and associate it with the sent command.

For transfer I first implemented Websockets for they already have a Bi-Directional data transport unlike other technologies like ServerSentEvents.

I then created a general object ConnectionBase that includes information like the subscriptions a connection has and how to talk to the client. That made it possible to implement the support for SSE and Polling afterwards.

The basic structure of the ConnectionBase looks like this:

public abstract class ConnectionBase : IDisposable
{
    public void Init(HttpContext context)
    {
        Id = Guid.NewGuid();
        Subscriptions = new List<CollectionSubscription>();
        HttpContext = context;
    }

    public Guid Id { get; set; }

    public HttpContext HttpContext { get; set; }

    public List<CollectionSubscription> Subscriptions { get; set; }

    public abstract Task Send(object message);
}

Every connection implementation has to implement the Send-method to send data to the client.

The implementation for Websockets for exampl looks like this:

public class WebsocketConnection : ConnectionBase, IDisposable
{
    public WebsocketConnection(WebSocket webSocket, HttpContext context)
    {
        Websocket = webSocket;
        Init(context);
    }

    public WebSocket Websocket { get; set; }


    public override async Task Send(object message)
    {
        await Websocket.Send(message);
    }
}

Subscription

Every value that a client wants to receive has to be requested from the server. If you want the client to receive changes of a collection you have to create a subscription for this collection.

When a client subscribes to a collection a new CollectionSubscription is created and added to the list of the connection.

If something changes the notifier will iterate through all connections and look for subscriptions that are affected by the changes. The changes will then get send to the client.

Client implementation

The server implementation for data synchronization already works. Now I had to create the client implementation. I decided to use RxJs for this job because the concept of data synchronization is pretty reactive and it would do the perfect job.

I will skip the boring part of creating a connection etc and just concentrate on the key part of the synchronization. If you are interested in the exact implementation you can check out the sources.

The main part of the client implementation is this piece of code:

const subject = new ReplaySubject<T[]>();

const wsSubscription = this.connectionManagerService.sendCommand(subscribeCommand).subscribe(response => {
    if (response.responseType === 'QueryResponse') {
      subject.next((<QueryResponse>response).result);
    } else if (response.responseType === 'ChangeResponse') {
      // Find and update the value
    } else if (response.responseType === 'UnloadResponse') {
      // Remove the value
    } else if (response.responseType === 'LoadResponse') {
      // Insert the value
    }
});

The main part of this is the ReplaySubject. It holds the value of the collection.

When the client requests a value it sends the SubscribeCommand to the server. The server stores the id associated with this command and every time something related to the subscription changes, it sends a notification with the same id back to the client. That means that everything that is related to the subscription will appear in the subscribe block of this statement.

The client only has to merge the changes into one list and has its value. The cool thing about using RxJs is, that you only have to emit a new value using .next(newValue) and the changes will get emitted where you need it.

The final API can be used like this:

export class DemoComponent implements OnInit {
  values$: Observable<Entry[]>;

  constructor(private db: SapphireDb) { }

  ngOnInit() {
    this.values$ = this.db.collection<Entry>('entries').values();
  }
}

CRUD

Now it was time to implement the basic CRUD operations to make development a lot easier afterwards. This operations are basically just normal calls on the DbContext to manage the data. I just created a reusable interface to call this methods.

The API for the client looks like this:

// add
this.collection.add({
  content: 'test value'
});

// delete
this.collection.remove(value);

// update
this.collection.update({
  ...value,
  content: v
});

Actions

The last requirement I had were the actions I can define on server side and easily call them through an easy API on client side.

I created an ActionHandler that you can generally understand like a controller. It contains different methods that will be callable through an interface.

An example of an action handler looks like this:

public class ExampleActions : ActionHandlerBase
{
  private readonly ExampleDb db;

  public ExampleActions(ExampleDb db)
  {
  this.db = db;
  }

  public async Task<int> AsyncTask()
  {
    for (int i = 0; i <= 100; i++)
    {
      Thread.Sleep(10);
    }

    return 33;
  }
}

You can call the method by using this code on client side:

this.db.execute('example', 'AsyncTask')
  .subscribe(console.log);

The implementation itself is not very interesting but what is definitly worth a thought is the fact, that you can also use async Methods in the action handler.
Because the connection is Bi-Directional the call does not block the communication between client and server until the execution finished. Instead the result will get pushed to the client when ready.

The result

With this results I already was pretty satisfied but I did not stop improving the features. The final project has some cool and interesting features, for example:

• 💾 Easy CRUD operations
• 🔑 Authentication/Authorization included
• ✔️ Database support
• 🔌 Actions
• 🌐 NLB support

The final result is called SapphireDb and you can find it on Github. I am thankful for every kind of feedback.

Also check out the documentation to learn more.

PM> Install-Package SapphireDb