Micro Nightmares

The Bad Old Days

In this article I am going to present what might look superficially like something you may be familiar with in the bad old days of monolithic applications. The monolithic application was a single application coded against a static database schema. It is monolithic because the data structures, business logic and interfaces are all defined in the one binary. Modification of any part of the application could potentially have impact on some other part.

Without a clear scope often these monolithic applications would become the 'do everything' application of the company. The increasing complexity of both the application and database schema meant that as time progressed modification would become increasingly difficult.

The orthodox approach to such applications was to break them into controllers, model and services. The controllers contained the API contract. They would depend on the model and the services to interact with the back end database. The services would contain the business logic of the application.

The difficulty of this approach is that effort required for changes increases disproportionately as the system complexity increases.

Microservices to the Rescue

Microservices were an architectural approach to break up monolithic applications into well defined smaller services which would, in theory, be self contained. The idea is that the larger system would be composed of many smaller applications with a clear scope and purpose. These separate services would have their own API contract, rather than sharing schema and databases. They would own their own database and control their own schema, thus insulating them from change in other systems. Lets call this approach to microservices the 'domain driven' approach because they are defined by the domain segments they contain.

While monolithic and microservice based applications can both be clustered it is the microservice approach which has benefited most from container based application hosting. Multiple clusters can be configured so that your application has a high level of redundancy and reliability. High levels of reliability and up-time are expected in modern web applications.

However, the fundamental structure looks very much the same as old monolithic applications, with controllers, model and services contained within the binary. The separation into independent applications did provide some benefit in terms of allowing independent teams to work on systems, but not without drawbacks.

Cross Cutting Concerns

While certain domain areas might be independent there are areas where there must be consistency across all applications. The classic example of this is security. Microservices all need to implement authorization. Authentication can be performed on the router before reaching the microservice, but the implementation of specific rules around what users are authorized to do must be incorporated into each service. Where with a monolithic application there might be one service dealing with authorization with a microservice you might see many separate and potentially inconsistent approaches taken to security.

High Latency

While using the microservice approach will not necessarily mean high latency, there is the risk that a architecture which involves deep call chains between related services due to services dependencies will result in high latency due to the number of hops and potential outages.

Inconsistencies across Services

Because each domain driven microservice has control over its own schema and there is no common database or agreed global schema we end up with separate services having subtly different representations. It also means where we might have had to maintain the code for one schema we will need to maintain code for slightly different schema.

Take an example were we want to add a field to the customer. It might be that the customer is maintained in one service, but is used across several. Each one will now need to have the schema changed and updates rolled out. While on the surface each service is independent there is a foundational coupling with the domain.

Another Object, Another Service, Another Database

Imagine you need to add another business object to your application. Under a domain driven microservice approach this means writing another application with its own domain, services, controllers. It means another cluster of servers, and more database resource.

Inefficiency Scalability

In a domain driven microservice application each service might come under high load potentially. In this case if it is dynamic the number of pods will grow to meet demand, even as other servers are idle because each server is only responsible for a specific service.

Capability Driven Microservices

A capability driven microservice approach is differentiated by how the services are broken down. The first most important principle is that the domain is expelled from code. They should each have an explicit capability of function.

Rather than having twenty different microservices all performing similar data storage and query functions with similar cross cutting concerns such as security, auditing, metrics and so forth, we have a single microservice which is capable of storing and querying data. The schema definitions for the data types are expelled into a meta-data store in the database itself.

Because microservices are not bound to the domain any node can execute any API call. This means less waste as increases in servers only occurs when the overall load demands it.

Dynamic Schema and Validation

If we introduce a new business object now it can be done at runtime, adding it to the meta-data via an administrators API call. We can then start saving data right away without any coding or new server infrastructure.

Though this approach the core capabilities of validation, complex querying, security and auditing are all present by default. Because the code is well tested and consistent you don't need to worry about potential variation between service.

Dynamic Business Rules

Applications require more than just storage and query. They will usually require the application of server side business rules which evaluate and calculate. Currently I've built this into the same node used for handling the API, but logically this could be a separate microservice concerned with business rule enforcement and orchestration. Again, the core principle here is separation of applications based on what they do rather than on the domain they handle. In fact we need to divorce the services from specific domains as far as possible in order to maximize broad reuse.

Isn't this a Monolithic Data Store?

The high level network architecture of capability driven microservices looks suspiciously like a monolithic application, in that there is a single application and database cluster. I'm avoiding SQL databases however as we want to ensure easy scalability.

There should be a single scalable data store for the system. My current approach allows for tables to be members of multiple schema, and schema to be owned by virtual 'apps'. Each schema has its own permissions. While it looks superficially monolithic there is logical separation of data between services and applications.

All for One and One for All

While there is a huge advantage in being able to quickly deliver API for applications with virtually no code there is another advantage to capability driven architecture. Capabilities added to a microservice are available to all. If you need to add some new capability, say doing a complex data transformation, this feature is made available to all users of the service, not just a single narrow application.

Collapsing Complexity

Capability driven microservices are perhaps a danger to software developers in that the need for a large number of independent small code bases for each domain area go away, replaced by a single flexible service which can handle common data storage and query use cases. Where the business needs other capabilities the same approach of domain separation means that they can be broadly applicable across the business.

Are there any downsides?

One downside is that these universal services are more complex than the domain coupled microservice approach. Furthermore most modern software development tools have been developed under the assumption of domain coupling, making life harder for those trying to develop using a more dynamic approach.

Another downside is version control of configurations. While the ability to quickly create and modify data structures is an awesome power it also comes with great responsibility. My approach does not yet enforce the same kind of code control as we do with software. This is a work in progress.