System Design

“Here’s the simple truth: you can’t innovate on products without first innovating the way you build them.” - Alex Schleifer, Airbnb

What is system design ?
System design is the process of defining the architecture, interfaces and data for a system that satisfies specific requirements​. Once your business or organization determines its requirements, you can begin to build them into a physical system design that addresses the needs of your customers.

What is the need of system design ?

“Programming without an overall architecture or design in mind is like exploring a cave with only a flashlight: You don’t know where you’ve been, you don’t know where you’re going, and you don’t know quite where you are.” – Danny Thorpe

1. It allows us to scale applications and create efficient architectures that are resilient and performant. Enables us to communicate effectively and helps in designing an architecture that meets those requirements.
2. We can trade-offs between scalability, cost, performance, reliability, and maintainability. It also involves understanding the best practices and architectural patterns to use in order to make the system as robust as possible.
3. When designing a system, it is important to consider the underlying infrastructure, such as databases, networks, and other services. We always needs to consider the design for scalability, security.

System design fundamentals

1. Scaling

Scalability refers to an application’s ability to handle and withstand an increased workload without sacrificing latency.​
Two types ​

1. Horizontal Scaling​ = adding more hardware to the existing hardware resource pool​
2. Vertical Scaling = adding more power to your server.

2. Microservice

Microservice architecture is a distinctive method of developing software systems that tries to focus on building single-function modules with well-defined interfaces and operations.

3. Proxy

A proxy server is a system or router that provides a gateway between users and the internet. In other words proxy server is a computer on the internet that accepts the incoming requests from the client and forwards those requests to the destination server.

4. CAP Theorem

In computer science, the CAP theorem, sometimes called CAP theorem model or Brewer’s theorem after its originator, Eric Brewer, states that any distributed system or data store can simultaneously provide only two of three guarantees: consistency, availability, and partition tolerance (CAP).

5. Storage

Scalability of your system design highly depends on the way you are storing the data. Different types of data storage mechanisms are suitable to manage your data for different use cases of your system, to achieve high scalability and better response time.
Some of the factors affecting the choice of data storage mechanisms are structure of your data, query frequency, query pattern, response time required, and scale needed.

• Caching

Getting data from in-memory cache is much faster than getting the data from database, file system, remote call to a third party service, or generating the data from complex computation.

• File Storage

For handling static asserts like product images for an ecommerce solution, videos for a video streaming solution, or file in an online directory service like google drive, it is ideal use a file storage or blob storage solution. The core difference between file or blob storage and databases is, in databases you can query-on and modify individual fields of database records, whereas in file storage you can query-on file’s metadata and only query or modify the file as a whole.

• Time Series Databases

A Time Series Database (TSDB) is a database specifically designed and optimised for time-stamped sequential data. It is suitable for measurements or events data that is tracked, monitored, and aggregated sequentially over a period of time.

• Data Warehouse

Data warehouse is a centralised and consolidated large amount of historical data solely intended to perform queries and analysis, this data is derived from a wide range of sources like application logs, transactions, and user events. It’s analytics capabilities allow organisations to derive valuable business insights from their data and improve decision making.

• Data Lakes

Data lakes are used when organisations need low-cost storage of unformatted, unstructured data from multiple sources that they intend to use for some purpose in the future. The structure, integrity, selection, and format of the various dataset is derived at the time of analysis by the person doing the analysis, at a later point in time in future.

• Relational database

It is a collection of data entities with pre-defined relationships between them, mostly based on their domain language. These entities are organised as tables with rows and columns. Each column holds a certain kind of data called a field or attribute about the data record, and each row represents a collection of related fields. attributes of one data record.

• NoSQL database

NoSQL databases are build for flexible schemas, and are recognised for their performance at scale. Key difference between NoSQL and relational databases is that NoSQL is for unstructured storage. This means NoSQL databases do not have fixed table structures, they are schema free

• Database combinations

In a modern microservices based application we use a combination of data storage mechanisms. Microservices architecture fundamentally supports it as it allows you to choose different databases for different services based on the use cases.

to be continued …