Compatibility comparison of GitLab on YugabyteDB and TiDB For PostgreSQL

Testing Background

GitLab is a very popular source code management tool around the world. In earlier versions, users could choose to use MySQL or PostgreSQL databases, but since version 12.1.0, the official support for MySQL has been completely abandoned.

Many functions in the new version of GitLab are developed based on the characteristics of PostgreSQL, which is a benchmark product for many using PostgreSQL as the underlying data storage.

Let's imagine this user scenario. A large group is divided into many divisions. Each division or even a small team may maintain its own GitLab. How to manage these warehouses at the group level becomes a difficult problem. for example:

• Version issues (open source version and commercial version, high version and low version)
• Fine-grained permission control
• data backup
• Infrastructure utilization

If there is a unified GitLab environment with good scalability and high availability, it is undoubtedly the best solution. However, the traditional stand-alone PostgreSQL database cannot meet the above requirements. Can you consider running GitLab on a distributed database?

In previous articles, we have compared the compatibility performance of GitLab on CockroachDB and YugabyteDB. References:

• Compatibility comparison of GitLab on CockroachDB and YugabyteDB (1) - System initialization
• GitLab's Compatibility Comparison on CockroachDB and YugabyteDB (2) - Read and Write Scenario Test

The test results of the above two distributed databases compatible with the PG protocol are that YugabyteDB has better compatibility, while the domestic distributed database TiDB's community project TiDB For PostgreSQL ) is also compatible with most GitLab functions, reference materials:

• TiDB For PostgreSQL is compatible with GitLab

In this article, we will select some core scenarios of GitLab on YugabyteDB and TiDB For PostgreSQL to compare the compatibility and performance, so as to provide more reference value for GitLab running on distributed databases.

Test Environment

1. GitLab

2. YugabyteDB

3. TiDB For PostgreSQL

4. Jmeter

Test Scenarios

For each read scenario, we will stress test 4 different concurrency levels, namely 10, 20, 50, and 100, and run continuously for 5 minutes.

For each write scenario, 1,000 sample data will be tested under the premise of the previous four concurrency levels.

Test Results

1、Project List

（99% response time trend comparison, higher means longer time）

Result analysis:

• The response time of TiDB For PostgreSQL increases significantly with the increase of concurrency, but the throughput cannot be improved. 10 concurrency is the best performance in this test.
• YugabyteDB also shows the same situation, and the error rate gradually increases.
• In this scenario, the performance and stability of GitLab running on TiDB For PostgreSQL is slightly better.

2、Project View

（99% response time trend comparison, higher means longer time）

Result analysis:

• TiDB For PostgreSQL still shows the best performance under 10 concurrency pressure.
• YugabyteDB also reflects the same situation, but the response time is slightly smaller, the throughput is larger, and the gap is relatively stable.
• GitLab runs better on YugabyteDB in this scenario.

3、Branch List

（99% response time trend comparison, higher means longer time）

Result analysis:

• TiDB For PostgreSQL also has the best performance at 10 concurrency.
• The overall trend of YugabyteDB is consistent, but the response time and throughput are significantly better than TiDB4PG.
• GitLab runs better on YugabyteDB in this scenario.

4、Issue List

（99% response time trend comparison, higher means longer time）

Result analysis:

• Similar to the previous scenario, GitLab runs better on YugabyteDB in this scenario, but the difference is not significant.

5、Merge Request List

（99% response time trend comparison, higher means longer time）

Result analysis:

• The trend is the same on both sides. YugabyteDB's test results are better than TiDB4PG, and the gap is not very large.

6、New Commit

（99% response time trend comparison, higher means longer time）

Analysis results:

• Concurrency has little effect on both sides, response time and throughput are stable, and there is almost no difference in performance between the two.

7、Create Branch

（99% response time trend comparison, higher means longer time）

Result analysis:

• When the concurrency is relatively small, the performance of TiDB For PostgreSQL is significantly better than that of YugabyteDB, and the error rates on both sides increase rapidly after the concurrency is increased.

8、Create Issue

Result analysis:

• TiDB For PostgreSQL cannot yet support the creation of Issues, and the error rate is 100%.
• YugabyteDB can create Issues, but in the case of concurrency, the error rate is also very high, and the response time is very long.

Summarize

In this test result, in most scenarios, GitLab running on YugabyteDB has slightly better performance than TiDB For PostgreSQL, which can be used as a performance reference for these two databases to a certain extent. At the same time, TiDB For PostgreSQL has some differences in compatibility. Supported features.

However, there are some reasons that may also affect the test results, including differences in the underlying data on both sides, differences in warehouse size, etc., because the above test scenarios involve not only database reading and writing, but also a large part of file operations.

On the other hand, from the perspective of the underlying architecture of the two, YugabyteDB's computing and storage layers are both in the TServer node. Compared with the deployment structure of TiDB For PostgreSQL, it is equivalent to at least 2 fewer network requests, which is also a factor that affects the stress test results. one.

TiDB For PostgreSQL is still in the process of continuous improvement, and this test result can also provide guidance for product development.