Testing Background
GitLab is a very popular source code management tool around the world. In earlier versions, users could choose to use MySQL or PostgreSQL databases, but since version 12.1.0, the official support for MySQL has been completely abandoned.
Many functions in the new version of GitLab are developed based on the characteristics of PostgreSQL, which is a benchmark product for many using PostgreSQL as the underlying data storage.
Let's imagine this user scenario. A large group is divided into many divisions. Each division or even a small team may maintain its own GitLab. How to manage these warehouses at the group level becomes a difficult problem. for example:
- Version issues (open source version and commercial version, high version and low version)
- Fine-grained permission control
- data backup
- Infrastructure utilization
If there is a unified GitLab environment with good scalability and high availability, it is undoubtedly the best solution. However, the traditional stand-alone PostgreSQL database cannot meet the above requirements. Can you consider running GitLab on a distributed database?
In previous articles, we have compared the compatibility performance of GitLab on CockroachDB and YugabyteDB. References:
- Compatibility comparison of GitLab on CockroachDB and YugabyteDB (1) - System initialization
- GitLab's Compatibility Comparison on CockroachDB and YugabyteDB (2) - Read and Write Scenario Test
The test results of the above two distributed databases compatible with the PG protocol are that YugabyteDB has better compatibility, while the domestic distributed database TiDB's community project TiDB For PostgreSQL ) is also compatible with most GitLab functions, reference materials:
In this article, we will select some core scenarios of GitLab on YugabyteDB and TiDB For PostgreSQL to compare the compatibility and performance, so as to provide more reference value for GitLab running on distributed databases.
Test Environment
1. GitLab
| Configuration item | Description |
|---|---|
| Version | 12.1.0-ee |
| Deployment Environment | Ubuntu 18.04 x86 16c 32G hdd |
2. YugabyteDB
| Configuration item | Description |
|---|---|
| Version | 2.13.2.0 |
| Deployment Environment | 3 Node CentOS 7.9 x86 4c 8G ssd |
| Deployment Architecture | 3 replicas, 3 nodes each for Master and Tserver |
3. TiDB For PostgreSQL
| Configuration item | Description |
|---|---|
| Version | TiDB-v5.3.0 , PostgreSQL 12 compatible |
| Deployment Environment | 3 Node CentOS 7.9 x86 4c 8G hdd |
| Deployment Architecture | 3 replicas, 1 PD, 1 TiDB, 3 TiKV |
4. Jmeter
| Configuration item | Description |
|---|---|
| Version | 5.4 |
| Deployment Environment | Windows 10 12c 16G |
Test Scenarios
| Scene Type | Scene name |
|---|---|
| read (5) | - Project List - Project View - Branch List - Issue List - Merge Request List |
| write (3) | - New Commit - Create Branch - Create Issue |
For each read scenario, we will stress test 4 different concurrency levels, namely 10, 20, 50, and 100, and run continuously for 5 minutes.
For each write scenario, 1,000 sample data will be tested under the premise of the previous four concurrency levels.
Test Results
1、Project List
| Threads | TiDB For PostgreSQL | YugabyteDB | ||||
|---|---|---|---|---|---|---|
| 99 Line(s) | Throughput(/sec) | Error(%) | 99 Line(s) | Throughput(/sec) | Error(%) | |
| 10 | 1.901 | 10.7 | 0 | 1.685 | 11.8 | 0 |
| 20 | 4.283 | 9.9 | 0 | 4.05 | 11.6 | 0.14 |
| 50 | 8.254 | 9.6 | 0 | 14.955 | 11.1 | 0.44 |
| 100 | 12.617 | 9.6 | 0 | 21.186 | 10.6 | 0.46 |
Result analysis:
- The response time of TiDB For PostgreSQL increases significantly with the increase of concurrency, but the throughput cannot be improved. 10 concurrency is the best performance in this test.
- YugabyteDB also shows the same situation, and the error rate gradually increases.
- In this scenario, the performance and stability of GitLab running on TiDB For PostgreSQL is slightly better.
2、Project View
| Threads | TiDB For PostgreSQL | YugabyteDB | ||||
|---|---|---|---|---|---|---|
| 99 Line(s) | Throughput(/sec) | Error(%) | 99 Line(s) | Throughput(/sec) | Error(%) | |
| 10 | 4.164 | 5.9 | 0 | 3.014 | 6.1 | 0 |
| 20 | 8.745 | 5.4 | 0 | 5.272 | 5.8 | 0 |
| 50 | 13.718 | 5.4 | 0.42 | 11.557 | 6.3 | 0 |
| 100 | 21.998 | 5.4 | 1.45 | 17.912 | 6.5 | 0 |
Result analysis:
- TiDB For PostgreSQL still shows the best performance under 10 concurrency pressure.
- YugabyteDB also reflects the same situation, but the response time is slightly smaller, the throughput is larger, and the gap is relatively stable.
- GitLab runs better on YugabyteDB in this scenario.
3、Branch List
| Threads | TiDB For PostgreSQL | YugabyteDB | ||||
|---|---|---|---|---|---|---|
| 99 Line(s) | Throughput(/sec) | Error(%) | 99 Line(s) | Throughput(/sec) | Error(%) | |
| 10 | 2.687 | 7.3 | 0 | 1.541 | 11.8 | 0 |
| 20 | 6.101 | 6.8 | 0 | 3.515 | 11.2 | 0 |
| 50 | 11.043 | 6.6 | 0 | 6.642 | 11.2 | 0 |
| 100 | 18.3 | 6.7 | 0 | 12.076 | 11 | 0 |
Result analysis:
- TiDB For PostgreSQL also has the best performance at 10 concurrency.
- The overall trend of YugabyteDB is consistent, but the response time and throughput are significantly better than TiDB4PG.
- GitLab runs better on YugabyteDB in this scenario.
4、Issue List
| Threads | TiDB For PostgreSQL | YugabyteDB | ||||
|---|---|---|---|---|---|---|
| 99 Line(s) | Throughput(/sec) | Error(%) | 99 Line(s) | Throughput(/sec) | Error(%) | |
| 10 | 1.696 | 11.8 | 0 | 1.56 | 12.2 | 0 |
| 20 | 3.613 | 10.8 | 0 | 2.871 | 13.5 | 0 |
| 50 | 7.155 | 11.1 | 0 | 5.966 | 12.7 | 0 |
| 100 | 12.06 | 11 | 0 | 9.769 | 12.8 | 0 |
Result analysis:
- Similar to the previous scenario, GitLab runs better on YugabyteDB in this scenario, but the difference is not significant.
5、Merge Request List
| Threads | TiDB For PostgreSQL | YugabyteDB | ||||
|---|---|---|---|---|---|---|
| 99 Line(s) | Throughput(/sec) | Error(%) | 99 Line(s) | Throughput(/sec) | Error(%) | |
| 10 | 1.975 | 10.9 | 0 | 1.355 | 13.5 | 0 |
| 20 | 3.838 | 9.9 | 0 | 3.222 | 12.7 | 0 |
| 50 | 8.057 | 9.9 | 0 | 6.34 | 11.7 | 0 |
| 100 | 13.437 | 9.7 | 0 | 10.426 | 11.9 | 0 |
Result analysis:
- The trend is the same on both sides. YugabyteDB's test results are better than TiDB4PG, and the gap is not very large.
6、New Commit
| Threads | TiDB For PostgreSQL | YugabyteDB | ||||
|---|---|---|---|---|---|---|
| 99 Line(s) | Throughput(/min) | Error(%) | 99 Line(s) | Throughput(/min) | Error(%) | |
| 10 | 2.357 | 52.2 | 0 | 2.304 | 52.7 | 0 |
| 20 | 2.323 | 52.3 | 0 | 2.271 | 53.8 | 0 |
| 50 | 2.391 | 51.3 | 0 | 2.366 | 53.2 | 0 |
| 100 | 2.361 | 51 | 0 | 2.338 | 52.1 | 0 |
Analysis results:
- Concurrency has little effect on both sides, response time and throughput are stable, and there is almost no difference in performance between the two.
7、Create Branch
| Threads | TiDB For PostgreSQL | YugabyteDB | ||||
|---|---|---|---|---|---|---|
| 99 Line(s) | Throughput(/sec) | Error(%) | 99 Line(s) | Throughput(/sec) | Error(%) | |
| 10 | 4.08 | 3.9 | 0 | 15.186 | 3.2 | 0 |
| 20 | 8.386 | 4.1 | 0 | 27.249 | 3.2 | 1.5 |
| 50 | 59.681 | 1.8 | 51.3 | 67.882 | 1.8 | 42.6 |
| 100 | 100.633 | 3.2 | 75.9 | 60.061 | 7.4 | 82.6 |
Result analysis:
- When the concurrency is relatively small, the performance of TiDB For PostgreSQL is significantly better than that of YugabyteDB, and the error rates on both sides increase rapidly after the concurrency is increased.
8、Create Issue
| Threads | TiDB For PostgreSQL | YugabyteDB | ||||
|---|---|---|---|---|---|---|
| 99 Line(s) | Throughput(/sec) | Error(%) | 99 Line(s) | Throughput(/sec) | Error(%) | |
| 10 | - | - | 100 | 17.284 | 6.4 | 69.1 |
| 20 | - | - | 100 | 10.818 | 11 | 79.5 |
| 50 | - | - | 100 | 5.877 | 20 | 92.60 |
| 100 | - | - | 100 | 10.084 | 21.8 | 93.4 |
Result analysis:
- TiDB For PostgreSQL cannot yet support the creation of Issues, and the error rate is 100%.
- YugabyteDB can create Issues, but in the case of concurrency, the error rate is also very high, and the response time is very long.
Summarize
In this test result, in most scenarios, GitLab running on YugabyteDB has slightly better performance than TiDB For PostgreSQL, which can be used as a performance reference for these two databases to a certain extent. At the same time, TiDB For PostgreSQL has some differences in compatibility. Supported features.
However, there are some reasons that may also affect the test results, including differences in the underlying data on both sides, differences in warehouse size, etc., because the above test scenarios involve not only database reading and writing, but also a large part of file operations.
On the other hand, from the perspective of the underlying architecture of the two, YugabyteDB's computing and storage layers are both in the TServer node. Compared with the deployment structure of TiDB For PostgreSQL, it is equivalent to at least 2 fewer network requests, which is also a factor that affects the stress test results. one.
TiDB For PostgreSQL is still in the process of continuous improvement, and this test result can also provide guidance for product development.
Top comments (2)
I see an increasing
Error(%). What are those errors? I guess that the GitLab workload was built for PostgreSQL using Read Committed isolation level and you are running with Serializable. With CockroachDB and TiDB there's no choice but YugabyteDB is PostgreSQL compatible and then provides all isolation levels.Can you share the code or run it on YugabyteDB started with
--yb_enable_read_committed_isolation=trueas in dev.to/yugabyte/how-to-set-read-co...?where is the test code ???