Sugarfina was migrated from M1 to M2 Cloud by Corra Magento Agency and today we will check the real performance of the Magento Expansive Cloud.
Sugarfina candy shop has traffic around 100-140K visitors per month.
Pretty small, however for our test it is a good candidate.
We will test Page generation time By Magento in different throughput/concurrency load. 1, 5, 10, 20, 40 concurrent requests to ensure that multiple users can harmoniously use a Magento eCommerce website without increasing TTFB (Time to First Bite) latency, which badly affecting user experience.
TTFB is a measurement of how long it takes the first bits to reach the browser from the initial request to the Magento Cloud server.
TTFB is a very key metric because the browser cannot begin rendering Magento Page content until it has the initial markup. Google recommending to have TTFB around 300 ms.
For the test, I will use the NodeJS scripting and GO K6:
/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/ \ | ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __________ \ |__| \__\ \_____/ .io
k6 is a developer-centric, free and open-source load testing tool built for making Magento 2 performance testing a productive and enjoyable experience.
Install Magento k6 load test globally as root:
wget https://bintray.com/loadimpact/rpm/rpm -O bintray-loadimpact-rpm.repo
sudo mv bintray-loadimpact-rpm.repo /etc/yum.repos.d/
sudo yum install k6
or Docker
docker pull loadimpact/k6
Magento load test usage
The set of basic options are not compatible with Apache ab toll which I have used before in my previous posts. But while ab can only set a concurrency level and lets the server adjust to it, K6 allows you to make scripting.
K6 is also quite extensible. Using the provided API it is very easy to integrate K6 with your package and run programmatic Magento 2 load tests. K6 makes it very easy to run load tests as part of systems tests, before deploying a new version of your software. The results include mean response times and percentiles so that you can abort deployment e.g. if 99% of the requests don't finish in 10 ms or less.
Calculating Magento 2 Requests Per Second with k6
With k6, you can test Magento 2 in terms of * requests per second (RPS) * by limiting how many requests each VU (virtual user) is able to make per unit of time. The formula to calculate the RPS is as follows:
Request Rate = (VU * R) / T
Request Rate: measured by the number of requests per second (RPS)
VU: the number of virtual users
R: the number of requests per VU iteration
T: a value larger than the time needed to complete a VU iteration
The above formula has can be thought of as a way to calculate the required number of virtual users you need for your script. The number of requests R are already known to you, since you have already defined them in your script. In or case it is 1. We are testing just a single Magento page per user. Then you need to calculate the time T needed to complete a VU iteration. Suppose you have 10 requests per virtual user, and you expect each to take 10 s to complete under the load (http_req_duration). The overall time T to complete the whole VU iteration is the number of requests multiplied by the supposed round-trip of requests/responses. It is also good practice to add one or more seconds to T to account for delays.
T = (R * http_req_duration) + (delay/sleep time) →
T = (1 * 10s) + 0s = 10s
Server response time is important.
Now that you know the values for R and T, you need to decide on the RPS you want to achieve. Suppose you want to achieve 1000 RPS, i.e. you want your system to handle 1000 requests per second. Given the previous formula:
Request Rate = (VU * R) / T
The VU calculation formula is:
VU = (Request Rate * T) / R →
VU = (1000 * 10) / 1 = 10000
10000 virtual users (threads) we need to generate 1000 requests per second load.
However, using my I7 CPU I can achieve only 100 Request per second using 1000 VUs Also with a higher RPS rate Response time will increase and you will need more VUs. basically, you need to find a point where response time is constant and this number is your optimal throughput. For the Magento Cloud, it is 1 request per second. See the results below.
You can check this in the results:
http_reqs..................: 1000 61.712989/s
iteration_duration.........: min=200.27µs med=8.73s avg=8.38s max=16.05s p(95)=13.24s
vus_max....................: 1000
where 1000 is VUs and 61 requests per second
max iteration duration = 16 sec.
RPS = 1000 / 16 = 62.5
We will test only two main use cases of the Magento eCommerce functionality.
- Catalog Category Page Generation Performance
- Catalog Product Page Generation Performance
We will test the same scenarios multiple times to calculate an average and 95 percentile of the Magento 2 page generation performance.
1. Magento cloud Product Page Generation performance
let's start by running a simple local script. Copy the code below, paste it into your favourite editor, and save it as "magento.js":
import http from 'k6/http';
import { sleep } from 'k6';
export default function () {
http.get('http://test.k6.io');
sleep(1);
}
Then run k6 using this command:
# k6 run magen.js -e url=https://www.sugarfina.com/sugar-skull-3-piece-candy-bento-box -u 1 -i 1 --include-system-env-vars=false
We will use 95% percentile - latency for 95% of the users without maximum extremums.
A percentile is a measure used in statistics indicating the value below which a given percentage of observations in a group of observations fall.
*** Why to use 95% percentile with Magento performance test***
The 95th percentile encompasses the experience of almost all of your eCommerce users, with only the most severe outliers excluded. This makes it perfect for spotting short-term trends and anomalies.
export function setup(data) {
console.log(JSON.stringify(__ENV));
return {'url': __ENV.url}
}
export default function (data) {
// The order of precedence is: defaults < config file < exported script options < environment variables < command-line flags.
// Options from each subsequent level can be used to overwrite the options from the previous levels, with the CLI flags having the highest priority.
var res = http.get(data.url);
console.log('Response time : ' + String(res.timings.duration) + ' ms');
console.log('Response status : ' + String(res.status) + ' ms');
console.log('Response TTFB : ' + String(res.timings.waiting) + ' ms');
}
Ok, it is our K6 Magento test results:
Concurrency 1
data_received..............: 2.7 MB 141 kB/s
data_sent..................: 16 kB 814 B/s
http_req_blocked...........: min=1.33µs med=1.44µs avg=43.76ms max=875.23ms p(95)=43.76ms
http_req_connecting........: min=0s med=0s avg=421.72µs max=8.43ms p(95)=421.72µs
http_req_duration..........: min=763.87ms med=924.78ms avg=917.88ms max=1.13s p(95)=1.05s
http_req_receiving.........: min=5.64ms med=8.58ms avg=10.68ms max=38.65ms p(95)=25.34ms
http_req_sending...........: min=63.09µs med=100.17µs avg=121.9µs max=533.39µs p(95)=151.09µs
http_req_tls_handshaking...: min=0s med=0s avg=2.13ms max=42.78ms p(95)=2.13ms
http_req_waiting...........: min=754.7ms med=915.74ms avg=907.07ms max=1.12s p(95)=1.05s
http_reqs..................: 20 1.033262/s
iteration_duration.........: min=249.36µs med=925.82ms avg=875.22ms max=1.66s p(95)=1.13s
iterations.................: 20 1.033262/s
vus........................: 1 min=1 max=1
vus_max....................: 1 min=1 max=1
Average TTFB is: 907 ms
95 Percentile: 1.05 ms
Let's check Magento 2 self-hosted on AWS Graviton2 medium (cheapest one) processor TTFB to feel the difference.
data_received..............: 1.3 MB 198 kB/s
data_sent..................: 2.2 kB 322 B/s
http_req_blocked...........: min=8.07µs med=8.98µs avg=3.69ms max=73.64ms p(95)=3.69ms
http_req_connecting........: min=0s med=0s avg=3.67ms max=73.42ms p(95)=3.67ms
http_req_duration..........: min=317.29ms med=320.63ms avg=323.99ms max=388.13ms p(95)=329.85ms
http_req_receiving.........: min=36.75ms med=37.33ms avg=39.35ms max=76.4ms p(95)=41.19ms
http_req_sending...........: min=34.94µs med=49.95µs avg=58.19µs max=207.18µs p(95)=73.37µs
http_req_tls_handshaking...: min=0s med=0s avg=0s max=0s p(95)=0s
http_req_waiting...........: min=279.81ms med=283.13ms avg=284.58ms max=311.51ms p(95)=288.75ms
http_reqs..................: 20 2.994795/s
iteration_duration.........: min=229.61µs med=321.17ms avg=298.58ms max=462.92ms p(95)=327.35ms
iterations.................: 20 2.994795/s
vus........................: 1 min=1 max=1
vus_max....................: 1 min=1 max=1
Average TTFB hosted on AWS without silly Magento Clouds is - 284ms with 95 percentile - 288.75 ms vs 1.05s on the Adobes Commerce Cloud.
Magento 2 Open Source is not so bad as its Commerce Cloud.
Concurrency 5
data_received..............: 1.4 MB 491 kB/s
data_sent..................: 10 kB 3.6 kB/s
http_req_blocked...........: min=1.53µs med=26.98ms avg=28.69ms max=60.71ms p(95)=60.69ms
http_req_connecting........: min=0s med=3.51ms avg=3.55ms max=7.19ms p(95)=7.17ms
http_req_duration..........: min=777.66ms med=1.08s avg=1.16s max=1.58s p(95)=1.53s
http_req_receiving.........: min=3.25ms med=16.62ms avg=17.04ms max=31.84ms p(95)=31.5ms
http_req_sending...........: min=96.09µs med=213.98µs avg=219.29µs max=344.63µs p(95)=337.84µs
http_req_tls_handshaking...: min=0s med=12.36ms avg=14.08ms max=31.48ms p(95)=31.46ms
http_req_waiting...........: min=751.53ms med=1.06s avg=1.14s max=1.57s p(95)=1.52s
http_reqs..................: 10 3.513193/s
iteration_duration.........: min=209.78µs med=1.11s avg=993.06ms max=1.58s p(95)=1.52s
iterations.................: 10 3.513193/s
vus........................: 4 min=4 max=5
vus_max....................: 5 min=5 max=5
Average: 993.06 ms however
95 percentile: 1.52 s
Concurrency 10
data_received..............: 2.8 MB 658 kB/s
data_sent..................: 21 kB 4.9 kB/s
http_req_blocked...........: min=1.15µs med=430.42ms avg=431.66ms max=865.56ms p(95)=864.4ms
http_req_connecting........: min=0s med=3.68ms avg=4.02ms max=8.85ms p(95)=8.84ms
http_req_duration..........: min=743.42ms med=1.01s avg=1.15s max=2s p(95)=1.93s
http_req_receiving.........: min=3.5ms med=19.7ms avg=20.08ms max=52.83ms p(95)=36.34ms
http_req_sending...........: min=64.98µs med=201.84µs avg=239.58µs max=571.64µs p(95)=510.48µs
http_req_tls_handshaking...: min=0s med=16.26ms avg=18.19ms max=38.56ms p(95)=37.81ms
http_req_waiting...........: min=732.85ms med=990.26ms avg=1.13s max=1.95s p(95)=1.9s
http_reqs..................: 20 4.711487/s
iteration_duration.........: min=236.87µs med=1.64s avg=1.44s max=2.86s p(95)=2.79s
iterations.................: 20 4.711487/s
vus........................: 2 min=2 max=10
vus_max....................: 10 min=10 max=10
Average: 1.13 s
95 Percentile: 1.9 s
Concurrency 20
data_received..............: 5.6 MB 1.1 MB/s
data_sent..................: 42 kB 8.4 kB/s
http_req_blocked...........: min=1.15µs med=27.47ms avg=30.02ms max=66.41ms p(95)=63.32ms
http_req_connecting........: min=0s med=3.62ms avg=4.17ms max=9.68ms p(95)=9.61ms
http_req_duration..........: min=771.83ms med=1.65s avg=1.72s max=3.28s p(95)=2.7s
http_req_receiving.........: min=2.27ms med=20.44ms avg=17.4ms max=32.78ms p(95)=30.48ms
http_req_sending...........: min=95.26µs med=188.4µs avg=235.17µs max=511.37µs p(95)=447.4µs
http_req_tls_handshaking...: min=0s med=18.08ms avg=20.18ms max=44.44ms p(95)=43.85ms
http_req_waiting...........: min=748.61ms med=1.63s avg=1.71s max=3.26s p(95)=2.68s
http_reqs..................: 40 8.063447/s
iteration_duration.........: min=272.16µs med=1.64s avg=1.67s max=3.28s p(95)=2.69s
iterations.................: 40 8.063447/s
vus........................: 5 min=5 max=20
vus_max....................: 20 min=20 max=20
Average: 1.71 s
95 Percentile: 2.69 s
Concurrency 40
data_received..............: 11 MB 1.9 MB/s
data_sent..................: 83 kB 14 kB/s
http_req_blocked...........: min=1.23µs med=40.84ms avg=43.5ms max=99.45ms p(95)=89.57ms
http_req_connecting........: min=0s med=3.48ms avg=4.24ms max=10.41ms p(95)=9.64ms
http_req_duration..........: min=789.82ms med=2.35s avg=2.2s max=3.17s p(95)=2.95s
http_req_receiving.........: min=2.87ms med=24.22ms avg=24ms max=61.64ms p(95)=41.88ms
http_req_sending...........: min=57.8µs med=208.3µs avg=215.18µs max=538.94µs p(95)=420.11µs
http_req_tls_handshaking...: min=0s med=32.07ms avg=33.33ms max=76.46ms p(95)=67.97ms
http_req_waiting...........: min=760.18ms med=2.32s avg=2.18s max=3.14s p(95)=2.93s
http_reqs..................: 80 13.485888/s
iteration_duration.........: min=223.3µs med=2.36s avg=2.19s max=3.27s p(95)=3.04s
iterations.................: 80 13.485888/s
vus........................: 13 min=13 max=40
vus_max....................: 40 min=40 max=40
Avarage: 2.18 s
Percentile 95: 3.05s
And this results in measuring only initial page rendering performance because of Magento 2 bad architecture your website also needs several other slow Ajax calls to fully rendered pages (Cart, Banners, Customer Data, Customisations requests). Order and checkout pages need 10-15 HTTP requests to be fully functional. Nothing bad to have additional requests however they should be fast and don't affect user experience.
In my next post, I will add that HTTP requests to the test scenarios to show AJAX functionality impact on Magento 2 performance. Thanks to the K6 performance tool we can achieve that without using legacy Jmeter scripts.
Magento 2 Product Page Performance result Pivot Table:
+----+------+---------------+
| | AVG | Percentile 95 |
+----+------+---------------+
| 1 | 907 | 1005 |
+----+------+---------------+
| 5 | 993 | 1520 |
+----+------+---------------+
| 10 | 1130 | 1900 |
+----+------+---------------+
| 20 | 1710 | 2690 |
+----+------+---------------+
| 40 | 2180 | 3050 |
+----+------+---------------+
Magento 2 Cloud Category Page Performance
Concurrency 1
data_received..............: 3.8 MB 356 kB/s
data_sent..................: 20 kB 1.9 kB/s
http_req_blocked...........: min=1.37µs med=1.6µs avg=4.26ms max=42.59ms p(95)=23.43ms
http_req_connecting........: min=0s med=0s avg=818.34µs max=8.18ms p(95)=4.5ms
http_req_duration..........: min=780.04ms med=928.68ms avg=1.05s max=2.01s p(95)=1.62s
http_req_receiving.........: min=18.25ms med=22.32ms avg=25.97ms max=50.4ms p(95)=41.3ms
http_req_sending...........: min=88.63µs med=99.24µs avg=143.06µs max=480.58µs p(95)=333.4µs
http_req_tls_handshaking...: min=0s med=0s avg=2.31ms max=23.14ms p(95)=12.73ms
http_req_waiting...........: min=750.57ms med=905.09ms avg=1.02s max=1.99s p(95)=1.6s
http_reqs..................: 10 0.936201/s
iteration_duration.........: min=202.77µs med=880.62ms avg=883.05ms max=2.01s p(95)=1.54s
Avarage: 1.02 s
95 Percentile: 1.6 s
Cuncurancy 5
data_received..............: 5.7 MB 1.2 MB/s
data_sent..................: 32 kB 6.5 kB/s
http_req_blocked...........: min=1.38µs med=2.03µs avg=14.44ms max=46.7ms p(95)=45.25ms
http_req_connecting........: min=0s med=0s avg=2.78ms max=9.59ms p(95)=9.55ms
http_req_duration..........: min=913.24ms med=1.32s avg=1.42s max=2.03s p(95)=1.99s
http_req_receiving.........: min=8.16ms med=23.72ms avg=29.28ms max=76.4ms p(95)=57.03ms
http_req_sending...........: min=93.49µs med=129.68µs avg=177.4µs max=401.33µs p(95)=361.66µs
http_req_tls_handshaking...: min=0s med=0s avg=7.77ms max=25.51ms p(95)=24.12ms
http_req_waiting...........: min=868.41ms med=1.31s avg=1.39s max=1.96s p(95)=1.95s
http_reqs..................: 15 3.088035/s
iteration_duration.........: min=226.43µs med=1.24s avg=1.27s max=2.07s p(95)=2s
Avarage: 1.39 s
95 percentile: 1.95s
Concurrency 10
http_req_blocked...........: min=1.42µs med=426.01ms avg=427.6ms max=860.05ms p(95)=856.9ms
http_req_connecting........: min=0s med=3.52ms avg=3.68ms max=8.55ms p(95)=8.08ms
http_req_duration..........: min=864.14ms med=1.19s avg=1.57s max=3.69s p(95)=3.14s
http_req_receiving.........: min=12.09ms med=36.62ms avg=43.11ms max=128.41ms p(95)=78.15ms
http_req_sending...........: min=68.59µs med=202.28µs avg=205.02µs max=398.16µs p(95)=335.29µs
http_req_tls_handshaking...: min=0s med=11.38ms avg=12.67ms max=28.67ms p(95)=27.09ms
http_req_waiting...........: min=835.87ms med=1.17s avg=1.53s max=3.65s p(95)=3.13s
http_reqs..................: 20 3.825479/s
iteration_duration.........: min=232.35µs med=1.8s avg=1.82s max=4.55s p(95)=3.34s
Average: 1.53 ms
95 percentile: 3.13 s
Concurrency 20
http_req_blocked...........: min=55.08ms med=59.98ms avg=60ms max=64.07ms p(95)=63.9ms
http_req_connecting........: min=6.84ms med=7.97ms avg=8.08ms max=9.16ms p(95)=9.16ms
http_req_duration..........: min=1.04s med=2.28s avg=2.17s max=3.36s p(95)=3.23s
http_req_receiving.........: min=34.65ms med=52.98ms avg=68.49ms max=296.89ms p(95)=109.76ms
http_req_sending...........: min=208.84µs med=311.28µs avg=328.6µs max=473.67µs p(95)=470.09µs
http_req_tls_handshaking...: min=35.95ms med=40.02ms avg=39.95ms max=42.3ms p(95)=42.19ms
http_req_waiting...........: min=1s med=2.23s avg=2.1s max=3.29s p(95)=3.17s
http_reqs..................: 20 5.661847/s
iteration_duration.........: min=259.99µs med=2.18s avg=2.02s max=3.42s p(95)=3.27s
Average: 2.1 s
95 Percentile: 3.17 s
Concurrency 40
http_req_blocked...........: min=1.27µs med=445.03ms avg=448.82ms max=901.91ms p(95)=900.54ms
http_req_connecting........: min=0s med=3.59ms avg=4.83ms max=15.95ms p(95)=11.08ms
http_req_duration..........: min=810.1ms med=2.22s avg=2.43s max=5.91s p(95)=4.06s
http_req_receiving.........: min=11.92ms med=55.08ms avg=64.52ms max=500.94ms p(95)=106.56ms
http_req_sending...........: min=60.61µs med=179.17µs avg=194.95µs max=575.51µs p(95)=366.95µs
http_req_tls_handshaking...: min=0s med=31.87ms avg=33.64ms max=69.3ms p(95)=68.84ms
http_req_waiting...........: min=765.03ms med=2.13s avg=2.37s max=5.41s p(95)=4s
http_reqs..................: 80 9.819241/s
iteration_duration.........: min=231.72µs med=2.79s avg=2.81s max=6.82s p(95)=4.94s
iterations.................: 80 9.819241/s
vus........................: 1 min=1 max=40
vus_max....................: 40 min=40 max=40
Average: 2.37 s
95 Percentile: 4 s
Magento 2 Category Page Performance result Pivot Table:
+----+------+---------------+
| | AVG | Percentile 95 |
+----+------+---------------+
| 1 | 1020 | 1600 |
+----+------+---------------+
| 5 | 1390 | 1950 |
+----+------+---------------+
| 10 | 1530 | 3130 |
+----+------+---------------+
| 20 | 2100 | 2690 |
+----+------+---------------+
| 40 | 2370 | 4000 |
+----+------+---------------+
How about Corra eCommerce implementation?
The backend performance is pretty awful. However, it can be a Magento Cloud mistake, not Corra's implementation.
Letch check theme frontend performance
Corra Magento theme Google Page Speed Frontend performance
Mobile:
You can check it there: https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fwww.sugarfina.com%2F3-piece-haunted-house-candy-bento-box
First Contentful Paint: 5.3 s
Speed Index: 19.6 s
Largest Contentful Paint: 55.9 s
Time to Interactive: 58.3 s
Total Blocking Time: 2,440 ms
Cumulative Layout Shift: 0.534
58 seconds to fully load the page on a mobile device is Absolutely not acceptable...
How about the desktop?
Desktop:
Pretty decent result.
So, we can see that Magento 2 is not the best for eCommerce websites and Magento Agencies should spend 100000 hours and merchants should pay for that 1 00000 000 000$$$ for the Magento and Adobe architectural and design failures.
As we can see, Magento Commerce Cloud and Magento 2 itself (no big difference hosting can't improve results so much even if TTFB will be 2 times faster) can generate 1-2 simultaneous requests without affecting user experience.
Top comments (0)