Discovering UI Performance testing with XCTest. Scrolling performance.

Mobile app performance is a critical aspect of the user experience. Poor performance can lead to frustration, negative reviews, and ultimately, loss of users. During app development it is crucial to detect performance issues as early as possible. In 2020 Apple introduced new tools to build Performance tests for UI interactions and currently this is one of recommended methods of improving your apps performance. In this article I tested these tools and described my experience with them. This article is focused on scrolling performance specifically.

Setup

I prepared the demo project. This is simple app which shows you a list of plants, you can select a plant and open details screen:

For all measurements I used MacBook M2 Pro with Ventura 13.0 and iPhone 13 Pro with iOS 16.0, XCode version is 14.2.
Here is the test I used for testing (it is based on content from here). It tests scrolling performance on the main screen of the app.

final class ScrollingPerformanceTests: XCTestCase {
    let app = XCUIApplication()

    func testScrollAnimationPerformance() {
        app.launch()
        let collection = app.collectionViews.firstMatch
        let measureOptions = XCTMeasureOptions()
        measureOptions.invocationOptions = [.manuallyStop]

        measure(metrics: [XCTOSSignpostMetric.scrollingAndDecelerationMetric], options: measureOptions) {
            collection.swipeUp(velocity: .fast)
            stopMeasuring()
            collection.swipeDown(velocity: .fast)
        }
    }
}

Test runner performs 5 iterations of measure (there is also additional first iteration when measuring does not happen) and shows average values. The number of iterations can be changed via XCTMeasureOptions object. Also for the test scheme I disabled all diagnostic features and switched to release configuration as mentioned here.
It is time to start our testing, let’s launch it on a simulator.

Unfortunately the simulator supports only Duration metric. You have to use a real device to get all available metrics. Let’s try it with a real device.

There are five new metrics: frame count, frame rate, number of hitches, hitches total duration, hitches time ratio. While frame count and frame rate are more or less obvious, there are three metrics mentioning the term “hitch”. According to this: a hitch is any time a frame appears on screen later than expected. Thus hitch time ratio is the total hitch time in an interval divided by its duration.

Baseline setup

Now for our performance tests to work we need to set up baselines. To set a baseline you need to tap on the performance measurement status icon (gray icon with a dot) and tap on Set Baseline button. Also on that screen you can see performance of the test through different iterations:

After you set a baseline you will see a new folder with .baseline suffix inside of xcshareddata folder (it is inside of .xcodeproj package). Inside this folder you can see two files: Info.plist and file with generated name and .plist suffix. First file stores information about a device used to record baseline. That will be used to compare test results, you can’t use test results with baseline if baseline was recorded on another device model. For our case the device model is iPhone 14,2 (iPhone 13 Pro), device model for simulator is the device model of the current computer. Second file stores class names and baseline values.
During baseline recording I encountered with two issues:

1. In most of the cases the test fails because of exceeding maximum standard deviation during one of the iteration for hitch related metrics (Number of Hitches, Hitches Total Duration and Hitches Time Ratio).

2. Test fails sometimes because of exceeding max allowed deviation for average value (this also relates only to above mentioned metrics).

Those issues show that currently performance tools don’t work perfectly with hitches related metrics and we have to search for a workaround to make it work stably. As a workaround for the first case I increased maximum standard deviation to 400% to basically ignore this check and focus only on average values. For the second case I did 5 runs to see if the current baseline is stable and if the test fails I accepted test values as a baseline, after that test had become stable and it was green without any changes in code.

Testing

For the testing purposes I put following line right before cellForItemAt in collection view data source:

/* x is number of microseconds, for example usleep(10000)
blocks current thread for 10000 microseconds
or 0.01 second and usleep(1000000) blocks for 1 second */

usleep(x)

For test delays I tried to find turning values when test starts to fail. Here is the table of results:

This table uses following abbreviations: HTR — Hitch Time Ratio, HTD — Hitch Total Duration, NoH — Number of Hitches, FR — Frame Rate, FC — Frame Count.

From the table you see that tests start to fail on 25ms delay and all tries fail when there is 75 ms delay which I believe is a good result. Also as you can see from the table metrics related to hitches are more sensitive than others: frame rate and frame count failed only on 100ms when hitches are noticeable visually.

Conclusion

To sum up, tests are quite sensitive and easy to build. Unfortunately mandatory use of real devices imposes restrictions, to automate the process you need a device farm. One time fails because of max standard deviation without any code changes impairs efficiency. Also worth mentioning that baseline recording can be complicated if we talk about using CI.
In my opinion this tool is powerful and easy to use, it is great for controlling scrolling performance in your app. It also can be automated as a part of your release process along with your other unit and UI tests. But there are still a lot of things to improve, first of all in terms of stability and I hope to see such improvements in the next version of Xcode.

Useful links:
Eliminate animation hitches with XCTest
Ultimate application performance survival guide
Explore UI animation hitches and the render loop