From virtual classrooms to business meetings, shopping to dating apps, video is quickly becoming the de-facto communication mode online.
Innovative developers and product thinkers are looking to create engaging live experiences in their applications. So naturally, it's critical that the audio-video SDK they build these experiences on top of provides a stable, extensible, and scalable bedrock.
Among the many factors to consider before purchasing an audio/video SDK, network reliability stands out. After all, nobody enjoys running a twenty-minute monolog on a video call only to realize your network was down the entire time…
Testing Network Reliability for Real-World Scenarios
In this article, we've downloaded, deployed, and tested the reliability of the 100ms React SDK. To do so, we designed a series of tests that simulate common scenarios in real-life. Of course, since that's not fun enough, we decided to unleash our “full crazy” by battle testing each round against extreme conditions.
The tests verify how the 100ms SDK fares across three parameters that define network reliability: low bandwidth, network blips & network switching.
Why Network Reliability Matters
In the real world, individuals often have to deal with unstable or less-than-ideal network conditions. This happens when:
moving from one network area to another while traveling
suddenly experiencing slow internet because of an expiring data pack
suddenly experiencing call disconnection for a few seconds due to issues in the larger infrastructure
Network connectivity issues occur more often than we think. Video SDKs need to, at best, be resilient to these issues, and, at worst, provide developers with tools to deal with them gracefully.
Deployment
100ms has a sample React app (100ms 2.0 Sample React App) meant to facilitate the testing of its SDK. We deployed it on Heroku and exposed it to a few commonly occurring end-user scenarios.
https://github.com/100mslive/100ms-web
We had to generate some credentials from the 100ms console and then deployed this example React app on Heroku.
The SDK was deployed and tested on the Chrome browser running on macOS Monterey.
Conditions and cases
All these tests were 1:1 calls, performed with 2 people in the room. A few details about each test before we get into the results:
- Low Bandwidth Test
Network speed varies across devices. For instance, users operating on 4G mobile data often experience a volatile network, as it tends to vary in speed and stability. In this test, we checked how 100ms handles calls with varying connection speeds on low bandwidth.
- Network Blip Test
Network crises can happen in the middle of a call. In this test, we checked how 100ms handles the sudden loss of network connectivity followed by automatic reconnection.
- Network Switching Test
It is common for users to switch between networks inadvertently. For example, they might be on a call while moving between state lines or from a city to the countryside, which may affect network strength.
Network switching usually occurs when you move away from the range of one network to another or when you switch between your available networks for a higher speed. In this test, we checked how 100ms handles a network switch.
1. Low Bandwidth Handling/Management Test
Audio/Video applications need to handle usage across varying network bandwidths. In this section, we monitor how 100ms handles calls for users with low bandwidth.
Testing Methodology for the Low Bandwidth Test
We used Network Link Conditioner to emulate different network conditions. We set the ideal resolution to 640x360, and tested the app on 4 different configurations: 300 Kbps, 500 Kbps, 800 Kbps, and 1 Mbps, switching from one to another in the middle of a call.
Test Results
The 100ms SDK handles the drop in bandwidth by prioritizing audio/video upload for other peers instead of audio/video download.
If the network is adequate (800 Kbps), the video of active or recent speakers continues to be visible. The audio remains perfectly functional.
If the network is poor, only peer audio is functional while their video degrades.
On the device facing poor network conditions, the video is somewhat degraded but not entirely non-functional. At lower bandwidths (500Kbps and 300Kbps), audio quality remains functional for all other peers in the meeting and only sees a drop for the attendee experiencing bandwidth constraints.
2. Network Blip Test
In this section, we check how 100ms handles call connectivity when a user’s network connection gets switched off, or drops for several seconds.
Testing Methodology for the Network Blip Test
First, we check the call by switching off the internet connection for 10 seconds. This is done by toggling the connected wifi network from the menu bar and connecting back by re-toggling the same.
Then, we iteratively repeat the same test for 20, 30, 45, and 60 seconds. While doing so, we observe the state of the call connection and how the app behaves during disconnection.
Test Results
The 100ms SDK reconnects every time when internet is disabled for 10, 20 and 30 seconds. When switched off for 45 and 60 seconds, the app tries to reconnect for 35s before disconnecting entirely.
3. Network Switching Test
Apps are often exposed to different network conditions in the real world. In this case, we’ve tested how the 100ms SDK reacts when the app moves from one network strength to another.
Testing Methodology for the Network Switching Test
This test checks how 100ms handles connection when switching from one network to another. We tested the app in 3 Wi-Fi networks:
2.5G and 5G from the same router, and a mobile hotspot.
To start the call, we connected to the Wifi 2.5G network.
Then, we switched from Wifi 2.5G to Wifi 5G.
Then, we switched back to Wifi 2.5G.
Then, we repeated the same process, switching to and from Wifi 2.5G and the Mobile Hotspot.
We waited for the call to reconnect during every network switch and monitored the time (in seconds) it took for the reconnection to occur.
Some of the flawed behavior in the ‘Wifi 2.5G to Hotspot’ test section might be due to the unstable 4G network connection we experienced while testing.
Test Results
The 100ms SDK manages to reconnect after every network switch. Sometimes the video reconnects after the audio. The average reconnection time when switching within the same network is 9.1s for audio and 10s for video. The time for reconnection between 2 different networks is 19.2s for audio and 13.8s for video.
Closing Notes
Given the centrality of reliability when it comes to choosing an audio-video SDK, we decided to lay all our cards on the table and reveal exactly how we fare in diverse network, bandwidth and end-user circumstances. Across all tests 100ms fared well under regular usage conditions. In some cases, like bandwidth drops, the SDK allows for graceful handling of degradation issues.
Of course, as an SDK provider, we pride ourselves for making 100ms even more bullet-proof, so we can’t wait to elegantly solve across all these conditions and meet you again with even more aggressive scenarios.
Top comments (0)