One of the things that boggle the mind is WebRTC performance testing. It stems from the fact that WebRTC has many moving parts AND that it incorporates different disciplines of computer science.
Let’s see what we need to know about WebRTC performance testing.
The moving parts in WebRTC
Here is a slide I like using to begin an explanation of WebRTC:
To build a real production-ready WebRTC application, you need to have multiple server types in place as well as cater to different types of clients.
On the server side, you need to have application servers, signaling servers, TURN and STUN servers as well as media servers. Each of these has its own unique characteristics that require different ways to handle its development, scaling, upgrading, etc.
The thing is, that each has its own effect on WebRTC performance in the final application. And this, in turn, makes testing WebRTC performance rather tricky.
WebRTC and development disciplines
One other important aspect to remember about WebRTC is that building a WebRTC application involves quite a few development disciplines.
A few years back, I asked a few people about WebRTC developers and got some interesting answers. One of the main themes in the answers was that there’s no such thing as a WebRTC developer, but rather different types of developers (and trust me – it is really hard to find a jack of all trades when it comes to WebRTC). Things to deal with here include frontend, backend, telecom, and mobile. The frontend and backend can be further split to generic and WebRTC specific, and in the same token, mobile can be split into iOS and Android as well as generic and WebRTC specific.
On the frontend, backend, and telecom you’ll have different means of planning, designing, thinking, developing, and testing for scale.
This leads us to the following conclusions:
- Different developers will look for different aspects of WebRTC when it comes to answering their scale questions
- Each server type will have its own scaling factors, implementation, and behavior
- Performance is going to mean different things to different people and activities
This leads us to what exactly is performance in WebRTC applications.
WebRTC metrics, statistics, and damn lies
The other day, we had a client call where the developer explained which metrics he uses to understand video quality. Since there are practically tens of different metrics to look at, picking one or two leading data points is important. The thing is, for me, it seemed like the decision on the metrics was rather esoteric in nature. It was one of the metrics you use when you want to drill down and debug an issue causing you a serious headache – not something I’d use as a generic parameter to determine video quality in general.
For me, WebRTC metrics related to performance and quality start with the basics:
- Bitrate – especially for video sessions
- Packet loss, jitter, and RTT – all give the immediate first impression of quality
- Machine CPU and memory – hugely important for client-side optimizations for user experience in group video calls (and other types of sessions as well)
Anything else related to the media itself, and you need to start asking yourself are you dealing with performance or with debugging and optimization?
That’s because almost every other metric you can think of would simply be a derivative of the baseline metrics in WebRTC – you’ll have more intra-frame requests if there’s high packet loss for example.
On the signaling side, you’ll be looking at other metrics as well, such as the speed it takes for certain actions to take place (the time it takes for a user to join a session and be fully connected for example).
The dynamic nature of WebRTC sessions
WebRTC is dynamic in nature. This makes running any kind of WebRTC performance test a challenge. With so many moving parts and unknowns, how do you even settle on an agreeable baseline?
The main 2 aspects affecting WebRTC media performance?
- Network performance and availability
- Device resources, especially CPU and to a slightly lesser extent available memory
What does it mean to conduct WebRTC performance testing anyways? It suggests that you are going to run a test to establish a baseline. And then you run again to check for performance – either improvements or setbacks from that baseline.
The intent here is to establish some kind of a framework where you take all the loose variables, make them static somehow, and then change only one or two main variables – like a configuration change or a new code piece that should improve performance. You then test to see if the changes made affected things for good or for worse.
To do it properly, you have to assume that only the things you want to change – change. The name of the game here is control. Or what I like to call it: predictable and repeatable testing.
Without this level of control and predictability, you will not be able to conduct WebRTC performance tests properly or efficiently.
What we need for a great WebRTC performance testing experience
When conducting WebRTC performance testing, here are the main things you need from your WebRTC testing framework. I’ve covered a few of them in my article about lessons learned in WebRTC stress testing.
- Predictable and repeatable is king: Make sure you can easily run the exact same scenario over and over again with predictable results. All aspects of the platform being tested (including CPU, memory, and network) need to be controlled by you so that you can focus on a given modification in configuration and code and see the performance changes due to it
- Collect everything: The WebRTC test framework you use should collect everything it can. You can never know what will come in handy – especially when you will need (and want) to move back and forth between performance to optimization to debugging and troubleshooting
- Visualize what’s important: You are running the same scenario over and over again. Being able to visualize the exact metric or value you are after quickly, and see how it changes dynamically across test runs and different changes to configuration and code is critical to improving your efficiency. Oh, and we have just the tool for that performance testing
Open source or commercial. Internal or third-party – whatever you decide to use for your WebRTc application, you will get to a point in time when WebRTC performance is going to be important to you. Be sure to plan ahead for the challenge. And if you are looking for a rock-solid solution, check out our testingRTC solution – it is geared towards WebRTC performance testing.
April 7, 2023
Phone System 