Introduction & setup

The environment for this QoE Snapshot consisted of a Witbe Video & Media on-the-go robot, connected to two Apple iPhone 8’s running the latest version of iOS.

Installed in a lab van or a regular car, the Video & Media on-the-go robot controls mobile devices, connected to the operator’s mobile network, in drive-test conditions. Instead of measuring the quality of the network by looking at QoS indicators, it is connected to real end-user test devices, and looks at true QoE KPIs. This allows operators to map out the true quality of experience of their mobile networks.

The specially chartered Witbe lab van drove four times around north London, on Friday April 5th, from 12pm to 8pm. Four runs were performed using two phones at the same time to compare a Mobile Operator with one of its associated MVNOs, by playing the same video in the BBC iPlayer app.

Witbe on-the-go robot set-up at the back of a lab van.

The following Mobile Operators and MVNOs were tested:

  • Run #1: Vodafone and Lebara Mobile

  • Run #2: EE and BT Mobile

  • Run #3: O2 and Sky Mobile

  • Run #4: Three and ID Mobile.

The following KPIs were measured and verified:

  • App Availability: percentage of viewing sessions for which the application successfully launched

  • Video Availability: percentage of viewing sessions for which the video both successfully started to play & successfully played for a minute

  • Video Initial Buffering Time: time between the Robot clicking on the play button and the video starting to play

  • Witbe VQ-MOS Score: a score representing the video quality of the first 60 seconds of video, measured with our famous Video Quality Mean Opinion Score algorithm.

All graphs and other visual representations of KPIs in this QoE Snapshot are generated with Witbe Datalab, our restitution interface for fault finding, root cause analysis and advanced analytics.

Global Overview

First of all, because each run was done at a different time of the day, it is important to state that they cannot be compared directly. We can only compare each pair of results, and this QoE Snapshot was designed mostly to analyse the performance of British mobile operators with their associated MVNOs.

The first conclusion that can be drawn, by looking at the Datalab dashboard is that the MVNOs compare quite nicely to their associated Operator.

Indeed, there were no significant difference in performance or quality between each Operator and its matching MVNOs: every time a degradation was reported on an MVNO, it was also observed on its associated Operator. Both Operators and MVNOs have a Video Initial Buffering Time lower than 5s and a reasonable Witbe VQ-MOS Score, averaging at 3.2 out of 5.

As already observed in our very first QoE Snapshot, customers appear to be benefiting from an overall satisfactory video viewing experience on mobile devices.

Performance during peak time

Run #4 (Three and ID Mobile) was performed at around 6pm, during peak hour, which resulted in a very congested network with lower median KPIs than previous runs. The chart above displays the Video Initial Buffering Time KPI for each run and shows a noticeable degradation on the 4th run.

Because the video on-the-go robot is used for drive tests and records its location as it makes its measurements, we are able to display the same KPI on a map of London. This allows us to identify the geographical areas were the performance was the most affected by peak hour usage (Finsbury Park and Crouch End).

We can also create a heatmap for video quality KPIs: the Witbe VQ-MOS score and the duration of Blurriness events which usually represent upscaling artefacts.

Witbe VQ-MOS is a ten-year R&D effort in psycho-acoustic and psycho-visual analysis of video streams. It works without referential, or previous knowledge of the video, and in real-time. It is based on three main artefacts: jerkiness, blurriness and blockiness.

Because all tests were using the same video content, it is another opportunity for us to play our favourite game of “spot the difference” and compare the video quality of the same frame on two different viewing sessions.

Example of low Witbe VQ-MOS score and high blurriness that lasted 54 seconds during peak hour.

Example of a satisfactory Witbe VQ-MOS score and blurriness level

Influence of network type & signal strength

When looking at the influence of network type & signal strength on Video Initial Buffering Time & Witbe VQ-MOS scores, an interesting thing we noticed is that a 4G connection with a poor signal (1 or 2 bars) outperformed on average a 3G connection with a good signal (3 or 4 bars).

By looking at the performance per signal strength for viewing sessions in 4G, it appears that signal strength itself did not really affect the quality of experience. Both Video Initial Buffering Time & Witbe VQ-MOS scores were not correlated to signal strength (i.e. number of bars).

This can be explained by the fact that nowadays, mobile video applications are built to deliver the best service possible even in poor connectivity conditions. This can explain why the quality of experience was roughly the same with a 1-bar signal strength and a 4-bar signal strength. This shows that using only QoS tools will not inform on the true QoE delivered to end users and is not enough for operators and MVNOs to provide the best possible service to their end-users.

The Blind Spot

It is interesting to note that the App availability measurements revealed a blind spot.

In a specific geographical area, the BBC iPlayer app couldn’t load properly, during peak or off-peak time, and with two different Operators. This blind spot would require a closer look to identify the cause of the problem. We could speculate that it is due to the presence of a park nearby, reminding us that environmental factors (trees, tunnels, etc) can also influence the quality of experience perceived by end users.

Conclusion

To conclude, it was interesting to measure no major difference between a mobile carrier and its associated MVNO, in terms of quality of experience.

This QoE Snapshot also shows, once again, a good overall quality of experience when watching videos on mobile devices in the UK, highlighting the continuous efforts and massive investments made by mobile operators to accommodate new user behaviors.

More information

To talk to us about your QoE challenges and hear how we provide and support Witbe’s QoE Monitoring solutions to Mobile & Wireline Operators, Broadcasters and large Enterprises, please fill out the short form below and we’ll get straight back to you.

  • The more information you provide, the better and more quickly we can respond.


  • This field is for validation purposes and should be left unchanged.