The following Mobile Operators and MVNOs were tested:
Run #1: Vodafone and Lebara Mobile
Run #2: EE and BT Mobile
Run #3: O2 and Sky Mobile
Run #4: Three and ID Mobile.
The following KPIs were measured and verified:
App Availability: percentage of viewing sessions for which the application successfully launched
Video Availability: percentage of viewing sessions for which the video both successfully started to play & successfully played for a minute
Video Initial Buffering Time: time between the Robot clicking on the play button and the video starting to play
Witbe VQ-MOS Score: a score representing the video quality of the first 60 seconds of video, measured with our famous Video Quality Mean Opinion Score algorithm.
All graphs and other visual representations of KPIs in this QoE Snapshot are generated with Witbe Datalab, our restitution interface for fault finding, root cause analysis and advanced analytics.
First of all, because each run was done at a different time of the day, it is important to state that they cannot be compared directly. We can only compare each pair of results, and this QoE Snapshot was designed mostly to analyse the performance of British mobile operators with their associated MVNOs.
The first conclusion that can be drawn, by looking at the Datalab dashboard is that the MVNOs compare quite nicely to their associated Operator.
Indeed, there were no significant difference in performance or quality between each Operator and its matching MVNOs: every time a degradation was reported on an MVNO, it was also observed on its associated Operator. Both Operators and MVNOs have a Video Initial Buffering Time lower than 5s and a reasonable Witbe VQ-MOS Score, averaging at 3.2 out of 5.
As already observed in our very first QoE Snapshot, customers appear to be benefiting from an overall satisfactory video viewing experience on mobile devices.
Performance during peak time
Run #4 (Three and ID Mobile) was performed at around 6pm, during peak hour, which resulted in a very congested network with lower median KPIs than previous runs. The chart above displays the Video Initial Buffering Time KPI for each run and shows a noticeable degradation on the 4th run.
Because the video on-the-go robot is used for drive tests and records its location as it makes its measurements, we are able to display the same KPI on a map of London. This allows us to identify the geographical areas were the performance was the most affected by peak hour usage (Finsbury Park and Crouch End).
We can also create a heatmap for video quality KPIs: the Witbe VQ-MOS score and the duration of Blurriness events which usually represent upscaling artefacts.
Witbe VQ-MOS is a ten-year R&D effort in psycho-acoustic and psycho-visual analysis of video streams. It works without referential, or previous knowledge of the video, and in real-time. It is based on three main artefacts: jerkiness, blurriness and blockiness.
Because all tests were using the same video content, it is another opportunity for us to play our favourite game of “spot the difference” and compare the video quality of the same frame on two different viewing sessions.
Example of low Witbe VQ-MOS score and high blurriness that lasted 54 seconds during peak hour.
Example of a satisfactory Witbe VQ-MOS score and blurriness level
Influence of network type & signal strength
When looking at the influence of network type & signal strength on Video Initial Buffering Time & Witbe VQ-MOS scores, an interesting thing we noticed is that a 4G connection with a poor signal (1 or 2 bars) outperformed on average a 3G connection with a good signal (3 or 4 bars).
By looking at the performance per signal strength for viewing sessions in 4G, it appears that signal strength itself did not really affect the quality of experience. Both Video Initial Buffering Time & Witbe VQ-MOS scores were not correlated to signal strength (i.e. number of bars).
This can be explained by the fact that nowadays, mobile video applications are built to deliver the best service possible even in poor connectivity conditions. This can explain why the quality of experience was roughly the same with a 1-bar signal strength and a 4-bar signal strength. This shows that using only QoS tools will not inform on the true QoE delivered to end users and is not enough for operators and MVNOs to provide the best possible service to their end-users.
The Blind Spot
It is interesting to note that the App availability measurements revealed a blind spot.
In a specific geographical area, the BBC iPlayer app couldn’t load properly, during peak or off-peak time, and with two different Operators. This blind spot would require a closer look to identify the cause of the problem. We could speculate that it is due to the presence of a park nearby, reminding us that environmental factors (trees, tunnels, etc) can also influence the quality of experience perceived by end users.
To conclude, it was interesting to measure no major difference between a mobile carrier and its associated MVNO, in terms of quality of experience.
This QoE Snapshot also shows, once again, a good overall quality of experience when watching videos on mobile devices in the UK, highlighting the continuous efforts and massive investments made by mobile operators to accommodate new user behaviors.
To talk to us about your QoE challenges and hear how we provide and support Witbe’s QoE Monitoring solutions to Mobile & Wireline Operators, Broadcasters and large Enterprises, please fill out the short form below and we’ll get straight back to you.