Customer Experience: If You Can't Measure It, You Can't Improve It

<![CDATAA few weeks back, I met an executive from a European Tier1 operator to talk about our QoE solution. His opening line was “We are a mobile operator; our customers know that we are a mobile operator and therefore expect a s**t service”. I am not sure whether he meant that or whether this was his way of politely saying “I am not interested in hearing about your product”, but it made me think, do operators really care about customer experience?
The answer is probably no: it is not that operators don’t care, operators do care about their customers, but it is often the case that they care about the wrong things. An operator might give their customers unlimited minutes and unlimited SMS messages and huge download speeds and amazing 4G coverage in every square inch of the country because that’s what the customer wants, right?
Not really! To most customers, a perfect service is a service that works smoothly and reliably every time. Building faster, bigger pipes in the core network and using complex modulation techniques to squeeze more bites into every Hz will go a long a way towards achieving that perfect service, but it will not guarantee it. Why? Because there are 7 layers in the OSI model and the service will be as good as the weakest layer, i.e. the one where most problems occur.
As an operator, detecting what the problems are is relatively easy – just listen in on your call center or look at your Twitter feed and/or Facebook page. However, figuring out the cause of said problems is surprisingly more difficult, even if they end up being surprisingly simple.
Let’s look at an example. An operator we worked with was getting a large number of calls from their customers complaining about slow loading web pages. When customers were asked to perform speed tests from their devices, the results were always adequate, so the conclusion was that it must be something to do with the device, and the usual blanket advice “update your browser and shut down all the applications you are not using” was dished out. Surprisingly enough, this never solved the problem. It was not until someone looked at the DNS resolution times and noticed that they were 10’s of seconds, rather than milliseconds, simply because the requests were routed through a convoluted and congested path.
Sometimes a spot measurement is not enough; repeated and frequent measurements are needed over a long period of time to figure out that something is amiss. A customer who installed our probes realized after two weeks that the quality of the video stream mysteriously degraded during the day and that the degradation of the service did not coincide with peaks in network congestion. The mystery was resolved when it was discovered that the issue was not due to the number of data packets getting around the network, but because the local content cache simply could not cope with the demand. To give an analogy it is like being late, not because the traffic was bad, but because there were not enough taxis.
As a former academic researcher and a system optimization specialist, my guiding mantra has always been “if you can’t measure it, you can’t improve it”. In the case of Quality of Experience, before I can improve it, I need to measure it. To measure QoE, I need to know the what and where to measure.
Customers experience the service through what Layer 7 displays on their screens. Consequently, with QoE measurement, you should start at Layer 7 not only because that is where your customer experience starts, but also because Layer 7 is where flaws in the lower layers will be laid bare. This is not to say that you should only focus on Layer 7 because the underlying layers are just as important. Similarly, relying solely on the lower layers of the OSI stack to judge the quality of experience at the higher layers is not the answer either. After all, we judge a painting by looking at the canvas, not at the artist’s paint brushes.
On a final note, data networks are complex beasts with an increasing amount of self-healing, self-optimising capabilities, and a fair amount of intelligence and autonomy. While this is great from a network management point of view, the hidden and out of reach complexity also makes ideal ground for the infamous Butterfly Effect, where a seemingly insignificant change in a hidden corner could have far-reaching repercussions further upstream. Years working in Revenue Assurance taught me a new mantra, “if you detect it early enough, you can fix the issue before it becomes a problem”.
Some battles are won before they start! Learn more about how to measure quality of service]>