Intro to VoIP quality measurements

Voice over Internet Protocol (VoIP) has many advantages, chief of which its cost-effectiveness. However, to compete with traditional telecommunications technologies, the quality of speech over a VoIP connection must be comparable to or better than that of the Public Switched Telephone Network (PSTN). Several objective testing methods are available to measure, monitor and analyze the speech quality of VoIP services. These include Perceptual Speech Quality Measure (PSQM), Perceptual Analysis/Measurement System (PAMS), Perceptual Evaluation of Speech Quality (PESQ), ITU-T P.563 and the E-model. This article explains the technical principles of these methodologies, shows how they are applied, and compares their performance characteristics.

Introduction
VoIP refers to the real-time transmission of voice signals as packetized data across networks by using the Internet Protocol. The main advantage of VoIP is its low cost structure compared to traditional telephone services, especially for long distance calls. The IP network was originally designed for non-real-time data communications, offering a "best-effort" service with no Quality of Service (QoS) guarantee. Speech quality is a very important aspect of voice communication. On the one hand, service providers must continually assess the quality of the service they offer to maintain their competitiveness. On the other hand, subscribers are constantly comparing the quality and cost structure of various voice services.

Compared to the traditional PSTN, new impairments including packet loss, temporal clipping, delay, delay jitter and codec distortion are introduced in VoIP. The commonly used Signal-to-Noise Ratio (SNR) metric, which was developed for Linear Time-Invariant (LTI) systems, is inadequate with VoIP, mainly because the use of codecs and delay jitter buffers invalidates the LTI assumption.

Overview of Subjective Speech Quality Test Methods
Subjective testing is the most reliable approach to assess speech quality because it measures the way humans perceive speech. The most widely accepted test method is the Mean Opinion Score (MOS), which was defined in the International Telecommunications Union's specification, ITU-T Rec. P.800. MOS testing is conducted in elaborate settings. Speech samples are played to a group of listeners who rate the quality of the speech using an integer opinion score. The scores are averaged to produce a MOS value. To minimize the variability of the rating, as many as 60 listeners may be required in a particular MOS test.

Several rating methods can be used in subjective MOS tests. Absolute Category Rating (ACR) is the most commonly used. It is a five-point scale where listeners are told to judge the "absolute" quality of the speech without comparison to a reference, in Listening Quality (LQ) and Listening Effort (LE) scales, as summarized in Table 1 below. The MOS in ACR LQ are often simply referred to as MOS. The highest MOS score a call can achieve is about 4.5. A rating of 4.0 or higher is considered toll quality.

Although the subjective MOS test is the most reliable, it is time-consuming, expensive and unrepeatable. The design of the test is strongly influenced by both the human subjects taking part in the test and the elaborate test setting. In sum, the subjective MOS test cannot be automated or performed frequently.

(Click to enlarge)
Table 1. MOS absolute category ratings.