Early digital video compression solutions primarily focused on applications that do not require real-time interaction, such as in TV broadcast, video-on-demand and DVD playback. In these applications the latency between the source and the decoded video is not important and can easily extend to several seconds. However, in applications where there is a closed feedback loop, such as video conference and videophone, latency is the most crucial aspect of the system, as it determines whether the system will be stable or not. Keeping the latency of a video codec in such systems as minimal as possible is the proper approach. In many such applications latency measured in sub 10 milliseconds is crucial and it takes a radically different approach from traditional ones to achieve a low latency implementation of the popular H.264/MPEG-4 AVC (Part 10) video coding standard.
Latency and zero latency defined
Simply put, video codec latency is defined here as the time lapse between the first pixel of video appearing in the source and the first pixel of decoded video appearing at the destination. Latency-sensitive video applications require that the time lapse between source and decoded video is extremely small. How small depends on the application, but as a guideline, keeping latency down to sub 10ms is a good idea. For convenience we will call such low latency "zero" latency. This is in contrast with the orders of magnitude higher latency found in non latency-sensitive applications.
Figure 1: Latency between source and decoded video
Latency sensitive video codec applications
In video conferencing and video telephony, noticeable delay makes a conversation impossible, unless a "walky-talky" like protocol is strictly followed. This makes the conversation unnatural and cumbersome. In these applications sub 33ms latency for the video codec is required.
Figure 2: Implications of latency in video conferencing
Next: Home networks, other latency sensitive video codec applications, unexpected benefits