Video codecs tutorial: Trade-offs with H.264, VC-1 and other advanced codecs

Video compression is an essential enabler for all these exciting new video products. Compression-decompression (codec) algorithms make it possible to store and transmit digital video. Typically, codecs are either industry standards such as MPEG-2, MPEG-4, H.264/AVC and AVS or proprietary algorithms, such as On2, Real Video, Nancy and Windows Media Video (WMV). WMV is an exception as it was originally a Microsoft proprietary algorithm that is now also standardized by SMPTE as VC-1. Codec technology has continuously improved in the last decade. The most recent codecs, H.264/AVC and VC-1, represent the third generation of video compression technology. Both codecs are capable of squeezing very high compression ratios utilizing the available processing horsepower in low-cost ICs such as programmable DSPs and fixed-function ASICs. However choosing the right codec and optimizing its real-time implementation for a specific application remains a tough challenge. The optimal design must trade-off between compression efficiency and the use of available computational horse-power. Obtaining the optimal compression efficiency with limited computational horse-power is a tough science.

In this paper, we first provide an overview of key concepts in video coding and describe the legacy compression standards. Next, we focus on the capabilities of the latest generation of codecs including H.264/AVC, WMV9/VC-1 and AVS and provide insights into compression and complexity trade-offs that each offers. Finally, we discuss real-time implementations and key trends in end-equipment segments that may influence choices between the popular video codecs.

The Video Compression Challenge
A major challenge for digital video is that raw or uncompressed video requires lots of data to be stored or transmitted. For example, standard definition NTSC video is typically digitized at 720x480 using 4:2:2 YCrCb at 30 frames per second, which requires a data rate of over 165 Mbps. Storing one 90-minute video requires over 110 GBytes or more than 25x the storage capability of a standard DVD-R. Even lower resolution video such as CIF (352x288 4:2:0 at 30 frames/second), which is often used in video streaming applications, requires over 36.5 Mbits/s. This is many times more than what can be sustained on broadband networks such as ADSL or 3G wireless. Today, broadband networks offer between 1-10 Mbps of sustained throughput. Clearly, compression is needed to store or transmit digital video.

The main goal for video compression is to encode digital video using as few bits as possible while maintaining visual quality. Codecs are based on the mathematical principles of information theory. However, building practical codec implementations requires making delicate trade-offs that approach being an art form.

Compression Tradeoffs
There are many factors to consider when selecting the codec in a digital video system. The most important ones are the visual quality requirements for the application, the environment (speed, latency and error characteristics) of the transmission channel or storage media and the format of the source content. Also highly important are the desired resolution, target bitrate, color depth, the number of frames per second and whether the content and/or display are progressive or interlaced.

Compression often involves trade-offs between the visual quality requirements and other needs of the application. Firstly, is the application storage, uni-cast, multi-cast, two-way, or broadcast? For storage applications, how much storage capacity is available and what is the recording duration? For non-storage applications, what is the maximum bit rate? For two-way video communication, what is the latency tolerance or allowable end-to-end system delay? If not two-way, is the content that must be encoded available in advance off-line or does it require to be encoded in real-time? How error-prone is the network or storage media? The various compression standards handle these trade-offs differently depending on the primary target application.

Another trade-off is the cost of real-time implementation of the encoding and decoding. Typically newer algorithms such as H.264/AVC or WMV9/VC-1 that achieve higher compression require increased processing, which can impact the cost for encoding and decoding devices, system power dissipation and system memory.

Standards Bodies
There have been two primary standards organizations driving the definition of video codecs. The International Telecommunications Union (ITU) is focused on telecommunication applications and has created the H.26x standards for low bitrate video telephony. These include H.261, H.262, H.263 and H.264. The International Standards Organization (ISO) is more focused on consumer applications and has defined the MPEG standards for compressing moving pictures. MPEG standards include MPEG-1, MPEG-2 and MPEG-4. Figure 1 illustrates the history of video codecs standardization.

MPEG and ISO often make slightly different tradeoffs based on its primary target applications. On occasions, the groups have worked together such as in the Joint Video Team (JVT) to define the H.264 codec also known as MPEG-4 Part 10 or MPEG-4 Advanced Video Coding (AVC) in the MPEG family. In this paper, we refer to this joint standard as H.264/AVC. Similarly H.262 and MPEG-2 are identical, while H.263 Baseline Profile technology has a significant overlap in techniques with the MPEG-4 Part 2 Simple Profile codec.

Standards have been critical for the widespread adoption of codec technology. Consumers find products based on standards affordable because of economies of scale. The industry is willing to invest on standards given their assurance of interoperability between vendors. Content providers are attracted to standards given the long life and broad demand their content would see. While almost all video standards are targeted for a few specific applications, they are often used to advantage in other applications when they are well suited.

Figure 1: Chronological progression of ITU and MPEG standard [10]

ITU and MPEG continue to evolve compression techniques and define new standards for better compression and newer market opportunities. China has recently defined a national video coding standard called AVS, which we also describe later in this paper. Standards currently in the works include ITU/MPEG Joint Scalable Video Coding, an amendment to H.264/ AVC, and MPEG Multi-view Video Coding. Meanwhile, existing standards are continually evolving to satisfy newer applications. For example, H.264 has recently defined a new mode called Fidelity Range Extension to address upcoming markets such as professional digital editing, HD-DVD and lossless coding.

In addition to industry standards from the ITU and ISO, several popular proprietary solutions have emerged particularly for Internet streaming media applications. These include Real Networks Real Video (RV10), Microsoft Windows Media Video 9 (WMV9) Series, ON2 VP6, and Nancy. Due to the installed base of content in these formats, proprietary codecs can become de facto standards. In September 2003, Microsoft proposed to the Society for Motion Picture and Television Engineers (SMPTE) that the WMV9 bitstream and syntax be standardized under the aegis of that organization. This proposal was accepted and WMV9 is now standardized in SMPTE as VC-1.

Next: Video Coding Principles