H.264 "zero" latency video encoding and decoding for time-critical applications

Home networks and other applications
An emerging application with high sensitivity to latency is wireless video networking in the home. This application has recently gained a lot of interest from CE manufacturers and aims to eliminate the HDMI cable between the HDTV set and video source, such as a settop box, DVD player or game box. A similar compelling case exists for the computer industry, where the link between laptop and flat panel monitor is replaced by a wireless connection.

In these applications user interaction with the remote control, game pad, keyboard or mouse, must result in instant screen updates, otherwise the solution is rendered entirely useless. Since transmission at multi-gigabit per second rates over a highly unpredictable RF link is impractical, video compression is required. In these applications sub 10ms latency for the video codec is a critical requirement.


View full size
Figure 3: Implications of latency in wireless video networking applications

Another example with a high emphasis on the importance of low latency is digital video surveillance for mission critical applications. The challenge here is to match the inherently low latency of analog video surveillance systems, as their digital counter parts replace them. In case of securing valuables, such as money in a bank, priceless artifacts in a museum, or merchandise in a store, it is important that the area or building where the intrusion occurs is instantly secured.

In multiple-camera-tracking, video feeds from several cameras are stitched together chronologically into a single feed, which tracks one or more moving objects of interest. Too much latency in the video feeds makes stitching these together a complicated task and renders the application useless for rapid response action. In all these surveillance applications sub 10ms latency for the video codec is a critical requirement.


View full size
Figure 4: Implications of latency in video surveillance applications

Lastly, a less obvious example is in electronic newsgathering or ENG. In these applications cameras in the field capture live action and transmit the video for live broadcast to a nearby satellite uplink truck, where it is edited in real-time prior to up linking. Video feeds from multiple cameras and camera panning/zooming actions need to be interpreted in real-time by the production crew.


Figure 5: Implications of latency in ENG applications

Very low latency in the video feeds is necessary to provide inherent synchronization between all the different video feeds and with panning/zooming actions of the cameras. In this application sub 33ms latency for the video codec is highly desirable.

Next: Unexpected benefits of "zero" latency video codecs, quality