Manufacturers of signal processing technology are addressing this problem by providing development environments that integrate processors, development tools, software and systems expertise to enable designers to work at a high system level of abstraction in developing video applications. This allows designers to focus on the application functionality and implement video, audio, voice and imaging technology through simple calls to an application programming interface (API) that handles details such as the implementation of specific codec engines and matching screen resolution.

A new generation of video applications
As the cost of implementing digital video has continued to drop, more embedded developers are incorporating video and audio technology into a wide range of products. For example, Figure 1 illustrates how set-top boxes (STBs) are integrating home surveillance security systems to enable homeowners to view visitors on their TV screens. At the same time the quality bar is rising for both traditional and nontraditional video applications. Improved computational algorithms are being developed to intelligently adjust contrast and focus. Flexible processing options will enable users to capture images using a variety of settings, including black/white, color, infrared, heat and sensor. Intelligent cameras are being developed to only capture the image when everyone in the frame has their eyes open.

View full size

Figure 1: Example of how a set-top box can be integrated with home surveillance security systems to identify a visitor at the front door and post a notification on the television.

Traditional codec implementation methods
The increasing use of video and increasing sophistication of new video applications is focusing attention on the limitations of traditional video development methods. One of the greatest obstacles is the need for hand coding of codec algorithms that compress a digital stream for transmission or storage and decompress it for viewing and editing. Today's codec standards are very complex and they are continually being improved to enhance video quality and reduce bandwidth consumption.

Consequently, many man-months are typically required for engineers to implement them in embedded video applications. Generally codecs are initially developed in a higher-level language such as C/C++ and later parts of the code are rewritten in assembly to optimize their performance and power consumption based on the hardware implementation. This second stage of the programming effort requires considerable expertise in the specific hardware platform.

While earlier generation codecs such as MPEG2 have stabilized, newer higher-efficiency codecs such as MPEG4 and H.264 are still undergoing refinements so consideration must be given to the potential for changes during the development process and in future upgrades. Newer codecs also have larger toolsets that provide developers with more options in characterizing the video stream. For example, one codec stream may allow the use of interlaced profiles while another does not.

Hardware considerations
The codec usually runs on either an application specific integrated circuit (ASIC) or a digital signal processor (DSP). An ASIC is custom design for the application so it typically offers lower power consumption and a smaller footprint. The disadvantage of using an ASIC is that the nonrecurring engineering expenses are high and implementing changes, such as to accommodate evolving codec standards, can be expensive. The implementation of a new codec usually requires increasing the die size of the ASIC device.

The DSP provides full software programmability including field reprogramming capabilities, which enables a user to roll out an MPEG 2 product and later upgrade to H.264 video codec. The DSP also makes it easy to deliver a product that supports multiple codecs and enables updating of codecs in the field in response to changes in the standards. Additionally, the DSP may be incorporated into a system on chip (SoC) that provides the opportunity to reduce part count by combining it with additional functionality, such as a general purpose processor (GPP) and accelerators.

What about the application?
Digital video implementation is only one part of most of today's innovative applications. The application, which typically runs on a GPP, provides the user interface and specific features that differentiate the product from others in its category. One of the critical roles of the application is bringing the hardware and the software together, such as buffering video between the codec and the video ports. Additionally, the application requires an operating system, such as Linux or Windows CE, which must also handle Ethernet stacks, wireless stacks, input/output device drivers, etc.

Next: Hardware abstraction through software
Hardware abstraction through software
Developers of new video-enabled products need to get to market quickly with advanced features that set them apart from competitors. To do this, they need to devote as many of their resources as possible to the application itself, rather than on video implementation details.

Developers of video hardware are responding by providing an integrated environment consisting of hardware, software, tools and support that greatly simplifies the implementation of digital video technology. This environment reduces video system complexity by providing an open platform in which multiple companies have developed and integrated the components that make up a digital video system. These high-performance signal processing platforms now include production-ready software, including hardware drivers, hand-optimized codecs and application code for managing audio-video synchronization and stream over the network.

The APIs provided by the new architecture make it possible for developers to focus on developing their application without having to spend much time with video implementation issues and without having to program a DSP. By separating hardware implementation details from their use in applications, developers can port applications to new platforms and operating systems much more easily, typically by changing the drivers behind the APIs.

Developers writing the application can simply call on standard library functions to implement a wide range of video, imaging, speech and audio codecs as well as other video and audio functionality. As an example, Figure 2 shows how the developer can access a video stream with a function call such as ProcessVideo() without having to be concerned with the details of the actual codec implementation.

Figure 2: Illustrates how developers can simplify development by using the API framework to call on standard library functions to easily implement video, imaging and speech codecs.

The APIs themselves, which are provided by the hardware manufacturer, are already optimized for the hardware implementation and, if a switch to new hardware is made, a new driver can be substituted without changing the application code. The codecs are updated by the hardware manufacturer when a standard changes or a new standard is released so that application developers can just download a new software library rather than having to rewrite code. This approach can substantially reduce the cost and lead-time of video development.

Accessing video from any source
A key advantage of this approach is that the application can access a video stream from a variety of sources without having to make major changes to the application code. For example, the following code can be used to access video from an ATA peripheral such as a Hard Disk Drive, but this example could be modified to access video data from any source.

InitPeripherals()
InitEthernet();
InitUSB();
InitATA();
InitDisplay();
InitH264Decoder();
OpenCodec()
OpenH264Channel();
StartDecode(ATA.Address, H264.Channel, Display.Address);

Through the APIs, developers can configure the hardware based video engines to match the specific needs of a particular application with complex programming so as to match image size, bits-per-pixel, manage frame buffers and built-in on screen display capabilities, among many other functions.

Next: Changing display size
Changing display size
The example code below shows how easy it is to change the display size once the video stream has been opened and configured. This code makes it possible for a set-top box to use the same code to, for example, connect directly to a home theatre monitor and download content to a personal video player.

InitPeripherals()
// Display driver is passed arguments to move away from default values
InitDisplay(QCIF | CIF | D1 | 720p | 1080i);

The new system level abstraction approach simplifies the implementation of advanced video features. The code in Figure 3 shows how this approach can be used to implement time-shifting to store real-time video to disk rather than playing it immediately so that the user can pause playback without missing any of the program.

Figure 3: This code demonstrates how a system level abstraction approach can be used to manage set-top box playback by implementing time-shifting to store real-time video to disk, rather than playing it immediately, so as to improve the user experience.

I/O layer drivers
The complex details behind each of these features is handled by low level drivers that are accessed through common API and thus are completely transparent to developers. The use of familiar and widely-supported Linux drivers simplifies the integration of video technology in embedded applications. The Video for Linux Two (V4L2) API on the capture side and use of the common FBDev APIs, well known in the Linux community, make it easy to migrate from a PC development environment to an embedded SOC environment.

The computational resources of the hardware devices are implemented in an optimal fashion without any assembly programming. This includes complex operations such as optimized utilization of DSP resources and hardware based acceleration engines, use of enhanced direct memory access peripherals in chained mode for more efficient data transfers, and packet processing in Interrupt vs. Tasklet modes to flexibly meet different application requirements.

In summary, this design methodology enables developers to deliver full video functionality by accessing APIs without being concerned about the details of video processing. The low-level specific details of video processing can easily be modified to accommodate codec or hardware changes with no changes to the upper layer of application code. This approach eliminates the need to program the DSP and instead lets the development team focus nearly all their energy on developing the application itself. This approach makes it possible to deliver products with integrated leading-edge video capabilities in far less time and at a lower cost, while reducing the time and expense of future product maintenance and improvement.

About the author
John "J.B." Fowler is a member of Texas Instruments' (TI) system-on-chip (SoC) product marketing team and has been with the company since 2000. During his tenure at TI, J.B. has served as an applications engineer and gained extensive experience with TI's DaVinci technology and the TMS320C6000 families of products, with an emphasis in video security applications. J.B. graduated from Colorado School of Mines with a Bachelor of Science degree in Electrical Engineering and a Master's Degree in System Engineering. For more information, visit ti.com.