Mobile handsets have evolved from the bulky luggable for voice only communications to the most recent ultra slim and ultra compact devices capable of MP3 playback, still and video image capture, high speed data transmission (rivaling DSL services) and GPS capabilities with television services already on the way. Integration of so many functions into a small form factor is driven by the evolving lifestyle of consumers in today's society. This presents design challenges for the system designer. Trends are similar in all other markets that have to integrate imaging and video applications. All consumer devices (digital still cameras, video cameras, MP3 devices) supporting video and imaging functions are getting smaller by the day and similar trends are happening in other non-consumer markets where video and image processing is required.
System designers need real-time video and image processing in a small form factor while consuming low power. This challenges IC designers to develop solutions that meet these extreme system design requirements, as each of these requirements tends to drive the solution in the opposite direction. For example, a high performance IC design will drive the cost and power to a high number. Solutions developed for these markets tend to be balanced towards one of the parameters more than the other.
System Block Diagram of a Camera Application
To determine the design challenges let's begin by looking at the system design requirements. The typical digital image/video capture system will have optics, image sensor (CCD or CMOS), analog front end, timing generation, "raw" image processing (taking raw image sensor with Bayer formatted color filters and processing all the way to YUV or YCbCr formats), and still or video compression engine (if required). CPU may be implemented at a system level to give the system designer some flexibility of system level software to get a better control of how the camera operates. In addition system level control functions are required via the support of PWMs, GPIOs, I2C and/or SPI interfaces to control other system level mechanics.
Nethra's NI-20x0 simplifies this system level design by integrating most of these system level functions into a SOC in an 8 mm x 8mm package. The device includes the following functionality on the same die:
ARM7TDMI 32-bit RISC CPU
___o Debugging via JTAG or debug UART
32KB of SRAM for program execution
64KB of embedded Flash for program storage
Support for up to 16MB SDRAM (16/32-bit operation)
PWMs, GPIOs, I2C and SPI interfaces
Programmable Timing Generator allowing the image processor to operate in a master or slave mode
Full color processing engine (capable of operating at 54 million pixels/second) for image sensors up to 3 mega-pixel resolution
8/10-bit CCIR 656 Interface (capable of operating at 108 million pixels/second in YUV 4:2:0 format)
This high level of integration in a small form factor has been implemented without sacrificing performance requirements.
View figure 1 full size
Figure 1. NI-20x0 detailed block diagram showing various functional blocks.
One of the biggest concerns of video/imaging system designers is the quality of algorithms they want to implement. This is one of the single biggest reasons for adopting DSP or FPGA-based architectures for camera designs. High volume image processing applications are driven toward mainstream DSPs from Texas Instruments, and for niche applications designers tend to use FPGAs in their designs.
Nethra's color processing algorithms are implemented using ASIC design methodology. All the memory required during the color processing is implemented as pipeline memory and the color processing algorithm hardware is tightly coupled to this memory. This design implementation permits the operation of the pixel processing at 54 million pixels per second.
The device family is capable of taking the raw image sensor data (ADC output from the CMOS image sensor or the Analog Front End in case of CCD image sensors); perform the entire image processing steps (listed below) and send CCIR-656 data out of the chip.
The products support following image processing algorithms:
Bad Pixel Correction
Lens Shading Correction
Fixed pattern noise correction
Adaptive Light Processing
___o Enhanced Low Light Processing
___o Electronic Image Stabilization
___o Red Eye Detection in Hardware
___o Auto Focus
___o Auto Exposure
___o Auto White Balance
The color processing algorithms have the potential to produce print-quality still images or crisp, vivid video depending on the system requirements. Histogram and statistics data are collected from image frames using multiple zones while the image data is flowing through the pipeline. This data is analyzed and used for implementing the camera system algorithms (AE, AF and AWB) to control sensors, motors, mechanics of the camera system, and image pipeline parameters.
Adaptive Light Processing
All imaging devices face the problem of having to handle image scenes that tend to have extreme lighting conditions at the same time (high intensity in one region of the image and low intensity in another region). Any auto exposure algorithm will try to adapt to one area or the other depending on the implementation (area vs. spot metered). NI-20x0 family of image processors supports compensation for such extreme lighting exposures in the same scene by implementing an adaptive lighting algorithm.
Adaptive light processing is implemented to support two modes of operation " backlit and normal lit scenes. Histogram and statistics engine are used to determine the profile of the image scene. This information determines the necessary adjustments to be made in the Luma and/or chroma so that the resulting image achieves a desired profile. The adaptive light logic then programs the optimum image adaptive adjustments to the hardware, which will perform the computation at very high speed. The following image shows the advantage of such an algorithm in capturing images in extreme lighting conditions:
View figure 2 full size
Figure 2. Adaptive lighting algorithm brings in details (top image) which may otherwise be hard to capture (bottom image).