Posted: 9:00 p.m. EST, 4/13/98

As the market for digital still cameras climbs from curiosity toward viability, a fierce debate is flaming up inside the design community. By its nature, the digital camera requires a large amount of image processing. Microprocessor vendors, ASIC vendors and even DSP chip vendors all see this need as a potential gold mine — a huge market poised to buy their wares. And each camp thinks it has a lock on the price/performance sweet spot.

Vendors of 32-bit micr ocontroller cores are sure they have the perfect solution to digital-camera design. "We believe that with the TriCore processor, we can do the image preprocessing and compression for a megapixel camera at four frames per second," said Tapan Joshi, manager of Siemens' imaging peripherals group. "This will reduce the cost of electronics in the camera significantly."

But other chip vendors come up with entirely different answers. "In our first-generation design we used a general-purpose CPU and augmented it with a fairly simple data path," said Allen Rush, chief technology officer at Sierra Imaging Inc. "That approach is OK — you can do it that way. But you find yourself moving around 300,000 pixels per frame. By the time you've turned them into real pixels, you are talking about a megabyte. That gets very uninteresting for any RISC processor you could afford to put in a camera — to get both good speed and good image quality, you have to have some kind of fast coprocessor."

Disagreeme nts of this magnitude are common in the digital-camera world. And no wonder: The digital still camera proffers a bewildering maze of trade-offs, beginning at the lens and not ceasing until you reach the storage medium. Each step along the way presents the designer with the opportunity to save money and make some other steps harder, or to spend money and make something else easier — to trade speed, cost, battery life and image quality against one another in a massive, multivariable search for a perhaps imaginary global optimum. And this in a market that has yet to clearly state its preferences.

"There is no right or wrong way to do this," Rush emphasized. "You have a philosophy about how to get the best image quality out of the system. Different people have different philosophies."

Walking the pipeline
To examine just one decision in digital-camera design — the choice between an embedded CPU, embedded digital signal processor or ASIC — it's useful to follow the image all the way through the camera, from lens to memory card. Each step in the process will influence the architectural choices for the ICs.

(It is best to narrow the discussion by talking about just one category of digital cameras. Cameras now on the market range from toys to scientific instruments, with price tags running from a few hundred dollars to tens of thousands. They may be monochrome or color, self-contained or permanently tethered to a computer. Here we are looking specifically at self-contained consumer cameras, recording color images of approximately 640 x 480 to 1,024 x 1,024 pixels and costing less than $1,000 retail.)

Fundamentally, all digital cameras work the same way. A lens focuses an image on a sensor chip. The chip converts the image into an analog pixel stream, which is digitized, processed to improve apparent picture quality, compressed and stored in some sort of memory. Every step in this p rocess includes variables that affect the choice of processor electronics.

The first step with an important bearing on the question is, surprisingly, the lens. This would not be the case if Leitz or Nikon were building the lenses, perhaps. But reality is that the budget for a $500 camera doesn't buy a superb lens. Often, it doesn't even buy a glass one.

Reality for inexpensive digital cameras in the next year or two will be a multi-element plastic lens, often of fixed focal length. These lenses are subject to astigmatism, chromatic aberrations, curvature of field — all the problems that the film-camera industry solved in the 1940s. In addition, because the image sensors are so small, apertures tend to be small enough to diffraction-limit the resolution of the better lenses.

Some of these problems must be accepted. Others can be put to rest by paying for a better lens. Alternatively, some problems can be repaired with image-processing software in the digital domain — but at the cost of more hardware, longer processing delay after pushing the button or both.

The lens forms an image on the surface of an image-sensor IC. At this stage in the industry, that chip will usually be a charge-coupled device (CCD). CMOS image sensors are gradually coming into use, but they face a multitude of problems. The CMOS chips promise lower power, higher integration and, eventually, lower cost. But today, they offer limited availability, several types of noise not present in CCD sensors and no economies of scale.

Again, there are trade-offs. There are two basic types of CCD sensors — with and without in-line transfer. Each type uses a photodiode in each pixel to convert the photons that strike it into electrons in a potential bucket. With in-line transfer, it is possible to electronically empty all the buckets and then, after an interval, dump them all at once into storage bins. This permits the CCD driver to control the length of an "exposure" without actually opening and clos ing a mechanical shutter.

CCDs without in-line transfer essentially must have a mechanical shutter. But in-line transfer means including two charge-coupled buckets in each pixel — one connected to the photodiode and one next to it, into which the charge is dumped. This reduces the surface area available for the photodiode, and hence the sensor's sensitivity.

From the buckets, the charge is shifted, bucket-brigade fashion, to the edge of the CCD chip, where it is generally buffered and sent to an external analog-to-digital converter. "Most CCDs have an optimal shift rate that minimizes both dark-current noise and readout noise," Rush explained. "You have to match the sample rate of your A/D to that. Typically, it's around 14 MHz, so you are looking for a good 20-Msample A/D."

Converter resolution is also an issue. If the CCD is giving something like a 60-dB maximum signal-to-noise ratio, you want 10 real data bits out of the converter. "But if you buy a 10-bit converter and run it close to maximum speed, you are going to give up at least half a bit," Rush said. "If you buy a fast 12-bit converter, you are paying for overkill, and in this price range the difference can show up real fast."

Another problem the converter and the downstream electronics must face is that the data from a CCD is inherently bursty. You get a line of pixels at maybe 14 MHz. Then you sit around a while, and then you get another line. In this cost category, putting in the necessary analog electronics to smooth out the data flow would be out of the question.

In theory, much of this complexity could be avoided by using a CMOS sensor and integrating amplifiers, an A/D converter and timing logic onto the chip. Such devices are under development and are occasionally discussed at conferences, but they are not yet a major factor in the industry.

In addition, CMOS sensors have, among other problems, pattern noise — noise caused by the CMOS circuit's inability to move charge without interfere nce from neighboring circuits. Such noise shows up as a permanent pattern "burned in" to the sensor. It can be removed by post-processing in the digital domain, at increased cost or reduced speed. Or it can be removed by circuitry integrated onto the sensor chip — again, at the cost of increased complexity.

From the converter, the data typically flows into a frame buffer, where as often as not it remains organized in lines, just as it was on the sensor. From now on the game is digital. But it is far from simple.

First, you have to get the right exposure and focus — most of these cameras are auto-everything. If you are trying to save processing cycles, this can be done the way Kodak has always done it in box Brownies: a switch to pick scenery or portrait, and a switch for sunny or cloudy conditions or to turn on the flash.

A more sophisticated approach, however, will use the image-processing hardware to scan a trial exposure. The system can build a histogram and estimate the correct exposure, even employing some educated guesses about the subject matter to distinguish, for instance, between a photo of a dark object and an accidentally back-lit photo of a fair-skinned person. In addition, relatively known techniques can guess the area of interest in the image and estimate the correct focus. This information can be used to make a for-real exposure based on the trial data — but you only have a few tens of milliseconds to make the trial exposure and process it.

The next problem the digital circuitry must handle is called "de-mosaicing." This goes back to a little, shall we say, simplification the image-sensor folks made. In real life, there aren't 480 lines in the image sensor, each containing 640 color pixels. Photodiodes are essentially color-blind. To get a color image, the sensor designer must lay a color filter mask over the chip, putting red filters over one-third of the photodiodes, green filters over another third and blue filters over the rest. (Actually, some sensors use subtractive, rather than additive, filters, such as green, cyan and magenta, with a fourth color to get better luminance data. But they are more common in camcorders.)

So instead of 300,000 three-color pixels, you have about 100,000 each of red, green and blue pixels. The first image-processing task is to put these 10-bit or 12-bit monochrome pixels back together so that the image kind of looks as if it came from 300,000 24-bit, three-color pixels.

Clearly there's not enough information. So the designer must interpolate, figuring out from two or three neighboring green pixels what the green content of the pixel in the middle would have been, if it hadn't had a red filter over it. How you go about doing this has a major impact on the computational load of your system.

"A lot of work has gone into de-mosaicing algorithms, and we at least have several patents in the area," Rush said. "You can do a simple bilinear interpolation to guess at what the missing colors should have b een. But the best you are going to do that way is a quarter of the Nyquist resolution of the original sample. That's not good enough for most camera designers, so there are lots of more clever approaches."

Whatever approach you choose, you now have a 640 x 480 (or whatever) array of 24-bit pixels. They are estimated pixels, mind you, but at least they are all there. At this point, the real crunching starts. Gamma corrections have to be applied, matching the highly linear transfer function of the photodiodes to the wildly logarithmic response of the human eye. Known errors in the optics, the sensor, the color filters, the A/D converter and so forth have to be reversed with signal-processing techniques.

"The idea is to make the pixels in the buffer more closely correspond to what we think was in the original image," Rush said. "A good job will take about 35 to 50 percent of the total processing time."

The amount of work to be done in this step is of course dependent on how many problems have been passed on by previous decisions. For example, this is where the buck stops for cheap optics, poor color filters on the CCD or marginal A/Ds. Money saved up front turns into memory and processing cycles here.

Finally, the processed image has to be compressed — usually in some variant of JPEG — so that it will fit into the available miniature memory cards. "The JPEG compression is fairly simple," Rush said. "Generally, people think a compression ratio of 10 or 20 to one will still give a just-perceptible loss of information."

Implementation views
And now the arguments may commence. Depending on the price point, the image quality desired and the all-important "button-to-button" delay, different chip vendors can take wildly different views on the processing task.

Sierra, for example, believes that a very substantial amount of processing is essential to adequate image quality in what it feels is a tolerable 1.5-second delay. Accordingly, the current generati on of Sierra electronics does most of the processing in a dedicated set of data paths. A 32-bit RISC core is included, but primarily for control purposes.

"We have a debate in the company about whether we have moved so much of the work into the data paths that we could get by with a 16-bit core," Rush said. "We probably could, but 32-bit RISC cores have grown so small that it's just as easy to use one of them. And some of our code is much happier with a 32-bit compiler."

Siemens, not unexpectedly, is at the other end of the debate. "A quantitative answer about how much processing is required is very dependent on the kinds of algorithms you choose," said Xiaoping Li, systems-design engineer with Siemens. "If you put a lot of work into filtering, noise reduction and white balance, the preprocessing task can easily get larger than the compression task. For instance, inexpensive fixed lenses, such as you find on most under-$500 cameras, require more preprocessing. So do today's CMOS image sensors, compared to CCDs."

But for some portion of the market, it makes sense to slash away at the tasks until a single CPU can handle them all. "We think that a software-only approach with a TriCore chip can put a $299 camera on the street late this year," said Siemens' Joshi.

Ironically, one of the problems in today's designs may turn into a partial solution tomorrow. "As they learn to do integration, a lot of preprocessing can be moved into the CMOS image sensor," Li said. "It is possible to do white balance, noise reduction, even autofocus on the sensor chip."

Sierra and Siemens stand at the two poles of the implementation debate. But there is a lot of ground in between. For instance, it is possible to augment a software-only approach with a limited amount of external hardware. Li suggested that if you could put one external hardware core into the system with the TriCore chip, it would probably be something to do 2-D convolution. "Tasks like noise reduction, image sharpening, maybe deblur ring — these things take up a lot of cycles," he said. "It might be possible to handle them externally."

That sort of observation perks up the ears of another semiconductor-industry segment hungry for growth: the DSP business. Texas Instruments Inc., for instance, has noticed that the processing load in digital cameras looks an awful lot like digital signal processing — just two-dimensional instead of unidimensional. This has led TI to suggest that its new 320C2700 DSP/microcontroller combination might be ideal for the purpose. Claiming C2XX-level signal-processing capabilities, reasonable general-purpose speed and better code density than a Motorola M-Core chip, TI would like to see the combo device used as an alternative to DSP-bearing embedded CPUs like TriCore and full-up ASICs.

Vendors of DSP ASIC cores can't be far behind. Already, the combination of a stock DSP core and a stock CPU core has established itself in a number of markets, most notably GSM phone handsets. As the ca mera market totters toward take-off, it will present an irresistible target.

So, in a not unfamiliar spectacle, we will see ASIC specialists, CPU vendors, DSP vendors and ASIC library vendors all converging on what might turn out to be a major market opportunity. As the players themselves say, there is no one right answer. There is philosophy, art, self-interest and, no doubt, prejudice. Makes for good engineering.

  Go to This Week's News

	eeProductCenter Launches SpecSearch®, New Parametric Parts Search Engine In our continuing effort to enhance our site, eeProductCenter introduces SpecSearch® powered by GlobalSpec. Click here.
Free Subscription to EE Times First Name Last Name Company Name Title Email address   Click here for your Free Subscription to EETimes Europe