If this happens to you, beware. It can certainly be done. In fact, very effective capture of moving or still images can be achieved at relatively low cost. But it is not simple, and it requires some understanding of the protocols and standards employed by the imaging industry.

This article outlines the key issues from the point of view of a design engineer who has not previously worked with video.

To illustrate the issues, this article will use as its example an imaginary project to add moving video-capture capability to a simple PIR intruder detector. In a conventional burglar alarm system, the sensor detects the presence of an unauthorised person, and triggers an audible alarm, and often also transmits an alert to the police or a security company.

The design example illustrated in this article shows how, in addition, the sensor device would be able to capture images of the burglar for later use by the police and courts of law.

The camera will need particular attributes in such a system:

- Ultra-low cost " since there are multiple sensors and multiple cameras in a building, a low unit cost for the camera is essential if the overall system cost is to be kept to a reasonable level;

- Small size " the devices must not be visually intrusive, particularly for residential use;

- Adequate image quality " sufficient to enable identification of an intruder in a wide variety of lighting conditions, to a standard required by the law. The system must offer high dynamic range in order to work as well in bright sunlight as in near-darkness.

These attributes dictate the use of a low-cost image sensor. While the latest consumer cameras boast image sensors with more than 10Mpixels, this design can use a device with as few as 1Mpixels. Such devices are available at very low cost, thanks to the proliferation of image sensors in mobile phones and vehicles.

As the article will show, however, this presents a number of challenges to the system designer, particularly in:

- Interfacing the sensor's output to a host processor or microcontroller;

- Capturing a suitable image under lighting conditions that are far from ideal. How an image sensor works
On the surface of a fingernail-sized silicon image sensor are millions of photosensitive elements " often diodes " each of which captures a single pixel. The term used to describe the circuitry and other elements that generate a pixel is photosite.

Image sensors come in two types, monochrome and colour. The description below covers both types, but the main difference is that the interface for a monochrome sensor is much simpler as the data per pixel relates only to light intensity.

When a camera's shutter is operated and the exposure begins, each photosite is uncovered to collect and store photons in a cavity. Once the exposure finishes, the camera closes each photosite, and assesses how many photons fell into each. The relative quantity of photons in each cavity translates into the level of light intensity; the precision of this information (the number of levels of light intensity) is determined by the bit depth (eg 255 levels for an 8-bit image) and the signal-to-noise ratio.

Each cavity is unable to distinguish how much of each colour has fallen into it, so the example in figure 1 would only be able to create greyscale images.

Fig. 1: How an image sensor captures incoming light.

To capture colour images, each cavity has to have a filter placed over it that only allows penetration of a particular colour. The most common type of colour filter array is called a 'Bayer Array', shown in figure 2.

Fig.2: Bayer Array, the most common filter type used in colour image sensors.

A Bayer array consists of alternating rows of red-green and green-blue filters. Notice how the Bayer array contains twice as many green as red or blue sensors, because the human eye is more sensitive to green light than to red or blue light.

The information can then be presented to the outside world in, for example, RGB format using a standard camera interface.

The raw data from an image sensor, then, simply provides information about the light intensity at each pixel.

Here, however, we encounter the first complexity for the engineer who is not familiar with video: not all sensors use a Bayer Array, although it is by far the most common method. At the same time, there are many raw pixel data standards, so it will be necessary to get the specific data format from each camera sensor manufacturer.

In consequence, changing the image sensor at either the prototype stage or in the final product could require a substantial amount of reconfiguration to handle a different data output. So it will be important to try to ensure that the image sensor chosen is going to be available for the expected duration of the product's life cycle. Low cost image sensors
Many low-cost image sensors are on the market, each with its own combination of attributes and advantages. For the sake of our intruder-detection system, a suitable example is the new MLX75307 camera-on-a-chip from Melexis, which is priced to be attractive to the famously cost-conscious automotive market.

Featuring a snapshot shutter with high dynamic range (up to 127dB in a single frame), this device enables the capture of fast-moving scenes (up to 150 frames/s) in the presence of bright light sources or dark shadows. The device offers panoramic VGA resolution at 750x400 pixels (PVGA), in black-and-white or colour.

In order to accommodate extremes in contrast, it is possible to select a piecewise linear-response function with compression of high light levels (known as 'multiple slopes' response). Figure 3 demonstrates the differences between the different slope responses. Originally developed for the automotive industry, this feature means that this sensor will be particularly good for identifying people when photographed in low-light conditions.

Fig. 3: How the MLX75307 handles extremes of contrast with piecewise linear response. The top picture the image shows a normal linear response curve. The second picture shows the effect of the MLX75307's piecewise linear response. Handling data inputs from low-cost image sensors
Now we come up against the second major challenge for the design engineer: this image sensor provides a Raw Pixel Data output, which means that a microcontroller, ASIC or DSP will be needed to convert the raw information into a format suitable for compression, storage and transmission.

More expensive image sensors incorporate a DSP to do this processing work, but they will often also provide other features (such as more pixels) that this system does not require. Therefore, to meet the design objective of low cost, a low-cost image sensor such as the MLX75307 should be used.

On a colour image sensor, the following image processing steps will be required:

1. filter out defective pixels 2. demosaicing of the Bayer pattern

3. calculate R, G and B by correcting and white balancing

4. calculate Y (light intensity) according to the target standard

5. calculate U and V out of R, B and Y

6. optional de-noising

7. pack the data for transfer

8. resample the frames to the required frame rate

When the image sensor is read out, line by line, the pixel sequence comes out RGRGRG....RG etc., and then the alternate line sequence is GBGBGB....GB etc. This output is called sequential RGB (or sRGB). This is the format the MLX75307 has adopted for its Bayer output video stream.

(It is worth noting here that, since each pixel has been made sensitive by the Bayer Array to only one colour, the overall sensitivity of a colour image sensor is lower than that of a monochrome sensor. As a result, monochrome sensors are often better for low-light applications, such as security cameras.)

This sRGB data, then, must be converted to an industry-standard format such as BT601, which uses the YUV protocol. An explanation of the YUV protocol will help in the understanding of what is required.

YUV colour space differs from RGB, which is what the camera captures and what humans view. When colour signals were developed in the 1950s, it was decided to allow monochrome TVs to continue to receive and decode monochrome signals, while colour sets would decode both monochrome and colour signals.

The Y in YUV stands for 'luma', which is brightness; monochrome TVs decode only the Y part of the signal. U and V provide colour information and are 'colour difference' signals of blue minus luma (B-Y) and red minus luma (R-Y). Through a process called 'colour space conversion', the video camera converts the RGB data captured by its sensors into either composite analogue signals (YUV) or component versions (analogue YPbPr or digital YCbCr). For rendering on screen, all these colour spaces must be converted back again to RGB by the TV or display system.

Clearly, converting Raw Pixel Data into a format such as BT601 that image processors or standard microprocessors can handle is a complex processing task. The question for the design engineer is: what system architecture to adopt to perform this task while meeting the overall requirement for low cost and small size? High-end microprocessor
A high-end microprocessor is one possible solution for adding intelligence and functionality to an image sensor. A typical example is Freescale's range of i.MX microprocessors, which now incorporates a camera interface. On the i.MX27, for example, the camera interface has configurable logic to support a number of popular image sensors. The device's 8-bit port can interface to YCC, YUV, Bayer or RGB data.

Figure 4 shows a block diagram of an IP camera based on the i.MX27. The central processing unit, video codec and multimedia accelerator would process the image and compression algorithms. To handle the Melexis sensor's output, it will be necessary to configure the camera sensor interface for Raw Pixel Data using CSI Control Registers 1, 2 and 3. In particular, the design engineer must correctly determine the Bayer Output format from the Melexis sensor and ensure that CSI Control Register 2 on the i.MX27 has the BTS (Bayer Tile Start) bits set correctly for the device. Careful study of the CMOS sensor interface in the i.MX27 reference manual will be required!

To see a bigger version of this graphic click here.

Fig. 4: block diagram of IP camera using i.MX27 microprocessor.

For a low-cost application such as an intruder detection system, however, a high-end device such as the i.MX27 will often be over-specified, offering many features that are not required. There is, however, an alternative: the specialist image processor. Such a device carries out the eight steps for image processing described above, and interfaces to a host processor. It is also cheaper than a general microprocessor.

A typical example is the BU6566GVW/BU6568GV from Rohm: it offers interfaces respectively for VGA and 1.3Mpixel cameras. It also provides a JPEG codec for image compression. (Unlike the i.MX27, however, the BU6566/68 will not accept the MLX75307's Raw Pixel Data output " it only accepts RGB or YUV=4:2:2 encoded data, which means that the designer will need to use an external DSP to convert the Melexis sensor's data.)

Capturing a usable image
With the Rohm image processor, then, YUV data can be processed into frames and output to a host processor. But it is worth bearing in mind the requirements of this application: the subject of the image does not want to be photographed and could be moving, the pictures could need to be taken at night, and yet to be useful they must allow the subject to be clearly identified.

Unmodified, the signal from some image sensors under the harsh lighting conditions experienced by typical security applications will often be unusable. Areas of excessive brightness or darkness could cause the image sensor to bleach or black out the image. A type of device called an adaptive image enhancer can solve this problem. It uses hardware-based image-processing technology to raise or lower the brightness of only selected portions of the image which are too bright or too dark. This approach achieves much better image clarity under extreme lighting conditions. A suitable choice for this function is the BU1571KN adaptive image enhancer from Rohm.

If you choose to use Melexis' MLX75307 as your image sensor, however, its 'multiple slopes' response function might be sufficient to adjust for the prevailing lighting conditions; if not, the design should use the Rohm BU1571KN, as this can selectively enhance the image in both the dark and light areas to provide the optimal output across the complete image.

Conclusion: architecture of a low-cost video camera system Figure 5 shows the complete architecture for a low-cost video camera, using the Melexis image sensor's 'multiple slopes' function to adjust for expected poor lighting conditions. The DSP converts the image sensor's Raw Pixel Data into a YUV format.

The host microcontroller is not specified, as the choice will be determined by the requirements of the rest of the system. But in the case of the burglar alarm system, Ethernet, wireless communication, CAN and USB interfaces are likely to be useful. Certainly the microcontroller must have I2C, SPI and external bus control capabilities. The I2C and SPI interfaces are used to configure and control the image sensor; the external bus is used to control the image processor.

To see a bigger version of this graphic click here.

Fig. 5: architecture of a low-cost video camera for use in bad lighting conditions.

Hands-on experience with this design example shows that it is not a simple task to add a camera to an existing system unless you have previous video design experience. If this is your first design using an image sensor, be prepared to do a great deal of reading on the subject before you start. Melexis cameras are an excellent choice from the point of view of quality and cost, but be aware that you will need to do some work to interface them to an image processor.

Anthony Parkin is technical solutions manager with Future Electronics in the U.K.

Related links and articles:

High-Definition Surveillance Systems Using Low Cost FPGAs