# Nonlinear nets approach runway to wireless apps

HANCOCK, N.H. — Recent research is revealing how to harness the nonlinear operation of biological neural networks to create a more powerful architecture for applications in telecommunications and robotics.

The architectural network created by Herbert Jaeger in Germany and the mathematical model produced by Wolfgang Maass in Austria bring the promise of man-made devices with the speed and power of the brain one step closer to reality.

Although a great deal of progress has been made in understanding how the brain works and applying that knowledge to computer systems via neural networks, no one has been able to come close to producing a device that can handle the information-processing power available to even the most humble of creatures.

A major hurdle for computer scientists and engineers alike is the recurrent feedback architecture of natural neural networks, which puts them in the terra incognita of nonlinear dynamics. Thus, the most widely used architecture-the back-propagation of errors network-is a linear system with only feed-forward connections.

Now Jaeger at the International University Bremen (Bremen, Germany) has hit on a black-box tactic for getting around the need to model nonlinear networks. Nonlinear systems can be approximately simulated on computers, but most lack any comprehensive mathematical model, which makes it difficult to design them into systems.

Jaeger has developed a type of feedback architecture he terms Echo State Networks. His ESNs are powerful nonlinear networks that look like back-propagation networks to the user.

Using Jaeger's ESN model, a network satisfying certain minimal specifications is randomly connected with random synaptic weights specified at the connections. One specification is that the network must have feedback loops, turning it into a nonlinear system. The feedback loops produce an echo effect-a single input will cause the network to continue operating, outputting a reverberating signal over time. Hence the name "echo-state network."

"I developed ESNs when I was a member of the Fraunhofer Institute for Autonomous Intelligent Systems (Sankt Agustin, Germany). This institute develops autonomous mobile robots. Thus, there is ongoing development work on using ESNs for robot control, both low-level motor control and high-level behavior control," said Jaeger.

The black box contains a randomly connected echo-state network that remains fixed. This is connected to a number of inputs and a row of output neurons that have adjustable weights. The system is then trained in the same manner as a conventional back-propagation network with the output weights being adjusted to reproduce model input-output data sets.

Jaeger likens the operation of ESNs to the standard mathematical technique of representing a nonlinear curve with a linear combination of nonlinear basis functions-Taylor series, Fourier series or wavelets being some of the most common basis sets. "You start from a collection of nonlinear functions or signals and linearly combine them to approximate a target. The larger the collection of basis functions you choose from, the finer the approximation," he said.

Jaeger characterizes nonlinear networks as "beautiful beasts," since they would have a power far beyond current networks, but are essentially untamable.

Mathematicians have discovered that only a few general criteria are needed to qualify a set of functions as a basis set for approximation purposes, and the same is true of the ESN black-box network.

"In typical engineering applications in signal processing and control, both weights and connectivity pattern are random," he explained. A sparse connectivity is important for good performance. A regular connectivity pattern with a retinal structure makes sense for image or video-processing tasks, where the input is a sequence of two-dimensional patterns. The input neurons would then likewise be arranged in a two-dimensional grid, as well as one or two subsequent recurrent layers."

ESNs have an additional advantage over a fixed set of basis functions, since the output of the network can be fed back to the black-box network, which tends to shape the response of the nonlinear network. The mechanism is similar to an infinite impulse response filter, and has created some stability problems, he said.

"The internal basis signals inside the reservoir are automatically preshaped to look like modified versions of the target signal. For instance, if the target signal is periodic, all the basis functions will automatically take on the same period. This preshaping of the basis is another factor that leads to the superior approximation properties of ESNs.

In benchmark runs on predicting chaotic time series, Jaeger found that the ESNs were more accurate by a factor of 2,400 than standard techniques.

Most practical systems being designed with the ESN approach are implementing the networks digitally in field-programmable gate arrays. "A major design decision is whether one goes for an analog or digital realization of the ESN. The former would of course run much faster-conceivably in the high-frequency or very high-frequency front end directly. "Analog hardware is noisy, however, and some mathematical groundwork to make ESNs noise-resistant remains to be done. Digital chips could implement the algorithm in its current form," he said.

"I am planning to start a research line of ESN applications in telecommunications," said Jaeger, who is working with a group at the Fraunhofer Society to offer consulting services to companies wishing to capitalize on the approach. "One typical application example would be the use of ESNs for dynamic-channel assignment in next-generation (fourth generation) high-data-rate cellular networks-up to 1 Gigabit per second-which will operate in an ad hoc and self-organizing fashion," he explained.

Work done with ESNs so far seems to confirm that they will be both as easy to work with as back-propagation of error networks, but are much more powerful in predicting time series.

But will ESNs have a better chance of behaving more like neural networks do in biological systems than other approaches researchers have used to date? An answer to that question is being investigated by Maass at the Technical University of Graz (Graz, Austria).

Maass independently discovered ESNs while trying to mathematically model the behavior of feedback circuits in the brain. While Jaeger's black-box networks only use a highly simplified model of a neuron, Maass' model has more realistic neurons that communicate using trains of voltage spikes. Maass called these nonlinear systems "liquid-state networks," comparing them with the surface of a liquid.

For example, when a sugar cube is dropped into a cup of coffee, the surface begins to undulate in a complex pattern that gradually diminishes in amplitude until it reaches the original zero state. A similar phenomenon occurs when a reservoir of feedback neural nets are given a single-input set of data. Given a time-series of input events, the continual agitation of the liquid stores a running history of the input sequence, giving the network a built-in memory.

Maass has formulated a general mathematical model called a liquid-state machine, similar to the universal Turing machine model of digital computers.

Such liquid-state computers can be shown to be universal for time-series prediction in the sense that they can implement any time-invariant filter with fading memory. Those specifications cover any nonlinear filter that could be designed for real-world time series.

Actual liquids are not good models with which to work, however, when it comes to creating liquid-state computers. Although a convenient metaphor, liquids are modeled as a collection of nearest neighbors (the water molecules) acting on one another. Neural networks offer more complex interactions, which are needed to mimic the neural activity actually seen in nature. To solve that problem, Maass tried substituting some realistic models of neurons communicating with voltage spike trains along with models of synaptic connections. The input and output from the "liquid" network was also modeled as realistic spike trains. The goal was to see if the whole system would exhibit the type of signal processing found in the brain.

Experiments with quite simple neural nets revealed a wide variety of network activity. It was discovered that a small reservoir of neurons could implement any finite-state machine, a computational approach that is used to process time series in engineered control systems. To verify the generality of these networks, Maass chose a series of randomly selected finite-state machines and found that in each case, its operation could be reproduced by adjusting only the weights of the output neurons.