Dubbed echo state networks (ESNs) by their discoverer, Herbert Jaeger at the International University Bremen (Bremen, Germany), these feedback networks are able to operate in highly non-linear regions close to their performance limits. That turns out to be a key capability that distinguishes biological organisms from man-made systems.

"I am planning to start a research line of ESN applications in telecommunications," said Jaeger, who has set up a web site that is attracting interest from industrial groups interested in capitalizing on the approach. "One typical application example would be the use of ESNs for dynamic channel assignment in next, or fourth, generation high data-rate cellular networks — up to 1 Gigabit per second — which will operate in an ad hoc and self-organizing fashion," he explained.

While it is generally recognized that nonlinear networks would be far more efficient and powerful than the linear variety, there is a shortage of effective engineering methods for working with them. Only very simple non-linear systems can be modelled mathematically, making them of limited usefulness. Somehow the brain, and most biological systems, have found a way to employ complex nonlinear dynamics effectively, which allows organisms to tackle complex real-world tasks while expending only a minimal amount of energy.

Jaeger hit on a black-box tactic for getting around the need to model non-linear networks. A network satisfying certain minimal specifications is randomly connected with random synaptic weights specified at the connections.

Echo networks harness the power of nonlinear dynamics using a black box approach.

One specification is that the network must have feedback loops, turning it into a non-linear system. The feedback loops produce an echo effect - a single input will cause the network to continue operating, outputing a reverberating signal over time. Hence the name "echo state network."

The black box contains a randomly connected echo state network which remains fixed. This is connected to a number of inputs and a row of output neurons that have adjustable weights. The system is then trained in the same manner as a conventional back-propagation network with the output weights being adjusted to reproduce model input-output data sets.

In benchmark runs on predicting chaotic time series, Jaeger found that the ESNs were more accurate by a factor of 2400 over standard techniques. Most practical systems being designed with the ESN approach are implementing the networks digitally in FPGAs. "A major design decision is whether one goes for an analog or digital realization of the ESN. The former would of course run much faster — conceivably in the high-frequency or very high frequency front-end directly. However, analog hardware is noisy and some mathematical groundwork to make ESNs noise-resistant remains to be done. Digital chips could implement the algorithm in its current form," he said.