Loudspeakers: Objective evaluations - Part 3: Comparing the subjective and objective domains

In the 40 years since those first tests described in Chapter 17 some things have changed, and some things have not. Loudspeakers, in general, are much better, and quite acceptable sound quality is available at eminently affordable price levels if you do a little shopping.

None of it happened by accident; it is the result of better measurement systems being widely available and relatively inexpensive, and designers developing a healthy respect for good technical performance. The widespread availability of excellent raw transducers has made the design of good sounding loudspeaker systems a lot easier than it once was.

Yet, in other parts of the audio world, there are few signs of progress. Reviewers continue to ignore the scientific method, and a few even disparage those who follow it. Measurements are not a requirement for product reviews, and those that are seen cover a wide range from almost useless to quite impressive.

Sometimes there are amusing examples of evasive writing when the technical data suggest something that runs contrary to the subjective component of the review. In one memorable case, the technical reviewer commented that he thought his measurements may be faulty - the data described an inferior product, the subjective reviewer really liked it. The measurements were correct.

Listening tests continue to be of the "take-it-home-and-listen-to-it" kind, so many important variables are not controlled, and adaptation and bias are both factors. This was the basis of comments by pioneering audio journalist J. Gordon Holt in Section 17.5, lamenting the lack of scientific controls in subjective evaluations.

The following evaluation involved four high-end loudspeakers in the $8000 to $11,000 range per pair. All had been applauded by the audiophile press, with accolades like "Editor's choice," "Class A," "Product of the Year," and so on. Readers of these publications were led to believe that any one of these products would be a superb choice, and the prices only enhance such a belief.

Figure 18.14 is another example of the subjective and objective domains exhibiting harmony. The results of double-blind listening tests indicate that two loudspeakers tied for first place - one was a clear second and one brought up the rear. In looking at the corresponding measured data, it is not difficult to see the progressive decay in performance. There would seem to be no great mystery here, but two key elements underlying this display are very rare in the audio business: properly conducted double-blind listening tests and accurate, comprehensive measurements.

FIGURE 18.14 Subjective ratings and objective data on four high-end loudspeakers. The error bars on the top of each histogram bar show 95% confidence levels, confirming that loudspeakers "R" and "I" are in a statistical tie for first place. The other differences are highly significant. Loudspeakers "R," "I," and "B" are forward-firing cone-dome systems. "M" is a hybrid system with an electrostatic mid-high-frequency driver mated with a conventional woofer. Loudspeakers "R," "I," and "B" are free from obvious resonances and differ only in the constancy of their frequency responses and directivities. Loudspeaker "M" is very directional, so directional in fact that listeners seated across the width of a sofa will experience different spectra. It also shows evidence of multiple resonances, bumps that appear in all of the curves including the sound power.

This test was a precursor to a much more ambitious test conducted by Olive (2003) who used some of these same loudspeakers, and others, in a test that, by the time of his paper, had involved 268 listeners and, by the time it was disbanded, over 350 listeners. As reported in the paper, no group of these listeners, most of whom were visitors to the facility, differed in the order of their product ratings.

Those tests were not as well controlled as the normal tests, in that time constraints allowed for only two rounds of listening and the listeners were in small groups, not individuals. It does show, though, that if listening tests are blind, with multiple comparisons among products, with revealing program and equal loudness levels, then it is possible to get remarkably consistent opinions from ordinary listeners. As shown in Figure 17.6, the consistent performance of selected and trained listeners is an advantage when time is of the essence, but listeners with an interest in audio and some attentive listening experience can yield the same answers if they are given enough time for repetitions.

Looking at Figure 18.14, it appears that recognizing true excellence or true inferiority in the measurements would not require any special training. Interpreting the relative merits of the vast middle ground of loudspeakers that don't do everything well is more difficult.

Here the decisions come down to trading off problems and virtues, the audibility of which are likely to be dependent on program material. A listening test may settle such a debate. However, this is a debate over the relative merits of flawed products, and the winner of this kind of test is still not an example of excellence. It is common in listening tests for there to be little disagreement about the winners and losers, products with the highest and lowest ratings, but less certainty about the ratings of the intermediate products.

REFERENCES
Olive, S. (2003). "Difference in Performance and Preference of Trained versus Untrained Listeners in Loudspeaker Tests: A Case Study," J. Audio Eng. Soc., 51, pp. 806–825.

Toole, F.E. (1986). "Loudspeaker Measurements and Their Relationship to Listener Preferences," J. Audio Eng. Soc., 34, pt. 1, pp. 227–235; pt. 2, pp. 323–348.

Printed with permission from Focal Press, a division of Elsevier. Copyright 2008. "Sound Reproduction: The Acoustics and Psychoacoustics of Loudspeakers and Rooms" edited by Floyd Toole. For more information about this title and other similar books, please visit www.elsevierdirect.com.

Related links:
Loudspeakers: Objective evaluations - Part 1: Sound source radiation patterns | Part 2: Measuring the essential properties of loudspeakers
Acoustics and Psychoacoustics Applied - Part 1: Listening room design
Acoustics and Psychoacoustics: Introduction to Sound, Part 1: Pressure waves and sound transmission | Part 2: Sound intensity, power and pressure level | Part 3: Adding sounds together | Part 4: The inverse square law | Part 5: Sound Interactions | Part 6: Sound Interactions (cont.) | Part 7: Time and frequency domains
Using the Decibel - Part 1: Introduction and underlying concepts | Using the Decibel - Part 2: Expressing Power as an Audio Level