Why Comparisons between HiFi components usually suck

Have you ever wondered how companies are able to create CD players or digital sources that sound hugely different yet in every test magazine all of them exhibit low distortions and flat frequency responses? Yeah, me too. Don´t let that comment misguide you: there are indeed differences between CD players, a cheap one will probably have a lesser sound quality then a much more expensive one. But I´ll say that these differences are not as huge as magazines or manufacturers want us to believe. If you, dear reader, happen to be a DBT lover and a staunch supporter of the argument that CD players all sound the same then this article may be for you - or not. And if you, dear reader, happen to be a sighted listening test lover and a staunch supporter of huge differences between digital sources... well, then you may be in for a surprise.

For years I have wondered how so many distinct CD players can sound so different when reviewed in an audio magazine like Stereophile, Audio or Stereoplay (the latter ones are in Germany). I started to think about this when I replaced my trusty portable CD player Technics SL-XP 300 in 1995 with a Pioneer PD-S 802 only to find out that the bigger, more modern, Legato-Link-equipped Pioneer actually sounded worse. Don´t get me wrong: I´m not talking about personal preferences, I´m talking about objective sound quality differences (as much as a human being can be objective when it comes to audio). The Pioneer had better dynamic, more bass and treble and more precision. But everything else was better on the Technics: it had better fine dynamic, better staging, the instrument separation was far better, the stage was equally wide but also deep compared to the 2D-stage of the Pioneer. The Pioneer sounded like being on steroids, like it was playing with distortions all the time. Furthermore, its flat room response was shaky compared to the one of the Technics. That was the first time I thought about differences and how a thing that should be better because of improved parts and circuitry and a bigger price tag sounded worse instead.

Anyway, some months ago I started collecting vintage portable CD players - and that was when I finally knew the reason. It should have been clear to me for years now since I´ve read about this countless times at hydrogenaudio.org but sometimes one can´t see shit. The reason for different sounding CD players is mostly the lack of level-matching!

Level matching means that you change the output level of any source so that it matches the output level of another source - with the result that both play with the same volume. This is vital for comparing HiFi components because otherwise the louder component would sound better. Why? Because our ear perceives louder volume to have a better sound when compared to something else that has a lower volume. This is called loudness. It´s also the reason why we currently are in the midst of the Loudness War where music labels try to make their albums as loud as possible so that their newest album from artist XY will sound better then the album from artist YX from another label.

Source: Wikipedia

You can see the effect of Loudness in the picture above: the higher the sound pressure the better will bass and treble frequencies be perceived. Thing is that our ears are most sensitive at 2000-4000 Hz (the deepest canyons in the picture) because that´s the frequency band where we can hear the most important thing: crucial acoustic information in human speech communications (e.g. vowel formants). All other frequencies need much more sound pressure to be perceived at all which means: the louder something gets the more bass, treble and perceived dynamics it will have. These perceived or "fake" dynamics will be responsible for a subjective improvement of sound quality while in reality the signal is - apart from higher amplitudes - exactly the same.

The source of all evil: Line outputs, here shown with a Sony CDP-990
(I forgot who made that photo - if it´s yours write me a message!)

What has that to do with the way components are reviewed and compared in HiFi magazines? Because the magazines don´t do level-matched reviews, therefore the louder component will for sure be better sounding. Why aren´t they doing level-matched reviews? Because they can´t, they´d need a pre-amp or an amplifier that can adjust its input volume and most high end amplifiers (tube or solid stage doesn´t matter) cannot do this. And even if they could you´d need the possibility of precise adjustments to do level-matched comparisons. For that you´d need to measure the output voltage of a component because that voltage is directly related to the amount of volume that comes out of the component. The quasi-standard output voltage is 2.0 Volt. You may not know this but the output consists of a little more or less complex amp - the input of your amplifier is in 99% of all cases a passive, unpowered input, much like an amp powering up a loudspeaker. And exactly as the loudspeaker this input represents a load (with the resistance measured in Ohm) to the powered output of the component it´s connected to. In most cases the load is 50 kOhm. You can read all about line levels on Wikipedia. I´ll give two examples: a Technics SL-VP 50 and a Sony D-515. Both are portable CD players and both are equipped with a standard 3.5 mm line-output. Because both are portable players they need to conserve power so therefore their output voltage has been reduced compared to fully grown stereo components.

Sony D-515, a rare classic with supposedly brilliant sound (Picture made by myself).

Technics SL-VP 50, even rarer (and it can play Video-CD) and not so well

regarded soundwise (Picture made by myself).

The Sony D-515 is a highly regarded portable CD player one can sometimes find ridiculously overpriced on eBay. It´s very famous for its supposedly wonderful sounding line output, especially because someone at head-fi.org did a review of its sound quality. The Technics SL-VP 50 is not so well known and no one ever wrote an online review about it. Yet I still own the Technics after aquiring it for only 22,- Euros on eBay while I sold the Sony I aquired a month prior for roughly 120,- Euros, also on eBay. When they were released to the market they cost much more: the Sony was released in 1992 and cost 500,- Euros, the Technics was released in 1995 and cost 450,- Euro. But why did I keep the unknown Technics instead of the rare and beautiful Sony? I´ll tell you: the Sony sounded like crap. The sound quality of this little expensive gem is overrated: it sounded balanced and wide but also flat, uninvolving, thin and boring. The Technics instead sounds very similar to my ASUS Xonar Essence ST, the staging is a close match and equally stable, the room is equally deep but not as wide, the precision is roughly the same, instrument sizes are indistinguishable. It misses some bass and treble impact, the fine dynamic and articulation aren´t as good, it sounds a bit harsher and not so flexible. Remember: the Technics is 17 years old.

The Audioquest King Cobra used for recording

I compared the two players by recording their output in 24/192 with my ASUS Xonar Essence ST (both times with the same RCA-cable, an Audioquest King Cobra, pictured above). After that I matched the levels of the recorded material with a precision of +/- 0.1 dB. Then I compared the recorded results to the original files the CD for reviewing was derived from - very unfair because the players had to "fight" the original master file grabbed digitally with my LiteOn. Even more surprising that the Technics still sounded that good when one considers that the reference was a bit-perfect copy whereas the recording were converted to analogue by the CD players and back to digital by the ASUS. But there you have it: when level-matched the Technics was by far the superior player by being close to the original. But in every review I´ve read the Sony is the classic and the Technics almost forgotten. The reason for this can be found in their line output voltage. According to the respective manuals the Sony sports an output voltage of 0.9 Volts whereas the Technics has a lower output voltage of only 0.6 Volts. In effect this means that the Technics exhibits a lower volume when connected to the same input. Once recorded I had to raise the level of the Technics-recording by roughly 2.5 dB to match the one of the Sony. For most people not knowing about these things the Technics would appear to sound worse compared to the Sony and the same is valid even for "professional" audio magazines:

Stereoplay "Best-of-the-Best" list: see how the louder Sony fares better (45-47 Points)

compared to the Technics (41-43 Points). Oh, I don´t own the copyright of this picture.
No copyright infringement is intended here.

A shame, isn´t it? Imagine: you have trusted an audio magazine for years only to find out that they disregard a basic principle for comparing stereo components. They basically jerked you around - and they do it all the time. Level-matched comparisons? A little bit of objectivity? No, Sir! The same happens when you go into a store with the intention of buying a new stereo component: the louder component will win and will be bought by you, even though the sound could possibly be worse. Sometimes companies do this on purpose and design their components with a stronger-than-usual output voltage so that it will win sympathies in the store or with magazines. The stores are unaware about this but the magazines aren´t because most magazines measure the components thouroughly and sometimes they even publish those measurments, in rare cases you can even have a look at the output voltage. But even then they tend to ignore this and continue to do reviews with unmatched levels. One reason for this is that if they wouldn´t do so every subsequent review would be boring while another reason is that most people don´t care. Ask yourself: do you have the possibility of a level-matched test? I guess not. So you will do it just the way they do: connect a new stereo component to your amplifier and wonder how much better (louder) it sounds. Sure, you can adjust the volume but since you don´t know the exact volume differences it still is an unfair contest.

It´s not a very nice situation you´re in, dear reader. But I have one piece of advice for you: if you happen to read a review in an audio magazine about a component with awesome results try to look for a measurment of the output voltage. Maybe you´re in for a surprise. Also do some research before you go to a store to buy something new, maybe some magazine has measured the output voltage or the manufacturer has published them (be careful: boutique manufacturers may lie about this or don´t know how to do a proper measurement). If you´re careful you may avoid a purchase you may regret later. It´s not a guarantee though - but we can try, can we not?