This'N That (Review Conditions & Ratings)

For your convenienc I´ve seperated this page into several paragraphs,

just scroll to the one interesting you the most, dear Constant Reader:

1. Overture

2. Basic rationale

3. Review methodology (outline)

4. Review methodology (detailed)

5. The ratings

6. Timeline of changes

7. Pre-2013 articles

1. Overture

Do you know that I don´t like to read professional reviews of electronic gear most of the time? The reasons often are the persons reviewing the device. You see, those professional reviewers only hear what they want to hear, they don´t hear what´s in front of their ears. Example: if something is expensive, shiny and desirable, it´ll get a good review. Just because it´s expensive, shiny and desirable. Not because of its sound. What, you thought you were the only one to which that is happening? You´re mistaken, professional reviewers are Humans, they take a dump like you and I do (though I´m Marlene Dietrich, therefore beautiful flowers fly out of my ass).

Walker Audio Procenium Black Diamond (picture copyright: Walker Audio)

The turntable on the picture above is a perfect example of an expensive, shiny and desirable toy. Made out of wood and some allegedly exotic materials, it certainly looks like the equivalent of womans' precious jewellery, sporting highly polished black surfaces, accentuated with gold. This shiny bauble plays vinyl records and according to certain "reviews" it does so with immaculate quality. It costs around $ 100.000 so it must sound good, right? No. I firmly believe that the same sound can be achieved with a turntable costing far less - but people reviewing the overpriced stuff above love to be fooled by expressive marketing, overall looks, weight and reputation: they hear the looks of the device. And - who could forget - vinyl itself is a bad idea, it´s an anachronism, just like MD. Fuck, there you have it: I´m partly guilty myself. What a revelation!

Additionally, many professional reviewers use some dubious reference to compare other stuff with. This reference is the reference only because it was reviewed against another reference from which it then inherited the status "Reference". But who knows if this or the old reference are sounding like 'the truth'? Who tells them what is right or wrong? No one does.

I don´t want to be fooled, I also don´t want to fool you, dear Reader, when you´re reading reviews on this blog. I try to make sure to be as objective as possible without sounding too much like a moralizing bitch. Though I haven´t always been successful, read on and illuminate yourself on how it can be done.

2. Basic rationale

When people, on Amazon.com for example, review electronic gear, they do so using the music they usually listen to. If some unit or some cable sounds fantastic with, say, "Satisfaction" by the Stones, it must be very good. But is that so? 'Cause the thing is: they all expect it to have improved sound using this or that unit. Yet no one asks the question, how something that renders / transports music is able to know when to improve that sound. I can answer this: it doesn´t. Not one electronic device does, whether it is your smartphone or some cable. No, not even your precious tube amplifier. Those are all dead things, to them, your music is just a collection of electrical power. They all are like the T-1000, they only do what they were engineered to do. And if you find that view unromantic, even though it is a fact, you´re at the wrong place.

So let´s start with the rationale that any gear should play back exactly what´s on the disc / inside the file. It isn´t supposed to change the sound into something that has more bass, treble, improved spatiality or added "beauty". People get this wrong all the time (and often, boutique companies, too). More bass: "Awesome, dude!" More treble: "Sounds crisper, doesn´t it?" All of these statements point to a sonic change. Yes, it might indeed work for "Satisfaction". But what if you want to play something else, Voodoo Lounge, for example? And you suddenly realize that the changes the player did to "Satisfaction" all of a sudden sound like crap? One songs' dream is another songs' nightmare.

Because the ultimate goal is a very simple one: absolute neutrality, without changing anything, without euphonic colourization. There's more: a player that colours, might "improve" the sound of some mediocre or bad recording. But a very good recording won´t be a very good recording anymore, it´ll be mediocre as well. Do you want that? And what enables you to determine how something is supposed to sound? How can you know, if your CD player adds more bass, changes spatiality or sounds muffled?

3. Review methodology (outline)

My method of reviewing audio gear is rather simple: I compare against the music itself, meaning 'the reference' is the file I use for playback. For a diagram describing this, look above (enlarge for detail). The test files I recorded playing back on the to-be-reviewed unit have to compete against their own pure source files. Those source files are:

Often used:

• Zam the Assassin & The Chase through Coruscant (Star Wars: Episode II, John Williams)
• Banning Back Home (Hook, John Williams)
• Pinned and Mounted (The Girl with the Dragon Tattoo, Trent Reznor & Atticus Ross)
• Swim (Ray of Light, Madonna)
• Symphony No. 1 - Prélude (Louis Vierne, TELARC CD-80329)

Sometimes used:

• Planet Home (Synkronized, Jamiroquai)
• Kathy's Song (Welcome To Earth, Apoptygma Berzerk)
• Earth Song (HIStory, Michael Jackson)

For reviewing I employ the services of a common Desktop PC. Using a decent internal, semi-professional soundcard I'm able to not only play back music or testfiles with high quality, I can also record anything with the same quality. The Xonar Essence STX performs impeccably when measured, its input has a noisefloor as low as -116 dB and it lacks any distortion, both of which are traits of professional gear. It allows me to use it as a reliable recording device and as a poor-mans measurement equipment.

Asus Xonar Essence STX: a trusty workhorse & my main tool for reviewing stuff (photo copyright: Asus)

4. Review methodology (detailed)

Portable players (flash based)

1. Preparing reference files (matching samplerate/bitdepth, codec, etc. to the capabilities of the to-be-reviewed unit)
2. Copying reference files to the to-be-reviewed unit
3. Recording the output of the to-be-reviewed with the Xonar Essence STX (software used: Sound Forge (ASIO) & 24/192)
4. Editing the recorded files to match starting / end points of the reference files
5. Adjusting gain of the recorded files to match the gain of the reference files (using a 1 kHz sine)
6. Resampling to the samplerate of the reference files
7. Listening test: comparing reference files to recorded files (double blind & sighted)

CD players (portable & stationary)

1. Preparing reference files (matching samplerate/bitdepth to the capabilities of the CD (16/44.1)
2. Burning prepared reference files to CD 
3. Recording the output of the to-be-reviewed with the Xonar Essence STX (software used: Sound Forge (ASIO) & 24/192)
4. Editing the recorded files to match starting / end points of the reference files
5. Adjusting gain of the recorded files to match the gain of the reference files (using a 1 kHz sine)
6. Resampling to the samplerate of the reference files
7. Listening test: comparing reference files to recorded files (double blind & sighted)

MD recorders

1. Resampling reference files to 24/44.1
2. Recording reference files digitally with Sony MDS-JE 530 (playback on PC with Creative X-Fi HD)
3. Playback of recorded MD digitally with Kenwood DM-5090 to PC (Creative X-Fi HD)
4. Result on PC now contains ATRAC encoding artifacts -> new reference files
5. Recorded MD played back with the to-be-reviewed unit; output recorded with the Xonar Essence STX (software used: Sound Forge (ASIO) & 24/192)
6. Editing the recorded files to match starting / end points of the reference files
7. Adjusting gain of the recorded files to match the gain of the reference files (using a 1 kHz sine)
8. Resampling to the samplerate of the reference files
9. Listening test: comparing reference files to recorded files (double blind & sighted)

Amplifiers (headphones)

Since I already did some very lovely pictures describing the necessary alterations of my process so
that it would work for an amplifier review, I just use them again, slightly changing the accompanying text.
You can find that particular review here.

The FiiO E07K is a headphone amp and amplifies any audio signal it is fed. Therefore

I need it to amplify something and this something is my Creative Labs Soundblaster X-Fi HD USB.

Which also means I´ll require new reference files, I cannot use my original files anymore since

the recorded files then would include the combined sound signature of two units.

Because of that, new reference files are created by playing back my original files

and recording the line-out of the Soundblaster using the

input of the Xonar Essence STX.

To create files containing the sound signature of the E07K I simply loop it into the signal chain where

it then amplifies the signal coming out of the line-outs of the Soundblaster X-Fi HD USB.

The rest is similar to the three examples above.

Interconnects & data cables

• FiiO L16 (3.5 / 3.5)
• Audioquest Evergreen (RCA / 3.5)
• Audioquest Evergreen (3.5 / 3.5)
• Audioquest King Cobra (RCA)
• Oehlbach i-Connect (3.5 / 2x 3.5)
• Philips SWA7302S/10 (optical cable)
• Wireworld Chroma (coaxial digital RCA)
• Audioquest Cinnamon (USB)
• Audioquest Forest (USB)
• Cordial interconnects 

Software

• RMAA
• foobar2000
• SoX resampler
• Sound Forge
• LAME
• QTAACenc
• FLAC

NOTE A: for any objectivist the cables will obviously be a red rag as they are the epitomization of subjectivists' claims of cable sound. Who said I don´t fall for subjectivists dreams of shiny baubles? I like their looks, their haptic and soundwise I want to make sure that I capture the units under review with the best quality possible (and even their quality is a matter of debate). I simply don´t want to take any chances by using a cheap cable that might introduce unwanted artifacts of its own.

EDIT: Thanks to research done by others and myself I've found that many of the fancy cables I own change the sound signature in subtle ways. Likely to appeal to customers. I don't want that and I'm actually angry that I was so stupid and fell for it. For that reason, the fancy cables are gone, I don't use them anymore. Instead, I've bought no-nonsense cable from the German company Cordial. My requirements for cable now are: robust, shielded, sturdy plugs. Sounds simple, yet many companies producing boutique cables are unable to do that.

NOTE B: this Blog started out as a site using only sighted listening tests. On occasion, I had reverted to double blind listening tests (DBT) though I rarely made that fact transparent to you, dear Reader. In August 2013 I finally decided to employ them all the time in addition to sighted tests. Back then, I found that, on occasion, I fell prey to psychoacoustics wreaking havoc (more than usual) with my listening abilities. In short: I was imagining things and it was clear to me that I had reached the limit of my sighted listening approach. I don´t enjoy to use them, but they are a necessity, without them I wouldn´t be able to perform reliable reviews. Summing up, reviews written before August 2013 are, for the most part, sighted. Reviews after August 2013 are a combination of sighted and DBT.

NOTE C: Measurements are important, even if you, dear Reader, hate them. When I started this blog at the end of 2011, I felt it prudent to include them and I fully expected that the crowd would love them. Not so. Many people don´t understand nor do they like to read them. They say "My ears are the best equipment anyway, measurements don´t tell me how it sounds" or "I don´t listen to test signals"... or something of that ilk. To those I say: you´re fooling yourself... and you´re stupid. Your ears are prone to be unreliable. Why? Because you really listen with your brain. It interprets the data your ears are hearing. The science describing this is called 'psychoacoustic' and it details how our hearing mixes psychological with physiological responses all the time. We cannot avoid it and we cannot switch off one in favor of the other. Measurements are therefore far more reliable than our ears can ever be. And who could forget that a device playing music isn´t a living and breathing being. It´s a collection of ICs and cables. I can only repeat myself: It´s an unfeeling machine. So many people fail to grasp this simple fact, instead assigning some "magical" personalities to anything they are reviewing. But your network player isn´t some concoction by J.K. Rowling. And because it´s a machine, it follows its engineering, its specifications and behaviour are pre-determined. Measurements exploit this, illuminating how well something was engineered. While mine fall short of what professionals are expecting, I nevertheless want to offer a - hopefully - meaningful summary of how a device behaves. So I will continue to include them. To reflect my unprofessionalism in that matter, I usually call them "Fancy Graphs" because, basically, that´s all they are: pictures that look nice.

5. The ratings

In reviews prior to June 2012, I used something I called 'sound pictures'. They were an attempt of visualizing what I was hearing. I thought that an EQ curve would serve that purpose well. I was wrong, my readers didn´t understand the concept, they were mistaking them with actual frequency responses. So I had to swallow the bitter pill that I wasn´t able to explain them properly and introduced ratings that can be found in similar form in magazines dealing with audio gear. I came up with these circular ratings:

The circles represent how close a particular device matches the sound of my reference files. Zero points would mean that the reviewed unit has nothing in common with the reference, five points means that it sounds exactly like the reference. To make it easier for myself, I created five rows of circles, each row representing a specific sonic trait:

Sonic Balance:

Dynamics:

Resolution:

Stage / Ambiance:

Character:

Sonic Balance: describes the capability of some gear to present all frequency bands with the same perceived sound pressure level in respect to the reference files

Dynamics: represents how close perceived dynamic range, attack, decay, transient response or 'prat' come to the reference files

Resolution: defines the level of detail retrieval, crispness, etc.

Stage / Ambiance: outline spatiality, air, staging, size & placement of instruments, closedness or dryness

Character: not only contains if a unit is capable of transporting correct timbre & colour but also if it adds something of its own

The final specific trait 'Character' is an especially difficult thing to deal with. Shortly after introducing these ratings, I discovered that I used it more often than not to describe my personal preferences. If something sounded particularly beautiful (-> not like the reference files) it was apt to be rated highly. To accomodate this, I introduced a sixth, differently coloured rating circle:

Basically, it describes if a unit adds something of its own with the end result exceeding the quality of the reference files. Before all of you audiophiles have a field day (because you´re still searching for something that´s better than the truth, admit it) be advised that a unit "achieving" this, fails at other traits, often massively. As an example I´d like to point out the Kenwood DP-5090 I reviewed in October 2012:

Sonic Balance:

Dynamics:

Resolution:

Stage / Ambiance:

Character:

I awarded it six points for Dynamics and Resolution, features where it excelled and was superior to the reference files. Yet it paid for it by exaggerating treble so much that it sounded as if someone simply boosted it with an EQ. The Sonic Balance was so far away from the reference files that I awarded it zero points. When someone increases the gain of high frequencies, you usually achieve better detail retrievance by pronouncing, for example, wind noises made by woodwinds. Of course, not only woodwinds are audible now, everything else (cymbals, hi-hats) is too. Detail retrievance has been improved at the cost of sonic balance and aggressiveness. So when some unit excels at one point, it just fails at others. For some albums, the Kenwood was a good match. But on most other albums... fucked up sound. "Better than the truth" doesn´t exist and if it does, there´s something wrong with the gear you´re listening to.

6. Timeline of changes

• 2011: Blog created
• 2012: 'sound pictures' retired, ratings introduced
• 2013: E-MU 0202 USB retired (assumed it was broke -> cause for malfunction: NuForce USB cable)
• 2013: regular usage of DBT introduced
• 2013: major design overhaul
• 2015: consolidation of two articles into this one, principles now combined for convenience

7. Pre-2013 articles

All this wonderful talk about perfect sighted listening tests I did for years... in some cases it didn´t work out as I'd hoped. When I now re-read my first articles 3 to 4 years after I have written them, I hate them. They´re badly written, too long and talkative, hot air. What was until then only unnerving, turns into aggression upon encountering my ratings for something I reviewed. In several cases they are wrong and in hindsight I have to wonder what I was hearing, despite level matching and sample-precise editing. Three player / recorder I reviewed are especially striking:

1. Sony MZ-R 55
2. Sony NW-A 1000
3. Sony D-335

All of them were rated far too high. The Sony NW-A 1000 sounds boring, missing treble and bass. Much more than I admitted three years ago. The Sony MZ-R 55 is a similar case: while it has nice spatiality, it sucks at dynamics and balance. The Sony D-335 is very good at spatiality, too, though it does some other things very wrong (balance). Not good. There are several other cases where I was too positive. Still, it´s a comforting that their reviews are "only" bordering on being too unrealistic:

1. Sony MZ-R 37
2. Pioneer BDP-140
3. Pioneer DV-610

Don´t get me wrong: they are all OK. But the Sony MZ-R 37 sounds, while that can be very beautiful, far too lush. The Pioneer BDP-140 sounds very good, too, but it lacks a bit more mids than I initially wrote. Finally, the Pioneer DV-610 is very good with SACDs but worse with CDs. In each case, I would subtract one additional point for each of their weakest character traits. Let this also be a warning to you, dear Constant Reader: you should not even trust me and my reviews (at least not all of those written before 2013). So, if you have any doubts, write and comment like Hell.

Last update: 14.11.2016