Publisher Profile

Audio by Van Alstine ABX Comparator Review, Part 2: Trials

By: |

Read Part 1: Audio Store & Wiring

Read Part 3: New Twists & Conclusion

The components and speakers used

I completed the listening sessions for ABX testing by building four distinct systems, using two at a time and separating two large blocks of listening sessions over several days by a few weeks in between. The first pairing of systems was decidedly low end, with economical speakers and cables. I wished to determine if the use of some inexpensive gear would ruin the potential to discern between systems. For instance, would less expensive speakers have enough of their own sonic signature for me to pick them out blindly? Here in its entirety are the elements of the initial test systems being compared. As you can see, there is a combination of elegant and economical gear, in which you should note the nearly identical amps:

Mac Mini running iTunes

Clarity Cable Organic USB

Exasound e22 DAC

Clarity Cable Organic RCA Interconnects

AVA ABX Comparator

 

(System A)

TEO Audio Liquid Pre; Wells Audio Innamorata Signature; Insignia 6” two-way

Coaxial speakers

 

(System B)

Cambridge Audio Azur 840E Preamp; Wells Audio Innamorata; Boston Acoustics

CR-7 bookshelf speakers

 

For these systems the bulk of the wiring was inexpensive Radio Shack 16 gauge audio cable, and both signal paths between systems were wired identically, including the power cords. The two versions of Innamorata amps offered an opportunity to compare an upgraded amp for assessment of the upgrade’s efficacy.

I paid quite a bit of attention to the placement of the speakers for systems A and B, being careful to have them equidistant from the listening chair, and placing them as follows:

(Left) A/B and (Right) A/B. The speakers were very close to each other, about 1-inch apart, and the 1-inch gap between them was in alignment with each respective ear so as to make it more difficult to discern from the soundstage and center image which speaker was playing. I felt the pairing of Left and Right of the A and B speakers was a more challenging placement than if I had placed one pair outside and the other inside, like so: (Left) A/B and (Right) B/A. In my listening I was not able to detect a shift in the soundstage and center image, that could give away which speaker was playing, thus I considered the placement a success for the trials.

Initially I used only system A listed above to casually switch sighted between the preamps and speakers to gain familiarity with the ABX and the sonic signature of each system. I ran through some very familiar tracks such as Marc Cohn’s “True Companion”, Steve Oliver’s “Bend or Break” recorded live, Christina Aguilera’s “Walk Away”, and Simply Red’s “Holding Back the Years”. I spent time flipping between the preamps, amps and speakers. In a premonition of sorts, I found the amps to sound all but indistinguishable. At the time I thought perhaps other influences were at work, and I knew I would return to a proper ABX testing of amps, so I moved on.

The system which caught my ear, that is, seemed most high-end to me, was system B, and I initially determined that the speakers were the most critical in that conclusion. When I compared preamps, with only one speaker the TEO Liquid Pre seemed much more luxurious, and yet when the systems were compared in entirety the Cambridge Audio pairing with the Boston speakers was the favorite. This is not to suggest that a preamp is negligible, for I also swapped systems to create a “most favorable” and “least favorable” pairing by reversing the preamps, and in that case the TEO Liquid Pre with the Boston speakers was even more convincingly upscale. I was informally getting to do real time matching of gear and selecting the combos I felt were most worthy. This alone is a powerful use of the ABX switch, and an audiophile who is contemplating spending a lot of money on a source, preamp, amp or speakers could resolve burning questions in their mind about the worthiness of an upgrade.

 

Blind testing

Then it came time (Finally!) to test my ears by shutting off the front display, putting the ABX into Test Mode. Since the Owner’s Manual refers to each incident in blind testing mode as a “test”, I will retain that terminology, and will apply the term “Trial” to indicate an event of 8 tests preprogrammed into the ABX. I did not see in the Manual a means of altering the frequency of tests when in blind testing mode. Here is where things began to get interesting! I say interesting, but truthfully, they became a bit scary, too. I obtained some results which I was quite confident would turn out the way they did, but in a few situations I was shocked by the results.

I first did a sighted dry run, stepping through trials 1-8 while seeing the indication on the display of which system was playing. I perked up to see that Trial 1 was system A, Trials 2-7 were system B, and Trial 8 system A. I had read in the Manual that occasionally the random generating switching might string along several tests without changing, and it had happened in the walk through!

Sensing that I had a handle on the characteristics of the two systems, I jumped into Blind Testing mode and put on Steve Oliver’s “Bend or Break,” slowly working my way through the 8 tests and writing down my guess as to which system was playing. The results? I was correct 7 of 8 times! The error came in the first test, which I selected system A, but knew instantly when I proceeded to test 2 that I had erred; the fuller, richer sound went away which meant I had initially been hearing system B.

This mistake is what I came to think of as an Orientation Error, in which the first selection in a blind test comparing two systems is essentially a true guess. As there is no immediate context to compare the sound and gain assurance of the guess, there is a 50/50 chance of being correct. I assumed, then, that if one was not correct initially and the random program continued to select the same system, one could be wrong as many as three or four times until a test with a switch of systems did occur. That meant the Orientation Error could conceivably result in blowing the entire Trial of 8 tests by missing on the first three or four. This would not definitively demonstrate an inability to discern between the sounds of level-matched systems, but only that one cannot have certainty in selecting between them until an actual switch of systems occurs.

I found my hypothesis to be borne out over time, as whenever I misjudged the initial test and in test 2 or 3 the systems were switched, I was quickly able to get “back on track” and complete the testing with nearly perfect accuracy. However, the second Trial taught me some additional lessons! Feeling confident, I decided to switch speakers and place a time limit of 30 seconds on my tests. System A now consisted of the Liquid Pre/Innamorata Signature/Insignia speakers, and System B consisted of Cambridge pre/Innamorata/Boston speakers.

Feeling buoyed by my initial success, I thought I would “predict” my score! The first Trial seemed such a cakewalk that I thought I would offer a prognostication of the results. In this instance I did not “reacquaint” myself with the sound of the systems A and B; I just changed them and after testing for level matching, went to the blind testing. In blind test mode I again went through the same piece of music and watched the clock to switch tests.

The results were horrific, and I knew it! My notes regarding the outcome were prophetic, “I predict a mess, horrid score. Why? Systems are more closely matched tonally and detail.” Boy, was I right, as I scored zero; I got all 8 of the tests wrong. When returning the functionality of the front display to step again through the 8 tests, I knew it was a disaster when I saw the first test showed system B. I had confused the sonic properties of system A with system B. I had perfect consistency in selecting the wrong systems, because I had misjudged their sound.

In a way this was bad, but in another way this was good. It was bad because it showed how easily we could ascribe to a rig an inaccurate set of sonic characteristics. However, when in the actual testing, I was able to discern consistently when a switch between the systems occurred. I knew that I would have to pay far more attention to the nuances of a system or component to score well, but I also knew that hearing the changes between systems was not as impossible as some suggest.

9 Responses to Audio by Van Alstine ABX Comparator Review, Part 2: Trials


  1. Dan Kuechle says:

    Here is a list of the latest changes / additions to the ABX box:

    1) As mentioned in the article, the common ground requirement between speaker level components has been be removed, thus allowing bridged amp testing and no damage from an inadvertent reversed speaker connection.
    2) At any time now, during blind test, system A or system B can be selected (as well as going back to system X). This allows the listener, at any time, to reference either system and then go back to the unknown system under evaluation.
    3) In blind test mode it has always been possible to run thru the 8 tests as many times as you want. moving from test 8 to test 1 in the “up” direction and from test 1 to test 8 in the “down” direction.

    Dan Kuechle, designer – AVA ABX Box

  2. Anonymous says:

    I wouldn’t use a additional box and cables to compare audio equipment, it just adds more to the system that’s not required or normal to have and all it’s going to do is mitigate any differences there are between the equipment being compared.

  3. Werd says:

    Hello
    While in possession of the ABX comparator did you leave the comparator your main listening system?
    That is not in use as comparator but running, inserted and hooked up in your system?

  4. Dan,
    God’s Joy,
    Thank you for the updates to the unit!

    “Anonymous”
    Unless you have perfect hearing, which I highly doubt, and perfect recall for that matter, then the additional equipment IS necessary to ascertain whether one is able to pick out gear or systems in a blind test. Also, unless one has definitive proof that all such equipment actually does mitigate any differences, then your assertion is conjecture.

    Blessings,
    Douglas Schroeder

  5. Werd,
    The Joy of the Lord to you,

    No, I removed the ABX when building systems for other reviewing purposes.

    Blessings,
    Douglas Schroeder

  6. Bob Bashaw says:

    Was it possible to compare just two units, such as two amplifiers? I would love to know if you could hear the difference between two level matched amplifiers with everything else being identical.

    • Anthony says:

      Hah – of course that is what it was designed for and the way it should have been done and would have been the scientific way to test something – one change at a time and limiting all other variables, but that would be too sensible. A reviewer would never do that as it would show them up.

      The differences in speakers are many orders of magnitude bigger than the imperceptible differences in _properly_ designed and _compatible_ electronic components.

      Comparing two completely different systems with different speakers in different room positions as the reviewer has done, and then trying to attribute differences to any one component within that system is ridiculous. This is a botched attempt by someone trying to “play” science, rather than real science. Despite that it is a step in the right direction and is better than most audiophile reviewers – as long as he excepts that his testing was flawed and communicates this. If he thinks he did things correctly and that the testing is pointless it is yet another step backwards.

      If he had tested a cd player and a dac with the exact same source data, eg testing a non hdcp cd with a file ripped from that cd (rather than a download from a different remastering) or just using the digital output from the cd player into the dac, and played level matched (properly level matched – like with a multimeter rather than by ear), and ensured that the cd player and dac output impedances were suitable for the input impedance of the amplifier, and that both devices had been tested to have flat frequency response, and that their are no obvious issues/flaws with the system such as hum or weird tones due to ground loops etc. If he did _all_ that and had the rest of the system completely the same, and tested blind, he would not have a chance of doing better than random chance guessing.

      I suspect he could even have used this ABX tester to test just a change in interconnects with the rest of the system remaining the same. Perhaps even just a change in speaker cables.

      If you want to read a good write up of how this sort of comparison should be done, have a read of the “Audio Equipment Testing White Paper” by Roger Sanders over at Sanders Sound Systems website. It is quite accessible for the laymen to read and covers the essential concepts of what is required for valid scientific testing.

      What I am seeing from reviewers and audiophiles who are afraid of a system like this, is that that are going into it with the wrong idea. Rather than treat it as a test of you personally being able to detect differences in components, you should use it as a test of the components themselves and whether they actually sound different from each other or if they sound exactly the same. It is not a test of your manhood or your golden ears, it is a test of the component and whether it actually sounds different or not.

      With science you start with a theory and try and produce _valid_ tests to disprove or prove it and you are to accept the results whatever they may be, providing your tests were actually valid and can be shown to be so.

      With situations like this, you seem to have a presupposition that for example $30000 cd player “A” _must_ sound different than $3000 cd player “B” because that is just what everyone says and they say they can hear it so it _must_ be true. So your theory then becomes that since A is better than B, if I can’t hear the difference then I must be faulty. I don’t want to risk knowing that so I won’t do the test or at least I’ll set it up lop sided so I can’t be proven to be deficient. Unfortunately if the original presupposition is faulty (which is could be) then that invalidates your conclusion that your hearing is deficient.

      What you should be doing is saying ‘people say $30000 cd player “A” sounds better than $3000 cd player “B” but there is no valid proof – just subjective opinions’. So the theory is $30000 cd player “A” sounds better than $3000 cd player “B” so I will design proper valid tests to test this theory and if this is proven true or false – that result is what will be accepted. If the tests are valid and cd player A and B are proven to sound exactly the same, then the theory is disproven and we must accept that they do in fact sound the same and anyone hearing differences between them therefore are incorrect and there must be some other factor at play to account for why they are hearing differences or why they say they are hearing differences. For example expectational bias, invalid testing (not level matched for example), changing too many variables at a time, taking too long between comparisons and relying on flawed memory of what something sounds like (even a few seconds can be too long), lying so they don’t lose resale value when they offload it, lying to promote a product, something else.

      I really wish you audiophiles would wake up and accept this. You could save yourselves a lot of money and you could then spend that money where it actually makes an audible improvement and would probably cure your virulent upgraditus in no time. When you keep spreading misinformation, you not only do yourself a disservice, you also hurt other people that listen to your misinformation and act on it.

  7. Bob Bashaw says:

    Oops, my computer didn’t print out the entire article earlier! Sorry, thanks for comparisson of the amps. I too had a similar experience. It was single blind, but I lost the ability to hear any differences. I never knew what to make of it. New heights of humility!

  8. Anthony,
    God’s peace to you,

    I did compare using single changes. I did more comparisons than I wrote about. It’s obviously not a lab test when conducted in a listening room. Imo, your objections are overdone. Without any animus, you sound like you have a bias and agenda to defend electronics and systems priced below a certain point. But, I’m not interested in extended argument about the article, nor your methods.

    Blessings,
    Douglas Schroeder

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Popups Powered By : XYZScripts.com