Results of MP3 at 128kbps public Listening Test

Notes by Roberto Amorim

with lots of help from ff123

 

These are the summary results of the MP3 at 128kbit/s Public Listening test.

 

User comments are available here.

I would like to point out two very serious issues with this test: not using the latest version of Xing, bundled with Real Player, that has been reportedly extensively tuned since version 1.5; and forcing VBR on codecs that shouldn't be using them. I'm confident iTunes MP3 would perform better if it was featured at CBR 128, and the same might apply to FhG. I take full responsability on those mistakes, and for them, I apologize.

How to interpret the plots: Each plot is drawn with the five codecs on the x axis and the ratings given (1.0 through 5.0) on the y axis. N is the number of listeners used to compute the means (average ratings) and 95% confidence intervals. The mean rating given to each codec is indicted by the middle point of each vertical line segment, and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is different than the one used on ff123's 64kbps test. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.

One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the DaFunk plot below, AudioActive is rated better than Gogo with 95% confidence. And AudioActive is rated better than FhG with greater than 95% confidence.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.

Some other important notes:

  • All codecs featured here were tested in their VBR modes, except AudioActive, because it only features CBR.
  • The Xing codec tested here is the last version available from XingTech. It was later bought by RealNetworks and underwent further development. The latest version - which wasn't tested here - is available with RealPlayer 10/RealOne

  • ATTENTION: the "iTunes" codec featured here is the MP3 encoder featured in iTunes - NOT the AAC encoder.

     

    Plot Comment
    Artist: Counting Crows
    Title: Big Yellow Taxi
    Album: CD single
    Submitted by: 2Bdecided

    Easy Listening

    Results: Lame, FhG, Gogo, AudioActive and Xing are tied. iTunes loses quite badly.

    Artist: Daft Punk
    Title: DaFunk
    Album: Homework
    Submitted by: madah

    House (Electronic/Techno)

    Results: Lame and AudioActive tied at first place. Gogo and Xing tied at second place. FhG and iTunes tied at third & last place.

    Artist: OMD
    Title: Enola Gay
    Album: Organisation
    Submitted by: AngelGR

    New Wave

    Results: Lame seems tied to FhG and Gogo, while AudioActive and Xing seem tied at 2nd place. iTunes loses, although it's quite close to the second place.

    Artist: India
    Title: La voz de la experiencia (feat. Celia Cruz)
    Album: Sobre el fuego

    Salsa

    Results: AudioActive wins, followed closely by Lame. Everything else is tied at the bottom. .

    Artist: Die Apokalyptischen Reiter
    Title: Gone
    Album: All you need is Love
    Submitted by: Dibrom

    Extreme Metal/NWOSDM

    Results: In this sample, Lame wins by far. FhG comes at second place, close to the mess at the bottom where all other encoders are (with Gogo a little above).

    Artist: John Linnell
    Title: Illinois
    Album: State Songs
    Submitted by: phong

    organ

    Results: Lame, Gogo and Xing are tied at first place, with Gogo a little below the others. iTunes and AudioActive are tied at second place. FhG surprisingly is at the bottom.

    Artist: Always Outnumbered
    Title: My Blood Rusts
    Album: Always Outnumbered / Ransom Split
    Submitted by: dev0

    Emocore/Punkrock

    Results: FhG wins. AudioActive is more or less tied with iTunes, which is more or less tied with Lame, which, by its turn, is more or less tied with Xing. Gogo loses.

    Artist: They Might Be Giants
    Title: New York City
    Album: Factory Showroom
    Submitted by: phong

    Pop

    Results: Lame and AudioActive are quite tied, followed by FhG and Gogo, which also seem tied. Xing is second worst and iTunes loses again.

    Artist: Friderick Chopin
    Title: Polonaise in D moll Op. 71, No. 1
    Submitted by: AstralStorm

    Piano solo

    Results: Lame and AudioActive are tied at first place, followed closely by iTunes (!). Gogo comes at third, FhG at fourth and Xing fails badly.

    Artst: Igor Stravinsky
    Title: Le Sacre Du Printemps
    Album: L'Adoration de la Terre
    Submitted by: ff123

    Orchestral

    Results: AudioActive performs very well. Lame comes at second place, FhG, iTunes and Xing are tied at third. Gogo fails very badly.

    Artist: Yasunori Mitsuda
    Title: Scars Left by Time
    Album: Chrono Cross Original Soundtrack
    Submitted by: rjamorim

    Heavy orchestral with drums

    Results: Lame is more or less tied to AudioActive at first place, and AudioActive is more or less tied to Xing. The rest is at the bottom.

    Artist: Green Day
    Title: Waiting
    Album: Warning

    Intro consists of guitar in far left with male vocal in center.

    Results: Waiting wreaks havoc again. Xing is the lone winner. Lame, FhG, Gogo and AudioActive are tied at a shameful second place much below. iTunes gets close to AudioActive, but loses neverthless.



    This is the bitrate distribution table. These bitrates were obtained using foobar2000

    		Xing	Lame	iTunes	Gogo	FhG	AActive
    BigYellow	140	132	126	132	135	128
    DaFunk		128	138	120	134	120	128
    EnolaGay	131	126	120	123	123	128
    experiencia	121	133	124	134	127	128
    gone		117	125	117	122	120	128
    Illinois	151	123	119	123	103	128
    mybloodrusts	129	121	117	126	137	128
    NewYorkCity	133	124	120	129	135	128
    Polonaise	84	122	114	114	96	128
    riteofspring	123	122	119	120	109	128
    Scars		136	125	123	124	115	128
    Waiting		138	122	119	124	126	128
                    -----------------------------------------------
    Means		127,5	126,0	119,8	125,4	120,5	128

     

    Overall Ratings: The results for each sample were grouped together, without modifications.

    Then I performed an ANOVA analysis. The results are graphed below. Lame wins, followed by AudioActive, which is more or less followed by Xing, FhG and Gogo. Although iTunes is a little tied with Gogo, it's safe to say it lost.

    While there was no clear winner, it's interesting to notice how well AudioActive performed, even being restrained by CBR encoding. Also, it seems the constant criticism directed at Xing is exaggerated at best.

    I would like to point out two very serious issues with this test: not using the latest version of Xing, bundled with Real Player, that has been reportedly extensively tuned since version 1.5; and forcing VBR on codecs that shouldn't be using them. I'm confident iTunes MP3 would perform better if it was featured at CBR 128, and the same might apply to FhG. I take full responsability on those mistakes, and for them, I apologize.

     

    And here's a zoomed version, showing only the significative part of the graph:

    Back to Roberto's Listening Tests page