Results of the public MP3 listening test @ 128 kbps (October 2008)


These are the summary results of the public MP3 listening test @ 128 kbps.

User comments are available here. If your packing utility supports RAR archives, you can also download a signed, locked and solid RAR file containing all results for all samples.

Encryption key can be downloaded from here.

How to interpret the plots: Each plot is drawn with six codecs on the X axis and the rating given (1.0 to 5.0) on the Y axis. The number of listeners used to compute the means (average ratings) and 95% confidence intervals are given on each plot. The mean rating given to each codec is indicated by the middle point of each vertical line segment and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is identical to the one used in Roberto Amorim's listening tests.

One codec can be said to be better than another with 95% confidence if the bottom of its segment is at or above the top of the competing codec's line segment.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.


Plot Comments
Plot finalfantasy - instrumental (harpsichords)

iTunes, Fraunhofer and Helix are tied on first place, followed by LAME 3.98.2 on second place and LAME 3.97 on third place. As expected, the low anchor is last.
Plot Chariots Of Fire by Vangelis from Madagascar - instrumental (synth brass)

iTunes, LAME 3.98.2, Fraunhofer and Helix are tied on first place, followed by LAME 3.97 which comes second and finally the low anchor that is close on the third position.
Plot San Francisco Bay Blues by Eric Clapton from Unplugged - country / blues

All contenders are tied on first place, the low anchor is last.
Plot Waiting by Green Day from International Superhits! - rock

LAME 3.97 is statistically better than LAME 3.98.2 and is tied to iTunes, Fraunhofer and Helix which however are not significantly better than LAME 3.98.2. The low anchor is last.
Plot In The Night by Pet Shop Boys from Disco - electronic dance

Helix is statistically better than iTunes, Fraunhofer and LAME 3.97 and it is tied to LAME 3.98.2. LAME 3.98.2 however is statistically on par with iTunes, Fraunhofer and LAME 3.97. The low anchor loses badly once more.
Plot atrain - jazz

LAME 3.98.2, Fraunhofer, LAME 3.97 and Helix are tied on first place, followed by iTunes on second place and the low anchor l3enc on third place.
Plot Tom's Diner by Suzanne Vega from Solitude Standing - female a cappella

iTunes, LAME 3.98.2, Fraunhofer and Helix are tied on first place, followed by LAME 3.97 that comes out second and the low anchor that loses again.
BTW, did you know that this sample is also called the "The Mother of the MP3" since Karlheinz Brandenburg used it to tune the MP3 encoder he was woking on? :)
Plot Danse Macabre by Camille Saint-Saëns - symphonic orchestra

The two LAME encoders are statistically better than iTunes and are tied to Fraunhofer and Helix which are tied to iTunes. l3enc is again last.
Plot Hypnotize by White Stripes from Elephant - garage rock

This sample shows LAME's statistical superiority vs. Helix at rock and metal which other users also discovered in independ tests at 128 kbps. Helix is tied to iTunes and Fraunhofer which are tied to both LAME encoders.
Plot Layla by Eric Clapton from Unplugged - acoustig guitar and applause

All contenders are tied on first place. Once again, l3enc loses as expected.
Plot Linchpin by Fear Factory from Digimortal - hard rock / metal

This sample also demonstrates that Helix has some difficulties with hard rock / metal and is therefore only tied to Fraunhofer - iTunes and the two LAME encoders which all three are tied are statistically better. Fraunhofer is also tied to LAME as well as iTunes. The low anchor performs worst.
Plot Kalifornia by Fatboy Slim from You've Come A Long Way, Baby - electronic

We finally see some more variance on this track which also had by far the highest average bitrate. Helix and LAME 3.98.2 perform equally well and are tied on first place. LAME 3.97 is tied to Fraunhofer which is tied to iTunes, however, LAME 3.97 is statistically better than iTunes. No surprise: l3enc is last.
Plot Castanets_Original - instrumental (castanets)

Although hard to see from the graph, iTunes, Fraunhofer and Helix are tied and statistically better than LAME 3.97 which is tied to LAME 3.98.2. The new LAME version is tied to iTunes, Fraunhofer and Helix. The low anchor is last.
Plot velvet - electronic, stereo separation

Again some more variance... Helix is tied to the two LAME versions and beats Fraunhofer. Both LAME versions are tied to Fraunhofer which all three, together with LAME, beat iTunes. l3enc is last for a last time.

These are the bitrates used:

    Sample (Duration in Seconds)     iTunes   LAME 3.98.2   l3enc (Low Anchor)   Fraunhofer   LAME 3.97   Helix
    ---------------------------------------------------------------------------------------------------------
    finalfantasy (30)                118      107           128                  119          97          114
    Vangelis_Chariots_of_Fire (15)   117      149           128                  121          126         110
    linchpin (24)                    139      143           128                  139          138         126
    Waiting (20)                     145      140           128                  140          149         151
    Pet_Shop_Boys_In_The_Night (27)  133      138           128                  144          146         149
    atrain (19)                      125      143           128                  149          150         151
    TomsDiner (20)                   141      109           128                  134          95          131
    macabre (17)                     120      136           128                  128          147         138
    White_Stripes_Hypnotize (15)     126      118           128                  129          109         97
    Layla (20)                       148      152           128                  147          158         152
    sfbay (15)                       151      145           128                  134          149         117
    fatboy_30sec (29)                192      214           128                  212          194         228
    Castanets_Original (7)           159      146           128                  151          133         143
    velvet (12)                      158      156           128                  163          159         173
    ---------------------------------------------------------------------------------------------------------
    Average: (19)                    141      143           128                  144          139         141
    Encoding Speed:                  25x      27x           1.63x                45x          18x         90x
    

Overall rating: The results for each sample were grouped together without modifications.

Then I performed an ANOVA analysis. The results are graphed below. They show that all encoders are tied on first place, except l3enc which of course comes out last being the low anchor.
What is interesting to see is how the MP3 codec actually evolved since its first days (l3enc was the first MP3 software encoder back in 1994 when it was released) and how it is still competitive with newer formats like AAC or Ogg Vorbis.
Another very interesting thing, which was also one of the goals for this test, is that Fraunhofer and especially Helix, which both outperform LAME in terms of encoding speed, are still very competitive. While statistically being tied to LAME on first place, Helix actually even received a higher rating than LAME 3.98.2 - and this at 90x encoding speed! Even FhG received a slightly higher score at least against LAME 3.97 which was the recommended encoder by the Hydrogenaudio community for a long time. But again, statistically, they are all tied so there is no quality winner.

Plot

The quality at 128 kbps is very good and MP3 encoders improved a lot since the last test. This was the last test conducted by me at this bitrate. It's time to move to bitrates like 96 kbps or 80 kbps.

Here is a zoomed version of the plot showing the competitors only and leaving out the low anchor l3enc.

Plot

And here is a plot showing the rating distribution across all 14 samples.

Plot

Finally, I would like to thank everyone who participated!

Free Web Hosting