Results of the public MP3 listening test @ 128 kbps (October 2008)
These are the summary results of the public MP3 listening test @ 128 kbps.
User comments are available here. If your packing utility supports RAR archives, you can also download a signed, locked and solid RAR file containing all results for all samples.
Encryption key can be downloaded from here.
How to interpret the plots: Each plot is drawn with six codecs on the X axis and the rating given (1.0 to 5.0) on the Y axis.
The number of listeners used to compute the means (average ratings) and 95% confidence intervals are given on each plot.
The mean rating given to each codec is indicated by the middle point of each vertical line segment and the value is printed next to it.
Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is identical to the one used in Roberto Amorim's listening tests.
One codec can be said to be better than another with 95% confidence if the bottom of its segment is at or above the top of the competing codec's line segment.
Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.
| Plot |
Comments |
 |
finalfantasy - instrumental (harpsichords)
iTunes, Fraunhofer and Helix are tied on first place, followed by LAME 3.98.2 on second place and LAME 3.97 on third place. As expected, the low anchor is last.
|
 |
Chariots Of Fire by Vangelis from Madagascar - instrumental (synth brass)
iTunes, LAME 3.98.2, Fraunhofer and Helix are tied on first place, followed by LAME 3.97 which comes second and finally the low anchor that is close on the third position.
|
 |
San Francisco Bay Blues by Eric Clapton from Unplugged - country / blues
All contenders are tied on first place, the low anchor is last.
|
 |
Waiting by Green Day from International Superhits! - rock
LAME 3.97 is statistically better than LAME 3.98.2 and is tied to iTunes, Fraunhofer and Helix which however are not significantly better than LAME 3.98.2. The low anchor is last.
|
 |
In The Night by Pet Shop Boys from Disco - electronic dance
Helix is statistically better than iTunes, Fraunhofer and LAME 3.97 and it is tied to LAME 3.98.2. LAME 3.98.2 however is statistically on par with iTunes, Fraunhofer and LAME 3.97. The low anchor loses badly once more.
|
 |
atrain - jazz
LAME 3.98.2, Fraunhofer, LAME 3.97 and Helix are tied on first place, followed by iTunes on second place and the low anchor l3enc on third place.
|
 |
Tom's Diner by Suzanne Vega from Solitude Standing - female a cappella
iTunes, LAME 3.98.2, Fraunhofer and Helix are tied on first place, followed by LAME 3.97 that comes out second and the low anchor that loses again.
BTW, did you know that this sample is also called the "The Mother of the MP3" since Karlheinz Brandenburg used it to tune the MP3 encoder he was woking on? :)
|
 |
Danse Macabre by Camille Saint-Saëns - symphonic orchestra
The two LAME encoders are statistically better than iTunes and are tied to Fraunhofer and Helix which are tied to iTunes. l3enc is again last.
|
 |
Hypnotize by White Stripes from Elephant - garage rock
This sample shows LAME's statistical superiority vs. Helix at rock and metal which other users also discovered in independ tests at 128 kbps. Helix is tied to iTunes and Fraunhofer which are tied to both LAME encoders.
|
 |
Layla by Eric Clapton from Unplugged - acoustig guitar and applause
All contenders are tied on first place. Once again, l3enc loses as expected.
|
 |
Linchpin by Fear Factory from Digimortal - hard rock / metal
This sample also demonstrates that Helix has some difficulties with hard rock / metal and is therefore only tied to Fraunhofer - iTunes and the two LAME encoders which all three are tied are statistically better. Fraunhofer is also tied to LAME as well as iTunes. The low anchor performs worst.
|
 |
Kalifornia by Fatboy Slim from You've Come A Long Way, Baby - electronic
We finally see some more variance on this track which also had by far the highest average bitrate. Helix and LAME 3.98.2 perform equally well and are tied on first place. LAME 3.97 is tied to Fraunhofer which is tied to iTunes, however, LAME 3.97 is statistically better than iTunes. No surprise: l3enc is last.
|
 |
Castanets_Original - instrumental (castanets)
Although hard to see from the graph, iTunes, Fraunhofer and Helix are tied and statistically better than LAME 3.97 which is tied to LAME 3.98.2. The new LAME version is tied to iTunes, Fraunhofer and Helix. The low anchor is last.
|
 |
velvet - electronic, stereo separation
Again some more variance... Helix is tied to the two LAME versions and beats Fraunhofer. Both LAME versions are tied to Fraunhofer which all three, together with LAME, beat iTunes. l3enc is last for a last time.
|
These are the bitrates used:
Sample (Duration in Seconds) iTunes LAME 3.98.2 l3enc (Low Anchor) Fraunhofer LAME 3.97 Helix
---------------------------------------------------------------------------------------------------------
finalfantasy (30) 118 107 128 119 97 114
Vangelis_Chariots_of_Fire (15) 117 149 128 121 126 110
linchpin (24) 139 143 128 139 138 126
Waiting (20) 145 140 128 140 149 151
Pet_Shop_Boys_In_The_Night (27) 133 138 128 144 146 149
atrain (19) 125 143 128 149 150 151
TomsDiner (20) 141 109 128 134 95 131
macabre (17) 120 136 128 128 147 138
White_Stripes_Hypnotize (15) 126 118 128 129 109 97
Layla (20) 148 152 128 147 158 152
sfbay (15) 151 145 128 134 149 117
fatboy_30sec (29) 192 214 128 212 194 228
Castanets_Original (7) 159 146 128 151 133 143
velvet (12) 158 156 128 163 159 173
---------------------------------------------------------------------------------------------------------
Average: (19) 141 143 128 144 139 141
Encoding Speed: 25x 27x 1.63x 45x 18x 90x
Overall rating: The results for each sample were grouped together without modifications.
Then I performed an ANOVA analysis. The results are graphed below. They show that all encoders are tied on first place, except l3enc which of course comes out last being the low anchor.
What is interesting to see is how the MP3 codec actually evolved since its first days (l3enc was the first MP3 software encoder back in 1994 when it was released) and how it is still competitive with newer formats like AAC or Ogg Vorbis.
Another very interesting thing, which was also one of the goals for this test, is that Fraunhofer and especially Helix, which both outperform LAME in terms of encoding speed, are still very competitive. While statistically being tied to LAME on first place, Helix actually even received a higher rating than LAME 3.98.2 - and this at 90x encoding speed! Even FhG received a slightly higher score at least against LAME 3.97 which was the recommended encoder by the Hydrogenaudio community for a long time. But again, statistically, they are all tied so there is no quality winner.
The quality at 128 kbps is very good and MP3 encoders improved a lot since the last test. This was the last test conducted by me at this bitrate. It's time to move to bitrates like 96 kbps or 80 kbps.
Here is a zoomed version of the plot showing the competitors only and leaving out the low anchor l3enc.
And here is a plot showing the rating distribution across all 14 samples.
Finally, I would like to thank everyone who participated!
