Results of the public multiformat listening test @ 128 kbps (December 2005)
These are the summary results of the public multiformat listening test @ 128 kbps.
User comments are available here. If your packing utility supports RAR archives, you can also download a signed, locked and solid RAR file containing all results for all samples.
Encryption key can be downloaded from here.
How to interpret the plots: Each plot is drawn with five1 codecs on the X axis and the rating given (1.0 to 5.0) on the Y axis.
The number of listeners used to compute the means (average ratings) and 95% confidence intervals2 are given on each plot.
The mean rating given to each codec is indicated by the middle point of each vertical line segment and the value is printed next to it.
Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is identical to the one used in Roberto Amorim's listening tests.
One codec can be said to be better than another with 95% confidence if the bottom of its segment is at or above the top of the competing codec's line segment.
For example, in the Carbonelli plot below, iTunes is rated better than LAME or Shine with 95% confidence.
Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.
Another important note: iTunes = iTunes AAC, LAME = LAME MP3, Nero = Nero AAC, Shine = Shine MP3 (Low Anchor), AoTuV = Aoyumi's Tuned Version of the Ogg Vorbis encoder.
1: Nero, although tested by all participants, was excluded from the test because of a problem found during the testing phase.
The plots for the individual samples show only the five encoders iTunes, LAME, Shine, AoTuV and WMA Professional. Additionally, the score achieved by Nero is printed in the comments column.
The overall rating plot will show six encoders on the X axis (Nero is included here, although visually separated from its contenders).
Additional information is available here as text document.
2: Because of the mentioned Nero problem, the final graph (overall rating) will display Nero without confidence bars.
| Plot |
Comments |
 |
Artist: Counting Crows
Title: Big Yellow Taxi
Genre: Easy Listening
Submitted by: 2Bdecided
Nero score: 4.79
Difficult... AoTuV and WMA Professional are tied. WMA Professional and iTunes are tied, but iTunes is not as good as AoTuV. iTunes and LAME are tied. LAME is not as good WMA Professional and definitely not as good as AoTuV.
|
 |
Artist: James Brown
Title: Bodyheat
Genre: Soul
Submitted by: tigre
Nero score: 4.54
All competitors tied on first place. Shine is on last place.
|
 |
Artist: Giovanni Stefano Carbonelli
Title: Sonata Settima In La Minore
Genre: Baroque Chamber Music
Submitted by: guruboolez
Nero score: 4.90
iTunes, AoTuV and WMA Professional are tied on first place. LAME and Shine are tied on last place.
|
 |
Artist: Maná
Title: Coladito
Genre: Instrumental
Submitted by: jorsol
Nero score: 4.48
All competitors are tied on first place, Shine loses.
|
 |
Artist: Santa Esmeralda
Title: Don't Let Me Be Misunderstood
Genre: Spanish Rock
Submitted by: Sebastian Mares
Nero score: 4.63
iTunes, LAME, AoTuV and WMA Professional are tied on first place, Shine is on last place.
|
 |
Artist: Unknown
Title: Yello
Genre: Electronic
Submitted by: Alex B
Nero score: 4.82
Same picture - all encoders tied on first place except Shine, which loses.
|
 |
Artist: Eels
Title: Elizabeth On The Bathroom Floor
Genre: Slow / Quiet
Submitted by: jido
Nero score: 4.88
This is getting boring, what? All competitors tied on first place, Shine loses.
|
 |
Artist: Eric Clapton
Title: Unknown
Genre: Rock
Submitted by: Acid Orange Juice
Nero score: 4.65
All competitors are more or less tied on first place, while Shine is losing.
16. Jan. 2006 - ATTENTION: There was an error with the ranking for sample 8 caused by an invalid result. The plot has been updated - the old plot is available here. The old Nero ranking was 4.67.
|
 |
Artist: Lars Erstrand
Title: Reunion Blues
Genre: Blues
Submitted by: sehested
Nero score: 4.68
iTunes, LAME, AoTuV and WMA Professional are on first place, Shine on last.
|
 |
Artist: Yann Tiersen
Title: Les Jours Heureux
Genre: Instrumental (Harpsichord)
Submitted by: nyarlathotep
Nero score: 4.54
Don't know what to write so I don't repeat myself. :-D All competitors are tied on first place. Shine comes out last.
|
 |
Artist: Saint-Saëns
Title: Danse Macabre
Genre: Symphonic Orchestra
Submitted by: Wombat
Nero score: 4.65
Shine is last, the rest is tied on #1.
|
 |
Artist: Sash!
Title: Mysterious Times
Genre: Electronic
Submitted by: Sebastian Mares
Nero score: 4.64
iTunes, LAME, AoTuV and WMA Professional are tied on first place, Shine loses.
|
 |
Artist: Maurice Ravel
Title: Rapsodie Espagnole
Genre: Symphonic Orchestra
Submitted by: Alex B
Nero score: 4.62
Same picture folks. I am going to wake you up when something changes. Everything tied on first place except Shine that again performs worst.
|
 |
Artist: School
Title: Supertramp
Genre: Piano, Acoustic Guitars, Cymbals and Bass
Submitted by: sehested
Nero score: 4.71
All tied on #1 except Shine.
|
 |
Artist: Paris Combo
Title: Senor
Genre: Jazz
Submitted by: sTisTi
Nero score: 4.79
And again...
|
 |
Artist: Elton John
Title: Song For Guy
Genre: Piano. Synthesizer
Submitted by: naturfreak
Nero score: 4.54
And again...
|
 |
Artist: Opeth
Title: The Drapery Falls
Genre: Progressive Death Metal
Submitted by: HisInfernalMajesty
Nero score: 4.54
And again...
|
 |
Artist: Eminem
Title: White America
Genre: Rap
Submitted by: Gambit
Nero score: 4.91
And guess what... Again! Still tied on first place except Shine.
|
These are the bitrates used:
Sample (Duration in Seconds) iTunes LAME Nero AoTuV WMA Professional
--------------------------------------------------------------------------------------
BigYellow (24) 139 141 139 147 138
bodyheat (25) 136 146 138 139 143
Carbonelli (17) 128 121 142 143 92
Coladito (20) 145 152 140 152 162
DontLetMeBeMisunderstood (30) 143 162 137 163 165
yello (9) 142 160 152 175 115
Elizabeth (29) 128 109 137 112 117
eric_clapton (25) 141 153 138 146 153
ReunionBlues (30) 144 155 137 143 163
LesJoursHeureux (20) 136 146 141 180 119
macabre (17) 133 147 142 149 125
MysteriousTimes (28) 143 148 137 146 153
ravel (28) 140 149 137 131 157
School (19) 144 153 141 163 150
Senor (17) 135 137 142 132 131
SongForGuy (15) 133 144 144 161 126
TheDraperyFalls (30) 138 146 137 156 140
WhiteAmerica (30) 128 113 137 125 113
--------------------------------------------------------------------------------------
Average: 22.94 137.56 143.44 139.89 147.94 136.78
Overall rating: The results for each sample were grouped together without modifications.
Then I performed an ANOVA analysis. The results are graphed below. The results do not show anything new - all encoders are tied on first place, except Shine which of course comes out last.
16. Jan. 2006 - ATTENTION: There was an error with the ranking for sample 8 caused by an invalid result which also slightly affected the overall rating plot. The plot has been updated - the old plot is available here. The zoomed plots were not affected because the only change occurred to Shine. The minimal difference of 0.001 for the confidence bars had no impact on the plots.
I think this test shows that with the current encoders, the quality at 128 kbps is very good and improved a lot since the last test. IMHO, additional tests at this bitrate should stop unless we are testing codecs that didn't receive any major updates, like WMA Standard or ATRAC3. It's time to move to bitrates like 96 kbps or even lower (64 kbps).
Here is a zoomed version of the plot showing the competitors only and leaving out the low anchor Shine.
And here is a zoomed version without Shine and Nero.
Finally, I would like to thank everyone who participated!
