Results of the public multiformat listening test @ 128 kbps (December 2005)


These are the summary results of the public multiformat listening test @ 128 kbps.

User comments are available here. If your packing utility supports RAR archives, you can also download a signed, locked and solid RAR file containing all results for all samples.

Encryption key can be downloaded from here.

How to interpret the plots: Each plot is drawn with five1 codecs on the X axis and the rating given (1.0 to 5.0) on the Y axis. The number of listeners used to compute the means (average ratings) and 95% confidence intervals2 are given on each plot. The mean rating given to each codec is indicated by the middle point of each vertical line segment and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is identical to the one used in Roberto Amorim's listening tests.

One codec can be said to be better than another with 95% confidence if the bottom of its segment is at or above the top of the competing codec's line segment. For example, in the Carbonelli plot below, iTunes is rated better than LAME or Shine with 95% confidence.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.

Another important note: iTunes = iTunes AAC, LAME = LAME MP3, Nero = Nero AAC, Shine = Shine MP3 (Low Anchor), AoTuV = Aoyumi's Tuned Version of the Ogg Vorbis encoder.

1: Nero, although tested by all participants, was excluded from the test because of a problem found during the testing phase. The plots for the individual samples show only the five encoders iTunes, LAME, Shine, AoTuV and WMA Professional. Additionally, the score achieved by Nero is printed in the comments column. The overall rating plot will show six encoders on the X axis (Nero is included here, although visually separated from its contenders). Additional information is available here as text document.

2: Because of the mentioned Nero problem, the final graph (overall rating) will display Nero without confidence bars.


Plot Comments
Plot Artist: Counting Crows
Title: Big Yellow Taxi
Genre: Easy Listening
Submitted by: 2Bdecided

Nero score: 4.79

Difficult... AoTuV and WMA Professional are tied. WMA Professional and iTunes are tied, but iTunes is not as good as AoTuV. iTunes and LAME are tied. LAME is not as good WMA Professional and definitely not as good as AoTuV.
Plot Artist: James Brown
Title: Bodyheat
Genre: Soul
Submitted by: tigre

Nero score: 4.54

All competitors tied on first place. Shine is on last place.
Plot Artist: Giovanni Stefano Carbonelli
Title: Sonata Settima In La Minore
Genre: Baroque Chamber Music
Submitted by: guruboolez

Nero score: 4.90

iTunes, AoTuV and WMA Professional are tied on first place. LAME and Shine are tied on last place.
Plot Artist: Maná
Title: Coladito
Genre: Instrumental
Submitted by: jorsol

Nero score: 4.48

All competitors are tied on first place, Shine loses.
Plot Artist: Santa Esmeralda
Title: Don't Let Me Be Misunderstood
Genre: Spanish Rock
Submitted by: Sebastian Mares

Nero score: 4.63

iTunes, LAME, AoTuV and WMA Professional are tied on first place, Shine is on last place.
Plot Artist: Unknown
Title: Yello
Genre: Electronic
Submitted by: Alex B

Nero score: 4.82

Same picture - all encoders tied on first place except Shine, which loses.
Plot Artist: Eels
Title: Elizabeth On The Bathroom Floor
Genre: Slow / Quiet
Submitted by: jido

Nero score: 4.88

This is getting boring, what? All competitors tied on first place, Shine loses.
Plot Artist: Eric Clapton
Title: Unknown
Genre: Rock
Submitted by: Acid Orange Juice

Nero score: 4.65

All competitors are more or less tied on first place, while Shine is losing.

16. Jan. 2006 - ATTENTION: There was an error with the ranking for sample 8 caused by an invalid result. The plot has been updated - the old plot is available here. The old Nero ranking was 4.67.
Plot Artist: Lars Erstrand
Title: Reunion Blues
Genre: Blues
Submitted by: sehested

Nero score: 4.68

iTunes, LAME, AoTuV and WMA Professional are on first place, Shine on last.
Plot Artist: Yann Tiersen
Title: Les Jours Heureux
Genre: Instrumental (Harpsichord)
Submitted by: nyarlathotep

Nero score: 4.54

Don't know what to write so I don't repeat myself. :-D All competitors are tied on first place. Shine comes out last.
Plot Artist: Saint-Saëns
Title: Danse Macabre
Genre: Symphonic Orchestra
Submitted by: Wombat

Nero score: 4.65

Shine is last, the rest is tied on #1.
Plot Artist: Sash!
Title: Mysterious Times
Genre: Electronic
Submitted by: Sebastian Mares

Nero score: 4.64

iTunes, LAME, AoTuV and WMA Professional are tied on first place, Shine loses.
Plot Artist: Maurice Ravel
Title: Rapsodie Espagnole
Genre: Symphonic Orchestra
Submitted by: Alex B

Nero score: 4.62

Same picture folks. I am going to wake you up when something changes. Everything tied on first place except Shine that again performs worst.
Plot Artist: School
Title: Supertramp
Genre: Piano, Acoustic Guitars, Cymbals and Bass
Submitted by: sehested

Nero score: 4.71

All tied on #1 except Shine.
Plot Artist: Paris Combo
Title: Senor
Genre: Jazz
Submitted by: sTisTi

Nero score: 4.79

And again...
Plot Artist: Elton John
Title: Song For Guy
Genre: Piano. Synthesizer
Submitted by: naturfreak

Nero score: 4.54

And again...
Plot Artist: Opeth
Title: The Drapery Falls
Genre: Progressive Death Metal
Submitted by: HisInfernalMajesty

Nero score: 4.54

And again...
Plot Artist: Eminem
Title: White America
Genre: Rap
Submitted by: Gambit

Nero score: 4.91

And guess what... Again! Still tied on first place except Shine.

These are the bitrates used:

    Sample (Duration in Seconds)   iTunes     LAME     Nero     AoTuV     WMA Professional
    --------------------------------------------------------------------------------------
    BigYellow (24)                 139        141      139      147       138
    bodyheat (25)                  136        146      138      139       143
    Carbonelli (17)                128        121      142      143        92
    Coladito (20)                  145        152      140      152       162
    DontLetMeBeMisunderstood (30)  143        162      137      163       165
    yello (9)                      142        160      152      175       115
    Elizabeth (29)                 128        109      137      112       117
    eric_clapton (25)              141        153      138      146       153
    ReunionBlues (30)              144        155      137      143       163
    LesJoursHeureux (20)           136        146      141      180       119
    macabre (17)                   133        147      142      149       125
    MysteriousTimes (28)           143        148      137      146       153
    ravel (28)                     140        149      137      131       157
    School (19)                    144        153      141      163       150
    Senor (17)                     135        137      142      132       131
    SongForGuy (15)                133        144      144      161       126
    TheDraperyFalls (30)           138        146      137      156       140
    WhiteAmerica (30)              128        113      137      125       113
    --------------------------------------------------------------------------------------
    Average: 22.94                 137.56     143.44   139.89   147.94    136.78
    

Overall rating: The results for each sample were grouped together without modifications.

Then I performed an ANOVA analysis. The results are graphed below. The results do not show anything new - all encoders are tied on first place, except Shine which of course comes out last.

Plot

16. Jan. 2006 - ATTENTION: There was an error with the ranking for sample 8 caused by an invalid result which also slightly affected the overall rating plot. The plot has been updated - the old plot is available here. The zoomed plots were not affected because the only change occurred to Shine. The minimal difference of 0.001 for the confidence bars had no impact on the plots.

I think this test shows that with the current encoders, the quality at 128 kbps is very good and improved a lot since the last test. IMHO, additional tests at this bitrate should stop unless we are testing codecs that didn't receive any major updates, like WMA Standard or ATRAC3. It's time to move to bitrates like 96 kbps or even lower (64 kbps).

Here is a zoomed version of the plot showing the competitors only and leaving out the low anchor Shine.

Plot

And here is a zoomed version without Shine and Nero.

Plot

Finally, I would like to thank everyone who participated!

Free Web Hosting