Notes by Roberto Amorim
with lots of help from ff123
These are the summary results of the 128kbit/s Extension Public Listening test.
User comments are available here.
Let me try to be clear: there was no winner in this listening test. Musepack didn't win. All modern codecs are tied at first place, simple as that.
How to interpret the plots: Each
plot is drawn with the five codecs on the x axis and the ratings
given (1.0 through 5.0) on the y axis. N is the number of
listeners used to compute the means (average ratings) and 95%
confidence intervals. The mean rating given to each codec is
indicted by the middle point of each vertical line segment, and
the value is printed next to it. Each vertical line segment
represents the 95% confidence interval (using ANOVA analysis)
for each codec.
This analysis is different than the one used on ff123's 64kbps test. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.
One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the 41_30sec plot below, WMA is rated better than Lame with 95% confidence. And AAC is rated better than Lame with greater than 95% confidence.
Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.
Another important note: It's worth mentioning that the WMA codec used in this test is WMA PRO (introduced in Windows Media 9). It has better quality, but is not backwards compatible with WMA standard, that is the codec that is actually supported by portable players, DVD players, etc. It's expected that devices supporting WMA Pro start being manufactured soon.
|Artist: Dave Matthews Band
Submitted by: Filburt
Results: AAC, MPC and Vorbis are
tied, WMA is a little lower than the others. Lame gets
Title: Take the A-Train
Album: Stax Space-Sound CD
Submitted by: ff123
Results: AAC, MPC and WMA are tied,
with WMA outstanding a little. Vorbis is a little behind.
|Composer: Johann Sebastian Bach
English Suites Nr.5 in E minor - Part.2: Allemande.
Performer: Blandine Rannou
Submitted by: Guruboolez
Results: WMA and MPC tied at first
place, with WMA outstanding well. AAC and Vorbis nearly
perfectly tied at second.
Title: Blackwater Park
Album: Blackwater Park
Quiet intro with acoustic and electric guitars followed by loud metal riffs.
Results: MPC, AAC and WMA tied at first place. Vorbis at second place, Lame at third just behind. Blade is faaaar....
|Artist: Two Lone Swordsmen
Title: Death To All Culture Snitches
Album: Tiny Reminders
Submitted by: Dibrom
Results: MPC wins. AAC and Vorbis tied at second. Lame is at third place. Interestingly, WMA fails badly on this sample (and only on this sample), only winning compared to Blade (5th)
|Artist: Man With No Name
Submitted by: Karl Heyes
Results: Vorbis is tied to MPC at first place. WMA is at second and AAC at third. Lame at fourth, Blade last.
|Artist: Eric Clapton
Title: Layla (unplugged)
Album: The All Time Greatest Rock Songs
Submitted by: Jan S.
Applause at the beginning, followed by piano and acoustic guitar
Results: WMA, MPC, and Vorbis are tied at first, with Vorbis a little worse than the others. AAC at second, Lame at third, Blade in last position.
Title: Danse Macabre"
Submitted by: Wombat
Results: WMA wins. AAC, Vorbis and MPC are tied at second. Lame at third, Blade at fourth.
|Artist: Mamas and Papas
Title: Midnight Voyage
Submitted by: Jan S.
drums and bass in the far left, guitar in the far right. Female vocal in the center.
Results: AAC, WMA, MPC and Vorbis tied at first place, with Vorbis a little worse than the others. Lame at second place, Blade at third.
Album: A Deeper Kind Of Slumber
Submitted by: treech
Results: AAC wins, MPC and WMA tied at second. Lame and Vorbis tied at third, Blade is last.
|Artist: The Source feat. Candy Staton
Track: You've Got the Love
Submitted by: Karl Heyes
Drums and ride cymbals intro, followed by bass and female vocals.
Results: WMA at first place AAC and MPC almost perfectly tied at second Lame and Vorbis at third, Blade way down there.
|Artist: Green Day
Intro consists of guitar in far left with male vocal in center.
Results: MPC wins, AAC comes close at second place WMA and vorbis are tied at third place. Lame at fourth, Blade at fifth place.
This is the bitrate distribution table. AAC and Ogg bitrates were gathered using Winamp; Lame, Blade and MPC bitrates using EncSpot and WMApro using Windows Media Encoder.
AAC Lame MPC Ogg WMApro Blade 41_30sec 129 125 166 143 128 128 ATrain 129 121 144 139 128 128 Bachpsichord 129 125 197 173 128 128 Blackwater 129 124 130 130 128 128 death2 129 133 115 114 128 128 FloorEssence 129 129 145 147 128 128 Layla 129 124 150 155 128 128 macabre 129 122 156 149 128 128 MidnightVoyage 129 121 142 141 128 128 thear1 129 123 132 128 128 128 TheSource 129 127 130 126 128 128 Waiting 129 121 147 137 128 128 ---------------------------------------------- Average 129 124.5 146.1 140.1 128 128
Overall Ratings: The results for each sample were grouped together, without modifications.
Then I performed an ANOVA analysis.
The results are graphed below. There's a big tie at first place,
with MPC only a little above AAC, WMA and Vorbis.
Lame is at second place and Blade is far behind at third.
Here's a zoomed version showing only the most significative part of the graph, leaving out the anchor:
Here is the old zoomed plot, before error bars were corrected.
Back to Roberto's Listening Tests page