Results of 128kbps Extension Public Listening Test

Notes by Roberto Amorim

with lots of help from ff123

 

These are the summary results of the 128kbit/s Extension Public Listening test.

 

User comments are available here.

Let me try to be clear: there was no winner in this listening test. Musepack didn't win. All modern codecs are tied at first place, simple as that.

 

How to interpret the plots: Each plot is drawn with the five codecs on the x axis and the ratings given (1.0 through 5.0) on the y axis. N is the number of listeners used to compute the means (average ratings) and 95% confidence intervals. The mean rating given to each codec is indicted by the middle point of each vertical line segment, and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is different than the one used on ff123's 64kbps test. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.

One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the 41_30sec plot below, WMA is rated better than Lame with 95% confidence. And AAC is rated better than Lame with greater than 95% confidence.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.

Another important note: It's worth mentioning that the WMA codec used in this test is WMA PRO (introduced in Windows Media 9). It has better quality, but is not backwards compatible with WMA standard, that is the codec that is actually supported by portable players, DVD players, etc. It's expected that devices supporting WMA Pro start being manufactured soon.

 

Plot Comment
Artist: Dave Matthews Band
Title: #41
Album: Crash
Submitted by: Filburt

"Rock"

Results: AAC, MPC and Vorbis are tied, WMA is a little lower than the others. Lame gets third place.
Blade is the worst by far.
Artist: Jazz-Preachers
Title: Take the A-Train
Album: Stax Space-Sound CD
Submitted by: ff123

Brass band

Results: AAC, MPC and WMA are tied, with WMA outstanding a little. Vorbis is a little behind.
MP3 comes at third place. Blade is worst again.
Composer: Johann Sebastian Bach
English Suites Nr.5 in E minor - Part.2: Allemande.
Performer: Blandine Rannou
Submitted by: Guruboolez

Solo Harpsichord

Results: WMA and MPC tied at first place, with WMA outstanding well. AAC and Vorbis nearly perfectly tied at second.
Lame at third place, well behind, and Blade at fourth, close to Lame.
Artist: Opeth
Title: Blackwater Park
Album: Blackwater Park

Quiet intro with acoustic and electric guitars followed by loud metal riffs.

Results: MPC, AAC and WMA tied at first place. Vorbis at second place, Lame at third just behind. Blade is faaaar....
Artist: Two Lone Swordsmen
Title: Death To All Culture Snitches
Album: Tiny Reminders
Submitted by: Dibrom

IDM

Results: MPC wins. AAC and Vorbis tied at second. Lame is at third place. Interestingly, WMA fails badly on this sample (and only on this sample), only winning compared to Blade (5th)
Artist: Man With No Name
Title: Floor-essence
Submitted by: Karl Heyes

Electronic mix

Results: Vorbis is tied to MPC at first place. WMA is at second and AAC at third. Lame at fourth, Blade last.
Artist: Eric Clapton
Title: Layla (unplugged)
Album: The All Time Greatest Rock Songs
Submitted by: Jan S.

Applause at the beginning, followed by piano and acoustic guitar

Results: WMA, MPC, and Vorbis are tied at first, with Vorbis a little worse than the others. AAC at second, Lame at third, Blade in last position.
Artist: Saint-Saëns
Title: Danse Macabre"
Album:
Submitted by: Wombat

Orchestral

Results: WMA wins. AAC, Vorbis and MPC are tied at second. Lame at third, Blade at fourth.
Artist: Mamas and Papas
Title: Midnight Voyage
Submitted by: Jan S.

drums and bass in the far left, guitar in the far right. Female vocal in the center.

Results: AAC, WMA, MPC and Vorbis tied at first place, with Vorbis a little worse than the others. Lame at second place, Blade at third.
Artst: Tiamat
Album: A Deeper Kind Of Slumber
Track: 01
Submitted by: treech

rock/metal riffs

Results: AAC wins, MPC and WMA tied at second. Lame and Vorbis tied at third, Blade is last.
Artist: The Source feat. Candy Staton
Track: You've Got the Love
Album: ?
Submitted by: Karl Heyes

Drums and ride cymbals intro, followed by bass and female vocals.

Results: WMA at first place AAC and MPC almost perfectly tied at second Lame and Vorbis at third, Blade way down there.
Artist: Green Day
Title: Waiting
Album: Warning

Intro consists of guitar in far left with male vocal in center.

Results: MPC wins, AAC comes close at second place WMA and vorbis are tied at third place. Lame at fourth, Blade at fifth place.

 

This is the bitrate distribution table. AAC and Ogg bitrates were gathered using Winamp; Lame, Blade and MPC bitrates using EncSpot and WMApro using Windows Media Encoder.

		AAC	Lame	MPC	Ogg	WMApro	Blade
41_30sec	129	125	166	143	128	128
ATrain		129	121	144	139	128	128
Bachpsichord	129	125	197	173	128	128
Blackwater	129	124	130	130	128	128
death2		129	133	115	114	128	128
FloorEssence	129	129	145	147	128	128
Layla		129	124	150	155	128	128
macabre		129	122	156	149	128	128
MidnightVoyage	129	121	142	141	128	128
thear1		129	123	132	128	128	128
TheSource	129	127	130	126	128	128
Waiting		129	121	147	137	128	128
               ----------------------------------------------
Average		129	124.5	146.1	140.1	128	128

Overall Ratings: The results for each sample were grouped together, without modifications.

Then I performed an ANOVA analysis. The results are graphed below. There's a big tie at first place, with MPC only a little above AAC, WMA and Vorbis.
Lame is at second place and Blade is far behind at third.

 

Here's a zoomed version showing only the most significative part of the graph, leaving out the anchor:

Here is the old zoomed plot, before error bars were corrected.

Back to Roberto's Listening Tests page

 

Free Web Hosting