Results of 64kbit/s Listening Test

Results of 64kbps Public Listening Test

Notes by Roberto Amorim

with lots of help from ff123

These are the summary results of the 64kbit/s Public Listening test.

User comments are available here.

A hint: It's very interesting to compare these test results with ff123's 64kbps test results, that were conduced a little more than an year ago. This way, it's possible to take conclusions about how some codecs developed since then.

How to interpret the plots: Each plot is drawn with the five codecs on the x axis and the ratings given (1.0 through 5.0) on the y axis. N is the number of listeners used to compute the means (average ratings) and 95% confidence intervals. The mean rating given to each codec is indicted by the middle point of each vertical line segment, and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is different than the one used on ff123's 64kbps test. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.

One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the BigYellow plot below, MP3pro is rated better than WMA with 95% confidence. And Ahead HE AAC is rated better than QT AAC with greater than 95% confidence.

It's worth mentioning that, while Ahead HE AAC, Vorbis, MP3pro and WMA were encoded in VBR mode, Real Audio and QuickTime were encoded in CBR mode since these codecs don't offer a VBR mode. Lame MP3 was encoded at ABR mode because that's how Lame performs better at this bitrate.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.

Another important note: Keep in mind that Lame MP3 is at 128kbps in this test. There are two reasons for that:
1-To put results into perpective. In this aspect, Lame was a high anchor (and FhG MP3 the low anchor)
2-To test if some of these codecs deliver the marketing plot of "Same quality as MP3 at half the bitrates", specially concerning WMA and MP3pro.

Plot	Comment
	Artist: Counting Crows Title: Big Yellow Taxi Album: CD single Submitted by: 2Bdecided Easy Listening Results: Lame on top, with Ahead HE AAC just close to it, followed by MP3pro and Vorbis. WMA is on 5th place, and Real and QuickTime tied at 6th/7th. FhG MP3 is last on this sample and most others.
	Artist: Daft Punk Title: DaFunk Album: Homework Submitted by: madah House (Electronic/Techno) Results: Lame again in first place, with Vorbis on second and Ahead HE AAC tied to MP3pro at 3rd/4th. WMA is 5th and QuickTime 6th, followed by Real and then FhG MP3.
	Artist: OMD Title: Enola Gay Album: Organisation Submitted by: AngelGR New Wave Results: Lame first, with Vorbis, WMA and MP3pro more or less tied following. Ahead HE AAC does pretty bad, tied to Real and QuickTime. FhG MP3 last.
	Artist: India Title: La voz de la experiencia (feat. Celia Cruz) Album: Sobre el fuego Salsa Results: Lame is the winner by far. MP3pro, Ahead HE AAC and Vorbis are tied, with HE AAC a little lower than the others. WMA on 5th place, QT AAC on 6th and Real on 7th. FhG MP3 last, again..
	Artist: Die Apokalyptischen Reiter Title: Gone Album: All you need is Love Submitted by: Dibrom Extreme Metal/NWOSDM Results: Lame wins, with Ahead HE AAC and MP3pro tied at 2nd/3rd. Vorbis is at 4th, then QT AAC, and then WMA and Real tied. FhG MP3 loses.
	Artist: John Linnell Title: Illinois Album: State Songs Submitted by: phong organ Results: Lame first, Ahead HE AAC second. Then come MP3pro, Real, WMA and Vorbis more or less tied, with Vorbis quite lower than the others. Quick Time is a little above FhG, which loses again.
	Artist: Always Outnumbered Title: My Blood Rusts Album: Always Outnumbered / Ransom Split Submitted by: dev0 Emocore/Punkrock Results: Evil sample. Best rankings start pretty low, with Lame tied to Ahead He AAC. Vorbis comes at third, followed by QuickTime and Real tied. WMA, MP3pro and FhG are tied at the bottom.
	Artist: They Might Be Giants Title: New York City Album: Factory Showroom Submitted by: phong Pop Results: Lame wins. Ahead HE AAC is tied to MP3pro in 2nd/3rd place. Vorbis is tied to Real in 4th/5th place. WMA is tied to QT AAC in 6th/7th place. FhG loses.
	Artist: Friderick Chopin Title: Polonaise in D moll Op. 71, No. 1 Submitted by: AstralStorm Piano solo No, I didn't mess up the results. Believe it or not, Lame is at first, Ahead HE AAC second, QT AAC third, Real and FhG MP3 tied slighly above MP3pro (!), WMA is tied to Vorbis at last place.
	Artst: Igor Stravinsky Title: Le Sacre Du Printemps Album: L'Adoration de la Terre Submitted by: ff123 Orchestral Results: Lame wins, by far. Ahead HE AAC is in second place, followed by MP3pro. Real and Vorbis are tied at 4th/5th place, WMA and QT come tied just after, and FhG loses badly.
	Artist: Yasunori Mitsuda Title: Scars Left by Time Album: Chrono Cross Original Soundtrack Submitted by: rjamorim Heavy orchestral with drums Results: Lame wins again, Ahead HE AAC is tied to MP3pro at 2nd/3rd place, with MP3pro a little lower. Vorbis is tied to WMA that comes followed closely by Real. QuickTime is on 7th place, and FhG is last.
	Artist: Green Day Title: Waiting Album: Warning Intro consists of guitar in far left with male vocal in center. Results: The traditional evil sample. Lame starts badly at 3.75, followed by Ahead HE AAC tied to MP3pro. Real, WMA, QT and Vorbis follow in a big tie, with Real a little higher than the others and statistically better than Vorbis. FhG is last.

This is the bitrate distribution table. These bitrates were obtained using foobar2000 and EncSpot.

		MP3pro	Real	WMA	FhGMP3	Lame	HE AAC	QT AAC	Vorbis
BigYellow	75	64	79	64	126	64	64	69
DaFunk		67	64	75	64	127	64	64	64
EnolaGay	68	64	69	64	125	61	64	59
experiencia	71	64	64	64	131	60	64	67
gone		69	64	66	64	125	62	64	63
Illinois	60	64	65	64	122	69	64	58
mybloodrusts	78	64	64	64	123	67	64	60
NewYorkCity	76	64	79	64	124	63	64	63
Polonaise	58	64	46	64	122	72	64	51
riteofspring	67	64	84	64	123	70	64	61
Scars		66	64	88	64	125	68	64	62
Waiting		72	64	78	64	121	72	64	63
                --------------------------------------------------------------
Means		68.9	64	71.4	64	124.5	66	64	61.6

Overall Ratings: The results for each sample were grouped together, without modifications.

Then I performed an ANOVA analysis. The results are graphed below.

The first (obvious) conclusion is: No codec delivers the marketing plot of same quality as MP3 at half the bitrates.
Lame MP3 at 128kbps wins, followed by Ahead/Nero HE AAC on 2nd place, CodingTechnologies' MP3pro on 3rd place, Ogg Vorbis on 4th place, Real Audio, QuickTime AAC and WMA9 tied near the middle of the graph, and FhG MP3 definitely at the bottom.

Here's a zoomed version showing only the most significative part of the graph, leaving out the anchors:

Here is the old zoomed plot, before error bars were corrected.

Back to Roberto's Listening Tests page