Results of AAC at 128kbps v2 public Listening Test

Notes by Roberto Amorim

with lots of help from ff123

These are the summary results of the AAC at 128kbit/s v2 Public Listening test.

User comments are available here.

How to interpret the plots: Each plot is drawn with the five codecs on the x axis and the ratings given (1.0 through 5.0) on the y axis. N is the number of listeners used to compute the means (average ratings) and 95% confidence intervals. The mean rating given to each codec is indicted by the middle point of each vertical line segment, and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is different than the one used on ff123's 64kbps test. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.

One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the bodyheat plot below, iTunes is rated better than Nero with 95% confidence. And Faac is rated better than Compaact! with greater than 95% confidence.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.

Some other important notes:

Plot	Comment
	Artist: Counting Crows Title: Big Yellow Taxi Album: CD single Submitted by: 2Bdecided Easy Listening Results: iTunes wins, with Nero closely behind it, more or less tied. Faac is tied with Nero, and Compaact! and Real are tied just below Faac.
	Artist: James Brown Title: Body Heat (live) Album: Greatest Hits Submitted by: tigre Soul Results: iTunes clearly is the best here, with Faac, Nero and Real coming tied behind it. Compaact! gets worst place.
	Artist: Daft Punk Title: DaFunk Album: Homework Submitted by: madah House (Electronic/Techno) Results: iTunes and Nero tied at first place, Real and Compaact! tied with Nero, and Faac tied to Real and Compaact!.
	Artist: Die Apokalyptischen Reiter Title: Gone Album: All you need is Love Submitted by: Dibrom Extreme Metal/NWOSDM Results: iTunes is the winner; Faac, Nero and Real are tied at second place. Compaact! comes in last place.
	Artist: Johannes Brahms Title: Danse Hongroise 6 Performer: Kolacny Duo Album: Danses Hongroises Submitted by: guruboolez Piano Solo Results: In an interesting surprise, Compaact gets a first place almost breaking through the top of the graph. Real, Faac, iTunes and Nero are tied at second place, with Nero placed a little lower than the others..
	Artist: Gustav Mahler Title: Symphonie 3 fin Conductor: Pierre Boulez Submitted by: guruboolez Orchestral Results: iTunes is a clear first place. Nero, Faac and Compaact! are tied at second place. Real comes close behind, in last place.
	Artist: Always Outnumbered Title: My Blood Rusts Album: Always Outnumbered / Ransom Split Submitted by: dev0 Emocore/Punkrock Results: Nero and iTunes are tied at first place, with Nero a little above iTunes. Faac is tied to iTunes. Real and Compaact! come tied at last place.
	Artist: They Might Be Giants Title: New York City Album: Factory Showroom Submitted by: phong Pop Results: Faac, iTunes and Nero are tied at first place. Faac and iTunes are almost perfectly tied, and Nero comes a little below. Real is in fourth place, and Compaact! comes in last place.
	Artst: Duran Duran Title: Ordinary World Album: Greatest Submitted by: rjamorim New Wave / Art Rock Results: iTunes comes at first place, followed closely by Nero in second place, which is followed closely by Faac in third place. Faac is followed closely by Real and Compaact!, in last place.
	Artst: Omara Portuondo Title: Quizas, Quizas, Quizas Album: Veinte Años Submitted by: tigre Latino (Salsa) Results: Nero and iTunes tied at first place, with Nero a little above iTunes. Real and Compaact are quite tied together with iTunes. Faac takes the last position this time
	Artist: Green Velvet Title: Coitus (remix) Album: Green Velvet Submitted by: Roel van den Berghe Electronic Results: Compaact! surprises again coming in first place. It is followed by iTunes and Nero, in second place. Real and Faac are at last place.
	Artist: Green Day Title: Waiting Album: Warning Intro consists of guitar in far left with male vocal in center. Results: iTunes, Real and Nero are tied at first place. Compaact! and Faac are at second place.

This is the bitrate distribution table. These bitrates were obtained using foobar2000

		Nero	Real	Faac	Comp!	iTunes
BigYellow	136	128	142	131	128
bodyheat	140	128	133	120	128
DaFunk		141	128	126	130	128
gone		131	128	127	120	128
Hongroise	148	128	105	123	128
Mahler		149	128	118	125	128
mybloodrusts	149	128	135	115	128
NewYorkCity	125	128	145	120	128
OrdinaryWorld	133	128	145	120	128
Quizas		136	128	122	141	128
velvet		155	128	145	172	128
Waiting		138	128	133	127	128
                --------------------------------------
Averages	140	128	131,3	128,6	128

Overall Ratings: The results for each sample were grouped together, without modifications.

Then I performed an ANOVA analysis. The results are graphed below. iTunes is more or less tied to Nero, although the very small overlapping of their error margins indicate iTunes is indeed better, although with smaller than 95% confidence. Real, Faac and Compaact! are tied at third place. (or third place if you consider Nero as the second place)

This test showed some very interesting developments since the last AAC at 128kbps test. The most obvious one is surely Faac, that lost relatively badly before, and now became a very serious contender. The newcomers Real (actually CodingTechnologies) and Compaact! show lots of promise. Nero might also have been improved - it clearly lost to QuickTime/iTunes last time - although it can also be credit to the fact that Nero is using VBR, while iTunes is still CBR (it's actually impressive that a CBR-only encoder managed to be ranked so considerably better than VBR implementations, and that surely indicates the superior quality of Apple's encoder). It's probably the best choice for Mac and Windows users today, because it's highest quality, fast encoding, and the added advantage of being completely free

It's worth mentioning that according to a key developer of the QuickTime/iTunes AAC encoder, this codec nearly didn't get tuned since the last test. Makes one wonder how much it can improve once the engineers start tuning it again and if a VBR mode is eventually implemented

Anyway, it's safe to say that all codecs represented here are pretty mature and, no matter what your choice among them is, it's very likely you'll get very good results for your encodings.

And here's a zoomed version, showing only the significative part of the graph:

Back to Roberto's Listening Tests page