Notes by Roberto Amorim
with lots of help from ff123
These are the summary results of the AAC at 128kbit/s v2 Public Listening test.
User comments are available here.
How to interpret the plots: Each
plot is drawn with the five codecs on the x axis and the ratings
given (1.0 through 5.0) on the y axis. N is the number of
listeners used to compute the means (average ratings) and 95%
confidence intervals. The mean rating given to each codec is
indicted by the middle point of each vertical line segment, and
the value is printed next to it. Each vertical line segment
represents the 95% confidence interval (using ANOVA analysis)
for each codec.
This analysis is different than the one used on ff123's 64kbps test. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.
One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the bodyheat plot below, iTunes is rated better than Nero with 95% confidence. And Faac is rated better than Compaact! with greater than 95% confidence.
Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.
Some other important notes:
|Artist: Counting Crows
Title: Big Yellow Taxi
Album: CD single
Submitted by: 2Bdecided
Results: iTunes wins, with Nero closely behind it, more or less tied. Faac is tied with Nero, and Compaact! and Real are tied just below Faac.
|Artist: James Brown
Title: Body Heat (live)
Album: Greatest Hits
Submitted by: tigre
Results: iTunes clearly is the best here, with Faac, Nero and Real coming tied behind it. Compaact! gets worst place.
|Artist: Daft Punk
Submitted by: madah
Results: iTunes and Nero tied at first place, Real and Compaact! tied with Nero, and Faac tied to Real and Compaact!.
|Artist: Die Apokalyptischen Reiter
Album: All you need is Love
Submitted by: Dibrom
Results: iTunes is the winner; Faac, Nero and Real are tied at second place. Compaact! comes in last place.
|Artist: Johannes Brahms
Title: Danse Hongroise 6
Performer: Kolacny Duo
Album: Danses Hongroises
Submitted by: guruboolez
Results: In an interesting surprise, Compaact gets a first place almost breaking through the top of the graph. Real, Faac, iTunes and Nero are tied at second place, with Nero placed a little lower than the others..
|Artist: Gustav Mahler
Title: Symphonie 3 fin
Conductor: Pierre Boulez
Submitted by: guruboolez
Results: iTunes is a clear first place. Nero, Faac and Compaact! are tied at second place. Real comes close behind, in last place.
|Artist: Always Outnumbered
Title: My Blood Rusts
Album: Always Outnumbered / Ransom Split
Submitted by: dev0
Results: Nero and iTunes are tied at first place, with Nero a little above iTunes. Faac is tied to iTunes. Real and Compaact! come tied at last place.
|Artist: They Might Be Giants
Title: New York City
Album: Factory Showroom
Submitted by: phong
Results: Faac, iTunes and Nero are tied at first place. Faac and iTunes are almost perfectly tied, and Nero comes a little below. Real is in fourth place, and Compaact! comes in last place.
|Artst: Duran Duran
Title: Ordinary World
Submitted by: rjamorim
New Wave / Art Rock
Results: iTunes comes at first place, followed closely by Nero in second place, which is followed closely by Faac in third place. Faac is followed closely by Real and Compaact!, in last place.
|Artst: Omara Portuondo
Title: Quizas, Quizas, Quizas
Album: Veinte Aņos
Submitted by: tigre
Results: Nero and iTunes tied at first place, with Nero a little above iTunes. Real and Compaact are quite tied together with iTunes. Faac takes the last position this time
|Artist: Green Velvet
Title: Coitus (remix)
Album: Green Velvet
Submitted by: Roel van den Berghe
Results: Compaact! surprises again coming in first place. It is followed by iTunes and Nero, in second place. Real and Faac are at last place.
|Artist: Green Day
Intro consists of guitar in far left with male vocal in center.
Results: iTunes, Real and Nero are tied at first place. Compaact! and Faac are at second place.
This is the bitrate distribution table. These bitrates were obtained using foobar2000
Nero Real Faac Comp! iTunes BigYellow 136 128 142 131 128 bodyheat 140 128 133 120 128 DaFunk 141 128 126 130 128 gone 131 128 127 120 128 Hongroise 148 128 105 123 128 Mahler 149 128 118 125 128 mybloodrusts 149 128 135 115 128 NewYorkCity 125 128 145 120 128 OrdinaryWorld 133 128 145 120 128 Quizas 136 128 122 141 128 velvet 155 128 145 172 128 Waiting 138 128 133 127 128 -------------------------------------- Averages 140 128 131,3 128,6 128
Overall Ratings: The results for each sample were grouped together, without modifications.
Then I performed an ANOVA analysis. The results are graphed below. iTunes is more or less tied to Nero, although the very small overlapping of their error margins indicate iTunes is indeed better, although with smaller than 95% confidence. Real, Faac and Compaact! are tied at third place. (or third place if you consider Nero as the second place)
This test showed some very interesting developments since the last AAC at 128kbps test. The most obvious one is surely Faac, that lost relatively badly before, and now became a very serious contender. The newcomers Real (actually CodingTechnologies) and Compaact! show lots of promise. Nero might also have been improved - it clearly lost to QuickTime/iTunes last time - although it can also be credit to the fact that Nero is using VBR, while iTunes is still CBR (it's actually impressive that a CBR-only encoder managed to be ranked so considerably better than VBR implementations, and that surely indicates the superior quality of Apple's encoder). It's probably the best choice for Mac and Windows users today, because it's highest quality, fast encoding, and the added advantage of being completely free
It's worth mentioning that according to a key developer of the QuickTime/iTunes AAC encoder, this codec nearly didn't get tuned since the last test. Makes one wonder how much it can improve once the engineers start tuning it again and if a VBR mode is eventually implemented
Anyway, it's safe to say that all codecs represented here are pretty mature and, no matter what your choice among them is, it's very likely you'll get very good results for your encodings.
And here's a zoomed version, showing only the significative part of the graph:
Here is the old zoomed plot, before error bars were corrected.
Back to Roberto's Listening Tests page