Results of AAC at 128kbps public Listening Test

Notes by Roberto Amorim

with lots of help from ff123

These are the summary results of the AAC at 128kbit/s Public Listening test.

User comments are available here.

How to interpret the plots: Each plot is drawn with the five codecs on the x axis and the ratings given (1.0 through 5.0) on the y axis. N is the number of listeners used to compute the means (average ratings) and 95% confidence intervals. The mean rating given to each codec is indicted by the middle point of each vertical line segment, and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is different than the one used on ff123's 64kbps test. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.

One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the FloorEssence plot below, QT is rated better than Psytel with 95% confidence. And QT is rated better than Faac with greater than 95% confidence.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.

Another important note: The ATrain and Layla samples aren't mentioned because they were too "transparent" - people gave high scores to all codecs - therefore no significant conclusion can be drawn from them.

Plot	Comment
	Artist: Dave Matthews Band Title: #41 Album: Crash Submitted by: Filburt "Rock" Results: QT wins this test. Sor, Psytel and Nero are grouped together. Faac is by far the worst
	Artist: Cradle Of Filth Title: Beauty Slept In Sodom Submitted by: Frank Klemm Solo Harpsichord Results: QT a little better than Sor and Psytel, but tied to them in first place. Nero and Faac tied for second, with Nero in a slightly lower ranking than Faac.
	Artist: Opeth Title: Blackwater Park Album: Blackwater Park Quiet intro with acoustic and electric guitars followed by loud metal riffs. Results: QT wins. Sor and Psytel tied in second. Faac and Nero just behind..
	Artist: Two Lone Swordsmen Title: Death To All Culture Snitches Album: Tiny Reminders Submitted by: Dibrom IDM Results: Sor and QT tied at first place, Sor placed a little higher. Psytel and Nero share 3rd place. Faac loses very badly.
	Artist: Man With No Name Title: Floor-essence Submitted by: Karl Heyes Electronic mix Results: Qt wins again. Nero, Sor and Psytel are tied in 2nd place, with Nero a little higher. Faac closely follows the others in 3rd place
	Artist: Desultory Title: Life Shatters Album: Bitterness Track: 01 Submitted by: treech Metal, complete with screeching vocals Results: QT wins, as usual. Everything else is tied for second
	Artist: Mamas and Papas Title: Midnight Voyage Submitted by: Jan S. drums and bass in the far left, guitar in the far right. Female vocal in the center. Results: QT on first place, almost tied to Sor and Nero. Sor, Nero and Psytel tied at second. Faac at third, almost tied to Psytel.
	Artst: Tiamat Album: A Deeper Kind Of Slumber Track: 01 Submitted by: treech rock/metal riffs Results: Qt, Sor and Nero tied at first. Psytel and Faac tied at second.
	Artist: The Source feat. Candy Staton Track: You've Got the Love Album: ? Submitted by: Karl Heyes Drums and ride cymbals intro, followed by bass and female vocals. Results: QT and Psytel tied at first. Nero and Sor tied at second. Faac worst by far.
	Artist: Green Day Title: Waiting Album: Warning Intro consists of guitar in far left with male vocal in center. Results: QT, Psytel, Sor and Nero are tied, and perform quite bad on this sample. Faac is by far the worst

Overall Ratings: The results for each sample were grouped together, without modifications.

Then I performed an ANOVA analysis. The results are graphed below. QuickTime is a clear winner, performing much better than the competition. Sorenson Squeeze, Psytel AACenc and Nero are tied, with Nero slightly lower than the others. Faac is clearly the worst.

FAAC shouldn't be used for reasons other than testing at the moment. As this test demonstrates, it's a bad encoder. Thankfully, development and tuning by experts like Krzysztof Nikiel is increasing Faac's quality at a fast pace.
If you don't mind about illegality, I suggest using AACenc, since it has good quality and is free. If you care about that, I suggest using QuickTime or Nero, although they aren't free.
Sorenson is very good, but its price is prohibitive.

It's important to note that the QuickTime codec used in this test isn't the same one used in Apple iTunes. The samples were encoded in QuickTime's "Best" quality mode, while iTunes uses the "better" setting. This setting reportedly produces the same quality as "best" on 16-bit material. (Best is targeted at 24bit material)

Keep in mind This test was comparing CBR codecs only.* Some of these codecs (Psytel AACenc, Nero and Faac) have the hability of encoding in VBR mode too. Usually, VBR produces better quality than CBR, sometimes even surpassing CBR-only codecs like QuickTime.
*due to FAAC's nature, files encoded with Faac are ABR (Average Bitrate)
Here is a sample of how VBR codecs compare to QuickTime CBR. Thanks to Guruboolez for these results.

And here's a zoomed version, showing only the significative part of the graph:

Back to Roberto's Listening Tests page