Results of AAC at 128kbps public Listening Test

Notes by Roberto Amorim

with lots of help from ff123

 

These are the summary results of the AAC at 128kbit/s Public Listening test.

 

User comments are available here.

How to interpret the plots: Each plot is drawn with the five codecs on the x axis and the ratings given (1.0 through 5.0) on the y axis. N is the number of listeners used to compute the means (average ratings) and 95% confidence intervals. The mean rating given to each codec is indicted by the middle point of each vertical line segment, and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is different than the one used on ff123's 64kbps test. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.

One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the FloorEssence plot below, QT is rated better than Psytel with 95% confidence. And QT is rated better than Faac with greater than 95% confidence.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.

Another important note: The ATrain and Layla samples aren't mentioned because they were too "transparent" - people gave high scores to all codecs - therefore no significant conclusion can be drawn from them.

 

Plot Comment
Artist: Dave Matthews Band
Title: #41
Album: Crash
Submitted by: Filburt

"Rock"

Results: QT wins this test. Sor, Psytel and Nero are grouped together. Faac is by far the worst

Artist: Cradle Of Filth
Title: Beauty Slept In Sodom
Submitted by: Frank Klemm

Solo Harpsichord

Results: QT a little better than Sor and Psytel, but tied to them in first place. Nero and Faac tied for second, with Nero in a slightly lower ranking than Faac.

Artist: Opeth
Title: Blackwater Park
Album: Blackwater Park

Quiet intro with acoustic and electric guitars followed by loud metal riffs.

Results: QT wins. Sor and Psytel tied in second. Faac and Nero just behind..

Artist: Two Lone Swordsmen
Title: Death To All Culture Snitches
Album: Tiny Reminders
Submitted by: Dibrom

IDM

Results: Sor and QT tied at first place, Sor placed a little higher. Psytel and Nero share 3rd place. Faac loses very badly.

Artist: Man With No Name
Title: Floor-essence
Submitted by: Karl Heyes

Electronic mix

Results: Qt wins again. Nero, Sor and Psytel are tied in 2nd place, with Nero a little higher. Faac closely follows the others in 3rd place

Artist: Desultory
Title: Life Shatters
Album: Bitterness
Track: 01
Submitted by: treech

Metal, complete with screeching vocals

Results: QT wins, as usual. Everything else is tied for second

Artist: Mamas and Papas
Title: Midnight Voyage
Submitted by: Jan S.

drums and bass in the far left, guitar in the far right. Female vocal in the center.

Results: QT on first place, almost tied to Sor and Nero. Sor, Nero and Psytel tied at second. Faac at third, almost tied to Psytel.

Artst: Tiamat
Album: A Deeper Kind Of Slumber
Track: 01
Submitted by: treech

rock/metal riffs

Results: Qt, Sor and Nero tied at first. Psytel and Faac tied at second.

Artist: The Source feat. Candy Staton
Track: You've Got the Love
Album: ?
Submitted by: Karl Heyes

Drums and ride cymbals intro, followed by bass and female vocals.

Results: QT and Psytel tied at first. Nero and Sor tied at second. Faac worst by far.

Artist: Green Day
Title: Waiting
Album: Warning

Intro consists of guitar in far left with male vocal in center.

Results: QT, Psytel, Sor and Nero are tied, and perform quite bad on this sample. Faac is by far the worst

 

 

Overall Ratings: The results for each sample were grouped together, without modifications.

Then I performed an ANOVA analysis. The results are graphed below. QuickTime is a clear winner, performing much better than the competition. Sorenson Squeeze, Psytel AACenc and Nero are tied, with Nero slightly lower than the others. Faac is clearly the worst.

FAAC shouldn't be used for reasons other than testing at the moment. As this test demonstrates, it's a bad encoder. Thankfully, development and tuning by experts like Krzysztof Nikiel is increasing Faac's quality at a fast pace.
If you don't mind about illegality, I suggest using AACenc, since it has good quality and is free. If you care about that, I suggest using QuickTime or Nero, although they aren't free.
Sorenson is very good, but its price is prohibitive.

It's important to note that the QuickTime codec used in this test isn't the same one used in Apple iTunes. The samples were encoded in QuickTime's "Best" quality mode, while iTunes uses the "better" setting. This setting reportedly produces the same quality as "best" on 16-bit material. (Best is targeted at 24bit material)

Keep in mind This test was comparing CBR codecs only.* Some of these codecs (Psytel AACenc, Nero and Faac) have the hability of encoding in VBR mode too. Usually, VBR produces better quality than CBR, sometimes even surpassing CBR-only codecs like QuickTime.
*due to FAAC's nature, files encoded with Faac are ABR (Average Bitrate)
Here is a sample of how VBR codecs compare to QuickTime CBR. Thanks to Guruboolez for these results.

 

And here's a zoomed version, showing only the significative part of the graph:

Here is the old zoomed plot, before error bars were corrected.

Back to Roberto's Listening Tests page