Notes by Roberto Amorim
with lots of help from ff123
These are the summary results of the AAC at 128kbit/s Public Listening test.
User comments are available here.
How to interpret the plots: Each
plot is drawn with the five codecs on the x axis and the ratings
given (1.0 through 5.0) on the y axis. N is the number of
listeners used to compute the means (average ratings) and 95%
confidence intervals. The mean rating given to each codec is
indicted by the middle point of each vertical line segment, and
the value is printed next to it. Each vertical line segment
represents the 95% confidence interval (using ANOVA analysis)
for each codec.
This analysis is different than the one used on ff123's 64kbps test. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.
One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the FloorEssence plot below, QT is rated better than Psytel with 95% confidence. And QT is rated better than Faac with greater than 95% confidence.
Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.
Another important note: The ATrain and Layla samples aren't mentioned because they were too "transparent" - people gave high scores to all codecs - therefore no significant conclusion can be drawn from them.
|Artist: Dave Matthews Band
Submitted by: Filburt
Results: QT wins this test. Sor, Psytel and Nero are grouped together. Faac is by far the worst
|Artist: Cradle Of Filth
Title: Beauty Slept In Sodom
Submitted by: Frank Klemm
Results: QT a little better than Sor and Psytel, but tied to them in first place. Nero and Faac tied for second, with Nero in a slightly lower ranking than Faac.
Title: Blackwater Park
Album: Blackwater Park
Quiet intro with acoustic and electric guitars followed by loud metal riffs.
Results: QT wins. Sor and Psytel tied in second. Faac and Nero just behind..
|Artist: Two Lone Swordsmen
Title: Death To All Culture Snitches
Album: Tiny Reminders
Submitted by: Dibrom
Results: Sor and QT tied at first place, Sor placed a little higher. Psytel and Nero share 3rd place. Faac loses very badly.
|Artist: Man With No Name
Submitted by: Karl Heyes
Results: Qt wins again. Nero, Sor and Psytel are tied in 2nd place, with Nero a little higher. Faac closely follows the others in 3rd place
Title: Life Shatters
Submitted by: treech
Metal, complete with screeching vocals
Results: QT wins, as usual. Everything else is tied for second
|Artist: Mamas and Papas
Title: Midnight Voyage
Submitted by: Jan S.
drums and bass in the far left, guitar in the far right. Female vocal in the center.
Results: QT on first place, almost tied to Sor and Nero. Sor, Nero and Psytel tied at second. Faac at third, almost tied to Psytel.
Album: A Deeper Kind Of Slumber
Submitted by: treech
Results: Qt, Sor and Nero tied at first. Psytel and Faac tied at second.
|Artist: The Source feat. Candy Staton
Track: You've Got the Love
Submitted by: Karl Heyes
Drums and ride cymbals intro, followed by bass and female vocals.
Results: QT and Psytel tied at first. Nero and Sor tied at second. Faac worst by far.
|Artist: Green Day
Intro consists of guitar in far left with male vocal in center.
Results: QT, Psytel, Sor and Nero are tied, and perform quite bad on this sample. Faac is by far the worst
Overall Ratings: The results for each sample were grouped together, without modifications.
Then I performed an ANOVA analysis. The results are graphed below. QuickTime is a clear winner, performing much better than the competition. Sorenson Squeeze, Psytel AACenc and Nero are tied, with Nero slightly lower than the others. Faac is clearly the worst.
FAAC shouldn't be used for reasons other than testing at the
moment. As this test demonstrates, it's a bad encoder. Thankfully,
development and tuning by experts like Krzysztof Nikiel is
increasing Faac's quality at a fast pace.
If you don't mind about illegality, I suggest using AACenc, since it has good quality and is free. If you care about that, I suggest using QuickTime or Nero, although they aren't free.
Sorenson is very good, but its price is prohibitive.
It's important to note that the QuickTime codec used in this test isn't the same one used in Apple iTunes. The samples were encoded in QuickTime's "Best" quality mode, while iTunes uses the "better" setting. This setting reportedly produces the same quality as "best" on 16-bit material. (Best is targeted at 24bit material)
Keep in mind This test was comparing CBR codecs only.*
Some of these codecs (Psytel AACenc, Nero and Faac) have the
hability of encoding in VBR mode too. Usually, VBR produces
better quality than CBR, sometimes even surpassing CBR-only
codecs like QuickTime.
*due to FAAC's nature, files encoded with Faac are ABR (Average Bitrate)
Here is a sample of how VBR codecs compare to QuickTime CBR. Thanks to Guruboolez for these results.
And here's a zoomed version, showing only the significative part of the graph:
Here is the old zoomed plot, before error bars were corrected.
Back to Roberto's Listening Tests page