Notes by Roberto Amorim
with lots of help from ff123
These are the summary results of the 64kbit/s Public Listening test.
User comments are available here.
A hint: It's very interesting to compare these test results with ff123's 64kbps test results, that were conduced a little more than an year ago. This way, it's possible to take conclusions about how some codecs developed since then.
How to interpret the plots: Each
plot is drawn with the five codecs on the x axis and the ratings
given (1.0 through 5.0) on the y axis. N is the number of
listeners used to compute the means (average ratings) and 95%
confidence intervals. The mean rating given to each codec is
indicted by the middle point of each vertical line segment, and
the value is printed next to it. Each vertical line segment
represents the 95% confidence interval (using ANOVA analysis)
for each codec.
This analysis is different than the one used on ff123's 64kbps test. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.
One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the BigYellow plot below, MP3pro is rated better than WMA with 95% confidence. And Ahead HE AAC is rated better than QT AAC with greater than 95% confidence.
It's worth mentioning that, while Ahead HE AAC, Vorbis, MP3pro and WMA were encoded in VBR mode, Real Audio and QuickTime were encoded in CBR mode since these codecs don't offer a VBR mode. Lame MP3 was encoded at ABR mode because that's how Lame performs better at this bitrate.
Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.
Another important note: Keep
in mind that Lame MP3 is at 128kbps in this test. There are two
reasons for that:
1-To put results into perpective. In this aspect, Lame was a high anchor (and FhG MP3 the low anchor)
2-To test if some of these codecs deliver the marketing plot of "Same quality as MP3 at half the bitrates", specially concerning WMA and MP3pro.
|Artist: Counting Crows
Title: Big Yellow Taxi
Album: CD single
Submitted by: 2Bdecided
Results: Lame on top, with Ahead HE AAC just close to it, followed by MP3pro and Vorbis. WMA is on 5th place, and Real and QuickTime tied at 6th/7th. FhG MP3 is last on this sample and most others.
|Artist: Daft Punk
Submitted by: madah
Results: Lame again in first place, with Vorbis on second and Ahead HE AAC tied to MP3pro at 3rd/4th. WMA is 5th and QuickTime 6th, followed by Real and then FhG MP3.
Title: Enola Gay
Submitted by: AngelGR
Results: Lame first, with Vorbis, WMA and MP3pro more or less tied following. Ahead HE AAC does pretty bad, tied to Real and QuickTime. FhG MP3 last.
Title: La voz de la experiencia (feat. Celia Cruz)
Album: Sobre el fuego
Results: Lame is the winner by far. MP3pro, Ahead HE AAC and Vorbis are tied, with HE AAC a little lower than the others. WMA on 5th place, QT AAC on 6th and Real on 7th. FhG MP3 last, again..
|Artist: Die Apokalyptischen Reiter
Album: All you need is Love
Submitted by: Dibrom
Results: Lame wins, with Ahead HE AAC and MP3pro tied at 2nd/3rd. Vorbis is at 4th, then QT AAC, and then WMA and Real tied. FhG MP3 loses.
|Artist: John Linnell
Album: State Songs
Submitted by: phong
Results: Lame first, Ahead HE AAC second. Then come MP3pro, Real, WMA and Vorbis more or less tied, with Vorbis quite lower than the others. Quick Time is a little above FhG, which loses again.
|Artist: Always Outnumbered
Title: My Blood Rusts
Album: Always Outnumbered / Ransom Split
Submitted by: dev0
Results: Evil sample. Best rankings start pretty low, with Lame tied to Ahead He AAC. Vorbis comes at third, followed by QuickTime and Real tied. WMA, MP3pro and FhG are tied at the bottom.
|Artist: They Might Be Giants
Title: New York City
Album: Factory Showroom
Submitted by: phong
Results: Lame wins. Ahead HE AAC is tied to MP3pro in 2nd/3rd place. Vorbis is tied to Real in 4th/5th place. WMA is tied to QT AAC in 6th/7th place. FhG loses.
|Artist: Friderick Chopin
Title: Polonaise in D moll Op. 71, No. 1
Submitted by: AstralStorm
No, I didn't mess up the results. Believe it or not, Lame is at first, Ahead HE AAC second, QT AAC third, Real and FhG MP3 tied slighly above MP3pro (!), WMA is tied to Vorbis at last place.
|Artst: Igor Stravinsky
Title: Le Sacre Du Printemps
Album: L'Adoration de la Terre
Submitted by: ff123
Results: Lame wins, by far. Ahead HE AAC is in second place, followed by MP3pro. Real and Vorbis are tied at 4th/5th place, WMA and QT come tied just after, and FhG loses badly.
|Artist: Yasunori Mitsuda
Title: Scars Left by Time
Album: Chrono Cross Original Soundtrack
Submitted by: rjamorim
Heavy orchestral with drums
Results: Lame wins again, Ahead HE AAC
is tied to MP3pro at 2nd/3rd place, with MP3pro a little lower.
|Artist: Green Day
Intro consists of guitar in far left with male vocal in center.
Results: The traditional evil sample. Lame starts badly at 3.75, followed by Ahead HE AAC tied to MP3pro. Real, WMA, QT and Vorbis follow in a big tie, with Real a little higher than the others and statistically better than Vorbis. FhG is last.
This is the bitrate distribution table. These bitrates were obtained using foobar2000 and EncSpot.
MP3pro Real WMA FhGMP3 Lame HE AAC QT AAC Vorbis BigYellow 75 64 79 64 126 64 64 69 DaFunk 67 64 75 64 127 64 64 64 EnolaGay 68 64 69 64 125 61 64 59 experiencia 71 64 64 64 131 60 64 67 gone 69 64 66 64 125 62 64 63 Illinois 60 64 65 64 122 69 64 58 mybloodrusts 78 64 64 64 123 67 64 60 NewYorkCity 76 64 79 64 124 63 64 63 Polonaise 58 64 46 64 122 72 64 51 riteofspring 67 64 84 64 123 70 64 61 Scars 66 64 88 64 125 68 64 62 Waiting 72 64 78 64 121 72 64 63 -------------------------------------------------------------- Means 68.9 64 71.4 64 124.5 66 64 61.6
Overall Ratings: The results for each sample were grouped together, without modifications.
Then I performed an ANOVA analysis. The results are graphed below.
The first (obvious) conclusion is: No codec delivers
the marketing plot of same quality as MP3 at half the bitrates.
Lame MP3 at 128kbps wins, followed by Ahead/Nero HE AAC on 2nd place, CodingTechnologies' MP3pro on 3rd place, Ogg Vorbis on 4th place, Real Audio, QuickTime AAC and WMA9 tied near the middle of the graph, and FhG MP3 definitely at the bottom.
Here's a zoomed version showing only the most significative part of the graph, leaving out the anchors:
Here is the old zoomed plot, before error bars were corrected.
Back to Roberto's Listening Tests page