Results of AAC at 128kbps v2 public Listening Test

Notes by Roberto Amorim

with lots of help from ff123

 

These are the summary results of the AAC at 128kbit/s v2 Public Listening test.

 

User comments are available here.

How to interpret the plots: Each plot is drawn with the five codecs on the x axis and the ratings given (1.0 through 5.0) on the y axis. N is the number of listeners used to compute the means (average ratings) and 95% confidence intervals. The mean rating given to each codec is indicted by the middle point of each vertical line segment, and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is different than the one used on ff123's 64kbps test. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.

One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the bodyheat plot below, iTunes is rated better than Nero with 95% confidence. And Faac is rated better than Compaact! with greater than 95% confidence.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.

Some other important notes:

  • Nero, FAAC and Compaact! were tested in VBR mode. Real and iTunes were tested in CBR mode.
  • iTunes is exactly the same as QuickTime in "Better" quality mode (which should give same results as "Best" on 16bit material). Both use the same encoding routines.
  •  

    Plot Comment
    Artist: Counting Crows
    Title: Big Yellow Taxi
    Album: CD single
    Submitted by: 2Bdecided

    Easy Listening

    Results: iTunes wins, with Nero closely behind it, more or less tied. Faac is tied with Nero, and Compaact! and Real are tied just below Faac.

    Artist: James Brown
    Title: Body Heat (live)
    Album: Greatest Hits
    Submitted by: tigre

    Soul

    Results: iTunes clearly is the best here, with Faac, Nero and Real coming tied behind it. Compaact! gets worst place.

    Artist: Daft Punk
    Title: DaFunk
    Album: Homework
    Submitted by: madah

    House (Electronic/Techno)

    Results: iTunes and Nero tied at first place, Real and Compaact! tied with Nero, and Faac tied to Real and Compaact!.

    Artist: Die Apokalyptischen Reiter
    Title: Gone
    Album: All you need is Love
    Submitted by: Dibrom

    Extreme Metal/NWOSDM

    Results: iTunes is the winner; Faac, Nero and Real are tied at second place. Compaact! comes in last place.

    Artist: Johannes Brahms
    Title: Danse Hongroise 6
    Performer: Kolacny Duo
    Album: Danses Hongroises
    Submitted by: guruboolez

    Piano Solo

    Results: In an interesting surprise, Compaact gets a first place almost breaking through the top of the graph. Real, Faac, iTunes and Nero are tied at second place, with Nero placed a little lower than the others..

    Artist: Gustav Mahler
    Title: Symphonie 3 fin
    Conductor: Pierre Boulez
    Submitted by: guruboolez

    Orchestral

    Results: iTunes is a clear first place. Nero, Faac and Compaact! are tied at second place. Real comes close behind, in last place.

    Artist: Always Outnumbered
    Title: My Blood Rusts
    Album: Always Outnumbered / Ransom Split
    Submitted by: dev0

    Emocore/Punkrock

    Results: Nero and iTunes are tied at first place, with Nero a little above iTunes. Faac is tied to iTunes. Real and Compaact! come tied at last place.

    Artist: They Might Be Giants
    Title: New York City
    Album: Factory Showroom
    Submitted by: phong

    Pop

    Results: Faac, iTunes and Nero are tied at first place. Faac and iTunes are almost perfectly tied, and Nero comes a little below. Real is in fourth place, and Compaact! comes in last place.

    Artst: Duran Duran
    Title: Ordinary World
    Album: Greatest
    Submitted by: rjamorim

    New Wave / Art Rock

    Results: iTunes comes at first place, followed closely by Nero in second place, which is followed closely by Faac in third place. Faac is followed closely by Real and Compaact!, in last place.

    Artst: Omara Portuondo
    Title: Quizas, Quizas, Quizas
    Album: Veinte Aņos
    Submitted by: tigre

    Latino (Salsa)

    Results: Nero and iTunes tied at first place, with Nero a little above iTunes. Real and Compaact are quite tied together with iTunes. Faac takes the last position this time

    Artist: Green Velvet
    Title: Coitus (remix)
    Album: Green Velvet
    Submitted by: Roel van den Berghe

    Electronic

    Results: Compaact! surprises again coming in first place. It is followed by iTunes and Nero, in second place. Real and Faac are at last place.

    Artist: Green Day
    Title: Waiting
    Album: Warning

    Intro consists of guitar in far left with male vocal in center.

    Results: iTunes, Real and Nero are tied at first place. Compaact! and Faac are at second place.



    This is the bitrate distribution table. These bitrates were obtained using foobar2000

    		Nero	Real	Faac	Comp!	iTunes
    BigYellow	136	128	142	131	128
    bodyheat	140	128	133	120	128
    DaFunk		141	128	126	130	128
    gone		131	128	127	120	128
    Hongroise	148	128	105	123	128
    Mahler		149	128	118	125	128
    mybloodrusts	149	128	135	115	128
    NewYorkCity	125	128	145	120	128
    OrdinaryWorld	133	128	145	120	128
    Quizas		136	128	122	141	128
    velvet		155	128	145	172	128
    Waiting		138	128	133	127	128
                    --------------------------------------
    Averages	140	128	131,3	128,6	128

     

    Overall Ratings: The results for each sample were grouped together, without modifications.

    Then I performed an ANOVA analysis. The results are graphed below. iTunes is more or less tied to Nero, although the very small overlapping of their error margins indicate iTunes is indeed better, although with smaller than 95% confidence. Real, Faac and Compaact! are tied at third place. (or third place if you consider Nero as the second place)

    This test showed some very interesting developments since the last AAC at 128kbps test. The most obvious one is surely Faac, that lost relatively badly before, and now became a very serious contender. The newcomers Real (actually CodingTechnologies) and Compaact! show lots of promise. Nero might also have been improved - it clearly lost to QuickTime/iTunes last time - although it can also be credit to the fact that Nero is using VBR, while iTunes is still CBR (it's actually impressive that a CBR-only encoder managed to be ranked so considerably better than VBR implementations, and that surely indicates the superior quality of Apple's encoder). It's probably the best choice for Mac and Windows users today, because it's highest quality, fast encoding, and the added advantage of being completely free

    It's worth mentioning that according to a key developer of the QuickTime/iTunes AAC encoder, this codec nearly didn't get tuned since the last test. Makes one wonder how much it can improve once the engineers start tuning it again and if a VBR mode is eventually implemented

    Anyway, it's safe to say that all codecs represented here are pretty mature and, no matter what your choice among them is, it's very likely you'll get very good results for your encodings.

     

    And here's a zoomed version, showing only the significative part of the graph:

    Here is the old zoomed plot, before error bars were corrected.

    Back to Roberto's Listening Tests page