Results of Multiformat at 128kbps public Listening Test

Notes by Roberto Amorim

with lots of help from ff123

These are the summary results of the Multiformat at 128kbit/s Public Listening test.

You are invited to subscribe to the Listening Test newsletter, where new tests being conduced and test results are announced.

User comments are available here.

 

How to interpret the plots: Each plot is drawn with the five codecs on the x axis and the ratings given (1.0 through 5.0) on the y axis. N is the number of listeners used to compute the means (average ratings) and 95% confidence intervals. The mean rating given to each codec is indicted by the middle point of each vertical line segment, and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is different than the one used on ff123's 64kbps test. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.

One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the chanchan plot below, Lame is rated better than Atrac3 with 95% confidence. And iTunes is rated better than Lame with greater than 95% confidence.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.

Some other important notes:

  • The Vorbis version tested here isn't the standard one offered by Xiph.org. Some vorbis enthusiasts, frustrated with Xiph's slow release schedule, decided to take matters on their hands and create code branches with better quality tunings. A listening test comparing branched versions to the standard Xiph version was conduced by Vorbis enthusiasts, and the winner was Aoyumi's aoTuV. That's why this unusual version is being featured here.
  • iTunes is exactly the same as QuickTime in "Better" quality mode (which should give same results as "Best" on 16bit material). Both use the same encoding routines.
  • iTunes is being tested at version 4.2 instead of 4.5, because of quality concerns raised by listeners
  • The custom Lame command line was chosen in a similar fashion - in a listening test conduced by enthusiasts.
  • The unusual quality settings for MPC and Vorbis were chosen after testing several qualities over a wide range of albums and styles, and picking the setting that generated results closer to 128kbps.
  •  

    Plot Comment
    Artist: Bela Bartok
    Title: String Quartet, No. 5 in B flat Major, Sz. 102, BB 110 - Allegro
    Performer: Juilliard String Quartet
    Album: The Six String Quartets
    Submitted by: harashin

    Classical - String Quartet

    Results: All codecs are tied

    Artist: Counting Crows
    Title: Big Yellow Taxi
    Album: CD single
    Submitted by: 2Bdecided

    Easy Listening

    Results: Vorbis aoTuV at first, tied to iTunes AAC and Lame. Lame and MPC are tied to iTunes. Atrac3 and WMA std. are tied to Lame and MPC.

    Artist: Charanga Habanera
    Title: Chan Chan
    Album: Various Artists - Salsa Timba
    Submitted by: tigre

    Latin

    Results: Vorbis aoTuV wins. MPC and iTunes AAC are tied at second place. Lame MP3 is at third place. WMA and Atrac3 share last place.

    Artist: Daft Punk
    Title: DaFunk
    Album: Homework
    Submitted by: madah

    House (Electronic/Techno)

    Results: Vorbis and MPC are tied at first place. Lame, iTunes and WMA are tied together and to MPC. Atrac3 loses.

    Artist: Claude Debussy
    Title: Une ForÍt
    Album: ?
    Submitted by: Guruboolez

    Classical with vocals

    Results: Vorbis breaks through the roof, in first place, tied to iTunes and Atrac3. Wma Std. is in second, tied to Lame, which is tied to MPC.

    Artist: Marvin Gaye
    Title: Let's Get It On
    Album: The Master compilation
    Submitted by: music_man_mpc

    Soul

    Results: Vorbis breaks through the roof again, in first place. MPC, Lame and iTunes are tied together at second place. Atrac3 and Wma std. share last place, more or less tied to Lame MP3.

    Artist: Die Apokalyptischen Reiter
    Title: Gone
    Album: All you need is Love
    Submitted by: Dibrom

    Extreme Metal/NWOSDM

    Results: Vorbis is tied to MPC and iTunes. Lame and WMA come just after, also tied to MPC and iTunes. Atrac3 loses.

    Artist: Johannes Brahms
    Title: Danse Hongroise 6
    Performer: Kolacny Duo
    Album: Danses Hongroises
    Submitted by: guruboolez

    Piano Solo

    Results: Vorbis aoTuV, MPC and Lame MP3 share first place. WMA, iTunes and Atrac3 share second (last) place..

    Artist: Portishead
    Title: It Could Be Sweet
    Album: Dummy
    Submitted by: ff123

    Orchestral

    Results: Vorbis, iTunes and WMA break through the roof sharing first place. MPC is at second place, followed closely by Lame in third place. Atrac3 loses again, badly.

    Artist: Kraftwerk
    Title: The Robots
    Album: The Man Machine
    Submitted by: ff123

    Electronic

    Results: MPC gets first place, Vorbis and iTunes share second place. Lame is on third place, more or less tied to WMA Std. Atrac3 loses badly. Quite interesting sample.

    Artist: Leahy
    Title: B Minor
    Album: ?
    Submitted by: Gecko

    Celtic

    Results: All codecs are tied

    Artist: Gustav Mahler
    Title: Symphonie 3 fin
    Conductor: Pierre Boulez
    Submitted by: guruboolez

    Orchestral

    Results: All codecs are tied

    Artist: They Might Be Giants
    Title: New York City
    Album: Factory Showroom
    Submitted by: phong

    Pop

    Results: Vorbis aoTuV, MPC and Lame share first place. iTunes is tied to MPC and Lame, but not to Vorbis. WMA is tied to AAC. And Atrac3 is tied to WMA.

    Artist: Duran Duran
    Title: Ordinary World
    Album: Greatest
    Submitted by: rjamorim

    New Wave / Art Rock

    Results: All codecs are tied

    Artist: Suzanne Vega
    Title: Rosemary
    Album: The Best of Suzanne Vega - Tried and True
    Submitted by: harashin

    Folk/Pop

    Results: Musepack is tied to Lame MP3, which is tied to Vorbis aoTuV, which is tied to iTunes AAC, which is tied to WMA Std, which is tied to Atrac3. Wheww.

    Artist: Give Up The Ghost
    Title: Since Always
    Album: We're Down 'Til We're Underground
    Submitted by: dav0

    Punkrock / Hardcore

    Results: Vorbis is at first place, more or less tied to MPC. WMA and Lame are tied to MPC. And iTunes and Atrac3 are tied to WMA and Lame

    Artist: The Harlem Gospel Singers
    Title: Put your trust in Jesus
    Album: Live at the Cologne Philharmonic Hall

    Gospel with applause

    Results: Vorbis, Lame and MPC are tied together. iTunes is tied to Vorbis. Atrac3 and WMA Std. are tied at last place.

    Artist: Green Day
    Title: Waiting
    Album: Warning

    Rock. Intro consists of guitar in far left with male vocal in center.

    Results: MPC wins by far. The rest is more or less tied together, except Atrac3 that can be considered losing to vorbis.



    This is the bitrate distribution table. These bitrates were obtained using foobar2000 and ff123's Python bitrate calculator

    			iTunes	MPC	Vorbis	Lame	WMA	Atrac3
    Bartok_strings2		128	153	148	146	126	132
    BigYellow		128	148	145	136	128	132
    chanchan		128	148	146	143	129	132
    DaFunk			128	135	124	142	128	132
    Debussy			128	98	120	108	129	132
    getiton			128	128	128	129	132	132
    gone			128	132	135	132	129	132
    Hongroise		128	104	128	117	129	132
    ItCouldBeSweet		128	91	110	93	128	132
    kraftwerk		128	152	135	141	127	132
    Leahy			128	155	149	133	128	132
    Mahler			128	146	134	144	129	132
    NewYorkCity		128	144	138	132	129	132
    OrdinaryWorld		128	153	146	142	128	132
    rosemary		128	135	136	130	128	132
    SinceAlways		128	146	123	153	128	132
    trust			128	144	154	146	131	132
    Waiting			128	153	144	148	131	132
    -----------------------------------------------------------------------
    Averages		128	136	135	134	128	132

     

    Overall Ratings: The results for each sample were grouped together, without modifications.

    Then I performed an ANOVA analysis. The results are graphed below. Vorbis aoTuV is tied to Musepack at first place, Lame MP3 is tied to iTunes AAC at second place, WMA Standard is in third place and Atrac3 gets last place.

    This test showed some very interesting developments since the last Multiformat at 128kbps test. Lame seems to have improved a lot, getting tied to it's technological successor AAC. Vorbis got much better, thanks to the independent tunings performed by Aoyumi. And Atrac3 surprised by it's bad performance.

     

    And here's a zoomed version, showing only the significative part of the graph:

    Back to Roberto's Listening Tests page