Results of Dial-up bitrate public Listening Test

Notes by Roberto Amorim

with lots of help from ff123

These are the summary results of the Dial-up bitrate Public Listening test.

You are invited to subscribe to the Listening Test newsletter, where new tests being conduced and test results are announced.

User comments are available here.

 

How to interpret the plots: Each plot is drawn with the five codecs on the x axis and the ratings given (1.0 through 5.0) on the y axis. N is the number of listeners used to compute the means (average ratings) and 95% confidence intervals. The mean rating given to each codec is indicted by the middle point of each vertical line segment, and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using Tukey Parametric HSD analysis) for each codec.

One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the chanchan plot below, Lame is rated better than Atrac3 with 95% confidence. And iTunes is rated better than Lame with greater than 95% confidence.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he/she prefers.

Some other important notes:

  • The Nero Digital codec featured here is actually HE AAC + Parametric Stereo, also known as aacPlus.
  • The built-in resampler in Oggenc wasn't used. It has been brought to the test conducer's attention that it doesn't work optimally compared to other resampling solutions (E.G, SSRC, used in this test).
    Thanks to dev0 for pointing this issue out.
  • The QDesign encoder used in this test is the demo Pro encoder available at ReallyRareWares from 1999.
  • It's worth reminding that Lame is the bottom anchor in this test, and the Lowpass at 7kHz is the top anchor
  •  

    Plot Comment
    Artist: Bela Bartok
    Title: String Quartet, No. 5 in B flat Major, Sz. 102, BB 110 - Allegro
    Performer: Juilliard String Quartet
    Album: The Six String Quartets
    Submitted by: harashin

    Classical - String Quartet

    Results: WMA and QDesign are tied at first place. Vorbis is at second place. Lowpass, Nero and Real are tied at third place, with Nero a little below the others. MP3pro is at 4th place, and Lame loses

    Artist: Charanga Habanera
    Title: Chan Chan
    Album: Various Artists - Salsa Timba
    Submitted by: tigre

    Latin

    Results: MP3pro, QDesign and Nero are tied at first place, with Nero a little below the others. Vorbis, WMA and Real seem to be all tied at 2nd place. Lowpass is at 3rd place, and Lame is at last place.

    Artist: Claude Debussy
    Title: Une ForÍt
    Album: ?
    Submitted by: Guruboolez

    Classical with vocals

    Results: WMA gets a virtual first place, with some codecs tying to the bottom of it's error margin. Everything else is more or less tied, except lame, that is at the bottom again.

    Artist: ?
    Title: Female speech sample
    Source: EBU SQAM material
    Submitted by: [proxima]

    speech

    Results: Nero wins alone here. WMA, Lowpass, MP3pro and Real are more or less tied at second place. Vorbis is a little tied to WMA, but I would give it 3rd place. Lame is at 4th place, and QDesign loses, starting a trend (the QDesign codec has always been known for terrible quality in human speech streams)

    Artist: Marvin Gaye
    Title: Let's Get It On
    Album: The Master compilation
    Submitted by: music_man_mpc

    Soul

    Results: Nero and MP3pro are perfectly tied at first place. QDesign and WMA are tied at second place. By their turns, Real and Vorbis are tied to WMA. Lowpass comes after Vorbis, and Lame loses.

    Artist: Johannes Brahms
    Title: Danse Hongroise 6
    Performer: Kolacny Duo
    Album: Danses Hongroises
    Submitted by: guruboolez

    Piano Solo

    Results: QDesign wins, with lowpass tied to it. Nero comes in second place, a little above (but tied to) Vorbis, WMA and MP3pro. Real loses to Nero, getting 3rd place. Lame is last again..

    Artist: Kraftwerk
    Title: The Robots
    Album: The Man Machine
    Submitted by: ff123

    Electronic

    Results: Lowpass, Nero and MP3pro are neatly tied at first place. WMA comes in second place, tied to Real, which is tied with QDesign, which is tied to Lame and Vorbis.

    Artist: Leahy
    Title: B Minor
    Album: ?
    Submitted by: Gecko

    Celtic

    Results: Nero, MP3pro and QDesign are tied at first place. WMA comes at second place. Vorbis, Lowpass and Real come at third place (although Real can be considered sharing WMA's second place). Lame loses.

    Artist: ?
    Title: Male speech sample
    Source: EBU SQAM material
    Submitted by: [proxima]

    speech

    Results: Nero at first, tied to lowpass, which is tied to Real, which is tied to MP3pro, which is tied to WMA, which is tied to Vorbis. Lame follows, and QDesign, not surprisingly, loses.

    Artist: Fatboy Slim
    Title: Ya Mama
    Album: Halfway Between the Gutter and the Stars
    Submitted by: Rafanetz

    Electronic (Acid House)

    Results: Nero and MP3pro are way ahead, sharing first place and leaving everyone else eating dust. The other codecs are more or less tied together except Real, that is winning over QDesign. Lame loses.

    Artist: They Might Be Giants
    Title: New York City
    Album: Factory Showroom
    Submitted by: phong

    Pop

    Results: Nero wins, MP3pro comes in second place, QDesign comes in third, a little tied to others. Vorbis, WMA and real are tied together at 4th place, and Lowpass is tied to Lame on last place.

    Artist: Duran Duran
    Title: Ordinary World
    Album: Greatest
    Submitted by: rjamorim

    New Wave / Art Rock

    Results: Nero wins. MP3pro comes in second tied to WMA and QDesign, which are a little below. Vorbis comes in third, more or less tied to Lowpass and Real. Lame loses.

    Artist: Suzanne Vega
    Title: Rosemary
    Album: The Best of Suzanne Vega - Tried and True
    Submitted by: harashin

    Folk/Pop

    Results: Nero and MP3pro are again sharing first place. There's a big messy tie at second place throwing Vorbis, Lowpass, Real and QDesign together. WMA gets third place, and Lame loses.

    Artist: Give Up The Ghost
    Title: Since Always
    Album: We're Down 'Til We're Underground
    Submitted by: dav0

    Punkrock / Hardcore

    Results: Nero and MP3pro shine on this one, getting, respectively, first and second place, far away from the other competing codecs. Vorbis, WMA, Real and QDesign are tied at third place, lowpass gets 4th, and Lame gets 5th.

    Artist: Suzanne Vega
    Title: Tom's Diner
    Album: Retrospective: The Best of Suzanne Vega
    Submitted by: harashin

    A Capella

    Results: Nero is at first place, MP3pro is at second place, WMA is at third, Lowpass and Real share fourth, Vorbis gets fifth, Lame gets sixth and QDesign gets seventh. But you can also say everything under MP3pro is tied, except QDesign.

    Artist: The Harlem Gospel Singers
    Title: Put your trust in Jesus
    Album: Live at the Cologne Philharmonic Hall Submitted by: tigre

    Gospel with applause

    Results: Probably due to the applause at the beginning, all codecs fail badly on this one. Lowpass shares first place with Nero. Everything else is tied at second place (although Lame seems to be losing here)

    Artist: Peter Heppner
    Title: Twelve
    Album: Die Flut
    Submitted by: LigH

    Darkwave

    Results: Nero is tied to WMA and MP3pro in first place. Next comes QDesign which is more or less tied to Vorbis, Lowpass and Real. Lame loses.

    Artist: Green Day
    Title: Waiting
    Album: Warning

    Rock. Intro consists of guitar in far left with male vocal in center.

    Results: MP3pro wins tied to Nero and Real. Then comes QDesign, more or less tied to Vorbis, WMA and Lowpass. Lame is tied to WMA.



    There's no bitrate distribution table for this test, since all codecs are being tested at 32kbps CBR

     

    Overall Ratings: The results for each sample were grouped together, without modifications.

    Then I performed a Tukey Parametric HSD analysis. The results are graphed below. Nero Digital Audio wins, tied to CodingTechnologies' MP3pro. Ogg Vorbis, WMA Std., 7kHz lowpass, Real Audio and QDesign Music Codec come in second place, with Vorbis a little below the others. Lame loses.

    There are some impressive findings in this test. For starters, how well codecs evolved since mid-2000, when EBU conduced their acclaimed MUSHRA formal listening test. You can compare the results of this test with that one using the 7kHz lowpass. Back then, the anchor clearly won over all other codecs. This time, it often lost to most of them in each individual sample.

    It's also interesting to notice what a wonderful technology is SBR, since it managed to bring the deprecated MP3 format to compare well against the state-of -the-art in audio coding.

    QDesign also was a pleasant surprise. Considering it's an encoder that hasn't been developed since mid-1999, it managed to compete very well against modern encoders

    Although Vorbis didn't perform well, almost getting under the big tie at second place, part of it can be credited to the fact that it was being tested at 22kHz. The severe lowpass at this bitrate seems to have introduced serious quality issues in the samples tested. One can hope Xiph enables 32kHz sampling for 32kbps, and further tunes such small bitrates.

     

    And here's a zoomed version, showing only the significative part of the graph, and without the anchors:

    It's also interesting to notice how the codecs score when only the music samples are taken into account (I.E, I ignored the samples female_speech, male_speech and TomsDiner). QDesign clearly performs better, demonstrating how badly tuned it is for human voice:

    Back to Roberto's Listening Tests page