Results of Dial-up bitrate (32kbit/s) Listening Test

Results of Dial-up bitrate public Listening Test

Notes by Roberto Amorim

with lots of help from ff123

These are the summary results of the Dial-up bitrate Public Listening test.

You are invited to subscribe to the Listening Test newsletter, where new tests being conduced and test results are announced.

User comments are available here.

How to interpret the plots: Each plot is drawn with the five codecs on the x axis and the ratings given (1.0 through 5.0) on the y axis. N is the number of listeners used to compute the means (average ratings) and 95% confidence intervals. The mean rating given to each codec is indicted by the middle point of each vertical line segment, and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using Tukey Parametric HSD analysis) for each codec.

One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the chanchan plot below, Lame is rated better than Atrac3 with 95% confidence. And iTunes is rated better than Lame with greater than 95% confidence.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he/she prefers.

Some other important notes:

The Nero Digital codec featured here is actually HE AAC + Parametric Stereo, also known as aacPlus.

The built-in resampler in Oggenc wasn't used. It has been brought to the test conducer's attention that it doesn't work optimally compared to other resampling solutions (E.G, SSRC, used in this test).
Thanks to dev0 for pointing this issue out.

The QDesign encoder used in this test is the demo Pro encoder available at ReallyRareWares from 1999.

It's worth reminding that Lame is the bottom anchor in this test, and the Lowpass at 7kHz is the top anchor

Plot	Comment
	Artist: Bela Bartok Title: String Quartet, No. 5 in B flat Major, Sz. 102, BB 110 - Allegro Performer: Juilliard String Quartet Album: The Six String Quartets Submitted by: harashin Classical - String Quartet Results: WMA and QDesign are tied at first place. Vorbis is at second place. Lowpass, Nero and Real are tied at third place, with Nero a little below the others. MP3pro is at 4th place, and Lame loses
	Artist: Charanga Habanera Title: Chan Chan Album: Various Artists - Salsa Timba Submitted by: tigre Latin Results: MP3pro, QDesign and Nero are tied at first place, with Nero a little below the others. Vorbis, WMA and Real seem to be all tied at 2nd place. Lowpass is at 3rd place, and Lame is at last place.
	Artist: Claude Debussy Title: Une Forêt Album: ? Submitted by: Guruboolez Classical with vocals Results: WMA gets a virtual first place, with some codecs tying to the bottom of it's error margin. Everything else is more or less tied, except lame, that is at the bottom again.
	Artist: ? Title: Female speech sample Source: EBU SQAM material Submitted by: [proxima] speech Results: Nero wins alone here. WMA, Lowpass, MP3pro and Real are more or less tied at second place. Vorbis is a little tied to WMA, but I would give it 3rd place. Lame is at 4th place, and QDesign loses, starting a trend (the QDesign codec has always been known for terrible quality in human speech streams)
	Artist: Marvin Gaye Title: Let's Get It On Album: The Master compilation Submitted by: music_man_mpc Soul Results: Nero and MP3pro are perfectly tied at first place. QDesign and WMA are tied at second place. By their turns, Real and Vorbis are tied to WMA. Lowpass comes after Vorbis, and Lame loses.
	Artist: Johannes Brahms Title: Danse Hongroise 6 Performer: Kolacny Duo Album: Danses Hongroises Submitted by: guruboolez Piano Solo Results: QDesign wins, with lowpass tied to it. Nero comes in second place, a little above (but tied to) Vorbis, WMA and MP3pro. Real loses to Nero, getting 3rd place. Lame is last again..
	Artist: Kraftwerk Title: The Robots Album: The Man Machine Submitted by: ff123 Electronic Results: Lowpass, Nero and MP3pro are neatly tied at first place. WMA comes in second place, tied to Real, which is tied with QDesign, which is tied to Lame and Vorbis.
	Artist: Leahy Title: B Minor Album: ? Submitted by: Gecko Celtic Results: Nero, MP3pro and QDesign are tied at first place. WMA comes at second place. Vorbis, Lowpass and Real come at third place (although Real can be considered sharing WMA's second place). Lame loses.
	Artist: ? Title: Male speech sample Source: EBU SQAM material Submitted by: [proxima] speech Results: Nero at first, tied to lowpass, which is tied to Real, which is tied to MP3pro, which is tied to WMA, which is tied to Vorbis. Lame follows, and QDesign, not surprisingly, loses.
	Artist: Fatboy Slim Title: Ya Mama Album: Halfway Between the Gutter and the Stars Submitted by: Rafanetz Electronic (Acid House) Results: Nero and MP3pro are way ahead, sharing first place and leaving everyone else eating dust. The other codecs are more or less tied together except Real, that is winning over QDesign. Lame loses.
	Artist: They Might Be Giants Title: New York City Album: Factory Showroom Submitted by: phong Pop Results: Nero wins, MP3pro comes in second place, QDesign comes in third, a little tied to others. Vorbis, WMA and real are tied together at 4th place, and Lowpass is tied to Lame on last place.
	Artist: Duran Duran Title: Ordinary World Album: Greatest Submitted by: rjamorim New Wave / Art Rock Results: Nero wins. MP3pro comes in second tied to WMA and QDesign, which are a little below. Vorbis comes in third, more or less tied to Lowpass and Real. Lame loses.
	Artist: Suzanne Vega Title: Rosemary Album: The Best of Suzanne Vega - Tried and True Submitted by: harashin Folk/Pop Results: Nero and MP3pro are again sharing first place. There's a big messy tie at second place throwing Vorbis, Lowpass, Real and QDesign together. WMA gets third place, and Lame loses.
	Artist: Give Up The Ghost Title: Since Always Album: We're Down 'Til We're Underground Submitted by: dav0 Punkrock / Hardcore Results: Nero and MP3pro shine on this one, getting, respectively, first and second place, far away from the other competing codecs. Vorbis, WMA, Real and QDesign are tied at third place, lowpass gets 4th, and Lame gets 5th.
	Artist: Suzanne Vega Title: Tom's Diner Album: Retrospective: The Best of Suzanne Vega Submitted by: harashin A Capella Results: Nero is at first place, MP3pro is at second place, WMA is at third, Lowpass and Real share fourth, Vorbis gets fifth, Lame gets sixth and QDesign gets seventh. But you can also say everything under MP3pro is tied, except QDesign.
	Artist: The Harlem Gospel Singers Title: Put your trust in Jesus Album: Live at the Cologne Philharmonic Hall Submitted by: tigre Gospel with applause Results: Probably due to the applause at the beginning, all codecs fail badly on this one. Lowpass shares first place with Nero. Everything else is tied at second place (although Lame seems to be losing here)
	Artist: Peter Heppner Title: Twelve Album: Die Flut Submitted by: LigH Darkwave Results: Nero is tied to WMA and MP3pro in first place. Next comes QDesign which is more or less tied to Vorbis, Lowpass and Real. Lame loses.
	Artist: Green Day Title: Waiting Album: Warning Rock. Intro consists of guitar in far left with male vocal in center. Results: MP3pro wins tied to Nero and Real. Then comes QDesign, more or less tied to Vorbis, WMA and Lowpass. Lame is tied to WMA.

There's no bitrate distribution table for this test, since all codecs are being tested at 32kbps CBR

Overall Ratings: The results for each sample were grouped together, without modifications.

Then I performed a Tukey Parametric HSD analysis. The results are graphed below. Nero Digital Audio wins, tied to CodingTechnologies' MP3pro. Ogg Vorbis, WMA Std., 7kHz lowpass, Real Audio and QDesign Music Codec come in second place, with Vorbis a little below the others. Lame loses.

There are some impressive findings in this test. For starters, how well codecs evolved since mid-2000, when EBU conduced their acclaimed MUSHRA formal listening test. You can compare the results of this test with that one using the 7kHz lowpass. Back then, the anchor clearly won over all other codecs. This time, it often lost to most of them in each individual sample.

It's also interesting to notice what a wonderful technology is SBR, since it managed to bring the deprecated MP3 format to compare well against the state-of -the-art in audio coding.

QDesign also was a pleasant surprise. Considering it's an encoder that hasn't been developed since mid-1999, it managed to compete very well against modern encoders

Although Vorbis didn't perform well, almost getting under the big tie at second place, part of it can be credited to the fact that it was being tested at 22kHz. The severe lowpass at this bitrate seems to have introduced serious quality issues in the samples tested. One can hope Xiph enables 32kHz sampling for 32kbps, and further tunes such small bitrates.

And here's a zoomed version, showing only the significative part of the graph, and without the anchors:

It's also interesting to notice how the codecs score when only the music samples are taken into account (I.E, I ignored the samples female_speech, male_speech and TomsDiner). QDesign clearly performs better, demonstrating how badly tuned it is for human voice:

Back to Roberto's Listening Tests page