Results of Multiformat at 128kbps public Listening Test

Notes by Roberto Amorim

with lots of help from ff123

These are the summary results of the Multiformat at 128kbit/s Public Listening test.

You are invited to subscribe to the Listening Test newsletter, where new tests being conduced and test results are announced.

User comments are available here.

How to interpret the plots: Each plot is drawn with the five codecs on the x axis and the ratings given (1.0 through 5.0) on the y axis. N is the number of listeners used to compute the means (average ratings) and 95% confidence intervals. The mean rating given to each codec is indicted by the middle point of each vertical line segment, and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is different than the one used on ff123's 64kbps test. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.

One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the chanchan plot below, Lame is rated better than Atrac3 with 95% confidence. And iTunes is rated better than Lame with greater than 95% confidence.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.

Some other important notes:

Plot	Comment
	Artist: Bela Bartok Title: String Quartet, No. 5 in B flat Major, Sz. 102, BB 110 - Allegro Performer: Juilliard String Quartet Album: The Six String Quartets Submitted by: harashin Classical - String Quartet Results: All codecs are tied
	Artist: Counting Crows Title: Big Yellow Taxi Album: CD single Submitted by: 2Bdecided Easy Listening Results: Vorbis aoTuV at first, tied to iTunes AAC and Lame. Lame and MPC are tied to iTunes. Atrac3 and WMA std. are tied to Lame and MPC.
	Artist: Charanga Habanera Title: Chan Chan Album: Various Artists - Salsa Timba Submitted by: tigre Latin Results: Vorbis aoTuV wins. MPC and iTunes AAC are tied at second place. Lame MP3 is at third place. WMA and Atrac3 share last place.
	Artist: Daft Punk Title: DaFunk Album: Homework Submitted by: madah House (Electronic/Techno) Results: Vorbis and MPC are tied at first place. Lame, iTunes and WMA are tied together and to MPC. Atrac3 loses.
	Artist: Claude Debussy Title: Une Forêt Album: ? Submitted by: Guruboolez Classical with vocals Results: Vorbis breaks through the roof, in first place, tied to iTunes and Atrac3. Wma Std. is in second, tied to Lame, which is tied to MPC.
	Artist: Marvin Gaye Title: Let's Get It On Album: The Master compilation Submitted by: music_man_mpc Soul Results: Vorbis breaks through the roof again, in first place. MPC, Lame and iTunes are tied together at second place. Atrac3 and Wma std. share last place, more or less tied to Lame MP3.
	Artist: Die Apokalyptischen Reiter Title: Gone Album: All you need is Love Submitted by: Dibrom Extreme Metal/NWOSDM Results: Vorbis is tied to MPC and iTunes. Lame and WMA come just after, also tied to MPC and iTunes. Atrac3 loses.
	Artist: Johannes Brahms Title: Danse Hongroise 6 Performer: Kolacny Duo Album: Danses Hongroises Submitted by: guruboolez Piano Solo Results: Vorbis aoTuV, MPC and Lame MP3 share first place. WMA, iTunes and Atrac3 share second (last) place..
	Artist: Portishead Title: It Could Be Sweet Album: Dummy Submitted by: ff123 Orchestral Results: Vorbis, iTunes and WMA break through the roof sharing first place. MPC is at second place, followed closely by Lame in third place. Atrac3 loses again, badly.
	Artist: Kraftwerk Title: The Robots Album: The Man Machine Submitted by: ff123 Electronic Results: MPC gets first place, Vorbis and iTunes share second place. Lame is on third place, more or less tied to WMA Std. Atrac3 loses badly. Quite interesting sample.
	Artist: Leahy Title: B Minor Album: ? Submitted by: Gecko Celtic Results: All codecs are tied
	Artist: Gustav Mahler Title: Symphonie 3 fin Conductor: Pierre Boulez Submitted by: guruboolez Orchestral Results: All codecs are tied
	Artist: They Might Be Giants Title: New York City Album: Factory Showroom Submitted by: phong Pop Results: Vorbis aoTuV, MPC and Lame share first place. iTunes is tied to MPC and Lame, but not to Vorbis. WMA is tied to AAC. And Atrac3 is tied to WMA.
	Artist: Duran Duran Title: Ordinary World Album: Greatest Submitted by: rjamorim New Wave / Art Rock Results: All codecs are tied
	Artist: Suzanne Vega Title: Rosemary Album: The Best of Suzanne Vega - Tried and True Submitted by: harashin Folk/Pop Results: Musepack is tied to Lame MP3, which is tied to Vorbis aoTuV, which is tied to iTunes AAC, which is tied to WMA Std, which is tied to Atrac3. Wheww.
	Artist: Give Up The Ghost Title: Since Always Album: We're Down 'Til We're Underground Submitted by: dav0 Punkrock / Hardcore Results: Vorbis is at first place, more or less tied to MPC. WMA and Lame are tied to MPC. And iTunes and Atrac3 are tied to WMA and Lame
	Artist: The Harlem Gospel Singers Title: Put your trust in Jesus Album: Live at the Cologne Philharmonic Hall Gospel with applause Results: Vorbis, Lame and MPC are tied together. iTunes is tied to Vorbis. Atrac3 and WMA Std. are tied at last place.
	Artist: Green Day Title: Waiting Album: Warning Rock. Intro consists of guitar in far left with male vocal in center. Results: MPC wins by far. The rest is more or less tied together, except Atrac3 that can be considered losing to vorbis.

This is the bitrate distribution table. These bitrates were obtained using foobar2000 and ff123's Python bitrate calculator

			iTunes	MPC	Vorbis	Lame	WMA	Atrac3
Bartok_strings2		128	153	148	146	126	132
BigYellow		128	148	145	136	128	132
chanchan		128	148	146	143	129	132
DaFunk			128	135	124	142	128	132
Debussy			128	98	120	108	129	132
getiton			128	128	128	129	132	132
gone			128	132	135	132	129	132
Hongroise		128	104	128	117	129	132
ItCouldBeSweet		128	91	110	93	128	132
kraftwerk		128	152	135	141	127	132
Leahy			128	155	149	133	128	132
Mahler			128	146	134	144	129	132
NewYorkCity		128	144	138	132	129	132
OrdinaryWorld		128	153	146	142	128	132
rosemary		128	135	136	130	128	132
SinceAlways		128	146	123	153	128	132
trust			128	144	154	146	131	132
Waiting			128	153	144	148	131	132
-----------------------------------------------------------------------
Averages		128	136	135	134	128	132

Overall Ratings: The results for each sample were grouped together, without modifications.

Then I performed an ANOVA analysis. The results are graphed below. Vorbis aoTuV is tied to Musepack at first place, Lame MP3 is tied to iTunes AAC at second place, WMA Standard is in third place and Atrac3 gets last place.

This test showed some very interesting developments since the last Multiformat at 128kbps test. Lame seems to have improved a lot, getting tied to it's technological successor AAC. Vorbis got much better, thanks to the independent tunings performed by Aoyumi. And Atrac3 surprised by it's bad performance.

And here's a zoomed version, showing only the significative part of the graph:

Back to Roberto's Listening Tests page