Results of MP3 at 128kbit/s Listening Test

Results of MP3 at 128kbps public Listening Test

Notes by Roberto Amorim

with lots of help from ff123

These are the summary results of the MP3 at 128kbit/s Public Listening test.

User comments are available here.

I would like to point out two very serious issues with this test: not using the latest version of Xing, bundled with Real Player, that has been reportedly extensively tuned since version 1.5; and forcing VBR on codecs that shouldn't be using them. I'm confident iTunes MP3 would perform better if it was featured at CBR 128, and the same might apply to FhG. I take full responsability on those mistakes, and for them, I apologize.

How to interpret the plots: Each plot is drawn with the five codecs on the x axis and the ratings given (1.0 through 5.0) on the y axis. N is the number of listeners used to compute the means (average ratings) and 95% confidence intervals. The mean rating given to each codec is indicted by the middle point of each vertical line segment, and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is different than the one used on ff123's 64kbps test. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.

One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the DaFunk plot below, AudioActive is rated better than Gogo with 95% confidence. And AudioActive is rated better than FhG with greater than 95% confidence.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.

Some other important notes:

All codecs featured here were tested in their VBR modes, except AudioActive, because it only features CBR.

The Xing codec tested here is the last version available from XingTech. It was later bought by RealNetworks and underwent further development. The latest version - which wasn't tested here - is available with RealPlayer 10/RealOne

ATTENTION: the "iTunes" codec featured here is the MP3 encoder featured in iTunes - NOT the AAC encoder.

Plot	Comment
	Artist: Counting Crows Title: Big Yellow Taxi Album: CD single Submitted by: 2Bdecided Easy Listening Results: Lame, FhG, Gogo, AudioActive and Xing are tied. iTunes loses quite badly.
	Artist: Daft Punk Title: DaFunk Album: Homework Submitted by: madah House (Electronic/Techno) Results: Lame and AudioActive tied at first place. Gogo and Xing tied at second place. FhG and iTunes tied at third & last place.
	Artist: OMD Title: Enola Gay Album: Organisation Submitted by: AngelGR New Wave Results: Lame seems tied to FhG and Gogo, while AudioActive and Xing seem tied at 2nd place. iTunes loses, although it's quite close to the second place.
	Artist: India Title: La voz de la experiencia (feat. Celia Cruz) Album: Sobre el fuego Salsa Results: AudioActive wins, followed closely by Lame. Everything else is tied at the bottom. .
	Artist: Die Apokalyptischen Reiter Title: Gone Album: All you need is Love Submitted by: Dibrom Extreme Metal/NWOSDM Results: In this sample, Lame wins by far. FhG comes at second place, close to the mess at the bottom where all other encoders are (with Gogo a little above).
	Artist: John Linnell Title: Illinois Album: State Songs Submitted by: phong organ Results: Lame, Gogo and Xing are tied at first place, with Gogo a little below the others. iTunes and AudioActive are tied at second place. FhG surprisingly is at the bottom.
	Artist: Always Outnumbered Title: My Blood Rusts Album: Always Outnumbered / Ransom Split Submitted by: dev0 Emocore/Punkrock Results: FhG wins. AudioActive is more or less tied with iTunes, which is more or less tied with Lame, which, by its turn, is more or less tied with Xing. Gogo loses.
	Artist: They Might Be Giants Title: New York City Album: Factory Showroom Submitted by: phong Pop Results: Lame and AudioActive are quite tied, followed by FhG and Gogo, which also seem tied. Xing is second worst and iTunes loses again.
	Artist: Friderick Chopin Title: Polonaise in D moll Op. 71, No. 1 Submitted by: AstralStorm Piano solo Results: Lame and AudioActive are tied at first place, followed closely by iTunes (!). Gogo comes at third, FhG at fourth and Xing fails badly.
	Artst: Igor Stravinsky Title: Le Sacre Du Printemps Album: L'Adoration de la Terre Submitted by: ff123 Orchestral Results: AudioActive performs very well. Lame comes at second place, FhG, iTunes and Xing are tied at third. Gogo fails very badly.
	Artist: Yasunori Mitsuda Title: Scars Left by Time Album: Chrono Cross Original Soundtrack Submitted by: rjamorim Heavy orchestral with drums Results: Lame is more or less tied to AudioActive at first place, and AudioActive is more or less tied to Xing. The rest is at the bottom.
	Artist: Green Day Title: Waiting Album: Warning Intro consists of guitar in far left with male vocal in center. Results: Waiting wreaks havoc again. Xing is the lone winner. Lame, FhG, Gogo and AudioActive are tied at a shameful second place much below. iTunes gets close to AudioActive, but loses neverthless.

This is the bitrate distribution table. These bitrates were obtained using foobar2000

		Xing	Lame	iTunes	Gogo	FhG	AActive
BigYellow	140	132	126	132	135	128
DaFunk		128	138	120	134	120	128
EnolaGay	131	126	120	123	123	128
experiencia	121	133	124	134	127	128
gone		117	125	117	122	120	128
Illinois	151	123	119	123	103	128
mybloodrusts	129	121	117	126	137	128
NewYorkCity	133	124	120	129	135	128
Polonaise	84	122	114	114	96	128
riteofspring	123	122	119	120	109	128
Scars		136	125	123	124	115	128
Waiting		138	122	119	124	126	128
                -----------------------------------------------
Means		127,5	126,0	119,8	125,4	120,5	128

Overall Ratings: The results for each sample were grouped together, without modifications.

Then I performed an ANOVA analysis. The results are graphed below. Lame wins, followed by AudioActive, which is more or less followed by Xing, FhG and Gogo. Although iTunes is a little tied with Gogo, it's safe to say it lost.

While there was no clear winner, it's interesting to notice how well AudioActive performed, even being restrained by CBR encoding. Also, it seems the constant criticism directed at Xing is exaggerated at best.

And here's a zoomed version, showing only the significative part of the graph:

Back to Roberto's Listening Tests page