Results of the public multiformat listening test @ 48 kbps (November 2006)

These are the summary results of the public multiformat listening test @ 48 kbps.

User comments are available here. If your packing utility supports RAR archives, you can also download a signed, locked and solid RAR file containing all results for all samples.

Encryption key can be downloaded from here.

How to interpret the plots: Each plot is drawn with six codecs on the X axis and the rating given (1.0 to 5.0) on the Y axis. The number of listeners used to compute the means (average ratings) and 95% confidence intervals are given on each plot. The mean rating given to each codec is indicated by the middle point of each vertical line segment and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is identical to the one used in Roberto Amorim's listening tests.

One codec can be said to be better than another with 95% confidence if the bottom of its segment is at or above the top of the competing codec's line segment. For example, in the Locomotive_Breath plot below, WMA Professional 10 is rated better than WMA Standard 9.2 with 95% confidence.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.

Plot	Comments
	Flute, some piano, cymbals, some electric guitar High anchor on top, followed by Vorbis, Nero and WMA Professional on second place. WMA Standard is third followed by the low anchor.
	Pretty loud mixture of symphony orchestra with percussion and electric guitar High anchor on top, very close to Vorbis, Nero and WMA Professional which are tied on second place. WMA Standard comes out on third place again followed by the low anchor.
	Low volume High anchor tied with Nero and WMA Professional, Vorbis tied with Nero and WMA Professional (but not with high anchor), WMA Standard tied with low anchor.
	Rap, strong English male voice Nero and high anchor tied on first place, Vorbis and WMA Pro tied on second place, WMA Standard comes out third, low anchor is last.
	English female voice All contenders tied on second place between high anchor and low anchor which are #1 and #3. WMA Standard is a very little bit worse than Nero, though.
	Stereo separation, electric guitar High anchor is first, followed by Nero which is tied with Vorbis and WMA Professional on second place. WMA Standard is third, low anchor is last.
	Electronic, pre-echo High anchor is clearly at top. Nero, WMA Standard and WMA Professional are tied on second place, Vorbis is third very close to WMA Pro and very close to the low anchor.
	Symphonic orchestra High anchor is first, followed by Nero and Vorbis which are tied on #2. The two WMA codecs are tied on third place followed by the low anchor which loses.
	80's electronic, pre-echo High anchor is first, Nero and the two WMA codecs are tied on #2, Vorbis is third followed by the low anchor.
	Bandwidth test High anchor is again first, followed by Vorbis which is tied with WMA Standard which is tied with Nero which is tied with WMA Pro which is tied with low anchor. Confused, what? :P
	Usual pop / trance High anchor is as always on #1, Nero is 2nd followed by Vorbis and WMA Pro which are tied on #3. WMA Standard is fourth, low anchor loses.
	High tones, sort of Indie music Nero is tied with high anchor on #1, Vorbis and WMA Pro are tied on #2, WMA Standard is third, low anchor loses.
	Simply weird :P High anchor is first, Vorbis, Nero and WMA Standard are tied on second place, WMA Pro is tied with WMA Standard (somewhat worse than Nero and Vorbis) and also tied with the low anchor.
	Jazz, French female voice High anchor is again first, Vorbis, Nero and WMA Pro are tied on #2, WMA Standard is third and low anchor loses once more.
	Classical, very dynamic Same picture as above.
	German male voice High anchor wins, Nero, the two WMA codecs and the low anchor are tied on second place, Vorbis loses!
	Instrumental (harpsichord) High anchor is on #1 followed by Nero on #2. Vorbis and WMA Pro are tied on #3, WMA Standard is tied with WMA Pro, but worse than Vorbis. Low anchor is last.
	Acoustic guitar High anchor is first, Nero is second, Vorbis and the two WMAs are third, low anchor is fourth.
	English male voice, easy listening / pop High anchor is first, followed by Nero on #2. WMA Pro is tied with Vorbis on #3, WMA Standard is on #4, low anchor is last.
	Stereo separation, violins For the last time, high anchor is first, Vorbis, Nero and WMA Pro are second, WMA Standard is third tied with the low anchor.

These are the bitrates used:

    Sample (Duration in Seconds)        AoTuV        Nero        WMA Standard        WMA Professional
    -------------------------------------------------------------------------------------------------
    Locomotive_Breath (43)              47           50          41                  48
    symphnoy_metal (29)                 50           54          47                  48
    debussy (30)                        44           42          30                  48
    WhiteAmerica (30)                   49           50          37                  48
    TomsDiner (19)                      47           35          39                  48
    Gypsy (30)                          53           49          64                  48
    eig (15)                            58           51          60                  48
    macabre (17)                        45           46          43                  48
    kraftwerk (29)                      51           45          60                  48
    bibilolo (25)                       48           48          32                  48
    MysteriousTimes (28)                56           52          57                  48
    Flyin___to_Fly (27)                 56           47          42                  48
    aquatisme (30)                      64           51          73                  48
    Senor (17)                          50           46          52                  48
    Paganini_Allegro_spirituoso (30)    47           43          40                  48
    spmg54_1 (16)                       45           31          36                  48
    LesJoursHeureux (20)                56           49          44                  48
    The_Wizard (28)                     52           44          53                  48
    BigYellow (24)                      51           52          51                  48
    Eleanor_Rigby (29)                  46           46          46                  48
    -------------------------------------------------------------------------------------------------
    Average: 25.8                       50.75        46.55       47.35               48
    

Overall rating: The results for each sample were grouped together without modifications.

Then I performed an ANOVA analysis. The results are graphed below.

The high anchor iTunes LC-AAC at 96 kbps is first, Nero follows on #2, Vorbis and WMA Professional are tied on #3, WMA Standard is on #4 and the low anchor iTunes LC-AAC at 48 kbps loses.

I think this test shows that in general, with modern encoders, the quality at 48 kbps is acceptable and should be good enough for Internet streaming or portable use with cell phones for example. However, one should also notice that the quality is very dependent on the input material and varies a lot from one sample to another. Also, many think that speech is very easy to encode, but the German male voice sample shows the contradictory. It's also interesting to see that WMA Professional performed quite well although it was the only contender that used CBR.

Here is a zoomed version of the plot showing the competitors only and leaving out the anchors.

Finally, I would like to thank everyone who participated!