RE: NDCG

[Stephen Robertson <ser@microsoft.com>]

Ian,

A good source which implements the original NDCG is available on the LETOR site http://research.microsoft.com/users/tyliu/LETOR/.  You should be able to run the perl script at item 3 on your own data, to get a comparison.  Alternatively, I think there are also some results on that site.

You might like to note that the original version has the rather curious property that rank 2 gets no discount.  A version commonly used in Microsoft is adjusted to give a discount even on rank 2.  How flexible are you making your implementation?  Any gain function/discount function/truncation point?

Stephen


> -----Original Message-----
> From: ireval@nist.gov [mailto:ireval@nist.gov] On Behalf Of Ian Soboroff
> Sent: 20 November 2007 20:01
> To: Multiple recipients of list
> Subject: NDCG
>
>
>
> Hi, all... I'm working on an implementation of NDCG.  While my math
> looks right (to me at least), I'd love to test it on some real retrieval
> runs.  Unfortunately, I can't find any runs with corresponding NDCG
> scores.  (I thought NTCIR did this but their online proceedings don't
> report NDCG.)
>
> Alternatively, I'd appreciate seeing a known-good implementation so I
> can be sure of my results.
>
> Thanks in advance,
> Ian
>