Re: NDCG

[Ian Soboroff <ian.soboroff@nist.gov>]

Stephen Robertson <ser@microsoft.com> writes:

> You might like to note that the original version has the rather
> curious property that rank 2 gets no discount.  A version commonly
> used in Microsoft is adjusted to give a discount even on rank 2.  How
> flexible are you making your implementation?  Any gain
> function/discount function/truncation point?

A few people are using such a variant that discounts by log(rank + 1).
I like this approach better myself, but at this point I want to make
sure I've caught all the bugs and to do that I want to make sure I match
the specification of the measure in the ToIS article.

My implementation is based on trec_eval, and will allow specifying gain
values per relevance level.  I think I will leave the log base as 2, as
I have not seen much research that shows much sensitivity in that
parameter.  Truncation will be as usual in trec_eval (using the -M
option).

I don't know if this will make it into the official trec_eval, but if
not I'll probably post my patch someplace.

Ian