Re: NDCG

["kando@nii.ac.jp" <kando@nii.ac.jp>]

Hi Ian,

NTCIR-6 CLIR used nDCG and other metrics for graded relevance
judgments like Q-measure, sliding ratio, modified sliding ratio,
generalized average precision. The scores of the top ranked runs
are included in the CLIR task overview  in the proceedings.

Submitted runs and actual scores of the above mentioned metrics
will be available for the purpose of IR evaluation research.
(Currently the user agreement form is under review by the IP Section,
and it won't take so long, I hope.)

For nDCG,  the log base that we used is b = 2 and  the gains for
highly relevant, relevant, partial relevant and irrelevant are
3,2,1, 0 respectively.

Noriko

Ian Soboroff wrote:

>Hi, all... I'm working on an implementation of NDCG.  While my math
>looks right (to me at least), I'd love to test it on some real retrieval
>runs.  Unfortunately, I can't find any runs with corresponding NDCG
>scores.  (I thought NTCIR did this but their online proceedings don't
>report NDCG.)
>
>Alternatively, I'd appreciate seeing a known-good implementation so I
>can be sure of my results.
>
>Thanks in advance,
>Ian
>
>
>
>
>  
>