Re: interassessor consistency data on TREC 06 Legal track ad hoc topics

["Gordon V. Cormack" <gvcormac@uwaterloo.ca>]

On Wed, Jan 31, 2007 at 05:13:53PM -0500, Dave Lewis (address for public mailing lists) wrote:
> 
> Only 9 of the topics have an expected agreement on positives of 0.70  
> or better, which is pretty worrisome from the standpoint of combining  
> relevance assessments from the TREC 06 and TREC 07 assessors.

I'm not so sure that is worrisome.  I'll have to dust
off my old numbers, and Ellen's, but that level of
disagreement might result in a whole lot less difference
in evaluation than you think.

It turns out that I'm right now doing some work on
the validity of various measuring techniques.  If I
could get the qrels and runfiles from you I'd welcome
more sample data and I can let you know what I come
up with.