Re: trecvid - general and known-item search evaluation details

["Arjen P. de Vries" <Arjen.de.Vries@cwi.nl>]

Hi Paul,

I have some minor comments on the proposed evaluation procedure.

Although I agree with the general idea of the proposal, I still have a serious
concern about the number of known-items known now, which - if our systems work
well - may be enlarged due to search results (admittedly a little bit
optimistic statement :) - but then they would not count as correct results.

I propose that groups themselves review their own results (which seems quite
reasonable effort to ask, considering the time most likely invested on getting
results to submit :), to check if there are more "correct" known-items in
their answer set. If so, they can notify the list, and people may vote against
if they disagree. Otherwise, it will count as an additional known-item and
added to the list.

What do you think, is that procedure implementable?

Arjen