Re: trecvid - general and known-item search evaluation details
- Subject: Re: trecvid - general and known-item search evaluation details
- From: "Arjen P. de Vries" <Arjen.de.Vries@cwi.nl>
- Date: Mon, 11 Jun 2001 13:52:06 +0200
- Content-Type: text/plain; charset=US-ASCII
- In-reply-to: Your message of "Fri, 08 Jun 2001 11:28:04 EDT." <3B20EDFA.4FE00901@nist.gov>
I have some minor comments on the proposed evaluation procedure.
Although I agree with the general idea of the proposal, I still have a serious
concern about the number of known-items known now, which - if our systems work
well - may be enlarged due to search results (admittedly a little bit
optimistic statement :) - but then they would not count as correct results.
I propose that groups themselves review their own results (which seems quite
reasonable effort to ask, considering the time most likely invested on getting
results to submit :), to check if there are more "correct" known-items in
their answer set. If so, they can notify the list, and people may vote against
if they disagree. Otherwise, it will count as an additional known-item and
added to the list.
What do you think, is that procedure implementable?
Date Index |
Thread Index |
Problems or questions? Contact email@example.com