TRECVID: known item comparison

[Paul Over <over@nist.gov>]

Hi Thijs, Karin, Milind, and other interested parties

After some discussion here, I've decided we will use the
overlap test in evaluating known-item searches here are NIST
this time around, but we encourage others to try other ways
and report the pros and cons at the workshop. As I said earlier,
the question of which measures are most useful is still an
open one. When we present results at TREC it will be with
this in mind. Here is the motivation for our choice:

1) The known-item search task output has long been defined 
as a shot rather than a starting point within some video. It 
seems too late to change this.

2) Similarly, the known items are defined to include ALL and, to 
the extent possible, ONLY the desired material, i.e. to be segments
not just starting points.  So there will be two end points and 
systems which do well on both, if there are any :-), should be 
rewarded in the metric versus systems that do well on only one. 

3) Systems, which for whatever reason, are not trying to
find an endpoint, can of course still participate by generating 
a stop frame, e.g., at some fixed distance form the start.

4) We'll experiment with the settings of RMIN and RMAX for 
evaluating known items. Since rather more frames are involved
than is the case with shot boundary, we expect to use values of 
RMIN and RMAX greater than 1/2 so that one segment matching two 
(Milind's case) should not, we think, occur. We also expect to 
define RMIN (or RMAX) with respect to the truth data (known-items 
in the topics) rather than with respect to the longer or shorter 
of the two segments to be compared. This will make them in effect
segment-level recall/precision knobs we can vary.

Hope this makes some sense :-)

- Paul

-- 
Paul Over - Retrieval Group
	    Information Access Division
	    Information Technology Laboratory
	    National Institute of Standards and Technology
	    Bldg. 225  Rm. A211  (Mailstop 8940)
	    Gaithersburg, MD  20899-8940   USA
	    Voice: 301 975-6784    Fax: 301 975-5287