Regarding Type A or Type B

["Milind Naphade" <naphade@us.ibm.com>]

HI Paul, and all,
      I just wanted to confirm that when the feature detection results will
be submitted for the search test set by those who volunteer, the
categorization of their systems as Type A or Type B should follow two
rules.

1) The use of the search test set is not permitted for building the feature
detector in the Type B system. This is very clear to most of us.

2) The search test set or performance of your detectors over it should not
be used to tune any of your Type B detectors. In other words if one  uses
the training set for training, then runs the detector over the search test
set, finds  by looking at the results of the test set  that performance can
be enhanced, and then retrains on the training set alone (even if  one does
not use the test set example for training) to get better detection on the
search test set, that no longer remains a Type B run and becomes a Type A
run.

The same also applies when providing results on the feature test set a
month from now.

Thanks,
Sincerely,
Milind


Milind R. Naphade
Research Staff Member
Pervasive Media Management Group
IBM T J Watson Research Center
19 Skyline Drive
Hawthorne, NY 10532
Phone: 9147847032
alt t/l 4812312


                                                                                                                       
                      Paul Over                                                                                        
                      <over@nist.gov>          To:       Multiple recipients of list <trecvid@nist.gov>                
                      Sent by:                 cc:                                                                     
                      trecvid@nist.gov         Subject:  Re: Question w.r.t. manual style B participation              
                                                                                                                       
                                                                                                                       
                      07/25/2002 07:59                                                                                 
                      AM                                                                                               
                      Please respond to                                                                                
                      trecvid                                                                                          
                                                                                                                       
                                                                                                                       




"Arjen P. de Vries" wrote:
>
> We had two different interpretations of the "rules" of participation in
the
> search task w.r.t. style A vs. B and manual vs. interactive participants,
> and would like to check your opinion before we are labelled as "cheaters"
:)
>
> While style A uses knowledge of the search test set, and style B does
not,
> there is also the distinction between manual and interactive, boiling
down to
> "manual does not have an interaction loop".
>
> Now the question is whether a manual run can try out a proposed query on
a
> comparable collection (like the feature training set), never looking at
the
> search test set; in one view, this remains a style B manual run as the
results
> have not been tried on the search test set, or automatically become an
> interactive run because a user has inspected search results (although on
a
> different collection).
>
> In other words, is a manual participant allowed to try out, during topic
> creation, its performance on a comparable collection, or does that
necessarily
> imply an interactive run?


Hi Arjen et al,

We could have a long and perhaps interesting group discussion about the
pros and
cons of various answers to your question, but frankly I think the time
would better
be spent now on other matters. I believe the answer follows from your
condensed
definition of "manual": no interaction loop. There is no qualification
based on
which search data is involved.

The manual participant is not allowed to interactively try out the topic on
any
collection; the interactive participant is.

- Paul


--
Paul Over - Retrieval Group
                 Information Access Division
                 Information Technology Laboratory
                 National Institute of Standards and Technology
                 Bldg. 225  Rm. A211  (Mailstop 8940)
                 Gaithersburg, MD  20899-8940   USA
                 Voice: 301 975-6784    Fax: 301 975-5287