Re: Question w.r.t. manual style B participation

["John R Smith" <jsmith@us.ibm.com>]

Hi Paul, Arjen,

We believe that Arjen raised an important point and request some
clarification.

The most recent definition of "manual" run (given in Paul's 7/25 note
below) allows the user given a search topic to interact with any collection
other than the search test collection to refine/produce a query expression
that is subsequently processed on the search test collection.  That query
expression shall be processed on the search test collection only once with
no chance for interaction.

This definition seems reasonable in potentially allowing query expansion
using the other collections, or for relevance feedback techniques to be
used on those collections to design the specific query expression.
However, at the extreme, this could allow use of other collections for
training and tuning according to the specific query topics, consider also
that relevance feedback itself can be viewed as a type of model building,
which seems less in the spirit of the search task.

Therefore, we ask if the point can be clarified as to what type/extent of
interaction on other collections shall be allowed given knowledge of a
search topic.

Thank you.

Best regards,

John R. Smith

---------------------------------------------------

Manager, Pervasive Media Management
IBM T. J. Watson Research Center
30 Saw Mill River Road
Hawthorne, NY 10532
(914) 784-7320; jrsmith@watson.ibm.com

---------------------------------------------------



|---------+---------------------------->
|         |           Paul Over        |
|         |           <over@nist.gov>  |
|         |           Sent by:         |
|         |           trecvid@nist.gov |
|         |                            |
|         |                            |
|         |           07/25/2002 10:52 |
|         |           AM               |
|         |           Please respond to|
|         |           trecvid          |
|---------+---------------------------->
  >-------------------------------------------------------------------------------------------------------------------------------|
  |                                                                                                                               |
  |       To:       Multiple recipients of list <trecvid@nist.gov>                                                                |
  |       cc:                                                                                                                     |
  |       Subject:  Re: Question w.r.t. manual style B participation                                                              |
  |                                                                                                                               |
  |                                                                                                                               |
  >-------------------------------------------------------------------------------------------------------------------------------|




"Arjen P. de Vries" wrote:
>
> | The manual participant is not allowed to interactively try out the
topic on any
> | collection; the interactive participant is.
>
> Ok, that's clear.
>
> Just to be overly certain, I assume that *also* disqualifies expanding
the
> topic description using a thesaurus like WordNet or performing a
websearch in Yahoo for
> expanding a query, and then manually selecting good expansion terms,
correct?
>
> Arjen
>
> PS: how about entering expansion terms without looking in other
resources, is
> that manual topic construction???

Arjen and others,

Your additional questions set off some discussion here and I think I want
to
reverse my initial opinion. Sorry for the reversal.

We need a simple rule which follows from the basic motivation for the
manual
type run. Here is my attempt.

1) Fully automatic translation of topic to query with no prior access to
the
test collection or subsequent access to results of the test collection
search is very hard.

2) Therefore we temporarily allow the inclusion of a human in this process
and call it a manual run.

3) Because we can imagine that a fully automatic system could use a
thesaurus
or even search a comparable collection and use the resulting information in

creating its one test query, we will allow the human in the manaul run to
so
as well. The list of resources consulted can be extended but cannot include
the search test collection.

Manual runs are different from interactive runs in that manual runs
only submit one query against the test collection. Interactive runs
may submit more than one query against the search test collection and
can use the results of such a search to guide the creation of subsequent
queries derived from the same topic.

- Paul


--
Paul Over - Retrieval Group
                 Information Access Division
                 Information Technology Laboratory
                 National Institute of Standards and Technology
                 Bldg. 225  Rm. A211  (Mailstop 8940)
                 Gaithersburg, MD  20899-8940   USA
                 Voice: 301 975-6784    Fax: 301 975-5287