Re: Question w.r.t. manual style B participation

[Paul Over <over@nist.gov>]

"Arjen P. de Vries" wrote:
> 
> | The manual participant is not allowed to interactively try out the topic on any
> | collection; the interactive participant is.
> 
> Ok, that's clear.
> 
> Just to be overly certain, I assume that *also* disqualifies expanding the
> topic description using a thesaurus like WordNet or performing a websearch in Yahoo for
> expanding a query, and then manually selecting good expansion terms, correct?
> 
> Arjen
> 
> PS: how about entering expansion terms without looking in other resources, is
> that manual topic construction???

Arjen and others,

Your additional questions set off some discussion here and I think I want to
reverse my initial opinion. Sorry for the reversal.

We need a simple rule which follows from the basic motivation for the manual 
type run. Here is my attempt.

1) Fully automatic translation of topic to query with no prior access to the
test collection or subsequent access to results of the test collection
search is very hard. 

2) Therefore we temporarily allow the inclusion of a human in this process
and call it a manual run. 

3) Because we can imagine that a fully automatic system could use a thesaurus 
or even search a comparable collection and use the resulting information in 
creating its one test query, we will allow the human in the manaul run to so 
as well. The list of resources consulted can be extended but cannot include
the search test collection.

Manual runs are different from interactive runs in that manual runs
only submit one query against the test collection. Interactive runs
may submit more than one query against the search test collection and
can use the results of such a search to guide the creation of subsequent
queries derived from the same topic.

- Paul
 

-- 
Paul Over - Retrieval Group
	    Information Access Division
	    Information Technology Laboratory
	    National Institute of Standards and Technology
	    Bldg. 225  Rm. A211  (Mailstop 8940)
	    Gaithersburg, MD  20899-8940   USA
	    Voice: 301 975-6784    Fax: 301 975-5287