Re: Question w.r.t. manual style B participation

[Paul Over <over@nist.gov>]

"Arjen P. de Vries" wrote:
> 
> We had two different interpretations of the "rules" of participation in the
> search task w.r.t. style A vs. B and manual vs. interactive participants,
> and would like to check your opinion before we are labelled as "cheaters" :)
> 
> While style A uses knowledge of the search test set, and style B does not,
> there is also the distinction between manual and interactive, boiling down to
> "manual does not have an interaction loop".
> 
> Now the question is whether a manual run can try out a proposed query on a
> comparable collection (like the feature training set), never looking at the
> search test set; in one view, this remains a style B manual run as the results
> have not been tried on the search test set, or automatically become an
> interactive run because a user has inspected search results (although on a
> different collection).
> 
> In other words, is a manual participant allowed to try out, during topic
> creation, its performance on a comparable collection, or does that necessarily
> imply an interactive run?


Hi Arjen et al,

We could have a long and perhaps interesting group discussion about the pros and 
cons of various answers to your question, but frankly I think the time would better 
be spent now on other matters. I believe the answer follows from your condensed 
definition of "manual": no interaction loop. There is no qualification based on 
which search data is involved. 

The manual participant is not allowed to interactively try out the topic on any 
collection; the interactive participant is.

- Paul


-- 
Paul Over - Retrieval Group
	    Information Access Division
	    Information Technology Laboratory
	    National Institute of Standards and Technology
	    Bldg. 225  Rm. A211  (Mailstop 8940)
	    Gaithersburg, MD  20899-8940   USA
	    Voice: 301 975-6784    Fax: 301 975-5287