Re: Number of runs - request for estimate today

[Paul Over <over@nist.gov>]

John R Smith wrote:
> I would like to ask if manual and interactive for SearchTest are counted as
> separate runs?  If so, then more runs may be needed to accommodate systems
> exploring ASR, X, X+ASR methods.  Otherwise, your suggestion for four runs
> (max) seems fine.

John and everyone,

For our estimates we assumed 4 search runs max, independent of whether they
were manual or interactive. I supposed groups would mostly do one or the
other.

Perhaps the track should require a single ASR-only baseline if any ASR is used
in any of the 4 runs from a group. This is weaker than last year's requirement
but could help answer the question: are the other factors adding anything 
significant in effectiveness beyond the ASR. What I don't know is whether
systems can run using only ASR mode.

Maybe I should get a count right now of how many groups plans to submit which
types of runs. We assumed about the same number of groups as last year but
let's replace the assumption with an better estimate. If you know how many
system variants (minimum) you will be interested in apart from any track
requirements, please use that in each category rather than just a Y or N.

	Yes (n), No  	manual search
	Yes (n), No  	interactive search
	Yes (n), No     feature extraction

Thanks,
Paul


	
-- 
Paul Over - Retrieval Group
	    Information Access Division
	    Information Technology Laboratory
	    National Institute of Standards and Technology
	    Bldg. 225  Rm. A211  (Mailstop 8940)
	    Gaithersburg, MD  20899-8940   USA
	    Voice: 301 975-6784    Fax: 301 975-5287