Evaluation procedure of the overlay text feature

[Christian Wolf <wolf@umiacs.umd.edu>]

Hi,

the university of maryland team participated on the overlay text feature
extraction task with a feature detector developped at INSA de Lyon,
France.

We received NISTS's evaluation results and were very surprised about the
unexpected bad results. We manually counter checked some of the results
against the mpg files and had problems to reproduce the precision
values. E.g., the results gave 20% precision at the first 5 results,
whereas, according to our evaluation scheme, the first 5 shots all
contained overlay text.

Could it be, that the evaluators at NIST used the original definition of
overlay text, which required text to have an (unrealistic) size of at
least a third of the screen height, vs. the new (changed) definition of
just being readable?

Best regards,

the UMD Team


P.S.: The first 5 overlay text shots our team submitted were:

1: video 255 shot 2: Shot boundary is 39s-50s, contains the text
"MIRACLES from AGRICULTURE" 
2. video 231 shot 55: Shot boundary is 9:4-9:12, "LECTURE TODAY, DR.
SUNNY SWEET"
3. video 213 shot 56: Shot boundary is 12:14-12:21, the whole screen is
filled with text "CONVENTIONAL SUBDIVISION CLUSTER SUBDIVISION" + a
table.
4. video 225 shot 1: Shot boundary is 0s-3s, "AVIS FILMS INC. PRESENTS"
5. video 198 shot 9: Shot boundary is 59s-1:10, "AMERICAN LOOK"

-- 
Christian Wolf
Language and Media processing Lab., Univ. of Maryland (Jul-Sep.2002) &
Laboratoire Reconnaissance de Formes et Vision, INSA de Lyon
Email: wolf@rfv.insa-lyon.fr  http://rfv.insa-lyon.fr/~wolf