Re: New ETR Format (LabGram#027)

[Olin Sibert <osibert@siliconkeep.com>]

At 08:43 AM 3/31/2004 -0500, you wrote:

> >a rumor that some validators never see ETRs
> >(particularly for ASE), but instead judge the evaluation by seeing the ST
> >directly by himself, and by checking WHO the ST author is.
>
>
>I have to strongly disagree with this suspicion. Review of the ETR is an
>essential and critical part of the validation effort. The process of
>cross checking between ERT, evaluation evidence, and CEM/CC is the
>process by which evaluation activities and results are verified. ETR
>reviews, particularly for ASE, are the methods that are used to inspect
>evaluation results.
>
>This holds true for CCEVS and I'm sure for the other schemes as well.
>
>-Paul Bicknell
>CCEVS Validator
>
>YOKOTA HIROFUMI wrote:
>
> >First, I appreciate the New ETR Format (LabGram #027).
> >
> >http://niap.nist.gov/cc-scheme/labgram027.html
> >
> >I think, the guidance is truly nice and, I believe, evaluators are
> >definitely need such kind of guidance.
> >
> >Second, however, I'd like to ask, is it possible to be provided with a bit
> >more help for the format?
> >
> >*********
> >Because, there is a portion in the guidance that is difficult to understand
> >and perform.
> >That is where it says: "to provide rationale as to why XXXXXX are
> >sufficient.
> >
> >In the guidance, that portion is classified to the category C.
> >
> >The category C is defined as:
> >Analysis - Used for more complex methodologies. A detailed description of
> >the methodology as it applies to the TOE under evaluation must be presented.
> >This description should include the evaluation team's procedures used in
> >carrying out the methodology on the TOE and its evidence. The evaluators
> >must produce a detailed work log of the procedure's application. All
> >procedures and logs must be included in the ETR.
> >
> >
> >Also, in the guidance, it is said that:
> > it should be noted that the following neither adds value to an ETR, nor
> >demonstrates understanding:
> >
> >1. Use of "stock phrases" from the CEM, or repeating the work unit (e.g.,
> >the evaluator examined..." or "the evaluator checked..."); or
> >2. Repetition of information from the evidence examined, or
> >3. Vague work descriptions (e.g., "the evaluator performed a mental
> >mapping").
> >*********
> >
> >Then, my questions:.
> >
> >1. Would it be possible to provide rationale (as to why XXXXXX are
> >sufficient to meet the CEM requirement) without restating(repeating) the CEM
> >phrases?
> >
> >2. Would it be possible to provide rationale (as to why XXXXXX are
> >sufficient to meet the CEM requirement) without mentioning(repetition) of
> >information from the evidence examined?
> >
> >Note) I think, it is impossible.
> >When one needs to provide a rational as to why A is sufficient to meet B,
> >how he/she can do this without mentioning A and B?
> > I think, reepetition of the phrases and repeating information are necessary
> >to describe rationales and have values for that purpose.
> >
> >3. So, I believe, that, usually, CEM phrases are restated and information
> >from the evidence are repeated in many of places in a ETR, aren't they?
> >
> >Furthermore, the guidance mandates evaluators to provide a detailed
> >description of the methodology as it applies to the TOE under evaluation
> >must be presented, and also to provide description of their detailed work
> >procedures and work logs.
> >
> >So, I can easily imagine and am afraid of the explosion of the size of ETR
> >(for ASE), which increases exponentially against the size of STs, exceeding
> >evaluators feasible efforts.
> >
> >ex.
> >(ST: 50 pages) => (ASE-ETR: 100 pages)
> >(ST: 100 pages) => (ASE-ETR: 200 pages)
> >(ST: 150 pages) => (ASE-ETR: 400 pages)
> >
> >Could such a concern be solved by referring to some good examples of ETR?
> >
> >So, questions 4 and 5 are:
> >
> >4. Is it possible for us to refer to some good ETRs (for ASE) and STs ( of
> >course, both sanitized ) ?
> >
> >5. If not, what would be the expected size of ETR (for ASE) in average for
> >STs of 50-100 pages around?
> >
> >Thanks and regards,
> >
> >Yokota
> >
> >+++++++++++++++++++++++++++++++
> >P.S.
> >
> >I agree that writing and validating ETR would not be an easy task.
> >
> >So, I halfly believe a rumor that some validators never see ETRs
> >(particularly for ASE), but instead judge the evaluation by seeing the ST
> >directly by himself, and by checking WHO the ST author is.
> >
> >Probably, they may have a valid reason for doing so, and I may agree with
> >the reason 100% by the cases.
> >
> >However, yet, it is a sad thing to think about the amount of effort of
> >evaluators who wrote the ETR sincerely attempting to comply with CEM.
> >It is sad to think about the cost and period for those efforts, being payed
> >by consumers.
> >
> >Although the new guidance is truly nice and I do not know any other scheme
> >has provided such a guidance, I think a bit more help ( - especially with
> >some good ETR samples provided - ) would greatly help many evaluators and
> >would be beneficial to entire CC community.
> >
> >++++++++++++++++++++++++++++++
> >
> >
> >
> >
> >
> >