tag:blogger.com,1999:blog-6042572809285034351.post4789766319511940186..comments2024-03-28T06:24:32.786-05:00Comments on atsec IT security blog: Programming the Evaluation RobotAndreashttp://www.blogger.com/profile/17368437730621711005noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-6042572809285034351.post-28435746599199339852012-10-24T14:00:09.046-05:002012-10-24T14:00:09.046-05:00Gerald,
Your blog makes some sense but you don'...Gerald,<br />Your blog makes some sense but you don't have any discussion of the reason for this change in the first place; the time, effort, cost of the current process; which leads to evaluations of products that are no longer being sold or supported. The problem NIAP is trying to solve is getting COTS in the hands of the acquires of a product that is actually still sold by the vendor. By the way, the one area that is not being compromised in the new approach is VAN and I would argue that is the one area where some subjectivity is ok.Josh the Occasional World Travelerhttps://www.blogger.com/profile/17942528610161174352noreply@blogger.comtag:blogger.com,1999:blog-6042572809285034351.post-82428479150533788832012-10-23T11:50:47.127-05:002012-10-23T11:50:47.127-05:00Dear Dan,
I fully agree with your comments. My cu...Dear Dan,<br /><br />I fully agree with your comments. My current concern is that assurance is lowered with the argument that evaluation activities are not objective and repeatable, using a very narrow definition of these terms.<br /><br />We shall strive for objectivity and repeatability, but we also shall accept that there will be differences in the details, and that such individual variations are not bad as long as they are documented and well-founded, rather than arbitrary, as you have said. I know that in many areas, when the evaluator's experience comes into play, this adds tremendous value to the evaluation, and I don't want this value to be thrown away with pseudo-formal arguments.<br /><br />Regards,<br />Gerald Gerald Krummeckhttp://www.atsec.comnoreply@blogger.comtag:blogger.com,1999:blog-6042572809285034351.post-44030205040273995072012-10-22T15:00:59.427-05:002012-10-22T15:00:59.427-05:00Reading this, I think that objectivity should be a...Reading this, I think that objectivity should be a goal -- and should already be achieved. The opposite of objectivity is bias towards a vendor or other organization, and there should be no trace of that in evaluations.<br /><br />I agree that repeatability is harder to acheive, and that a checklist approach is not the way to achieve that. Repeatability, however, can be achieved in a number of ways. It can be achieved through dictating the specifics of how something is to be tested. I'll argue that's more than repeatability -- that's standardized testing (akin to what we seen in US public schools), which doesn't always have the intended effects of creating quality.<br /><br />But repeatability can be achieved in other ways, such as through documentation of test plans and procedures, in such a way that another organization could come in and perform the exact same tests and, presuming the same product, get the same results. That also is repeatability... and that is something that is achievable without degredation of quality.Anonymousnoreply@blogger.com