Why engaging a test laboratory early pays off.

Over the last few years we have discussed industry criticism about the length of security testing and evaluation projects, we have pointed out that atsec regularly starts work while the product is still being developed. This win-win approach reduces the overall time and cost of the evaluation, shortens the time between product release and the final certification, and can enable the sponsor to be first on the market with a certified product.

Although designers of cryptographic modules are well aware that cryptographic modules require standards-based verification before they can be sold in the Federal marketplace, one of the problems we encounter is that many wait until their product is complete or well-along in development before considering engaging a laboratory to begin the required testing. Although a sequential approach might seem to make sense, pursuing development and testing in parallel can save both time and money. We have several examples where hardware needed design changes causing a full additional round of prototyping and approvals that could have been avoided. Of course the laboratory cannot advise you on your product design but our staff has deep knowledge of relevant standards and can tell you quickly if your design is likely to conform to the standard or not.

Here’s how an optimized approach can succeed. The vendor contacts the laboratory very early in the development cycle. The laboratory reviews the product designs for hardware design, protocols, algorithms and other relevant items before any development begins. If design choices have been made that would create issues with conforming to the standard (For example FIPS 140-2), they can be corrected early in the development project. It's well known that the earlier in the development cycle that problems are identified and fixed the cheaper and more efficient it is to do so [1]. Note that redesign of the case , hardware components or a printed circuit board will typically take more time than software fixes.

In addition, the test laboratory can develop the necessary functional testing in parallel during product development. In many cases, testing can be performed during the development phase identifying any problems while development is still ramped up. Final testing can be completed as soon as the final product is ready. If the Security Policy document has been developed along with the user documentation during the development phase the product can be submitted for validation by the CMVP very soon after the product is completed and will appear on the "In validation list" published by NIST. Assuming that the lab is experienced with FIPS 140-2 and has already a thorough understanding of your product that was gained during the development cycle, this approach reduces substantially the risk of late product modifications due to feedback from the CMVP.

Thus, provided that the lab has submitted a high-quality test report, the lag between the end of product development and receiving your certificate is restricted only to that of the final validation activities performed by the CMVP. So, it is important to work with an experienced lab when seeking to optimize the certification of your product.

Achieving certification as soon as possible after product release maximizes the return on the vendor’s investment in testing.

Developing an on-going relationship with a test laboratory is an advantageous and cost-effective investment. Most products that are tested for standards compliance also require repeat testing as the product changes and features are added over time. Repeat testing with the same laboratory is faster and less expensive for several reasons.

If the changes to the product are small, the laboratory might only need to update the associated documentation. In some scenarios the changes are not affecting the security functionality greatly and only the changes need to be reviewed, or perhaps a reduced set of regression testing can be employed . Even if the changed product needs full testing, the lab’s familiarity with the vendor’s product, documentation, development methods and personnel will help to optimize the validation process. Most significantly, the test team will already have a working knowledge of the product, including any special knowledge needed to finish the tests quickly. If re-testing is a regular event – often the case for major products with long lives – the laboratory and the vendor can work together to optimize the whole process, potentially resulting in a vastly-reduced (potentially greater than half) validation effort for each product iteration.

We would welcome a chance to talk to you about your IT security projects and discuss how we can help you streamline the validation process.

[1] Boehm, Barry W. and Philip N. Papaccio. 'Understanding and Controlling Software Costs,' IEEE Transactions on Software Engineering, v. 14, no. 10, October 1988, pp. 1462-1477
available from http://faculty.ksu.edu.sa/ghazy/Cost_MSc/R6.pdf

[2] Implementation Guidance for FIPS PUB 140-2 and the Cryptographic Module Validation Program, August 2010, NIST
available from http://csrc.nist.gov/groups/STM/cmvp/documents/fips140-2/FIPS1402IG.pdf