Deep Thought on PPs

Encouraged by Helmut's results from his visit to the oracle at Delphi, and having some questions of my own about PPs, I too petitioned for a trip to visit the oracle. Alas, atsec's travel budget is not infinite and I was encouraged to find a similar service closer to home.

Putting my thinking cap on, I decided to consult with Deep Thought. This amazing system is renowned for giving highly accurate answers, no matter the quality of the question, and access to its design team is available in almost every part of the world. So I sent my son to rummage behind our compost heap to bring forth the mice for further evaluation of this system and what assurance I could gain about the security functionality.

The mice stated that an evaluation of Deep Thought using the CC is planned but is currently awaiting the development of a suitable cPP by the Golgafrinchan Technical Community (GTC), who are currently studying information on this kind of evaluation.

Well, regardless of that, I do have some questions about PPs that I feel should be based on objective analysis of information rather than subjective means. This is especially important because the whole community is currently focusing their efforts on PPs. Some examples of questions are:

• Has the proportion of evaluations claiming PP conformance changed over the years?
• How many PPs on the CC portal have not been claimed in an evaluation listed on the CC portal?
• Are any evaluations listed on the CC portal claiming compliance to PPs that are NOT listed on the CC portal?
• Which technology areas tend to use PPs more than others?
• Which are the top 25 most claimed PPs?
• How many PPs have been claimed in a listed validation report only once?
• What is the meaning of life, the universe and everything?

So, the mice allowed me to pose these questions, and I was rewarded by a response from Deep Thought:

"The answers to all of these questions are on the CC Portal." 

Accordingly, I went to the CC portal and downloaded the publicly available CSV files about PPs and  the various certified products that are listed there. I reviewed the CSV files and discovered that the information I needed was not there! True, some PPs are listed in the CSV files, but the information was not there for earlier years and later information is far from complete and is sometimes even incorrect.

I asked Deep Thought, "You said this information was on the portal, but it is not of useful quality. How could you give me such a misleading answer?" Deep Thought replied: 

"The answers are on the CC Portal." 

The mice explained "These things take time. Just as Slartibartfast spent many aeons detailing the low level design of our next generation system for our proposed evaluation of The Earth, including the fjords and the crinkly bits, so you need to read every single validation report on the CC portal. It's a trivial task - there's not so many of them. Only a tad under 2000 of them."

So having great respect for the mice and for Deep Thought I did exactly that. Slipping a babel fish in my ear to help with my comprehension of a multitude of languages, I began the process using the following method.

Method for analysis of PP conformance claims in validation reports listed on the CC Portal

1. First I retrieved the summary CSV files from the CC portal (This was done on 2012/09/24).
2. Next, I combined the PP and archived PP files into one spreadsheet and similarly combined the data for the certified products and archived certified products into one spreadsheet.
3. Then I download every validation report related to the certified products and reviewed each validation report for PP conformance claim statements. In a few instances, the validation report was not available or corrupt, so in this case I reviewed the Security Target for claims.
4. Where multiple PPs were claimed, I duplicated the relevant entry in the CSV file to allow for a one PP per line claim and marked each claim with a "claim number."
5. I updated the new certified products CSV file, adding and correcting the PP claim with the information found by reviewing the validation report.
   
   In many cases the Portal file contained a name for a PP that is apparently an identifier that is of publicly undefined origin, and didn't match the identifier cited in the validation report, so I  added a new column to note the validation report identifier, as well as the CC Portal's identifier. For example, a PP cited in the validation report as BSI-PP-0002-2001 was given in the portal file as SVGPP01.
   I resolved any errors, omissions as best as I could. For example:
   
   • The portal reference "JSPP" could refer to  either  the JavaCard System Standard 2.1.1 Configuration Protection Profile, Version 1.0b or the one for the JavaCard System Standard 2.2. In this case, I modified (and added) the Portal name to include the version.
   • In some cases the Portal reference was not consistent. For example:
     
     • The SECURITY_IC_V1.0 is confused with a ref to what should be SVGPP01;
     • SVGPP01 and  SVGPP refer to the same PP; 
     • Both IEEE Std. 2600.1™-2009 and  PP_HCD_BR_V1.0 refer to the same PP. 
     I corrected these in line with the contents of the validation report.
   • In many cases the PP conformance is not listed in the Portal CSV file at all. In these cases I added the conformance claim to the CSV file.
   • In some cases the date of the PP in the certification report (sometimes called a "validation report") and the date of certification of the PP on the portal do not match by a long way. When no version or date was given in the validation report, a best guess was made.
6. Identify PPs claimed in a certification report on the portal that are not listed on the portal and add them, marking them as "not on the CC portal" in the CSV file.
7. Review the file and look for inconsistencies and attempt to resolve them through further reference to the validation reports, STs, or scheme PP lists as appropriate.

Best practices/improvements for citing PPs

As a result of this process, I devised some recommended best practices for certification bodies when preparing their certification reports. Of course some schemes already do follow these practices, but it is not the case that all schemes do so. Of course we have a problem of historic information  where it is not reasonable to update validation reports retrospectively. This is especially true  for reports related to archived products.

1. State PP claims clearly and concisely in the validation report. Include versions of the PP (even if there is only one version now, another may come along later) and PP certification date (where available).
2. If there is no PP claim, state that fact in the validation report. Just not mentioning PPs at all leaves one wondering if that information is buried in the document somewhere.
3. If a PP claim is made in the validation report state clearly if it is strict conformance or demonstrable conformance that is claimed. (Note that these conformance definitions first appeared in R3.1 of the CC. Before this PP conformance should be of the "strict" type.)
4. Clearly distinguish between a "complete" conformance claim and PPs that were used as reference, or which are partially claimed.
5. Even when a PP is updated, it is important to keep the previous versions for reference. This is especially true when certified products have made claims to that version. Removing an older version of a PP means that it is impossible to properly understand the claims made for that product.
6. Review the process for updating the CSV files on the CC Portal. Could the information being submitted about reported product evaluations or the processing of the information provided be improved? 
7. Review the requirements for the PP registry, including:
• The scope of the register (For example should all PPs, even if they are not used under the CCRA, certified  or packages be listed?);
• Management requirements and processes for operating for the register;
• The structure of the database and the types of information that are summarized;
• The method for allocating of a unique identifier to each PP in order to allow distinguishing PPs with the same or similar names from becoming confused and to identify equivalency between the same PP that has been certified under different national schemes.
• There currently seems to be a mixture of nomenclature methods in use to identify a PP. These include using the full text name, a common (CC portal issued?) identifier, a national scheme identifier, and the validation report number. None of these methods used alone uniquely identify a PP.   

  8.  Validate PPs before using them. 

Some of the detailed PP Questions answered:

Has the proportion of evaluations claiming PP conformance changed over the years?

The data shows that since 2001 the percentage of evaluations that do not claim conformance to any PP has remained between 50-60%.

Data before 2000 was omitted due to the low number of evaluation projects.

Certificate maintenance projects were not included in the data.

How many PPs on the CC portal have not been claimed in an evaluation listed on the CC portal?

Total Number of PPs listed on the CC Portal: 228

Total number of PPs on the CC portal that have been mentioned as claimed by one or more validation reports: 53
This would suggest that only about a quarter of the PPs listed on the CC portal have been used in an evaluation that is also listed on the portal.

Are any evaluations listed on the CC portal claiming compliance to PPs that are NOT listed on the CC portal?

Yes, I found 14 certified product listings  that claimed a PP that is not currently listed on the CC Portal. Some were certified PPs. These PPs are:

PP Ref ID	Name	Certification Date
ANSSI-PP/0304	JavaCard System Standard 2.1.1 Configuration Protection Profile- Version 1.0b	8/1/2003
BSI-CC-PP-0020-V3-2010-MA-01	Protection Profile for electronic Health Card (eHC)- elektronische Gesundheitskarte (eGK)- Version2.9- 19 April 2011	4/19/2011
BSI-CC-PP-0056-2009	Protection Profile for Machine Readable Travel Document with 'ICAO Application'- Extended Access Control- Version 1.10	3/25/2009
BSI-CC-PP-0057-2010	Digital Tachograph - Vehicle Unit (VU PP) version 1.0	7/13/2010
BSI-PP-0030-2008	Schutzprofil PC Client Specific Trusted Platform Module TPM Family 1.2; Level 2; Version 1.1
NIPS_PP_V1.0	Network Intrusion Prevention System Protection Profile- Version 1.0	12/21/2005
PP_CIMC_V1.0	Certificate Issuing and Management Components (CIMC) In Basic Robustness Environments V1.0	4/27/2009
PP_CIMC_V1.1	Certificate Issuing and Management Components Protection Profile- Version 1.1	12/27/2001
PP_FW_AL_BR_V1.0	U.S. Government Protection Profile for Application-level Firewall in Basic Robustness Environments- Version 1.1
PP_FW_TF_MR_V1.2	U.S. Government Traffic-Filter Firewall Protection Profile for Medium Robustness Environments- Version 1.2	1/30/2009
PP_PKE_V2.8	U.S. Government Family of Protection Profiles for Public Key-Enabled Applications for Basic Robustness Environments- Version 2.8	5/2/2007
PP-ACSE	Application de crΘation de signature Θlectronique	2/15/2005
PP-FW-KR_V1.2	Firewall Protection Profile for Government V1.2 (2006. 5.17)
PP-FW-KR_V2.0	Firewall Protection Profile V2.0

Which technology areas tend to use PPs more than others? 

The  use of PPs in the defined technology areas should be further analyzed. Perhaps it is necessary for the CC community to focus on some technology areas, or that existing PPs are de-facto cPPs and change might bring a risk of disruption in that area?

How many PPs listed have been claimed in a listed validation report only once?

Interestingly, the answer to this question is 42.

Which are the top 25 most claimed PPs?

In this analysis, any certificate maintenance activities for a product certificate were not included.

In the light of the new CC MB led technical communities, it's interesting to review the provenance of these PPs, which I am classifying below according to certain characteristics. I pay particular attention to the smart card technology area because this is widely held as a model for future technical communities.

General technology-area  (Smart cards)

The Smartcard IC platform PP V 1.0 (BSI-PP-0002-2001)  often referred to as SVGPP01_V1.0 was provided by a group of 4 smartcard vendors working under the auspices of an organization called "Eurosmart." It was certified by the German BSI scheme in 2001 and is still an active PP. It claims to be based on ANSSI PP/9806 and the Smartcard Protection Profile of the Smartcard Security User Group; SCSUG, Draft Version 2.1d, March 21, 2001.

ANSSI PP/9806 was certified by the French scheme, ANSSI in April 1999 and was claimed in evaluations between 2000 to 2007, with a couple of evaluation claims in 2008 and one in 2010. According to the link to the PP given on the portal  the PP was issued in September 1998 and is the work of a group of 5 vendors. No mention of a user group or organization is made. It is still an active PP on the CC portal but is no longer listed on the ANSSI website for PPs (ANSSI's archived PP page was not working at the time of writing).

The Security IC Platform PP V1.0 (BSI-CC-PP-0035-2007), sometimes known as SECURITY_IC_V1.0, was written in August 2007. The authors are cited as 5 vendors and the PP was certified by BSI. Its basis is claimed to be BSI-PP-0002-2001 and it is not archived status on the CC portal.

The other smartcard related PP on the list with product  evaluation claims is the JavaCard System Standard 2.1.1 Configuration Protection Profile (ANSSI-PP/0304) sometimes known as JCSPPC-2.1.1. This PP is not listed on the portal, although it's predecessor (V1.0b) is. According to this predecessor PP, the main input for this PP came from a single vendor, but what may be the latest version, version 3.0 (May 2010) was reviewed by the Java Card Forum – Common Criteria Subgroup.

This analysis of the general smartcard related PPs revealed that the most claimed PPs are developed by vendors, and that a lab (ITSEF) is engaged as editor of the PP. All are certified PPs by European schemes and a vendor group organization is involved in the PP development. Interestingly these groups are regional in nature, Eurosmart and the Java Card Forum in Europe. Interestingly no PPs that have been claimed in a product evaluation have emerged from the U.S./Latin America focused group, The Smart Card Alliance, nor from the Asia Pacific Smart Card Association. Eurosmart produced a survey of PPs relevant to the smartcard sector, the last version was made in July 2010. As an interesting side note, the Smart Card Security User group's  SCSUG PP V3 appears on the CC portal 4 times. One for each time it was validated and certified (France, Germany, U.S. and Canada) but was never claimed in an evaluation listed on the portal. According to the PP, the SCSUG were comprised of credit card companies (security assurance consumers) as well as the U.S. organizations, NIST and NSA. This class of PPs is perhaps the one best described by the vision for cPPs and the associated technical communities.

The PP for hardcopy devices, PP_HCD_BR_V1.0 was developed through  the IEEE and published as an IEEE standard. It is marked as archived on the CC portal and the U.S. CCEVS also falls into this class.

It is worth noting that the initial IEEE PP was superseded by a U.S. Government approved interim PP based on the IEEE version but that has been modified to meet the U.S. Government policy for low assurance. A CC MB sponsored technical community is working on a new PP for this technology.

Smart card (technology area) specific applications

This group of PPs are aimed at smart card applications, such as digital signature, machine readable travel documents and electronic health card. All have been certified by Germany's BSI.

• BSI-PP-0005-2002, and BSI-PP-0006-2002, Protection Profile - Secure Signature-Creation Device (Type 2 and Type 3): These PPs were  prepared for the European Electronic Signature Standardisation Initiative EESSI by CEN/ISSS area F on secure signature-creation devices (SSCDs). "The document is for use by the European Commission in accordance with the procedure laid down in Article 9 of the Directive 1999/93/ec of the European parliament and of the council of 13 December 1999 on a Community framework for electronic signatures as generally recognised standard for electronic-signature products in the Official Journal of the European Communities."
  In other words, the PP  describes the requirements of  regional international legislation.
  
• Machine readable travel documents for passports (BSI-_PP-0026-2007, BSI-PP-0017-2005 and BSI-CC_PP-0056-2009): These PPs were sponsored by a government organization (Germany's BSI) and drafted by a laboratory (ITSEF). They are based on the requirements and recommendations of the International Civil Aviation Organization (ICAO) and addresses the advanced security methods Basic Access Control and Extended Access Control given by ICAO.
  In other words, the PP  describes the requirements of international regulation.
• PP for electronic Health Card (eHC) - elektonische Gesundheitskarfe (eGK), BSI-PP-0020-V2-2007-MA02. This PP does not appear on the portal but is listed on BSI's website. The PP defines the security objectives and requirements for the electronic
  Health Card (German: “elektronische Gesundheitskarte”) based on the regulations for the
  German health care system. The PP was sponsored by the German ministry of health and drafted by a CC laboratory (ITSEF).
  In other words the PP describes the requirements of national legislation.

Translations of  criteria models existing before CC

The Controlled Access Protection Profile, PP_OS_CA_V1.D, (CAPP), Labelled Security PP, PP_OS_LS_V1.b, (LSPP), and PP-RBAC_V1.0 (RBAC) PPs were widely used in operating system evaluations, as well as occasionally in other technology areas with functionality relevant to this PP class (e.g., database). These PPs were derived from the requirements of the U.S. Department of Defense (DoD) Trusted Computer System Evaluation Criteria (TCSEC), dated December, 1985, and in the case of RBAC from the Federal Criteria. In other words they were translations from existing criteria as the Common Criteria were established. They are largely no longer used and are marked as archived in both the CC portal and the US CCEVS, although they are still listed as active in some other national schemes.

National policy PPs

Several other PPs appear in the top 25 most used PPs that are embodying national policy. 

U.S. Government PPs appearing in the list include a PP for Intrusion Detection Systems, PP_IDS_SYS_BR_V1.7, the U.S. Government PP for Application-level Firewall (Basic Robustness) PP and the U.S. Government  PP for Database Management Systems (Basic Robustness) PP_DBMS_BR_V1.2, and a PP for Peripheral Sharing Switch (PSS) For Human Interface Devices, PP_PSSHID_V1.2. The background for these PPs lies in the U.S. IA policy. These PPs were well used but have been recently retired in favor of those that meet the current U.S. legislative and regulative policies, mainly for the DoD. Originally developed by the U.S. Government in order to specify security requirementsfor COTS products the U.S. has begun developing newer PPs that meet the new policies through NIAP led technical communities. (These NIAP led TCs are distinct from the CC MB led TCs.) It is unclear to me if this method of PP development is an attempt to ensure that the developers' stakeholder voices are heard during the development of such policy and the related PPs.

Also in this class of PPs is Network Intrusion Prevention System Protection Profile, KECS-CR-2004-04 specifying an EAL 4 level of assurance. This PP is not available on the CC portal nor on the Korea national scheme PP list. However, the certification report reveals that both were written and sponsored by the Korean Information Security Agency (KISA).

Others

These include a PP for hardcopy devices, PP_HCD_BR_V1.0, which was developed through  the IEEE and published as an IEEE standard. It is marked as archived on the CC portal and by the U.S. CCEVS. 

The latter mention that it is superseeded by an U.S. Government approved PP based on this PP but that has been modified to meet the U.S. Government policy for low assurance. A CC MB sponsored technical community is working on a new PP for this technology.

We also see a PP named "Oracle Database Management System, PP_ODMS_V2.1, which was developed by a single vendor and has been used only by that vendor. It is worth noting that this is a historical case and that lately other PPs for that technology have been developed that have been used by other vendors, as well as the original vendor in the database technology group.

Discussion

I suggest that there are PPs that have emerged from several different types of communities. From generic technology PPs (such as what we see in the smart card industry) that are often developed by the vendor community at large, through those that specify the requirements for security found in broad regional legislation or industry regulation on a multinational basis to a means for specifying policy for a distinct stakeholder group. In some cases, a single vendor leads the field in developing an initial PP which can later broaden into a wider community. For some of these classes it is appropriate that wide representation from industry is made, but I contend that this may not be necessary for cases where existing legislation, regulation or policy is being translated into the language of CC. In these cases experts on the policy/legislation/regulation are needed alongside experts in the CC. Only if there is room for requirements negotiation is it necessary to have vendor representation.

Apart from some of the more recent U.S. PPs, it has been seen as best practice to have a national scheme certify a PP and it would also seem to be best practice to have an expert in CC draft the PP.

In other words, it is important to understand the high-level goals of a PP before specifying and pulling together an appropriate technical community.

What is the meaning of life, the universe and everything?

The answer is 42. However, one should note that it is vitally important to ask the right question.

The information file, derived from the published on the CC portal and updated with the PP data from the published validation reports, is made available to the CC community, with no conditions. This will hopefully enable my colleagues to answer their own questions. {MD5: 0b238e3d51bcc4663a91a0affe568809 , SHA256: 5312218c9db3c96225f728af614489934812166fea288532f45613689864c140)  However, please note that  I make no guarantees or warranties about the file or the information within it save that I used my best endeavors to prepare it accurately given the information sources available to me.

Conclusion

I have done my best to present the information about PP usage in a way that will enable the community to ask and answer questions about PPs in a more objective way. It is my hope that analysis of the information will enable informed consideration of the proposed cPP, technical communities, vision statements and scheme policy makers. 

Already, in this brief analysis I have shown that that the information can help ask and perhaps even answer some key questions relating to the use of PPs. That if we consider the classes of PPs, those purveying vendor-led minimum security functionality vs those enacting policy, regulation or legislation then we immediately see that the make up of a technical community must vary. For the former we might consider the "end user" to be those consuming the information assurance offered by vendors, i.e., those in the commercial sector as well as in government space. For the latter, the end user is in fact the COTS product developer who must ensure that the cited functionality and assurance is in place for their products. So including the end user in a technical community requires an understanding of who that end-user is. For some PP developments it may not be necessary to include an end user.

I believe that the information also shows the need for a very careful consideration of a PP registry, and the management and parameters of the information recorded. National schemes vary in the way in which the PP records are maintained and the CC Portal has not done a fantastic job in collating the data over the years. I would respectfully suggest though that as we move to the use of cPPs and consider the CC MB vision statement that we also consider what information and records we need now and in the future. As PPs are produced answering national, regional or other needs and cPPs are also produced, how we refer to these and keep track of them is very important in comprehending the assurance provided by product evaluations. This work supports the comments made in Double Vision by Sal la Pietra that the CC MB vision statement needs some informed refinement and by Helmut Kurth in Collaborative PPs in which the need for much enhanced guidance on the topic of PPs from the CCMB/CCDB is sorely needed.

By Fiona Pattinson