Maintaining Test Integrity during Private Testing

NSS does not utilize the same exploits in the Private test as in Public tests or certifications. This is to maintain the integrity of the public tests and ensure that vendors engaged in private testing are not given an unfair advantage.Attacks used in private testing target the same range of vulnerabilities used in the public tests. However, different versions of the exploits are used.

· The same range of vulnerabilities are represented in both the private and public tests. However, different exploit variances are used between the two types of test to ensure vendors are writing vulnerability-based signatures in order to adequately protect their customers, and not simply writing exploit-specific signatures to perform well in testing. For example, private tests utilize a higher number of Proof Of Concept (POC) exploits and PCAPs, whereas public testing and certification relies exclusively on NSS’ unique and comprehensive live exploit test harness.

· Vendors who write vulnerability based signatures rather than exploit specific ones will achieve similar results in both private and public tests

· Vendors that write signatures to catch POC PCAPs, but not real exploits and variants, may experience different test results between private and public tests.

Of course, this policy means we have to keep investing in keeping things fresh, accurate and relevant. It's a never-ending job, and we have some of the best people in the industry doing it.

Raising the Bar in Testing

It is important for NSS Labs to periodically raise the bar as the industry advances, both offensively and defensively. In other words, we increase the difficulty of the test to match the needs (as driven by cyber criminals' innovative new attacks), and the capabilities of vendors who have developed new approaches to countering the threats.

This is what NSS Labs has been doing over the past 2 years in particular: raising the bar. As such, our reports and the scores vendors receive on them deserve to be put into context. We perform a number of different types of tests over the past few years:

1. Product Certification - a full review of the whole product; including security effectiveness, performance, manageability and stability.

2. Group Tests - comparative testing of a class of products from leading vendors. Due to the volume of vendors, the depth of the testing may be abbreviated and focused. The browser security, endpoint protection, and Network IPS tests are great examples of these.

3. Security Update Monitor (SUM) Tests - unique to the industry, NSS Labs has been testing IPS products on a monthly basis. Every quarter, we tally the average scores and bestow awards. The attacks in this test set are focused on vulnerability disclosures made the previous month. As such, the volume of new additions is generally in the couple dozen range. Overall, there are currently 300 entries.

4. Individual Product Reports - these are brand new detailed reports that were created during group testing. They capture the nitty gritty details that are rolled up into the group test.

5. Exposure Reports - An industry and NSS first. These reports list specific vulnerabilities that are NOT shielded by an IPS. This information is critical in helping organizations knowing where their protection is and is not. Contact us for confidential demonstration. (Knowing can help identify appropriate mitigations, such as writing custom signatures, implementing new rules, resegmenting the network, or ultimately switching or adding a security product.)

By the Numbers:
Exploit selection and validation is a serious matter. Our engineers take care to identify the greatest risks to enterprises, based on commonly deployed systems, utilizing a modified CVSS.


Certifications performed in 2008 and 2009 tested 600 vulnerabilities. Compare this with other tests, e.g. ICSA IPS tests only 120 vulnerabilities (all server side). Our 600 is more than 480 better, because vendors did not know what they were, thus preventing the 'gaming of the test'.

SUM testing generally adds 20-30 vulnerabilities per month. Relatively small, directly reflective of popular attacks.

Our latest group test utilized 1,159 live exploits. More than 2x the number we used previously for certifications (and nearly 10x more than the next lab).

NSS Labs Tests are Harder Than The Other Guys'...
Difficult, real-world tests are an important part of raising the bar. Marketers like to have big numbers, and when it comes to scores, 100% is the target. Unfortunately, that's not the reality of information security products. We at NSS do not expect any product to catch 100% of the attacks in any of our tests. If they do, we probably are not working hard enough (or the bad guys gave up and went home - unlikely). There are more threats than can be protected against, and depending on the vendor, the margin can be acceptable, or pretty significant.

Take the difficulty of the test into consideration when comparing products. The lower the bar, the easier it is to score 100%. And the less meaningful it is. A 70% score on an NSS Labs Group Test (1,159 exploits) is still 6.7 times more validation than a 100% of 120 exploits in an ICSA Labs(r) test. And a 95% score on an NSS certification is 4.9 times more.

When we at NSS Labs raised the bar on this Q4 2009 IPS Group Test, we really cranked it up. So, if you're wondering why a vendor who previously scored in the 90's is scoring lower on the group test, it's not necessarily because they are slacking off. In most cases it is quite the opposite. Thus, one should be sure to compare products within the test set and methodology. Ergo, the 80% product definitely bested the 17% product by a wide berth. Be very cautious of the latter. And reward those vendors who submitted to this rigorous testing in the first place.

NSS Labs Mission Revisited

NSS Labs' mission is to help raise the bar of information security capabilities and practices, for enterprises and vendors alike.

For enterprises, that means helping them choose and implement better defenses. We do this by performing rigorous testing of leading products in various configurations and publishing test reports for purchase as an information service, much like other analyst firms, like Gartner, IDC, Forrester, etc. (However, we are the only ones that actually perform hands on, comprehensive testing of security products). There are several types of reports, individual certifications (full 360 reviews), comparative group tests, Security Update Monitoring, and our new Exposure Reports. The products in this information service help subscribers understand what is protected and what is not. Nothing protects 100%, so knowing the specifics is important.

For security product vendors, this means testing them against standardized evaluation criteria to establish a baseline, and drawing attention to key issues and requirements. We then test according to best practices methodologies. Our reports also reward those vendors that perform well, and they can use those for marketing. When it comes to improving products, vendors have great resources, and some of the smartest teams around. In addition, they often turn to outside experts for help. NSS is well equipped to assist with this type of private testing and consulting. However, we are always careful to maintain integrity during the process.

CISOs - the Wild List isn't

In a Forbes article this week, Analyst Charlotte Dunlap outlines questions CISOs should be asking when evaluating anti-malware products. There's a common misperception about the 'wild list' and how meaningful it is. If you're buying antivirus products because they're certified by one of the organizations that uses the Wild List in its testing, you're not exactly referencing the most rigorous, meaningful standards.

The Wild List:
- contains a couple hundred virus samples (922 in August 2009 to be exact)
- contains only viruses, except a couple worms (Koobface and Confiker - and dozens of variants of the same), only added a couple months ago. There are no rootkits, trojans, downloaders, or spyware! Note: Trojans and downloaders are arguably the most prevalent initial infectors (exploits are another story)
- contains viruses that have been agreed upon by at least TWO antivirus researchers, who in almost all cases work for AV companies
- is generally 2-3 months behind emerging threats by the time folks agree
Now, this was a good idea back when there were a hundred viruses a month. But, the volume and complexity has outpaced the organization's ability to keep up, and has become less relevant.

In our opinion, the Wild List is NOT representative of threats on the Internet, and it is extremely biased based on sharing and narrow definition of scope. Should you be basing your purchasing decisions off of certifications that use it? (ICSA Labs, VB100, West Coast Labs)

Live Testing, web malware and assumptions...

NSS labs just uploaded the video archive of the Live Testing Webinar we did on 3/31. This was a webinar with live Q&A as a follow up to the initial browser security test report we performed on 6 different web browsers' ability to block socially engineered malware. As we roll out this new test methodology we wanted to give readers a deeper, interactive look into the testing process. There were a few questions from readers about how we did it, why it's more relevant than static or 'in-lab' dynamic testing, and how to interpret the different measurements, etc.
Interestingly we are hearing from two different camps. A few bloggers/journalists are finding their assumptions challenged about their favorite programs; "how can that be?" Meanwhile, 'hard core' security researchers are telling us they are glad to see more comprehensive empirical validation of some of their own data points. Regardless of whether your assumptions were validated or challenged, the data can now drive the conversation - and future research.

Gartner lists NSS Labs certification as criteria for Magic Quadrant

In case anyone is wondering what the value of an NSS certification is, Gartner has recently recognized the value of NSS Labs certifications by adding them to the short list of criteria for products to achieve ranking in the coveted Gartner Magic Quadrant for Network IPS. NSS Labs pioneered the Network Intrusion Prevention Systems (IPS) standards and test methodologies as early as 2002, and these are globally recognized as the de facto gold standard for the industry. 3rd party testing such as NSS Labs group test certification is an important measure of product quality, which carries the highest weighting of all the evaluation criteria.

The fact that NSS was listed before Common Criteria was probably not accidental. The difference between the two evaluations is significant; NSS evaluates real-world security effectiveness and performance, whereas CC primarily evaluates the processes used to create a product.

Note:NSS Labs has completed a number of network IPS product evaluations this year on products from IBM, Juniper and others and are currently performing the industry's only 10 Gbps IPS group test.

We hear time and again from information security managers and CISOs that our reports are helping them make informed decisions that they couldn't make with less rigorous evaluations. Such acknowledgement makes what we do all that more rewarding. On behalf of all the staff and engineers at NSS Labs, I'd like to thank the gentlemen at Gartner for acknowledging the efforts of our product analysts.

P.S. We don't plan to stop at IPS...

How long is a product certification valid?

Recently we have been asked about some of our older product certification reports, whether or not they were still valid; what's changed, etc; some all the way back to 2001. So just how long is a product certification valid?

From an IT Security buyer's perspective, the question is really: how long after the certification does the product still offer similar effectiveness, performance and usability characteristics? How well do they still meet the essential criteria?

1. Unlike static applications, security products with updates (signatures, heuristics, code, patches) change frequently in order to remain effective. (IPS products generally release new signatures on a weekly or daily basis. Antivirus products are becoming increasingly dynamic: last year Kaspersky was pushing hourly updates, and recently McAfee and Symantec have boasted 'real-time' updates.) Thus, a product could increase or decrease effectiveness significantly even 6 months out.
2. Performance can change anytime the code is changed. Yes, even a 'little' maintenance patch can have pronounced effects on throughput, state tables, latency, etc. To be fair, the converse is true: a vendor could release a patch that improves performance. Oh, and the more signatures that are turned on by default generally consume more resources and thus negatively affect performance.
3. Unfortunately, management capabilities don't change often enough. So if an interface is 'so-so', you can probably count on having to live with it for a while. Intuitive, easy-to-use interfaces is one of the underserved areas of security products.
   

These are all things that buyers should check on, whether it is in an NSS Labs report, or some other evaluation. The short answer (which I saved for last) is that a certification can be leveraged by a vendor for one major release cycle. These are generally 18 months long. Any new major release, and buyers should really ask for an updated report. Beware of certifications that are 2, 3, or even 4 or more years old.

Here's a little-known trick! Carefully scrutinize products that have not changed the major version number in a loooong time. Some vendors keep the same major version and modify minor numbers only for years on end in order to circumvent recertification requirements of painful things like common criteria.

NSS Labs does not withdraw certifications after an arbitrary period of time. Perhaps we should; some other labs do, and we could likely make more money to be blunt. Instead, we rely on vendor willingness to 'step up and show their mettle.'

About Deep Packet Inspection

What is DPI? How can it be used effectively? What are the different use cases and requirements for such products?
We recently hosted a webinar in which we discuss this and the methodologies needed to properly evaluate the DPI functionality of network devices under the demanding network conditions in which they will be deployed. The webinar can be viewed here.