Moved

This blog has moved to www.nsslabs.com/blog
Please update your bookmarks.

Why you need to Test it like a hacker!

Some tests you don't want to be too hard. Like those we take in school that we don't think will mean too much to us in life later on. Say, for some it's abstract poetry of the middle ages, basket weaving in the precambrian era, etc. For these you just want to get by, so when an easy test comes along, the tested party generally breathes a sigh of relief.

In contrast, some tests are hard for our own good. Physical endurance tests before summiting Mt. Whitney or K2. Crash tests of car safety equipment like seat belts, air bags and brakes. You really want to make sure those things work as advertised so they'll function when you need them.

So it is with enterprise security testing, and security product testing in particular. In a world where virtually every antivirus (antimalware) or endpoint security product is 'certified' by two or three different labs, one would think they're all equally good. And especially if they've got a certification from the government, right? Dead wrong. They've all been 'certified' because they've been able to figure out how to pass the test, or because the test is not hard enough, not necessarily survive the crash.

In our experience, there's rarely such a thing as 'too hard' of a test. In order to know how well a product will defend you, you've got to TEST IT LIKE A HACKER. You need to subject the products in your environment to the same stress and attacks that they will face against motivated, persistent adversaries sometimes even using advanced techniques. After all, fixing problems before a breach is always much less expensive than cleaning up the mess afterwards.

As more and more high-profile breaches are disclosed, securing our intellectual property and assets is no longer just a technical issue. NSS Labs makes a lot of its security research and educational content available for free. I encourage you to browse some of the results to find out more.

Redefining the security gateway

This week I'm at the Pacific Crest Emerging Technologies Summit in San Francisco. And security is hot.

Apparently, enterprise IT buyers are not the only ones interested in information security products. Investors - institutional, hedge funds, private equity, etc - are all trying to read the tea leaves of the marketing soup being slung by security vendors. It's a stark contrast to the crowds at BlackHat and Defcon. These investors want to understand which companies will outperform or under-perform their competitors in the marketplace. While they clearly posses great knowledge about the financials of these companies, several are admittedly struggling to understand the technology table-stakes and differentiators required to compete. Increasingly, they're realizing they need to understand the security tech a little better in order to formulate and justify their investment thesis. I'm fielding questions like: Why do we need new security gateways? What is application control about? How are enterprises buying/using the technology? Can opensource security compete? Which approach will win? Which companies have products vs. platforms? With 20 to 40 competing companies in security market segments, surely not all of them can 'perform' and survive long term as stand-alone entities.

In a few hours I'll be tackling some of these questions on a panel with some of the leaders in network security - Barracuda, Fortinet and Sourcefire. This should be a good debate, and we'll have to follow up with some of those larger players who aren't represented, like Check Point, Cisco, HP/TippingPoint, IBM/ISS, Juniper.

RSA update

Like most, I'm recovering from the annual pilgrimage to the RSA conference in San Francisco two weeks ago. As usual, it was a great mecca in which to reconnect with friends, clients, business partners and new folks in the community. I'd especially like to thank all the supporters in our enterprise and vendor advisory boards. The NSS team is grateful for getting 60 of the busiest people in the biz to actively participate in discussions about how to improve information security through testing. It’s a topic that’s garnering momentum. I thank all of you for your input and suggestions on how we can improve and continue to deliver meaningful, actionable information services.

What are your priorities and concerns for 2011? Let us know and you could win a $100 AMEX card. Respondents will receive complementary access to the research results. Take the survey.

New Research

• We have a number of endpoint protection platform (EPP) and network security reports we are rolling out, including EPP evasion, multi-vector attack protection, next generation firewall (NGFW), and firewall (FW). There will be a subsequent post on each of these.
• We have been busy coordinating remediation of an important security issue with a number of firewall vendors. Stay tuned for the full report.
• As Anti-malware continues to fail to protect endpoints, we have been investigating alternatives such as application control (application whitelisting), and secure browsing. While much of this has been performed for private clients in the financial services industry, we are gearing up for a proper group test of these technologies in Q2. Vendors, submit your products. Enterprise buyers, let us know what challenges you’re facing, your criteria and experiences. Contact us. 

Also New from NSS Labs

• We rolled out a new video explaining the use cases for our services and how we can help organizations make informed infosec decisions.
• We also have new collateral which goes into greater detail. See the services overview, or dive into our
• Research calendar
• Consulting and custom testing services for proof of concept and bake-off testing
• Modeling defense-in-depth and tracking risk exposure using our VulnScope service
• Understanding the balance between security and cost with our Security Value Map
• Finally, NSS is actively expanding to meet the demands of our growing Fortune 2000 client base. If you’re a talented, hands-on infosec professional who understands the value of testing and ethical hacking, and is passionate about improving information security, we should talk. Contact us about career opportunities.

AV Testing double standards and independence

NSS Labs’ innovative tests are designed to inform end-users about how products truly perform against today’s motivated attackers. We perform a test or gap analysis on security products, so organizations can understand what is and isn’t being protected, and accurately assess the risk and take steps to mitigate it. While enterprises and government organizations appreciate this valuable, independent analysis, many of the AV vendors do not.

When NSS Labs published its uncensored, real-world results of endpoint protection products (AV), some vendors used the anti-malware testing standards organization (AMTSO) to try to discredit the test. One of their objections was that we recommend against buying products that scored on the bottom third of our test. Sorry, we unabashedly believe malware protection should indeed be the key purchasing criteria for an AV product. And for vendors who claim their anti-spam on the corporate desktop will improve their protection against socially-engineered malware hosted on web sites, that’s just stretching it.

Rather than shoot the messenger, vendors with their customer’s best interests in mind should seek to learn from tests like these in order to improve their products. Unfortunately, that’s usually not the case in the AV world after too many years of self-congratulatory testing and certification.

AMTSO is an AV vendor-driven consortium, and while it can be a useful information sharing organization for AV insiders, it has demonstrated its utter failure as a credible independent organization. Throughout the 3-year history of this organization, AMTSO has failed to evaluate the tests and certifications that most of its vendor members sponsor and fund; e.g. VB100% awards, ICSA Labs and West Coast Labs certifications. These validations are important sales material in the $9B market place, but they wouldn't pass the same AMTSO guidelines that were supposedly applied to the NSS Labs test.

Such market validations are a part of the industry, but can be dangerous when they convey a false sense of security to buyers as they do now. Meanwhile, end-users can stay well informed about what products do - and more importantly - what they DO NOT do, by reading our subscriber-funded research and test reports. If a vendor is complaining about our test, chances are they did poorly on an important metric. Learn what some vendors don’t want you to see by reading our independent anti-malware test reports or the Google Aurora protection analysis report in particular (free to non-clients).

caveat emptor

Maintaining Test Integrity during Private Testing

NSS does not utilize the same exploits in the Private test as in Public tests or certifications. This is to maintain the integrity of the public tests and ensure that vendors engaged in private testing are not given an unfair advantage.Attacks used in private testing target the same range of vulnerabilities used in the public tests. However, different versions of the exploits are used.

· The same range of vulnerabilities are represented in both the private and public tests. However, different exploit variances are used between the two types of test to ensure vendors are writing vulnerability-based signatures in order to adequately protect their customers, and not simply writing exploit-specific signatures to perform well in testing. For example, private tests utilize a higher number of Proof Of Concept (POC) exploits and PCAPs, whereas public testing and certification relies exclusively on NSS’ unique and comprehensive live exploit test harness.

· Vendors who write vulnerability based signatures rather than exploit specific ones will achieve similar results in both private and public tests

· Vendors that write signatures to catch POC PCAPs, but not real exploits and variants, may experience different test results between private and public tests.

Of course, this policy means we have to keep investing in keeping things fresh, accurate and relevant. It's a never-ending job, and we have some of the best people in the industry doing it.

Raising the Bar in Testing

It is important for NSS Labs to periodically raise the bar as the industry advances, both offensively and defensively. In other words, we increase the difficulty of the test to match the needs (as driven by cyber criminals' innovative new attacks), and the capabilities of vendors who have developed new approaches to countering the threats.

This is what NSS Labs has been doing over the past 2 years in particular: raising the bar. As such, our reports and the scores vendors receive on them deserve to be put into context. We perform a number of different types of tests over the past few years:

1. Product Certification - a full review of the whole product; including security effectiveness, performance, manageability and stability.

2. Group Tests - comparative testing of a class of products from leading vendors. Due to the volume of vendors, the depth of the testing may be abbreviated and focused. The browser security, endpoint protection, and Network IPS tests are great examples of these.

3. Security Update Monitor (SUM) Tests - unique to the industry, NSS Labs has been testing IPS products on a monthly basis. Every quarter, we tally the average scores and bestow awards. The attacks in this test set are focused on vulnerability disclosures made the previous month. As such, the volume of new additions is generally in the couple dozen range. Overall, there are currently 300 entries.

4. Individual Product Reports - these are brand new detailed reports that were created during group testing. They capture the nitty gritty details that are rolled up into the group test.

5. Exposure Reports - An industry and NSS first. These reports list specific vulnerabilities that are NOT shielded by an IPS. This information is critical in helping organizations knowing where their protection is and is not. Contact us for confidential demonstration. (Knowing can help identify appropriate mitigations, such as writing custom signatures, implementing new rules, resegmenting the network, or ultimately switching or adding a security product.)

By the Numbers:
Exploit selection and validation is a serious matter. Our engineers take care to identify the greatest risks to enterprises, based on commonly deployed systems, utilizing a modified CVSS.


Certifications performed in 2008 and 2009 tested 600 vulnerabilities. Compare this with other tests, e.g. ICSA IPS tests only 120 vulnerabilities (all server side). Our 600 is more than 480 better, because vendors did not know what they were, thus preventing the 'gaming of the test'.

SUM testing generally adds 20-30 vulnerabilities per month. Relatively small, directly reflective of popular attacks.

Our latest group test utilized 1,159 live exploits. More than 2x the number we used previously for certifications (and nearly 10x more than the next lab).

NSS Labs Tests are Harder Than The Other Guys'...
Difficult, real-world tests are an important part of raising the bar. Marketers like to have big numbers, and when it comes to scores, 100% is the target. Unfortunately, that's not the reality of information security products. We at NSS do not expect any product to catch 100% of the attacks in any of our tests. If they do, we probably are not working hard enough (or the bad guys gave up and went home - unlikely). There are more threats than can be protected against, and depending on the vendor, the margin can be acceptable, or pretty significant.

Take the difficulty of the test into consideration when comparing products. The lower the bar, the easier it is to score 100%. And the less meaningful it is. A 70% score on an NSS Labs Group Test (1,159 exploits) is still 6.7 times more validation than a 100% of 120 exploits in an ICSA Labs(r) test. And a 95% score on an NSS certification is 4.9 times more.

When we at NSS Labs raised the bar on this Q4 2009 IPS Group Test, we really cranked it up. So, if you're wondering why a vendor who previously scored in the 90's is scoring lower on the group test, it's not necessarily because they are slacking off. In most cases it is quite the opposite. Thus, one should be sure to compare products within the test set and methodology. Ergo, the 80% product definitely bested the 17% product by a wide berth. Be very cautious of the latter. And reward those vendors who submitted to this rigorous testing in the first place.

NSS Labs Mission Revisited

NSS Labs' mission is to help raise the bar of information security capabilities and practices, for enterprises and vendors alike.

For enterprises, that means helping them choose and implement better defenses. We do this by performing rigorous testing of leading products in various configurations and publishing test reports for purchase as an information service, much like other analyst firms, like Gartner, IDC, Forrester, etc. (However, we are the only ones that actually perform hands on, comprehensive testing of security products). There are several types of reports, individual certifications (full 360 reviews), comparative group tests, Security Update Monitoring, and our new Exposure Reports. The products in this information service help subscribers understand what is protected and what is not. Nothing protects 100%, so knowing the specifics is important.

For security product vendors, this means testing them against standardized evaluation criteria to establish a baseline, and drawing attention to key issues and requirements. We then test according to best practices methodologies. Our reports also reward those vendors that perform well, and they can use those for marketing. When it comes to improving products, vendors have great resources, and some of the smartest teams around. In addition, they often turn to outside experts for help. NSS is well equipped to assist with this type of private testing and consulting. However, we are always careful to maintain integrity during the process.

Microsoft IE7 zero day exploit - patch released

Today, just 7 days after the discovery of a critical zero-day exploit in Microsoft's popular Internet Explorer (see Microsoft Security Advisory 961051), Microsoft has released its analysis and a public patch via various Windows Update services.

We at NSS Labs has been following this closely, as live exploits have been circulating and growing rapidly, reaching more than 10,000 infected sites (TrendMicro). There are different implementations, including java script and ActiveX that exploit the XML parser in IE versions 5.01 through IE8 beta 2. See the official description and analysis from Microsoft MS08-078 for a complete list of affected versions and systems. And on the more interesting side, HD Moore at BreakingPoint Systems describes his analysis.

Gartner lists NSS Labs certification as criteria for Magic Quadrant

In case anyone is wondering what the value of an NSS certification is, Gartner has recently recognized the value of NSS Labs certifications by adding them to the short list of criteria for products to achieve ranking in the coveted Gartner Magic Quadrant for Network IPS. NSS Labs pioneered the Network Intrusion Prevention Systems (IPS) standards and test methodologies as early as 2002, and these are globally recognized as the de facto gold standard for the industry. 3rd party testing such as NSS Labs group test certification is an important measure of product quality, which carries the highest weighting of all the evaluation criteria.

The fact that NSS was listed before Common Criteria was probably not accidental. The difference between the two evaluations is significant; NSS evaluates real-world security effectiveness and performance, whereas CC primarily evaluates the processes used to create a product.

Note:NSS Labs has completed a number of network IPS product evaluations this year on products from IBM, Juniper and others and are currently performing the industry's only 10 Gbps IPS group test.

We hear time and again from information security managers and CISOs that our reports are helping them make informed decisions that they couldn't make with less rigorous evaluations. Such acknowledgement makes what we do all that more rewarding. On behalf of all the staff and engineers at NSS Labs, I'd like to thank the gentlemen at Gartner for acknowledging the efforts of our product analysts.

P.S. We don't plan to stop at IPS...

Greasing the skids of commerce

"Commerce requires a meeting of the minds between buyer and seller, and it's just not happening. The sellers can't explain what they're selling to the buyers, and the buyers don't buy because they don't understand what the sellers are selling. There's a mismatch between the two; they're so far apart that they're barely speaking the same language." Bruce Schneier on the security industry.

Having been on both sides of the vendor-IT buyer fence, I can definitely relate to both parties frustration. In this vein, some have referred to NSS Labs reports as 'next generation sales collateral', bridging the gap between brochureware and a proof of concept test (and who has time, expertise and resources for all that anyways).

Testing, Testing, 1-2-3

A recent interview/article with NSS Labs' Vik Phatak on how enterprises can setup a test network to evaluate functionality, performance and interoperability of vendor products prior to purchase and deployment. Article in Processor.

To infinity and beyond!

Well, perhaps not as glamorous as Buzz Lightyear's famous launch slogan, but still exciting for the universe, er industry, is our move to 10Gbps testing and beyond. Today we are launching a mini-webinar series starting July 16, discussing high-speed deep packet inspection testing. More information.

Keep It In The Family

I am often asked why we only have single product certifications on our Web site, and why we don't certify an entire product family from each vendor. Well we do, but the problem for the vendor is that it gets very expensive to produce such a certification.

Let me explain.

NSS is ONLY prepared to certify any product after a thorough evaluation of that product. Our view is that performance and security effectiveness BOTH need to be evaluated completely for every product. If you have a range of seven products ranging from 100Mbps to 2Gbps, the vendor might claim that they are all using the same code base, but for them to receive an NSS Approved award we have to verify that fact. After all, if someone tried to convince you that Bart and Lisa were both identical because they are both Simpsons you would be more than a little skeptical, would you not?

We need to put every device in our test rig and subject each one to the same extensive battery of tests that we would for a single product certification. That is the ONLY way to ensure that you, the reader and eventual purchaser of these products, are getting the real information on how these devices will perform in your network. The only thing that stays constant across an entire product family (usually!) is the management interface and usability.

It pains me to see so called "product family certifications" from other sources, because we know how they are produced - after all, those same vendors are our clients also. We read the "reports" and note the lack of any valid performance figures for each of the products. We note the lack of any individual security effectiveness analyses for the individual products. We note also an abundance of "as reported by vendor" statements in some of these, indicating a willingness to take vendor claims on faith without verifying them. They read like a marketing or branding exercise rather than a technical evaluation - a waste of money for the vendor and a waste of time for the reader.

As a testing house, it may be painful but you DO need to test absolutely everything for every single product in the family. A "representative sample" just does not cut it.

You, dear reader, need to know individual performance details, for example. How can you rely on manufacturers performance figures? Isn't that why you read NSS reports in the first place? You need to know if the 1Gbps device is going to give you a true 1Gbps across the wire when you load it up or if you will need to budget for the 2Gbps device instead. If you were buying a TV, wouldn't you want to know why you should consider paying 20% more for the next model in the range? You also need to know that the 100Mbps device doesn't disable fragmentation reassembly or curtail the signature set, opening up huge security holes in the process of trying to get higher performance out of low-end hardware.

That is the value NSS provides with its detailed individual product reports.

Right now, two enlightened vendors are putting their entire UTM product range through our labs, and the results will appear later this year. The advantage for the vendor is that they receive a true NSS Approved award for every device in the product line. The end result for you, dear reader, will not be a single product family report, but one complete report for every device tested, allowing you to make your purchasing or short-listing decisions with absolute confidence.

Rest assured that when you read an NSS report, you will be getting a detailed evaluation of the device under test in terms of usability, security effectiveness and performance. For every single product in the range!

-Bob Walder, CTO/Founder

Toys for Geeks

One of the best things about working in a test lab like NSS is that we get play with all the latest, coolest stuff. Well, cool if you are a geek at heart, that is. It might not be an Aston Martin or a Playstation 4 but the new BP10K from BreakingPoint Systems does at least have white "go faster" stripes on the British racing green front panel....
And go faster it does. NSS has spent almost a year evaluating this equipment for use in its labs, and has been using it in earnest for the last few months. This has been a considerable commitment by NSS, given that our extensive methodologies consist of literally hundreds of different performance tests, and moving them to a new platform is no mean feat.

BreakingPoint has made this possible with a software architecture and GUI design that abstracts as much of the physical layer of the test rig from the logical requirements of the test. As just one example, converting an existing test between in-line layer 2 to routed layer 3 is the work of only a couple of mouse clicks - no need to go through hundreds of test scripts altering IP addresses and default gateways. And there are lots of new cool bells and whistles which will allow us to create incredibly complex tests.

But software isn't cool, is it guys? It's the hardware that gets us excited. And the BP10K can generate complex multi-protocol real-world traffic at line speeds - and that means at 20Gbps (40Gbps full duplex), with 7.5 million concurrent connections and rates of up to 750,000 connections per second from a single appliance with four fiber 10Gbps ports. And you can incorporate multiple appliances in a single test to scale up to hundreds of Gigabits.

In our lab, we have mixed 'n' matched BP10K's and the 2Gbps (4Gbps full duplex) BP1000's to provide us with a total of 60Gbps of traffic generation capability over both 10Gbps fiber and 1Gbps copper interfaces, and this will allow us to standardize on the BPS kit for our Layer 4-7 testing going forward.

All it needs now is a twin exhaust and flashy alloy wheels and we are all set...

-Bob Walder, CTO/Founder

RFI for leading network/test tools

NSS Labs continually evaluates and validates testing tools and best practices. This is a necessary step prior to selecting and implementing the best tools in our test methodologies, which result in our publicly published test reports. Our lab engineering team is thus requesting leading test tool, network infrastructure product and service providers to brief them on their offerings and roadmap. Best in class products will be selected for use in NSS Labs' next generation test facility. More info

Bankinfosecurity.com interview with Rick Moy on Product Testing

My interview with Tom Field of BankInfoSecurity.com at RSA about NSS Labs and how our product evaluations are helping the banking and payment card industry with security and compliance.

Listen to the interview
View page at bankinfosecurity.com

Interview with Martin McKeay at RSA

I had the pleasure of a brief chat with Martin of Networks Security Podcasts about what we do at NSS Labs. Martin is a prolific security blogger, podcaster, and QSA by day. Listen to the interview here: http://www.mckeay.net/2008/04/09/rsa-2008-rick-moy-nss-labs/