Dec 22, 2008

Which endpoint protection products stop IE Exploits?

During the week of Dec 15-18, NSS Labs conducted a series of tests of popular anti-malware and endpoint protection products to evaluate their ability to protect clients from exploits targeting the IE vulnerability. The results are somewhat surprising, showing a broad lack of protection from current enterprise products. Admins are advised to read this and address any gaps ASAP.

Tested antivirus/anti-malware/endpoint protection products include:

  • AVG Internet Security Network Edition v8.0
  • Kaspersky Total Space Security v6.0
  • McAfee Total Protection for Endpoint
  • Sophos Endpoint Security and Control v8.0
  • Symantec Endpoint Protection 11.0.2 MR2
  • Trend Micro Officescan 8.0 SP1 R3
Read the report here.

Exploits vs Drive-by Downloads

What's a "drive-by download" anyways? Recent discussions and the flurry of media articles about the recent Microsoft Internet Explorer vulnerability have given rise to some discussion. So, we at NSS Labs decided to provide this clarification of exploits vs drive-by downloads in response to some research and discussions we've had with a number of end-users and vendors. Our recent research into the Internet Explorer exploits revealed that some vendors and enterprises were not 'framing' the problem properly.

The "drive-by download" is the result of a successful exploit. It is worth noting that the exploit could have executed any arbitrary code, including returning a shell prompt, deleting or encrypting files, etc. But, more likely than not these days, the perpetrator prefers to go unnoticed so they can continue to leverage the newest memeber of their botnet in their quest for world domination. So, more frequently we see keyloggers, trojans, and other 'quiet' culprits. Come to think of it, drive-bys are usually pretty noisy with all the shooting and screeching of tires and such.

So, when vendors and end-users talk about the "download" it can unduly shift the focus towards the result and away from the cause. There are very few exploits compared to hundreds of thousands of pieces of malware. And the exploits are easier to detect - if you are looking in the right place... Network IPS and Host IPS (which can be part of an endpoint protection product) are two great solutions.

Exploits vs Drive-by Downloads.

Dec 17, 2008

Microsoft IE7 zero day exploit - patch released

Today, just 7 days after the discovery of a critical zero-day exploit in Microsoft's popular Internet Explorer (see Microsoft Security Advisory 961051), Microsoft has released its analysis and a public patch via various Windows Update services.

We at NSS Labs has been following this closely, as live exploits have been circulating and growing rapidly, reaching more than 10,000 infected sites (TrendMicro). There are different implementations, including java script and ActiveX that exploit the XML parser in IE versions 5.01 through IE8 beta 2. See the official description and analysis from Microsoft MS08-078 for a complete list of affected versions and systems. And on the more interesting side, HD Moore at BreakingPoint Systems describes his analysis.

Dec 15, 2008

IBM’s Proventia Server for Windows v2 passes NSS Labs PCI Suitability Testing

IBM’s Proventia Server for Windows v2 has successfully passed NSS Labs’ PCI Suitability testing for Host Intrusion Prevention Systems (HIPS). The security effectiveness of Proventia Server for Windows 2.0 was excellent. NSS Labs tested the product on numerous Windows platforms, and a wide range of applications. Proventia Server for Windows 2.0 detected and blocked a total of 64 exploits (98.5%) – all of which were Attacker Initiated. Support for PCI DSS requirements was excellent. Overall, out of 58 tested requirements, the product supports 57 (98%).

Read the complete report on IBM's Proventia Server

Nov 19, 2008

"Strategic" solutions vs. "pure play"

Vik Phatak of NSS Labs discussed the impact of running IPS within a router in this Network World article about integrated security.

Nov 17, 2008

Gartner lists NSS Labs certification as criteria for Magic Quadrant

In case anyone is wondering what the value of an NSS certification is, Gartner has recently recognized the value of NSS Labs certifications by adding them to the short list of criteria for products to achieve ranking in the coveted Gartner Magic Quadrant for Network IPS. NSS Labs pioneered the Network Intrusion Prevention Systems (IPS) standards and test methodologies as early as 2002, and these are globally recognized as the de facto gold standard for the industry. 3rd party testing such as NSS Labs group test certification is an important measure of product quality, which carries the highest weighting of all the evaluation criteria.

The fact that NSS was listed before Common Criteria was probably not accidental. The difference between the two evaluations is significant; NSS evaluates real-world security effectiveness and performance, whereas CC primarily evaluates the processes used to create a product.

Note:NSS Labs has completed a number of network IPS product evaluations this year on products from IBM, Juniper and others and are currently performing the industry's only 10 Gbps IPS group test.

We hear time and again from information security managers and CISOs that our reports are helping them make informed decisions that they couldn't make with less rigorous evaluations. Such acknowledgement makes what we do all that more rewarding. On behalf of all the staff and engineers at NSS Labs, I'd like to thank the gentlemen at Gartner for acknowledging the efforts of our product analysts.

P.S. We don't plan to stop at IPS...

Nov 2, 2008

Test in the "ether"

We at NSS Labs work pretty hard testing network, host and other information security products. Gruelling but rewarding work. Sometimes we get to have a little fun as well, like this recent "Air-Test."

Oct 28, 2008

RSA Conference: Short-term impact of the financial crisis

Here at the RSA Security Conference 2008 in London's ExCel Centre. In a recent interview with netevents I was asked -
Q: "What's the long-term security outlook?"
A: Long-term it’s good for several reasons.
1. Vendors are constantly developing new and improved products.
2. Users are getting more awareness and practical security training.
3. Companies derive competitive advantages by connecting with suppliers, customers and partners. It's increasingly understood by business managers that 'networking stuff' is needed to make money. And thanks to compliance mandates like PCI DSS, security is getting more attention and funding. Or at least it was.

Short-term there’s an increasing danger secondary ripple effects of the financial crisis. IT Security organizations, and other cost centers, will likely be squeezed to invest less time, resources and finances on solving security problems. This would be a dangerous win for the bad guys, who could have weaker, poorer funded defenses to contend with.

Contrast this with the time when governments on both sides of the axis had a clear focus and funding for cryptographic technologies as a lever in the information warfare of WWII.

Oct 16, 2008

Why doesn't NSS Labs have a report on Product X?

Just because you don't see a product evaluation report on our website, it does not mean we have not evaluated the product. There are several possible scenarios:
  • NSS Labs is in process of testing the product. However, due to NDA and confidentiality reasons we cannot disclose whether or not we are testing a given product until the vendor decides to make it public.
  • The product vendor is waiting to release a new major revision before having it (re-)certified.
  • The product was evaluated by NSS Labs, but issues were found that the vendor opted to fix before completing the public certification.
  • The product simply has not yet been evaluated. NSS Labs operates meaningful and rigorous product testing. Not every vendor wishes to subject their product to this process.
NSS Labs makes every effort to involve product vendors in our tests. However, for various reasons, we cannot always secure their participation. Since you as a reader may not know which of the above cases is true, we recommend you inquire with the product vendor's PR or product management team.

Oct 10, 2008

How long is a product certification valid?

Recently we have been asked about some of our older product certification reports, whether or not they were still valid; what's changed, etc; some all the way back to 2001. So just how long is a product certification valid?

From an IT Security buyer's perspective, the question is really: how long after the certification does the product still offer similar effectiveness, performance and usability characteristics? How well do they still meet the essential criteria?
  1. Unlike static applications, security products with updates (signatures, heuristics, code, patches) change frequently in order to remain effective. (IPS products generally release new signatures on a weekly or daily basis. Antivirus products are becoming increasingly dynamic: last year Kaspersky was pushing hourly updates, and recently McAfee and Symantec have boasted 'real-time' updates.) Thus, a product could increase or decrease effectiveness significantly even 6 months out.
  2. Performance can change anytime the code is changed. Yes, even a 'little' maintenance patch can have pronounced effects on throughput, state tables, latency, etc. To be fair, the converse is true: a vendor could release a patch that improves performance. Oh, and the more signatures that are turned on by default generally consume more resources and thus negatively affect performance.
  3. Unfortunately, management capabilities don't change often enough. So if an interface is 'so-so', you can probably count on having to live with it for a while. Intuitive, easy-to-use interfaces is one of the underserved areas of security products.
These are all things that buyers should check on, whether it is in an NSS Labs report, or some other evaluation. The short answer (which I saved for last) is that a certification can be leveraged by a vendor for one major release cycle. These are generally 18 months long. Any new major release, and buyers should really ask for an updated report. Beware of certifications that are 2, 3, or even 4 or more years old.

Here's a little-known trick! Carefully scrutinize products that have not changed the major version number in a loooong time. Some vendors keep the same major version and modify minor numbers only for years on end in order to circumvent recertification requirements of painful things like common criteria.

NSS Labs does not withdraw certifications after an arbitrary period of time. Perhaps we should; some other labs do, and we could likely make more money to be blunt. Instead, we rely on vendor willingness to 'step up and show their mettle.'

Oct 8, 2008

Greasing the skids of commerce

"Commerce requires a meeting of the minds between buyer and seller, and it's just not happening. The sellers can't explain what they're selling to the buyers, and the buyers don't buy because they don't understand what the sellers are selling. There's a mismatch between the two; they're so far apart that they're barely speaking the same language." Bruce Schneier on the security industry.

Having been on both sides of the vendor-IT buyer fence, I can definitely relate to both parties frustration. In this vein, some have referred to NSS Labs reports as 'next generation sales collateral', bridging the gap between brochureware and a proof of concept test (and who has time, expertise and resources for all that anyways).

Oct 6, 2008

North American PCI Community Meeting

We just got back from the North American PCI community meeting. The turnout was about double compared to the 2007 meeting, with all the major QSAs and many name brand retailers and banks in attendance. and the SSC has clearly achieved quite a bit in the last year. Changes to the new PCI DSS version 1.2 were discussed, the first in-person Special Interest Group (SIG) meetings took place, and there were even about 40 vendor exhibits. Branden Williams, Director of the PCI Practice at Verisign, and I sat down and talked about some of the trends and changes in DSS 1.2 (watch the video).

The exhibits were a great opportunity to meet face to face with top technical representatives from these vendors and QSAs. And for them they got direct access to key influencers and decision-makers in the PCI community. Interesting note about the marketing banners, just about all claimed to have an easy PCI Compliance solution. Of course the practitioners know there is no magical "PCI Compliance Solution" and that it is more of a process or journey where the multiple layers of details cannot be avoided. But clearly some marketers are going for the standard easy benefit-oriented taglines, because after all, a marketer's goal is to get you to stop and listen. We heard a lot of merchants and card brands talking about the challenge of getting that next layer of information, which was a great segue into what NSS Labs does to validate vendor product functionality and specifically how it relates to PCI DSS.

Vik and I are serving as secretary for the Wireless Security SIG and I was honored to be able to address the community and provide an update of the SIGs activities. The goal of the SIGs is to make recommendations to the council, which will then review the recommendations, ask questions and render the final decisions. Without revealing too much, it is important to know that we are not taking a technology-centric approach that will make life harder for merchants. Rather, the SIG has decided to take a problem-oriented approach to the task, by focusing first on the problems we are trying to solve for specific groups of users. Very similar to the methods taught by pragmatic marketing. So, Level 3 & 4 merchants who believe they do not have wireless in their network would be one use case; Level 1 & 2s with known use of WiFi would be another. Of course there are many details, and there are sub-groups working on implementation guides and advanced technologies (like BlueTooth and Satellite). If you're a participating organization and would like to 'participate' drop me a line - rmoy AT you know where.

Sep 17, 2008

How important is a user interface after all?

One important thing to consider when evaluating security products for any environment is manageability and usability. Having tested a vast array of products, it's probably safe to say we've seen a spectrum of good bad and ugly interfaces. But I'm not just talking about the look and feel. Far more important is the suitability to task: how well thought out are the most important and frequent tasks that a user will have to complete? Is critical information that I need to take action on represented effectively? How many clicks to get to it? Often times we get both excited and scared by large management frameworks. These can easily tend to present data in engineering terms of tables and lists without much thought to the objective. The last thing I want to see in a console is a lot of text in tables or generic plug-ins to meet some requirement to make data available. With so much R&D cost put into developing speeds, feeds and detection, are we as an industry investing appropriately in the equally important human interfaces?

Sep 12, 2008

Testing, Testing, 1-2-3

A recent interview/article with NSS Labs' Vik Phatak on how enterprises can setup a test network to evaluate functionality, performance and interoperability of vendor products prior to purchase and deployment. Article in Processor.

Aug 13, 2008

About Deep Packet Inspection

What is DPI? How can it be used effectively? What are the different use cases and requirements for such products?
We recently hosted a webinar in which we discuss this and the methodologies needed to properly evaluate the DPI functionality of network devices under the demanding network conditions in which they will be deployed. The webinar can be viewed here.

Jul 24, 2008

Got an opinion about IPS?

If you're currently using an IPS, or in the market for one, we want to hear from you.

As an exercise to accompany our 10Gbps Network IPS group test, we decided to ask end-users what they like and dislike about their current IPS products, how they use them, and what they'd wish for in their next go around.

Simply take the short survey, and you could also win a $50 Amazon gift certificate.

Jun 17, 2008

To infinity and beyond!

Well, perhaps not as glamorous as Buzz Lightyear's famous launch slogan, but still exciting for the universe, er industry, is our move to 10Gbps testing and beyond. Today we are launching a mini-webinar series starting July 16, discussing high-speed deep packet inspection testing. More information.

Jun 9, 2008

10 Gbps Intrusion Prevention - Finally?

Ladies and Gentlemen, start your engines...
The races are now officially on. After a couple of quiet and not so quiet announcements of 10Gbps network IPS products this last year, it appears the market has achieved a quorum. And NSS Labs is continuing its tradition by leading the industry's first group test of these speed-demons. We will be evaluating the security effectiveness of these products at various performance levels, as well as their stability, usability/management features. See the methodology (link below) if you're interested in the details.

Several vendors are offering appliances boasting true 10Gbps throughput, while yet others are offering solutions which combine a load balancer and multiple smaller NIPS appliances. There are operational and financial reasons for both approaches. Some of the trade-offs will be discussed in the final group report to be published in Q4. If you're a vendor, we'd like to hear from you. If you're a user, buyer, or otherwise interested, you may wish to sign up to be alerted when the results are out (newsletter sign-up).

More Info:
- Details of the test announcement
- The preliminary test methodology

May 29, 2008

PCI Research Survey

NSS Labs is collaborating with the Aberdeen Group on a benchmark study regarding best practices for achieving and sustaining PCI DSS compliance. In exchange for your participation in this 15-minute survey, you’ll receive a full copy of the final report when it publishes on 6/30/08 (a $399 value). Individual responses will be kept strictly confidential, and data will only be used in aggregate. Take the survey.

More research from NSS Labs.

May 21, 2008

Interview with TechTarget's Neil Roiter on PCI Suitability Reports

TechTarget's Neil Roiter and I discussed our new PCI Suitability reports, and how these help merchants seeking compliance to evaluate products before they face a PCI assessment. Listen to the podcast.

May 17, 2008

PCI Compliant Products

Kurt Roemer, CTO at Citrix recently discussed PCI Compliant Products on his blog, and I agree with his points thoroughly. So, since he mentioned us so kindly, I thought I'd offer some support and clarification.

I've written before in the NSS Labs blog , there's no such thing as a PCI compliant product . No product will make you compliant, but having the wrong product, or even the right product incorrectly configured could impede validation of compliance. From a terminology perspective, we prefer to say that products address or support compliance (to varying degrees).

That's right, there's no wholesale certification. Different aspects of a product support different requirements either completely, partially, or not at all. And in some cases, the requirements are not even directly applicable to a product. To get this "factual information" that Kurt is calling for, someone has to get their hands dirty with the details. This is what we are about at NSS Labs. Our reports only contain statements of a product's ability to support the specific individual requirements of the PCI DSS that we have empirically validated in the lab. Given that there is no official PCI certification for network/security products (other than PEDs), this is a pretty good start. Note: NSS Labs has been certifying network/security productsagainst our openly published standards since the 1990's. Our new reports focus on the suitability of a product for use in merchant networks, using the PCI DSS as a reference.

In this manner, I believe we're helping security and compliance professionals get beyond broad marketing claims and make more informed buying and implementation decisions. (So far, we've released 2 public PCI Suitability reports  and have a number of others in the queue.)

PS. Eventually I will have 'the talk' with my kids about Santa Claus, Unicorns and PCI compliance. But thankfully, no time soon. ;-)

Thanks Kurt!

May 9, 2008

Keep It In The Family

I am often asked why we only have single product certifications on our Web site, and why we don't certify an entire product family from each vendor. Well we do, but the problem for the vendor is that it gets very expensive to produce such a certification.

Let me explain.

NSS is ONLY prepared to certify any product after a thorough evaluation of that product. Our view is that performance and security effectiveness BOTH need to be evaluated completely for every product. If you have a range of seven products ranging from 100Mbps to 2Gbps, the vendor might claim that they are all using the same code base, but for them to receive an NSS Approved award we have to verify that fact. After all, if someone tried to convince you that Bart and Lisa were both identical because they are both Simpsons you would be more than a little skeptical, would you not?

We need to put every device in our test rig and subject each one to the same extensive battery of tests that we would for a single product certification. That is the ONLY way to ensure that you, the reader and eventual purchaser of these products, are getting the real information on how these devices will perform in your network. The only thing that stays constant across an entire product family (usually!) is the management interface and usability.

It pains me to see so called "product family certifications" from other sources, because we know how they are produced - after all, those same vendors are our clients also. We read the "reports" and note the lack of any valid performance figures for each of the products. We note the lack of any individual security effectiveness analyses for the individual products. We note also an abundance of "as reported by vendor" statements in some of these, indicating a willingness to take vendor claims on faith without verifying them. They read like a marketing or branding exercise rather than a technical evaluation - a waste of money for the vendor and a waste of time for the reader.

As a testing house, it may be painful but you DO need to test absolutely everything for every single product in the family. A "representative sample" just does not cut it.

You, dear reader, need to know individual performance details, for example. How can you rely on manufacturers performance figures? Isn't that why you read NSS reports in the first place? You need to know if the 1Gbps device is going to give you a true 1Gbps across the wire when you load it up or if you will need to budget for the 2Gbps device instead. If you were buying a TV, wouldn't you want to know why you should consider paying 20% more for the next model in the range? You also need to know that the 100Mbps device doesn't disable fragmentation reassembly or curtail the signature set, opening up huge security holes in the process of trying to get higher performance out of low-end hardware.

That is the value NSS provides with its detailed individual product reports.

Right now, two enlightened vendors are putting their entire UTM product range through our labs, and the results will appear later this year. The advantage for the vendor is that they receive a true NSS Approved award for every device in the product line. The end result for you, dear reader, will not be a single product family report, but one complete report for every device tested, allowing you to make your purchasing or short-listing decisions with absolute confidence.

Rest assured that when you read an NSS report, you will be getting a detailed evaluation of the device under test in terms of usability, security effectiveness and performance. For every single product in the range!

-Bob Walder, CTO/Founder

Toys for Geeks

One of the best things about working in a test lab like NSS is that we get play with all the latest, coolest stuff. Well, cool if you are a geek at heart, that is. It might not be an Aston Martin or a Playstation 4 but the new BP10K from BreakingPoint Systems does at least have white "go faster" stripes on the British racing green front panel....
And go faster it does. NSS has spent almost a year evaluating this equipment for use in its labs, and has been using it in earnest for the last few months. This has been a considerable commitment by NSS, given that our extensive methodologies consist of literally hundreds of different performance tests, and moving them to a new platform is no mean feat.

BreakingPoint has made this possible with a software architecture and GUI design that abstracts as much of the physical layer of the test rig from the logical requirements of the test. As just one example, converting an existing test between in-line layer 2 to routed layer 3 is the work of only a couple of mouse clicks - no need to go through hundreds of test scripts altering IP addresses and default gateways. And there are lots of new cool bells and whistles which will allow us to create incredibly complex tests.

But software isn't cool, is it guys? It's the hardware that gets us excited. And the BP10K can generate complex multi-protocol real-world traffic at line speeds - and that means at 20Gbps (40Gbps full duplex), with 7.5 million concurrent connections and rates of up to 750,000 connections per second from a single appliance with four fiber 10Gbps ports. And you can incorporate multiple appliances in a single test to scale up to hundreds of Gigabits.

In our lab, we have mixed 'n' matched BP10K's and the 2Gbps (4Gbps full duplex) BP1000's to provide us with a total of 60Gbps of traffic generation capability over both 10Gbps fiber and 1Gbps copper interfaces, and this will allow us to standardize on the BPS kit for our Layer 4-7 testing going forward.

All it needs now is a twin exhaust and flashy alloy wheels and we are all set...

-Bob Walder, CTO/Founder

May 8, 2008

RFI for leading network/test tools

NSS Labs continually evaluates and validates testing tools and best practices. This is a necessary step prior to selecting and implementing the best tools in our test methodologies, which result in our publicly published test reports. Our lab engineering team is thus requesting leading test tool, network infrastructure product and service providers to brief them on their offerings and roadmap. Best in class products will be selected for use in NSS Labs' next generation test facility. More info

May 7, 2008

Fastest Public Test of a Network IPS

As network traffic continues to grow, so too do the demands on network infrastructures. As a result, multi-gigabit network IPS devices are gaining traction, and providing essential protection in a switched core environment.

Yesterday, NSS Labs released a milestone report on what is the fastest independently verified Network IPS product on the market, to date - the IBM/ISS GX6116. (I say to-date because there are certainly a couple of 10Gig devices that have recently debuted, and we look forward to also testing these). What is notable here is that our tests are not based merely on RFC 2544 (UDP packet blasting), which can inflate a vendor’s performance metrics due to the stateless nature of UDP and typically large packet sizes used. (See our white paper on Pitfalls of Performance Testing). Rather, NSS Labs dedicates a lot of attention to creating real-world multi-protocol test suites across a wide range of use cases.

In our real world tests, we create a complex mix of protocols including HTTP, FTP, SMTP, DNS, etc and pass these through the device under (DUT) test at speeds up to 30 Gbps. This is a live test with deep packet inspection and default or recommended rules turned on. The Proventia GX6116 displayed excellent performance up to 6 Gbps coupled with extremely low latency under all normal traffic conditions.Security effectiveness was also impressive, with excellent coverage above 95% for the most critical vulnerabilities, out of a set of 579 – the largest set of exploits run in any public test.

Read the full report here:

May 6, 2008

PCI Self-Assessment Questionnaires Embrace Use-Case Philosophy!

I have been meaning to comment on this for a while, but better late than never. Earlier this year, the PCI SSC released an updated, and well-thought out collection of self-assessment questionnaires to replace the previous, single questionnaire. This is a very welcome enhancement for a number of reasons, not the least of which is because it shows a clear support for a use-case-based approach - something NSS Labs has been working towards in its own testing.

In fact, we've written a white paper outlining how use cases can help IT Security and Compliance professionals evaluate products for appropriate usage in their environments. In short, know your environment, and specifically what you're trying to protect, and this will help you define more granular (and thus more useful) protection requirements for your control selections (i.e. security products).

There is no silver bullet or magic product, and in fact, as products are increasingly differentiating themselves, defining the requirements early on in the process is increasingly important. For buyers, this means being better prepared, and more discerning in the evaluation process. For vendors, this should be a welcome opportunity to claim some higher ground (in terms of positioning and differentiation) in some very 'mushy' crowded markets where customers turn quickly to price as a differentiator when they can't tell the difference in benefits.

May 5, 2008 interview with Rick Moy on Product Testing

My interview with Tom Field of at RSA about NSS Labs and how our product evaluations are helping the banking and payment card industry with security and compliance.

Listen to the interview
View page at

Apr 27, 2008

Interview with Martin McKeay at RSA

I had the pleasure of a brief chat with Martin of Networks Security Podcasts about what we do at NSS Labs. Martin is a prolific security blogger, podcaster, and QSA by day. Listen to the interview here:

Apr 14, 2008

Rocking RSA

Last week's RSA Conference 2008 in San Francisco was one of the best one's I've ever been to. For purely selfish reasons! NSS Labs had a number of firsts.
  • It was our first time to have a booth at any trade show.
  • Over a dozen product vendors proudly displayed their NSS Approved logos at their booths. These large shiny plaques are about 5 times larger than the typical plastic sign you might otherwise see floating about.
  • Our debut was accompanied by the support of a broad ecosystem of test tool providers, security vendors, and others who shared our booth as partners.
  • We hosted two incredibly well attended Advisory Group sessions on testing and PCI.
  • We released a record number of product certification reports.
  • We threw the undisputed coolest party of all RSA and hung out with the heavy-lifters of the security industry, press, and analyst community. Where else could you get your groove on, and enjoy a shoe shine, shave and massage?
What could we possibly do next? I ask myself.