Network Intrusion Prevention Group Test Released

The security analysts at NSS Labs tested 13 different network IPS products, including stand-alone IPS and multi-function gateways, and one unified threat management product. If your organization is evaluating IPS solutions, or is looking to benchmark your current vendor, then this is the definitive report to read. Data and analysis are based on multiple man-years of complex, real-world testing that mimics how cyber-criminals are working to penetrate corporate defenses (see methodology). No surveys, interviews or soft trends. This is the hard test data upon which organizations base critical, big-dollar decisions.

The report includes valuable information not available anywhere else:

• Total cost of ownership analysis – are you getting the most security for your budget?

• Security effectiveness – how much effort is required to protect all your assets?
• Real-world performance benchmarks – can the device handle your traffic?
• Management and usability insights – how much time is really required to achieve results?

While the full breadth and depth of the research is available only to our subscribers, we are making a summary available to non-clients. Next week/year I will blog more about the key findings and what they mean for IT buyers in 2011.

Tested Products include (alphabetically):
1. CHECKPOINT POWER-1 11065
2. CISCO IPS 4260
3. ENDACE CORE-100 (IDS)
4. FORTINET FORTIGATE 3810A
5. IBM PROVENTIA NETWORK IPS GX6116
6. JUNIPER IDP-8200
7. JUNIPER SRX 3600
8. MCAFEE M-8000
9. NSFOCUS NIPS-1200
10. PALO ALTO NETWORKS PA-4020
11. SOURCEFIRE 3D 4500
12. STONESOFT IPS-1205
13. STONESOFT IPS 3205

Tales from the Trenches of IPS Testing

An update on our current 2010 IPS group test.


Those of you following network IPS know that the last NSS Labs IPS group test in Q4 2009 made quite an impact in the marketplace. The testing soundly destroyed the notion that an organization could buy one of the ‘leading products’ and rest easy
- Vendors with leading market share and analyst accolades were shown to have mixed to poor results on our robust exploit testing;
- Over half the vendors lacking coverage for well-known evasion techniques
- Many put performance ahead of security effectiveness

We were pleased to work collaboratively with many of the vendors in ensuing months to further identify and rectify issues in advance of our next group test. In the process, several vendors released new hardware and software. Excited as we were to test drive the new improved models, we were disappointed in more than a few situations to discover the latest and greatest versions had show-stopper level flaws that could cost their customers a great deal of money and time. A number of vendors requested additional time to remediate flaws, and it was clear that much would change very quickly.

Meanwhile, the size of the NSS Labs 2010 IPS group test has also grown in number of participants and complexity, making this by far the largest, most in-depth group test of its kind.
- We’ve added several new vendor products;
- We’ve refreshed the exploits used, changing and upgrading a third of the content;
- Added additional HTML evasions;

Determined to not let further delays keep our much awaited group test off the streets any longer, we decided to take a phased approach, releasing the test in two parts; Part I will contain five vendors, and Part II will contain the remaining vendors. While this phased release provides enterprises with much needed information on some of the top vendors, it arguably leaves them waiting to see the rest. This may be more or less relevant depending on one’s situation, and which products are under consideration. Certainly some vendor sales teams maybe playing catch up. In the meantime, NSS clients can utilize analyst advisory sessions to receive additional guidance and help fill in the gaps.

So, look for part one of the NSS Labs IPS group test to arrive middle of next week here, and part two in the middle of December. Clients will automatically be notified when it is posted via email alert. If you’re not a client, register for free here, or contact our advisory services group to learn more.

Network IPS Group Test Results Available

Over the past few years of working with clients and testing IPS products, we began to notice some troubling trends. Some vendors missed more attacks than seemed acceptable. A number of vendors refused to participate in our testing—even at no cost. Enterprise readers asked some tough questions about current products and issues. And breaches resulting in compromised data continued to increase. Meanwhile, we heard several opinions from our clients, industry analysts, and researchers. These beliefs can be summarized as follows:

· IPS is a mature market. There is relatively little difference between products; thus, management and price are the key purchasing factors

· Best-of-breed products are more effective than those from strategic vendors who provide a wider range of products

· The market leader (from an installed base perspective) provides the best protection

· Organizations are protected as long as they keep their IPS systems updated

This group test set out to determine if these beliefs were correct. In order to garner the greatest participation, and allay any potential concerns of bias, we invited all leading vendors to submit their products at no cost. Every vendor below brought and configured their best Network IPS products; all were generally available (GA), no Beta or otherwise unavailable products were included. The following is a current list of the products that were tested, sorted alphabetically:

1. Cisco IPS 4260 Sensor, version 444.0

2. IBM Proventia Network IPS GX4004, version 29.100

3. IBM Proventia Network IPS GX6116, version 29.100

4. Juniper Networks IDP-250, version 5.0.110090831

5. Juniper Networks IDP-600c, version 5.0.110090831

6. Juniper Networks IDP-800, version 5.0.110090831

7. McAfee M-1250, signature version 5.4.5.23

8. McAfee M-8000, signature version 4.1.59.23

9. Sourcefire 3D 4500, rules version 4.8.2.1

10. Stonesoft StoneGate IPS 1030, 5.0.1 build 5053 update package 261

11. Stonesoft StoneGate IPS 1060, 5.0.1 build 5053 update package 261

12. Stonesoft StoneGate IPS 6105, 5.0.1 build 5053 update package 261

13. TippingPoint (TP) 10 IPS, DV 2.5.2.7834

14. TippingPoint 660N IPS, DV 2.5.2.7834

15. TippingPoint 2500N IPS, DV 2.5.2.7834

Some vendors provided multiple test units of varying performance levels. Across the board, all vendors claimed identical protection across models.

The results will be shocking to most and have already generated plenty of buzz. 1,159 live exploits is the most ever tested. If you currently operate one of these products, or are considering investing in an IPS, the information in this exclusive test report is invaluable.

TippingPoint Tests

In response to inquiries about the blog posting made by TippingPoint President, Alan Kessler, we provide the following:

In August 2009, NSS Labs performed an independent test of the TippingPoint 10 product and determined it only blocked 39% of common exploits. Subsequently, TP came to our lab for private testing for further assistance, as they stated. TP customers can see a spike of hundreds of filters which appeared in October and November.

In early December, NSS Labs released its independent IPS group test of 15 different IPS products submitted by 7 vendors, including TP. The product improved marginally, but is rated ‘caution’ due to its subpar protection on our tests. Now TippingPoint has publicly complained in this tippingpointblog that the test must be inaccurate because it didn’t correspond with the results of their private testing with NSS, with 'customer experience', nor with their internal testing.

3 response points:
1st. These modern IPS products are so complex, that customers will rarely be in position to question or test a vendor properly. And they rarely do when it is a brand name. Very few enterprises have the sophisticated testing tools, expertise and access to exploits like the vendors and a professional security testing lab like NSS. So, having a lot of customers does not necessarily mean they are aware of the true protection they are receiving. In fact, not knowing is a liability in itself for all involved.

2nd. RE: Private testing results. At NSS we don’t use the same attack set in our private testing, as we do in public testing. That would be like getting a copy of the test and answers beforehand, and would give private clients an unfair advantage over other vendors. We do test the same vulnerabilities, but the specific exploits we use vary. This should underscore the integrity of NSS Labs testing principles and procedures. In general, differences in results could be attributable to signatures written too narrowly; e.g. for specific exploits vs vulnerabilities, or to signatures written for a test lab environment.

3rd. We certainly cannot account for any vendor's internal testing procedures. However, the findings of our two previous tests were ultimately corroborated.

As far as delaying the Network IPS Group Test Report. It would be unfair to enterprise readers all around not to disclose validated testing results that could help them mitigate threats that might not be stopped by their defenses as they expect (that could be considered irresponsible non-disclosure). A delay would also not be fair to the other 6 vendors who also participated. As with the previous tests, NSS took great care to validate the results. Deciding to act positively upon them to deliver the better customer protection is the next imperative.

NSS Awards First Gold in 5 Years

Yes, it's true. After a long 5 years of waiting for the next great product, at RSA Conference 2009 this year, we bestowed the prestigious NSS Labs Gold Award to IBM/ISS for it's Proventia Network IPS GX6116. IBM's was the first IPS to pass our new requirements for Gold, including the monthly recurring Security Update Monitor (SUM) program testing.

The GX6116 scored an average of 98.6% over the 3 consecutive months of testing. This new recurring testing program ensures that vendors are keeping up with current threat protection levels as advertised. Each month our engineers add new attacks to the test set according to our modified CVSS ranking of enterprise-relevant vulnerabilities. Unlike other tests, the vendors do not know which exploits will be used in this blind test. So 98.6% is pretty impressive. Most other products don't do nearly as well.

Also to be commended is the 8Gbps of real-world throughput achieved by the GX. Certainly, the IBM team worked hard and should be proud of their accomplishments on this rigorous test program. Here is Brian Truskowski, General Manager of IBM/ISS, accepting the NSS Gold Award; and his team: Dan Holden, John Pirc, Eric York, Greg Adams.

IBM isn't the only participant in the program. You can look forward to monthly testing from McAfee as well (coming soon).

First 10Gbps IPS certification: McAfee M-8000 receives NSS Labs Approved

10Gbps:
NSS Labs just released the first 10Gbps IPS certification as part of our 10Gbps IPS group test. A number of vendors are offering 10Gbps appliances: Juniper, McAfee, Enterasys, Force10, Sourcefire. McAfee's M-8000 was the first to pass our extensive testing and receive certification. In addition to meeting the rigorous performance requirements, the product scored exceptionally well on the security effectiveness testing. Read the full report here.

Still other vendors are taking the solution approach by including a load balancer and multiple IPS devices. It should be noted, these could use any reasonable switching approach to stack/VLAN multiple physical IPS devices into one logical unit. Think of products from the likes of: IBM, Cisco, Crossbeam (Chassis/Blade), Sourcefire, TippingPoint, TopLayer, etc. Depending on what a company already has installed, and what their growth/infrstructure plans look like, this model could also work well. It will come down to a TCO and effectiveness decision.

It should be noted that this was an award that was a long time in the making since we announced the testing in the summer of 2008; and many vendors had announced products well before that. Indeed there are many reasons why it takes so long. #1 - It's hard to get right. It is not necessarily easy for a vendor that has a 'successful' 1Gbps IPS to deliver the same quality product that truly meets 10Gbps requirements. We just held a technical webinar on the topic of 10Gbps IPS. We covered the challenges that vendors face when making a 10 Gbps IPS, as well as those faced by buyers who are evaluating these products. The webinar is recorded and available here. I was pleasantly surprised to receive several comments that this was the "best webinar ever," and very informative. If you don't have time to listen to the webinar, you can probably at least peruse the slides.

As we've seen in our testing, there are plenty of gotchas to look out for. And for this large and complex of a purchase, most of the potential buyers do NOT have the capabilities to adequately evaluate and test such a product. In such cases it should really behoove the vendors who have done a good job to have their products validated by a competent 3rd party. So be sure to ask your vendor what kind of testing and certification the product has gone through. (OK, somewhat of a trick question: I must confess I don't know of any other lab capable of doing the level of testing that we do, either in terms of throughput or security ;-)

/rick

Webinar: 10Gbps Intrusion Prevention

Are 10Gbps network IPS products mature enough for deployment? Depends... Join our upcoming webcast to find out why 10Gbps IPS is more than 10 times more difficult to get right than 1Gbps IPS. NSS Labs' Vik Phatak will also walk through a checklist of criteria to look for when evaluating products. We'll also give a behind the scenes look at how we implement our industry standard IPS test methodology using products like BreakingPoint.

You can also look forward to some real experiences culled from our 10Gbps IPS group test. The first results forthcoming end of January 09 and the full report end of Q1.

Which endpoint protection products stop IE Exploits?

During the week of Dec 15-18, NSS Labs conducted a series of tests of popular anti-malware and endpoint protection products to evaluate their ability to protect clients from exploits targeting the IE vulnerability. The results are somewhat surprising, showing a broad lack of protection from current enterprise products. Admins are advised to read this and address any gaps ASAP.

Tested antivirus/anti-malware/endpoint protection products include:

• AVG Internet Security Network Edition v8.0
• Kaspersky Total Space Security v6.0
• McAfee Total Protection for Endpoint
• Sophos Endpoint Security and Control v8.0
• Symantec Endpoint Protection 11.0.2 MR2
• Trend Micro Officescan 8.0 SP1 R3

Read the report here.

Exploits vs Drive-by Downloads

What's a "drive-by download" anyways? Recent discussions and the flurry of media articles about the recent Microsoft Internet Explorer vulnerability have given rise to some discussion. So, we at NSS Labs decided to provide this clarification of exploits vs drive-by downloads in response to some research and discussions we've had with a number of end-users and vendors. Our recent research into the Internet Explorer exploits revealed that some vendors and enterprises were not 'framing' the problem properly.

The "drive-by download" is the result of a successful exploit. It is worth noting that the exploit could have executed any arbitrary code, including returning a shell prompt, deleting or encrypting files, etc. But, more likely than not these days, the perpetrator prefers to go unnoticed so they can continue to leverage the newest memeber of their botnet in their quest for world domination. So, more frequently we see keyloggers, trojans, and other 'quiet' culprits. Come to think of it, drive-bys are usually pretty noisy with all the shooting and screeching of tires and such.

So, when vendors and end-users talk about the "download" it can unduly shift the focus towards the result and away from the cause. There are very few exploits compared to hundreds of thousands of pieces of malware. And the exploits are easier to detect - if you are looking in the right place... Network IPS and Host IPS (which can be part of an endpoint protection product) are two great solutions.

Exploits vs Drive-by Downloads.

"Strategic" solutions vs. "pure play"

Vik Phatak of NSS Labs discussed the impact of running IPS within a router in this Network World article about integrated security.

Gartner lists NSS Labs certification as criteria for Magic Quadrant

In case anyone is wondering what the value of an NSS certification is, Gartner has recently recognized the value of NSS Labs certifications by adding them to the short list of criteria for products to achieve ranking in the coveted Gartner Magic Quadrant for Network IPS. NSS Labs pioneered the Network Intrusion Prevention Systems (IPS) standards and test methodologies as early as 2002, and these are globally recognized as the de facto gold standard for the industry. 3rd party testing such as NSS Labs group test certification is an important measure of product quality, which carries the highest weighting of all the evaluation criteria.

The fact that NSS was listed before Common Criteria was probably not accidental. The difference between the two evaluations is significant; NSS evaluates real-world security effectiveness and performance, whereas CC primarily evaluates the processes used to create a product.

Note:NSS Labs has completed a number of network IPS product evaluations this year on products from IBM, Juniper and others and are currently performing the industry's only 10 Gbps IPS group test.

We hear time and again from information security managers and CISOs that our reports are helping them make informed decisions that they couldn't make with less rigorous evaluations. Such acknowledgement makes what we do all that more rewarding. On behalf of all the staff and engineers at NSS Labs, I'd like to thank the gentlemen at Gartner for acknowledging the efforts of our product analysts.

P.S. We don't plan to stop at IPS...

Got an opinion about IPS?

If you're currently using an IPS, or in the market for one, we want to hear from you.

As an exercise to accompany our 10Gbps Network IPS group test, we decided to ask end-users what they like and dislike about their current IPS products, how they use them, and what they'd wish for in their next go around.

Simply take the short survey, and you could also win a $50 Amazon gift certificate.

10 Gbps Intrusion Prevention - Finally?

Ladies and Gentlemen, start your engines...
The races are now officially on. After a couple of quiet and not so quiet announcements of 10Gbps network IPS products this last year, it appears the market has achieved a quorum. And NSS Labs is continuing its tradition by leading the industry's first group test of these speed-demons. We will be evaluating the security effectiveness of these products at various performance levels, as well as their stability, usability/management features. See the methodology (link below) if you're interested in the details.

Several vendors are offering appliances boasting true 10Gbps throughput, while yet others are offering solutions which combine a load balancer and multiple smaller NIPS appliances. There are operational and financial reasons for both approaches. Some of the trade-offs will be discussed in the final group report to be published in Q4. If you're a vendor, we'd like to hear from you. If you're a user, buyer, or otherwise interested, you may wish to sign up to be alerted when the results are out (newsletter sign-up).

More Info:
- Details of the test announcement
- The preliminary test methodology