TippingPoint Tests

In response to inquiries about the blog posting made by TippingPoint President, Alan Kessler, we provide the following:

In August 2009, NSS Labs performed an independent test of the TippingPoint 10 product and determined it only blocked 39% of common exploits. Subsequently, TP came to our lab for private testing for further assistance, as they stated. TP customers can see a spike of hundreds of filters which appeared in October and November.

In early December, NSS Labs released its independent IPS group test of 15 different IPS products submitted by 7 vendors, including TP. The product improved marginally, but is rated ‘caution’ due to its subpar protection on our tests. Now TippingPoint has publicly complained in this tippingpointblog that the test must be inaccurate because it didn’t correspond with the results of their private testing with NSS, with 'customer experience', nor with their internal testing.

3 response points:
1st. These modern IPS products are so complex, that customers will rarely be in position to question or test a vendor properly. And they rarely do when it is a brand name. Very few enterprises have the sophisticated testing tools, expertise and access to exploits like the vendors and a professional security testing lab like NSS. So, having a lot of customers does not necessarily mean they are aware of the true protection they are receiving. In fact, not knowing is a liability in itself for all involved.

2nd. RE: Private testing results. At NSS we don’t use the same attack set in our private testing, as we do in public testing. That would be like getting a copy of the test and answers beforehand, and would give private clients an unfair advantage over other vendors. We do test the same vulnerabilities, but the specific exploits we use vary. This should underscore the integrity of NSS Labs testing principles and procedures. In general, differences in results could be attributable to signatures written too narrowly; e.g. for specific exploits vs vulnerabilities, or to signatures written for a test lab environment.

3rd. We certainly cannot account for any vendor's internal testing procedures. However, the findings of our two previous tests were ultimately corroborated.

As far as delaying the Network IPS Group Test Report. It would be unfair to enterprise readers all around not to disclose validated testing results that could help them mitigate threats that might not be stopped by their defenses as they expect (that could be considered irresponsible non-disclosure). A delay would also not be fair to the other 6 vendors who also participated. As with the previous tests, NSS took great care to validate the results. Deciding to act positively upon them to deliver the better customer protection is the next imperative.