AV Testing double standards and independence

NSS Labs’ innovative tests are designed to inform end-users about how products truly perform against today’s motivated attackers. We perform a test or gap analysis on security products, so organizations can understand what is and isn’t being protected, and accurately assess the risk and take steps to mitigate it. While enterprises and government organizations appreciate this valuable, independent analysis, many of the AV vendors do not.

When NSS Labs published its uncensored, real-world results of endpoint protection products (AV), some vendors used the anti-malware testing standards organization (AMTSO) to try to discredit the test. One of their objections was that we recommend against buying products that scored on the bottom third of our test. Sorry, we unabashedly believe malware protection should indeed be the key purchasing criteria for an AV product. And for vendors who claim their anti-spam on the corporate desktop will improve their protection against socially-engineered malware hosted on web sites, that’s just stretching it.

Rather than shoot the messenger, vendors with their customer’s best interests in mind should seek to learn from tests like these in order to improve their products. Unfortunately, that’s usually not the case in the AV world after too many years of self-congratulatory testing and certification.

AMTSO is an AV vendor-driven consortium, and while it can be a useful information sharing organization for AV insiders, it has demonstrated its utter failure as a credible independent organization. Throughout the 3-year history of this organization, AMTSO has failed to evaluate the tests and certifications that most of its vendor members sponsor and fund; e.g. VB100% awards, ICSA Labs and West Coast Labs certifications. These validations are important sales material in the $9B market place, but they wouldn't pass the same AMTSO guidelines that were supposedly applied to the NSS Labs test.

Such market validations are a part of the industry, but can be dangerous when they convey a false sense of security to buyers as they do now. Meanwhile, end-users can stay well informed about what products do - and more importantly - what they DO NOT do, by reading our subscriber-funded research and test reports. If a vendor is complaining about our test, chances are they did poorly on an important metric. Learn what some vendors don’t want you to see by reading our independent anti-malware test reports or the Google Aurora protection analysis report in particular (free to non-clients).

caveat emptor

Questionable Questions (And Some Answers)

Normally, NSS Labs does not engage in public disputes over our test results. However, AVG’s recent blog post about our recent Operation Aurora test grossly misrepresents the facts in an apparent attempt to discredit the results and testers. We have chosen to respond:

The important fact for AVG’s 110 million users is: AVG Internet Security 9 did not stop the Aurora exploit. This was true when we tested on January 29, 2010. And it was still true when we re-tested with their latest version on March 12, 2010—nearly two months after the initial attack became public. See for yourself in this video (the exploit executes calc.exe as proof).

On AVG’s blog, they claim the following:

“This is a screenshot of AVG blocking the Aurora 0-day attack from the AVG Labs.”

• However if you look closely, the screenshot AVG presented shows they were using Firefox, not Internet Explorer. CVE-2010-0249 was a vulnerability in Internet Explorer, not Firefox. Showing Firefox being "protected" displays a fundamental misunderstanding of the nature of the Aurora attack.

"In fact, the exploit is blocked separately by three different security rules of AVG’s product"

• We don’t dispute that AVG has rules, but they did not prevent the exploit. This is why proper testing & QA is important. Further, as you can see in the video (using Internet Explorer), we found that AVG’s warning appears after the exploit successfully gained control of the computer and performed remote code execution (calc.exe).
  

AVG has failed to provide any credible evidence that our test results are incorrect.

From the moment that AVG contacted us with concerns, we sought to share the information required for them to reproduce the attack themselves. The Operation Aurora code was included within the report itself. We have since posted a video on YouTube, and we made it clear that the easiest way to reproduce the test was to use the Metasploit Framework's built-in (free) Aurora exploit and embed a payload of their choice (such as calc.exe). With this free, publicly available information, AVG engineers should have been able to reproduce this attack, as their peers at other vendors have.

However, AVG wanted us to do more…

During our years of testing, we have found that some vendors have abused the time and trust of testers by not doing their homework before making claims that test results are incorrect. We stand by our results. And in cases where vendors insist we have made a mistake, we will work with them to resolve any ambiguities. If it turns out that the vendor is incorrect, we expect to be compensated for our (consulting) time. If we made a mistake, we will publicly correct the error and the vendor bears no cost.

Under these conditions, AVG had nothing to lose if they were confident in their product. That they have chosen a different path speaks volumes.

Vulnerabilities, Exploits & Payloads, Oh My!

Over the past few years at NSS Labs we have interacted with some very smart people and been able to observe which technologies and approaches work better than others. And being immersed in information security product testing, it is easy to forget that not everyone has the same vantage point that we do.

I was having a conversation recently with a very smart Intel Hardware/Software Engineer when the Google / Aurora attack came up. He said that based upon the explanations from AV community, he assumed that the attack was not stoppable… I assured him that was not the case, and that there is a technical solution that can protect against attacks like Aurora. Host Intrusion Prevention technology exists and works by preventing the vulnerability (in this case CVE-2010-0249) from being exploited. If the vulnerability cannot be exploited, the payload (malware) cannot be delivered.

So the question is whether it is easier and better to protect against:

- One (1) vulnerability
- Six (6) exploits
- 55,000+ viruses, Trojans, rootkits and other malware PER DAY



I then shared that in our recent study of seven Endpoint Security (AV) products, only one product (McAfee) protected against Aurora and exploit variants. And that the AV industry as a whole is still focused on chasing malware and, as such, is ill equipped to deal with exploit-based attacks like Operation Aurora.

So the discussion turned to ways in which someone can be infected with malware from the web. Basically, there are two methods:

1. An attack on the User – i.e. tricking someone to download software that contains malware – video codecs, pirated software, fake AV, etc.
   
2. An attack on the computer – i.e. exploits containing malicious payloads (aka “drive-by” exploits).
   

The first is solved by a combination of user education and reputation systems like those provided in Internet Explorer 8, Firefox, Chrome, etc., which warn people that the software they are about to download is infected. Some AV products have this as well. The second is solved by HIPS. Not AV. Why HIPS? Because true HIPS products protect the vulnerability from exploit. They do this by operating in memory and inspecting data as it streams onto a computer as well as inspecting processes before allowing them to run. Traditional AV products do not do this; which is why AV vendors have purchased HIPS technology over the past few years (McAfee bought Entercept, TrendMicro bought ThirdBrigade, Cisco bought Okena, etc.).

But, in general, no vendor seems to have uniform integration of HIPS into their endpoint products either on the consumer or corporate side. It clearly takes some time to integrate these technologies. Unfortunately, the marketing message of “proactive protection” has gotten ahead of the technical reality. This is what we found in our Aurora test: http://nsslabs.com/test-reports/NSSLabs_Vulnerability-based%20Protection-Google-EPPv14.pdf

The bottom line is that the best way to solve the “drive-by” problem is for AV vendors to implement true HIPS which prevents a vulnerability from being exploited, and thus a malware payload cannot be delivered.

-----

However, nearly two weeks after the attack was made public, one product still does not stop the original exploit, and 85% failed to stop additional exploit variants. Few cyber-security attacks were better publicized than this one. It is concerning that such shallow coverage was delivered. Are lesser publicized but equally dangerous vulnerabilities better protected? Probably not.

In lieu of applying patches to thousands of machines, IT professionals rely on network and endpoint security products to provide a virtual shield for these vulnerabilities. Patching is usually prioritized and scheduled monthly. Deferring patching without good vulnerability coverage increases risk. When faced with these test results, organizations would be advised to install the Microsoft patch(es) as soon as possible.

AVG & The Aurora Exploit

Unfortunately, we have observed that some products rely heavily on file-based detection (of malware). These products are scanning for payloads once they hit the disk. The problem with this approach is that exploits occur in memory (e.g. Aurora) and might never touch the disk.

We do not know why AVG failed to prevent the Aurora exploit (which operates in memory). But we know that it did. We observed the exploit successfully gain control of the PC and perform arbitrary remote code execution (the exploit ran Calc.exe as proof). And we did observe AVG detect the attack in Internet Explorer's cache - after the fact.

This video was captured on March 11, 2010. We turned off automatic updates to preserve the version in time, which of course caused multiple red blinky warnings to be issued. We tested again (see 2nd video below) after updating to put to rest any FUD.




To be fair, AVG may have protection for other exploits.

However, as of today, nearly 2 months after the story of the attack first broke, a fully updated AVG still does not provide protection from the the original Aurora exploit. What is additionally concerning is that the product issues a pop-up message telling the user that the threat was detected and quarantined. See for yourself.

This video was captured on March 12, 2010. We turned on automatic updates, plus manually updated to ensure the latest protection was installed.

GroupThink and InfoSec - SecurityBSidesSF

At about 1:45h into the video, Vik Phatak begins his talk about group think in infosec. It is a well-reasoned argument for more aggressive testing in order to improve our defenses. We present methodologies and results from our testing of IPS and Anti-malware products to make the case that we have much work to do and must stay vigilant and innovative.

Protecting vulnerability CVE 2010-0249

On Thursday, 1/21 Microsoft released an out of band patch for CVE-2010-0249; this was the vulnerability that was exploited during the 'aurora operation' against Google and 30+ other companies over the last month. The press coverage and political context makes this a high profile attack, and a story rife with confusion and concern amongst CISOs.

So, we performed some initial testing. On Friday, 1/22 NSS Labs validated that the patch was effective on IE6 on Windows XP, SP2 and IE8 on Windows 7 against multiple variants of the exploit. This means, that the patch appears to cover the vulnerability and multiple variants, and should be applied as soon as possible. The downside is that if you have thousands of PCs, this will take a while, including your own test cycle. Many organizations schedule monthly updates to desktops and servers, so you could be waiting a while.

And in the mean time? That's what security products like Network IPS and endpoint protection (which should include Host IPS) are for. But depending on your vendor's release schedule, and your acceptance/deployment schedule, some waiting/exposure could be involved as well.

Network IPS Group Test Results Available

Over the past few years of working with clients and testing IPS products, we began to notice some troubling trends. Some vendors missed more attacks than seemed acceptable. A number of vendors refused to participate in our testing—even at no cost. Enterprise readers asked some tough questions about current products and issues. And breaches resulting in compromised data continued to increase. Meanwhile, we heard several opinions from our clients, industry analysts, and researchers. These beliefs can be summarized as follows:

· IPS is a mature market. There is relatively little difference between products; thus, management and price are the key purchasing factors

· Best-of-breed products are more effective than those from strategic vendors who provide a wider range of products

· The market leader (from an installed base perspective) provides the best protection

· Organizations are protected as long as they keep their IPS systems updated

This group test set out to determine if these beliefs were correct. In order to garner the greatest participation, and allay any potential concerns of bias, we invited all leading vendors to submit their products at no cost. Every vendor below brought and configured their best Network IPS products; all were generally available (GA), no Beta or otherwise unavailable products were included. The following is a current list of the products that were tested, sorted alphabetically:

1. Cisco IPS 4260 Sensor, version 444.0

2. IBM Proventia Network IPS GX4004, version 29.100

3. IBM Proventia Network IPS GX6116, version 29.100

4. Juniper Networks IDP-250, version 5.0.110090831

5. Juniper Networks IDP-600c, version 5.0.110090831

6. Juniper Networks IDP-800, version 5.0.110090831

7. McAfee M-1250, signature version 5.4.5.23

8. McAfee M-8000, signature version 4.1.59.23

9. Sourcefire 3D 4500, rules version 4.8.2.1

10. Stonesoft StoneGate IPS 1030, 5.0.1 build 5053 update package 261

11. Stonesoft StoneGate IPS 1060, 5.0.1 build 5053 update package 261

12. Stonesoft StoneGate IPS 6105, 5.0.1 build 5053 update package 261

13. TippingPoint (TP) 10 IPS, DV 2.5.2.7834

14. TippingPoint 660N IPS, DV 2.5.2.7834

15. TippingPoint 2500N IPS, DV 2.5.2.7834

Some vendors provided multiple test units of varying performance levels. Across the board, all vendors claimed identical protection across models.

The results will be shocking to most and have already generated plenty of buzz. 1,159 live exploits is the most ever tested. If you currently operate one of these products, or are considering investing in an IPS, the information in this exclusive test report is invaluable.