Passions of an assessor: Donde esta corazon?

Michelle is a passionate infosec pro and assessor. She gets some kudos today for expressing on a personal level the frustrations of many infosec practitioners whose job it is to audit, assess and help improve their clients' defenses. PCI DSS forces those who would do little or nothing for security to do something more. It also encourages those who would do more to do less because it is just enough to deal with a clear and present threat: the audit.

As Josh Corman at the 451 Group likes to say: “Why focus on compliance instead of security? I might be hacked, but I will be fined.” (if you handle cardholder data). Given the amount of client-side attacks and botnet infection data we see, the case could be made otherwise. Corporations are getting attacked daily. They might not be aware of it though, due to the holes in their security defenses, logs, and even alerting practices.

After all, security products can only alert and report on what they have detections for. Based on our testing, that leaves a significant gap with every vendor, between 12 and 83%. Do you know which holes matter on your network and where they are? Want to hear ideas on how to improve and not just pass?

I'm happy to echo Michelle's call for more heart and less check box.

Rick

Thanks for breaking it!

People hire us to break stuff (and lately we're pretty good at it). Well not just, but breaking is part of testing, as is validating You'd think folks wouldn't want to hire us for that, and a lot of times you'd be right. But, this week, we had a large networking vendor in the lab testing their product. On day 2 we discovered a significant vulnerability that we were able to exploit. We replicated it before their eyes. What did the vendor do? He gave our lead engineer a high five!

Why? because, after having visited several labs with this same product, we were the first to find something and not simply give it the 'rubber stamp.' This is why you test. Not just to validate features, but so you can find out what you still need to do to improve it. Good product developers like this one "get it." He just got tremendous value out of the engagement, and has already put in proposals on the spot for additional testing with us. And maybe his competitors have similar issues (which is often the case), so now this vendor is ahead of the game and will likely have it fixed very shortly.

IT Buyers: This is the attitude you want to see in your vendors.

Measuring Security

Vik Phatak, CTO participated on a panel discussion at SourceBoston conference titled "Measuring Security". This discussion explored the ins and outs of testing endpoint protection products, otherwise known as anti-virus/antimalware. Hosted by Andrew Jacquith of Forrester, and also with Peter Stelzhammer of AV-Comparatives, and Mario Vuksan of Reversing Labs. Watch the video link.

AV Testing double standards and independence

NSS Labs’ innovative tests are designed to inform end-users about how products truly perform against today’s motivated attackers. We perform a test or gap analysis on security products, so organizations can understand what is and isn’t being protected, and accurately assess the risk and take steps to mitigate it. While enterprises and government organizations appreciate this valuable, independent analysis, many of the AV vendors do not.

When NSS Labs published its uncensored, real-world results of endpoint protection products (AV), some vendors used the anti-malware testing standards organization (AMTSO) to try to discredit the test. One of their objections was that we recommend against buying products that scored on the bottom third of our test. Sorry, we unabashedly believe malware protection should indeed be the key purchasing criteria for an AV product. And for vendors who claim their anti-spam on the corporate desktop will improve their protection against socially-engineered malware hosted on web sites, that’s just stretching it.

Rather than shoot the messenger, vendors with their customer’s best interests in mind should seek to learn from tests like these in order to improve their products. Unfortunately, that’s usually not the case in the AV world after too many years of self-congratulatory testing and certification.

AMTSO is an AV vendor-driven consortium, and while it can be a useful information sharing organization for AV insiders, it has demonstrated its utter failure as a credible independent organization. Throughout the 3-year history of this organization, AMTSO has failed to evaluate the tests and certifications that most of its vendor members sponsor and fund; e.g. VB100% awards, ICSA Labs and West Coast Labs certifications. These validations are important sales material in the $9B market place, but they wouldn't pass the same AMTSO guidelines that were supposedly applied to the NSS Labs test.

Such market validations are a part of the industry, but can be dangerous when they convey a false sense of security to buyers as they do now. Meanwhile, end-users can stay well informed about what products do - and more importantly - what they DO NOT do, by reading our subscriber-funded research and test reports. If a vendor is complaining about our test, chances are they did poorly on an important metric. Learn what some vendors don’t want you to see by reading our independent anti-malware test reports or the Google Aurora protection analysis report in particular (free to non-clients).

caveat emptor

Questionable Questions (And Some Answers)

Normally, NSS Labs does not engage in public disputes over our test results. However, AVG’s recent blog post about our recent Operation Aurora test grossly misrepresents the facts in an apparent attempt to discredit the results and testers. We have chosen to respond:

The important fact for AVG’s 110 million users is: AVG Internet Security 9 did not stop the Aurora exploit. This was true when we tested on January 29, 2010. And it was still true when we re-tested with their latest version on March 12, 2010—nearly two months after the initial attack became public. See for yourself in this video (the exploit executes calc.exe as proof).

On AVG’s blog, they claim the following:

“This is a screenshot of AVG blocking the Aurora 0-day attack from the AVG Labs.”

• However if you look closely, the screenshot AVG presented shows they were using Firefox, not Internet Explorer. CVE-2010-0249 was a vulnerability in Internet Explorer, not Firefox. Showing Firefox being "protected" displays a fundamental misunderstanding of the nature of the Aurora attack.

"In fact, the exploit is blocked separately by three different security rules of AVG’s product"

• We don’t dispute that AVG has rules, but they did not prevent the exploit. This is why proper testing & QA is important. Further, as you can see in the video (using Internet Explorer), we found that AVG’s warning appears after the exploit successfully gained control of the computer and performed remote code execution (calc.exe).
  

AVG has failed to provide any credible evidence that our test results are incorrect.

From the moment that AVG contacted us with concerns, we sought to share the information required for them to reproduce the attack themselves. The Operation Aurora code was included within the report itself. We have since posted a video on YouTube, and we made it clear that the easiest way to reproduce the test was to use the Metasploit Framework's built-in (free) Aurora exploit and embed a payload of their choice (such as calc.exe). With this free, publicly available information, AVG engineers should have been able to reproduce this attack, as their peers at other vendors have.

However, AVG wanted us to do more…

During our years of testing, we have found that some vendors have abused the time and trust of testers by not doing their homework before making claims that test results are incorrect. We stand by our results. And in cases where vendors insist we have made a mistake, we will work with them to resolve any ambiguities. If it turns out that the vendor is incorrect, we expect to be compensated for our (consulting) time. If we made a mistake, we will publicly correct the error and the vendor bears no cost.

Under these conditions, AVG had nothing to lose if they were confident in their product. That they have chosen a different path speaks volumes.

Vulnerabilities, Exploits & Payloads, Oh My!

Over the past few years at NSS Labs we have interacted with some very smart people and been able to observe which technologies and approaches work better than others. And being immersed in information security product testing, it is easy to forget that not everyone has the same vantage point that we do.

I was having a conversation recently with a very smart Intel Hardware/Software Engineer when the Google / Aurora attack came up. He said that based upon the explanations from AV community, he assumed that the attack was not stoppable… I assured him that was not the case, and that there is a technical solution that can protect against attacks like Aurora. Host Intrusion Prevention technology exists and works by preventing the vulnerability (in this case CVE-2010-0249) from being exploited. If the vulnerability cannot be exploited, the payload (malware) cannot be delivered.

So the question is whether it is easier and better to protect against:

- One (1) vulnerability
- Six (6) exploits
- 55,000+ viruses, Trojans, rootkits and other malware PER DAY



I then shared that in our recent study of seven Endpoint Security (AV) products, only one product (McAfee) protected against Aurora and exploit variants. And that the AV industry as a whole is still focused on chasing malware and, as such, is ill equipped to deal with exploit-based attacks like Operation Aurora.

So the discussion turned to ways in which someone can be infected with malware from the web. Basically, there are two methods:

1. An attack on the User – i.e. tricking someone to download software that contains malware – video codecs, pirated software, fake AV, etc.
   
2. An attack on the computer – i.e. exploits containing malicious payloads (aka “drive-by” exploits).
   

The first is solved by a combination of user education and reputation systems like those provided in Internet Explorer 8, Firefox, Chrome, etc., which warn people that the software they are about to download is infected. Some AV products have this as well. The second is solved by HIPS. Not AV. Why HIPS? Because true HIPS products protect the vulnerability from exploit. They do this by operating in memory and inspecting data as it streams onto a computer as well as inspecting processes before allowing them to run. Traditional AV products do not do this; which is why AV vendors have purchased HIPS technology over the past few years (McAfee bought Entercept, TrendMicro bought ThirdBrigade, Cisco bought Okena, etc.).

But, in general, no vendor seems to have uniform integration of HIPS into their endpoint products either on the consumer or corporate side. It clearly takes some time to integrate these technologies. Unfortunately, the marketing message of “proactive protection” has gotten ahead of the technical reality. This is what we found in our Aurora test: http://nsslabs.com/test-reports/NSSLabs_Vulnerability-based%20Protection-Google-EPPv14.pdf

The bottom line is that the best way to solve the “drive-by” problem is for AV vendors to implement true HIPS which prevents a vulnerability from being exploited, and thus a malware payload cannot be delivered.

-----

However, nearly two weeks after the attack was made public, one product still does not stop the original exploit, and 85% failed to stop additional exploit variants. Few cyber-security attacks were better publicized than this one. It is concerning that such shallow coverage was delivered. Are lesser publicized but equally dangerous vulnerabilities better protected? Probably not.

In lieu of applying patches to thousands of machines, IT professionals rely on network and endpoint security products to provide a virtual shield for these vulnerabilities. Patching is usually prioritized and scheduled monthly. Deferring patching without good vulnerability coverage increases risk. When faced with these test results, organizations would be advised to install the Microsoft patch(es) as soon as possible.

AVG & The Aurora Exploit

Unfortunately, we have observed that some products rely heavily on file-based detection (of malware). These products are scanning for payloads once they hit the disk. The problem with this approach is that exploits occur in memory (e.g. Aurora) and might never touch the disk.

We do not know why AVG failed to prevent the Aurora exploit (which operates in memory). But we know that it did. We observed the exploit successfully gain control of the PC and perform arbitrary remote code execution (the exploit ran Calc.exe as proof). And we did observe AVG detect the attack in Internet Explorer's cache - after the fact.

This video was captured on March 11, 2010. We turned off automatic updates to preserve the version in time, which of course caused multiple red blinky warnings to be issued. We tested again (see 2nd video below) after updating to put to rest any FUD.




To be fair, AVG may have protection for other exploits.

However, as of today, nearly 2 months after the story of the attack first broke, a fully updated AVG still does not provide protection from the the original Aurora exploit. What is additionally concerning is that the product issues a pop-up message telling the user that the threat was detected and quarantined. See for yourself.

This video was captured on March 12, 2010. We turned on automatic updates, plus manually updated to ensure the latest protection was installed.