Why you need to Test it like a hacker!

Some tests you don't want to be too hard. Like those we take in school that we don't think will mean too much to us in life later on. Say, for some it's abstract poetry of the middle ages, basket weaving in the precambrian era, etc. For these you just want to get by, so when an easy test comes along, the tested party generally breathes a sigh of relief.

In contrast, some tests are hard for our own good. Physical endurance tests before summiting Mt. Whitney or K2. Crash tests of car safety equipment like seat belts, air bags and brakes. You really want to make sure those things work as advertised so they'll function when you need them.

So it is with enterprise security testing, and security product testing in particular. In a world where virtually every antivirus (antimalware) or endpoint security product is 'certified' by two or three different labs, one would think they're all equally good. And especially if they've got a certification from the government, right? Dead wrong. They've all been 'certified' because they've been able to figure out how to pass the test, or because the test is not hard enough, not necessarily survive the crash.

In our experience, there's rarely such a thing as 'too hard' of a test. In order to know how well a product will defend you, you've got to TEST IT LIKE A HACKER. You need to subject the products in your environment to the same stress and attacks that they will face against motivated, persistent adversaries sometimes even using advanced techniques. After all, fixing problems before a breach is always much less expensive than cleaning up the mess afterwards.

As more and more high-profile breaches are disclosed, securing our intellectual property and assets is no longer just a technical issue. NSS Labs makes a lot of its security research and educational content available for free. I encourage you to browse some of the results to find out more.

Stopping malware with a browser

This week we released another report on socially-engineered malware protections delivered by browsers. While most articles and blogs seemed to interpret the results properly, there were some inaccuracies that we wish to address. Security is a complex field and the terminology can sometimes be misinterpreted. This can be compounded when vendors who did poorly add their spin, or when the data challenges one’s own beliefs.

Stopping Malware vs. Security:
Some of the key security terms are clarified in a previous post, especially relating to “most secure” browser. Some articles incorrectly stated that we found IE to be the “most secure” browser. What we tested was browser effectiveness at stopping malware from reaching a user or their PCs, not the security of the browser itself or its plug-ins. Modern browsers are a wonderful, free additional layer of protection. They work well with your favorite antivirus software. Browsers however will not stop malware coming via email or USB drives.

The Malware threat:
Malware is arguably the largest security threat facing users today – with more than 60,000 unique, new samples entering circulation each day (source: McAfee). There’s a $14B industry addressing the problem of malware. These test results challenge the comfortable status quo of many of the vendors. The notion that a free product adds so much protection can easily upset the industry apple cart. The assertion that the focus of the test was narrow (made by Google and some others) flies in the face of all generally accepted data. To say one’s browser was “built with security in mind” is nice, but marketing speak. What we’ve offered is hard data about malware protection. The exploitability of the browser is also a very important topic. But, even in this case, data doesn’t support Chrome being ‘more secure’ (i.e. less vulnerable) than other browsers (see CVEs, Secunia disclosures, etc.). We do sincerely applaud the innovations and bug bounties though, and encourage all vendors to build more security in.

Versions tested:
The claim that we tested an “old” version of Chrome is patently false. As stated in the report, the test was run in late September, when version 6 was the current browser. Since then Google has released two other so-called “major” releases, none of which have claimed improvements to the tested SafeBrowsing functionality. Here is the Timeline for Chrome Version Release:
Stable Version Release Date
5.0.375      2010-05-25
6.0.472      2010-09-02
7.0.517      2010-10-21
8.0.552      2010-12-02
Furthermore, Chrome’s sandboxing is designed for exploits protection; it does not protect against socially-engineered malware. You click it, it runs.

At the time of the test, Opera’s website marketed protection from malware as a feature, yet our results showed no protection was available. AVG officials have separately acknowledged that the integration of its technology was not yet complete, confirming our results. Features should only be marketed after they are actually in the product.

Application Reputation:
In a world of dizzying information, there’s much rush to a sound bite of who “won”. The most significant technological message of this test may have been overlooked. This test benchmarked the world’s first implementation of an application reputation system within free web browsers that goes beyond simple black lists. Nascent stand-alone security products such as SolidCore (acquired by McAfee), Bit9 and CoreTrace utilize what is commonly referred to as white listing, and commercial endpoint security products are starting to include some form of this as well. Apple uses a “walled garden” approach to limit exposure to malware on its tightly controlled platforms by pre-approving apps.

In web browsers, so far we’ve seen just black listing. A URL or application is either known to be bad, or unknown. What’s unique about Microsoft’s approach in the IE9 browser is that applications have 3 states: known good (white), known bad (black) and unknown (grey). The combination of good and bad indicators is clearly powerful, stopping 99% of malware via the web download vector. The use of application reputation to identify good applications and bad ones is unique to IE, for now. Will other vendors follow Microsoft’s lead?

Methodology and Open Invitations:
No vendor has influence over what/how we test or where we get malware from. We constantly run our Live Test network with a variety of security products - antivirus, browsers, and other network devices.
Over the past 2 years we’ve been running these tests, NSS has discussed results and methodology (which is included in the report) with all of the browser vendors; even providing sample URLs for validation to them in past tests. Some had even privately acknowledged issues.

Our past invitations to become more involved in the ongoing testing and review of results still stands.

Threat Types and Terminology

Terminology used to describe attacks is often misunderstood by the broader public. Thus, we are providing this brief explanation of threat types and the terms we use in our reports.

End users and their computers face a number of different attack types. At a high level there are two: 1) Socially-engineered attacks target the user, and work only when the user is tricked into performing an action; running a malicious file or giving up personal data to a fraudulent site. 2) Other attacks target vulnerabilities in systems and applications. The following chart gives a rough breakdown of common threats against end user systems.

Layers of Security
These types of security threats can be mitigated by a range of security products; including IPS, UTM, SWG appliances, and on the endpoint: Internet security suites, most anti-malware products, and even web browsers. Modern browsers have implemented an additional layer of security to help users differentiate between good and bad web sites and downloads.

When selecting security products, either for home or business environments, it's often hard to tell from the marketing literature which products actually stop threats. And protection levels offered by products in the different categories can vary greatly. The above taxonomy should help you ask more specific questions of vendors. It also acts as a guide to terminology used in NSS Labs test reports.

Security products protecting users and their computers
When someone says “Product X stops more malware, exploits etc.” or “Product X offers better malware or exploit protection”, what they mean is that Product X inspects traffic passing through it and stops these attacks from reaching and/or affecting the end user or the operating system.

Security products themselves susceptible to threats
In addition, security suites and browsers (and their plug-ins) can be susceptible to exploits if the software has vulnerabilities in them. When someone says “browser X is more secure” what they are trying to say is that browser X has fewer vulnerabilities. Unfortunately, most software, and all browsers have vulnerabilities. For example in the first 9 months of 2010, Microsoft Internet Explorer had 43 new published vulnerabilities, while Google Chrome had 106, according to Secunia research.

For more exhaustive treatments on threat types including product test results, consult our research services at nsslabs.com.

Client-side expoit demonstration videos

NSS has been testing exploits and exploit-detection/protection products for the last 10 years or so, dating back to the early days of IDS and IPS . It's arguably one of our specialties. And we have written before about the differences between malware, exploits and vulnerabilities. Yet, as these pernicious threats are elevated into mainstream consumer and enterprise awareness, we are seeing quite a bit of terminology confusion. So, for our latest Host Intrusion Prevention System (HIPS) test of enterprise anti-malware products, we created some demonstration videos of client-side exploits used in the test.

Product	Vulnerability	Video Demonstration
Panda	CVE-2010-0249	http://www.youtube.com/watch?v=VPEzOZniuls
Panda	CVE-2010-0806	http://www.youtube.com/watch?v=_fstkScB7YM
F-Secure	CVE-2010-0249	http://www.youtube.com/watch?v=DdvtTuwfa5I
ESET	CVE-2006-0003	http://www.youtube.com/watch?v=U9WRkQyZvb8
Sophos	CVE-2006-4704	http://www.youtube.com/watch?v=F2gfNLzqBwg
Symantec	CVE-2010-0483	http://www.youtube.com/watch?v=RqQLmKI_R30

Client-side exploits

Client-side exploits are aggressive weapons used by cybercriminals that allow them to silently take control of computers that visit a web site. The infected widgets on nearly 5 million Network Solutions sites are a prime example (see Krebs' report on Armorize discovery). And NSS Labs' Q2 2010 test of 10 endpoint protection products for HIPS evaluates protection against client-side exploits.


In recent discussions with clients and journalists we've found the need to clarify some definitions of how these attacks work, and how they're different from typical socially-engineered malware campaigns. So, the following definitions and analogies are provided in an effort to provide clarification, as well as to bridge an ongoing communication gap between security vendors and their customers.

Vulnerability

Like a locked door that can be opened with the right key or combination, a vulnerability is a bug in software code that allows a product to be exploited. An example of a software vulnerability is improperly-defined memory usage within a function that enables content sent to a specific memory location to be run with privileged rights.

Exploit

An exploit is a specially crafted code sequence which can ”trigger” or ”unlock” a vulnerability within an application, such as a heap spray, buffer overflow attack, etc. An exploit can be hiding in an infected website (client-side attack) where it ambushes visiting computers or be launched from an another computer (remote attack).

Payload

The payload is the content that gets delivered once the vulnerable application has been exploited. Payloads are the actions that are performed on the compromised target computer, such as command execution, writing a file to disk, returning a reverse shell, etc. This may be malware, but does not have to be.

The Test

The test utilized 123 common and public vulnerabilities dating from 2006 to 2010. These vulnerabilities were exploited when a user visited an infected web page hosting the attack code. The attacks occurred in two stages:
1. The attacker caused a specially-crafted stream of data and code to be delivered to a precise location. This exploited the victim’s computer, gaining the attacker the ability to perform arbitrary code execution.
2. Malicious code was silently executed on the victim’s computer.

If the attack can be thwarted in Stage One (successful exploit), then it cannot progress to Stage Two, where a malicious payload can be delivered. As long as the exploit is not defeated, then installing malware is just one of many possible actions the attacker can take. And the choice of malicious code is nearly infinite. Since there are far fewer exploits than malware, it is imperative that attacks be defeated in the earliest possible stage. In other words, it is advantageous for AV suites to detect the exploit vs. chasing malware samples.

Results and Next Steps

Unfortunately, 75% of corporate users are under-protected, based on vendor market share and their respective scores, which ranged dramatically from 29% to 100%; and even worse for variants. Depending which product you have, you may have significant cause for concern. So, what's next?

For starters, if you're an organization with critical data behind any of these products, I suggest you buy and read the full report here. There has been a lot of press coverage of the report highlights, but not the details, generally reporting failure of AV suites in this area. While you may not be ripping out thousands of endpoint deployments, you may be asking your vendor some tough questions and setting expectations for improvement. Exploit detection & blocking is clearly an area that the security industry needs to focus on going forward. Got questions about other threat mitigation options, consult our other reports, or contact us.

Measuring Security

Vik Phatak, CTO participated on a panel discussion at SourceBoston conference titled "Measuring Security". This discussion explored the ins and outs of testing endpoint protection products, otherwise known as anti-virus/antimalware. Hosted by Andrew Jacquith of Forrester, and also with Peter Stelzhammer of AV-Comparatives, and Mario Vuksan of Reversing Labs. Watch the video link.

AV Testing double standards and independence

NSS Labs’ innovative tests are designed to inform end-users about how products truly perform against today’s motivated attackers. We perform a test or gap analysis on security products, so organizations can understand what is and isn’t being protected, and accurately assess the risk and take steps to mitigate it. While enterprises and government organizations appreciate this valuable, independent analysis, many of the AV vendors do not.

When NSS Labs published its uncensored, real-world results of endpoint protection products (AV), some vendors used the anti-malware testing standards organization (AMTSO) to try to discredit the test. One of their objections was that we recommend against buying products that scored on the bottom third of our test. Sorry, we unabashedly believe malware protection should indeed be the key purchasing criteria for an AV product. And for vendors who claim their anti-spam on the corporate desktop will improve their protection against socially-engineered malware hosted on web sites, that’s just stretching it.

Rather than shoot the messenger, vendors with their customer’s best interests in mind should seek to learn from tests like these in order to improve their products. Unfortunately, that’s usually not the case in the AV world after too many years of self-congratulatory testing and certification.

AMTSO is an AV vendor-driven consortium, and while it can be a useful information sharing organization for AV insiders, it has demonstrated its utter failure as a credible independent organization. Throughout the 3-year history of this organization, AMTSO has failed to evaluate the tests and certifications that most of its vendor members sponsor and fund; e.g. VB100% awards, ICSA Labs and West Coast Labs certifications. These validations are important sales material in the $9B market place, but they wouldn't pass the same AMTSO guidelines that were supposedly applied to the NSS Labs test.

Such market validations are a part of the industry, but can be dangerous when they convey a false sense of security to buyers as they do now. Meanwhile, end-users can stay well informed about what products do - and more importantly - what they DO NOT do, by reading our subscriber-funded research and test reports. If a vendor is complaining about our test, chances are they did poorly on an important metric. Learn what some vendors don’t want you to see by reading our independent anti-malware test reports or the Google Aurora protection analysis report in particular (free to non-clients).

caveat emptor

CISOs - the Wild List isn't

In a Forbes article this week, Analyst Charlotte Dunlap outlines questions CISOs should be asking when evaluating anti-malware products. There's a common misperception about the 'wild list' and how meaningful it is. If you're buying antivirus products because they're certified by one of the organizations that uses the Wild List in its testing, you're not exactly referencing the most rigorous, meaningful standards.

The Wild List:
- contains a couple hundred virus samples (922 in August 2009 to be exact)
- contains only viruses, except a couple worms (Koobface and Confiker - and dozens of variants of the same), only added a couple months ago. There are no rootkits, trojans, downloaders, or spyware! Note: Trojans and downloaders are arguably the most prevalent initial infectors (exploits are another story)
- contains viruses that have been agreed upon by at least TWO antivirus researchers, who in almost all cases work for AV companies
- is generally 2-3 months behind emerging threats by the time folks agree
Now, this was a good idea back when there were a hundred viruses a month. But, the volume and complexity has outpaced the organization's ability to keep up, and has become less relevant.

In our opinion, the Wild List is NOT representative of threats on the Internet, and it is extremely biased based on sharing and narrow definition of scope. Should you be basing your purchasing decisions off of certifications that use it? (ICSA Labs, VB100, West Coast Labs)

Evading Anti-malware Products

Anti-malware products are not 100% effective, as evidenced by our recent anti-malware tests. In fact, some seem to be falling behind the bad guys.

Why? Because the bad guys are smart and aggressive. And remember, cybercriminals need only find one open door to get in, whereas defenders need to protect all the doors.

Cybercriminals are employing a plethora of techniques in a highly automated fashion to evade detection. Gunter Ollman and the Damballa team have written a nice paper explaining malware evasion techniques. These automated methods allow bad guys to create massive amounts of unique malware that can circumvent AV software. Popular techniques include using:
1. Crypters
2. Protectors
3. Packers
4. Binders
5. Quality Assurance
See the well-written paper for a more complete discussion. This is why AV products are having to evolve, and quickly.

Are anti-malware products a commodity?

As any antivirus/security vendor will tell you: "No. we're not all the same. Mine is better." :-) Yet, in the Sept 2009 issue of Infosecurity Magazine, Forrester analyst Natalie Lambert suggests that all anti-malware products are essentially the same. "Generally speaking, antimalware is antimalware; what you get from one vendor is not much different than what you get from another."

It is perhaps understandable how one might believe this given all the marketing and the sheer difficulty in empirically discerning otherwise (but not really for an analyst). Much of the testing shows scores between 98 and 99%. And other long-standing organizations have essentially declared as much through their certifications. Dozens of products have achieved the Virus Bulletin VB100%(tm) award, and still others tout the Westcoast Labs Checkmark(tm) certification as a moniker of distinction. And ICSA Labs has certified 52 antivirus products to be up to snuff. So they must all be great, right?

Wrong. This is where real-world independent testing comes in that actually measures meaningful differences, like proactive protection (keeping malware off the machine), time to add protection, and protection over an extended period of time. In our recent Group Test of corporate and consumer endpoint protection products using our Live Testing methodology, we found a dramatic stratification of products' abilities to stop socially engineered malware (the kind that tricks users into clicking 'download and run'), currently the largest infection vector. Here are some key findings from the consumer report:

• Proactive 0-hour protection ranged from 26% to 70%
• Overall protection varied between 67% and 96% (over the course of 17 days of 24x7 testing)

Given such vast differences in real-world effectiveness, what value are these certifications anyways? In our opinion, not much... The threatscape has accelerated and some vendors are adapting faster than others. Unfortunately, testing was not adapting, resulting in a huge gap in trusted, real-world knowledge. The ensuing false sense of security creates greater risk for companies and consumers. We are filling that hole by delivering data based on our Live Testing methodology.

Since we performed these tests on our own, without any vendor funding, we are selling the group test of corporate endpoint protection products. See all the anti-malware product reports.

Which products we tested:

1. AVG Internet Security, version 8.5.364
2. Eset Smart Security 4, version 4.0.437
3. F-Secure Client Security version 8.01
4. Kaspersky Internet Security 2010, version 9.0.0.459
5. McAfee VirusScan Enterprise:8.7.0 + McAfee Site Advisor Enterprise:2.0.0
6. Norman Endpoint protection for Small Business and Enterprise
7. Sophos Endpoint Protection for Enterprise - Anti-Virus version 7.6.8
8. Symantec Endpoint Protection (for Enterprise), version 11
9. Panda Internet Security 2009, version 14.00.00
10. Trend Micro Office Scan Enterprise, version 10

What % of threats do you expect your anti-malware product to stop?

We are about to publish a new round of anti-malware testing data and would like to compare perceptions with reality. I'm expecting some interesting results to say the least.

7 simple questions here:
http://www.surveymonkey.com/s.aspx?sm=oiGBnkYL3i_2bBTEE4P24QNA_3d_3d

Thanks for your help

Q3 2009 Browser Security Tests Published

Today we published our 2nd round of live browser security tests. Two separate tests measured protection against phishing and socially engineered malware across 5 browsers: Apple Safari 4, Google Chrome 2, Mozilla Firefox 3, Opera 10 Beta and Windows Internet Explorer 8.

A key take away is that while the other browsers maintained or decreased protection between the two tests, Internet Explorer continued to improve its protection against cybercriminals.


Socially engineered malware is the most common and impactful threat on the Internet today, with browser protection averaging between 1% and 81%. Internet Explorer 8 caught 81% of the socially engineered malware sites over time, leading other browsers by a 54% margin. Safari 4 and Firefox 3 caught 21% and 27% respectively, while Chrome 2 blocked 7% and Opera 10 Beta blocked 1%.


Phishing protection over time varied greatly between 2% and 83% among the browsers. Statistically, Internet Explorer 8 at 83% and Firefox 3 at 80% had a two-way tie for first, given the margin of error of 3.6%. Opera 10 Beta, exhibited more extreme variances during testing and averaged 54% protection. Chrome 2 consistently blocked 26% of phishing sites, and Safari 4 offered just 2% overall protection. Firefox 3.5 crashing issues prevented it from being tested reliably.

The full text and analysis of these and other reports on browser security can be found at http://nsslabs.com/browser-security.

NSS Labs live testing methodology represents an accurate, real-world testing that can be performed on information security products.

- Newly discovered malicious phishing and malware sites were added to the test, which repeated every four hours 24x7 for a minimum of 12 days

- All five browsers tested URLs simultaneously

- All sites were validated before, during and after via multiple methods

Endpoint Protection Group Test Started

NSS Labs is continuing its testing of anti-malware products and has started its first group test of endpoint protection products. We are testing the ability to protect against socially engineered malware downloaded from the web. This is a continuous live test that will measure time to protect, and average protection over time. All systems are connected to the live internet and subjected to actual downloads of actual, fresh malware every 4 hours over a period of 12 days.

Both consumer and corporate products are being evaluated. Stay tuned for more information or contact me with any questions (rmoy).

Two acquisitions in two weeks!

Within the last 2 weeks, two young companies that NSS Labs did independent certifications on were acquired. ThirdBrigade, which makes Host Intrusion Prevention Software (HIPS) was acquired by TrendMicro, one of the major antimalware vendors. This product filled a server-side gap in their product line.
Solidcore Systems, which makes memory firewall/application white listing products, was acquired by McAfee. The #2 antimalware vendor cum security vendor has added whitelisting to its billion dollar portfolio of antimalware, vulnerability and intrusion prevention products. In Q3 of 2008, NSS Labs had evaluated and certified the S3 Control Embedded product as NSS Approved for Host Malware Protection.
In a down economy, strong vendors go shopping for technologies to round out their product lines so they're in positions of strength when the buyers recover. Note, even with all the cost cutting and layoffs, there's always money left for strategic purposes. And if you're a CEO who is going to make a purchase in this economy, there's not much room for forgiveness. So, you can bet they did their homework on all sides: technology, sales execution, management, margins, balance sheet, etc. I'm pleased NSS Labs was able to help these young companies grow their businesses and wish them well in the next stages of their evolution.

Live Testing, web malware and assumptions...

NSS labs just uploaded the video archive of the Live Testing Webinar we did on 3/31. This was a webinar with live Q&A as a follow up to the initial browser security test report we performed on 6 different web browsers' ability to block socially engineered malware. As we roll out this new test methodology we wanted to give readers a deeper, interactive look into the testing process. There were a few questions from readers about how we did it, why it's more relevant than static or 'in-lab' dynamic testing, and how to interpret the different measurements, etc.
Interestingly we are hearing from two different camps. A few bloggers/journalists are finding their assumptions challenged about their favorite programs; "how can that be?" Meanwhile, 'hard core' security researchers are telling us they are glad to see more comprehensive empirical validation of some of their own data points. Regardless of whether your assumptions were validated or challenged, the data can now drive the conversation - and future research.

CBS News covers Socially Engineered Malware

The lead story tonight on CBS News' 60 minutes show was about socially engineered malware pushed by cyber gangs. One can see a good example of how users are tricked into clicking on links sent to them from supposed friends via social networking sites. Symantec's Steve Trilling also explained the workings of the Confiker worm and a keylogger trojan to the CBS anchor, Leslie Stahl. Very timely given the upcoming April 1 'trigger date' for Confiker. NSS Labs of course recently published a report on socially engineered malware testing we performed in early March.

web browser security study - socially engineered malware

NSS Labs just released a study we did on 6 leading web browsers' ability to stop socially engineered malware attacks. We tested Safari, Chrome, IE7, IE8, Firefox and Opera. This is extremely relevant today since the majority of malware is currently being delivered via the web. Trend Micro research puts it globally at 53%, dwarfing email at just 12%. Oh how times have changed.

Read the full report here: http://nsslabs.com/anti-malware/browser-security

Also notable, this was the industry’s first live test of fresh malware sites. We pulled thousands of URLs off the web in real-time and fed them into 6 different browsers (84 unique machines) every 2 hours. A lot of work went into building this test harness and you'll certainly be hearing more about it shortly. Also keep in mind, while the highest score was Microsoft at 69%, this is nothing to sneeze at. All of the sites were extremely fresh, and the time between detection on the web and testing in the harness was between 30 minutes and 2 hours. Compare this to a VB100, ICSA, West Coast or other wild-list type test where the malware is generally 2+ months old. Our new Live Testing model yields a much more real-world assessment of anti-malware detection rates.

As far as the results, we were pleasantly surprised at just how well IE8 did. Browsers, and IE8 in particular, are becoming a viable extra layer of security on top of anti-malware/endpoint protection.

Note: NSS Labs developed the test methodology and infrastructure independently. Microsoft provided funding.

Which endpoint protection products stop IE Exploits?

During the week of Dec 15-18, NSS Labs conducted a series of tests of popular anti-malware and endpoint protection products to evaluate their ability to protect clients from exploits targeting the IE vulnerability. The results are somewhat surprising, showing a broad lack of protection from current enterprise products. Admins are advised to read this and address any gaps ASAP.

Tested antivirus/anti-malware/endpoint protection products include:

• AVG Internet Security Network Edition v8.0
• Kaspersky Total Space Security v6.0
• McAfee Total Protection for Endpoint
• Sophos Endpoint Security and Control v8.0
• Symantec Endpoint Protection 11.0.2 MR2
• Trend Micro Officescan 8.0 SP1 R3

Read the report here.