Enterprise Network Firewalls Leak

NSS Labs released yet another hard-hitting test report, not on the latest security innovations, but rather on decades old technology: network firewalls. They've been around long enough to take for granted and are starting to be replaced by next generation firewalls.

In our testing, five out of six, or 83%, of the enterprise network firewalls we tested in January leaked traffic using the default settings that the vendor ships to customers, letting external attackers become trusted insiders. Yes, let that sink in for a minute as there is no way to understate the importance of this. Several currently deployed enterprise firewalls are leaking traffic. And half are also failing stability testing, which jeopardizes integrity and continuity of operations. Everything is well documented in our Enterprise Network Firewall Group Test Report (client access required), FAQ, and Remediation Brief (free to registered users). Tested firewalls include: Check Point, Cisco, Fortinet, Juniper, Palo Alto Networks, Sonicwall.

This is not some new vulnerability. Rather, it's a well-known problem for which papers have been written, and attack code is available on the net. The bad guys have the info already. But apparently most of the vendors, and probably nearly all their customers who rely on firewall protection either don't know or have prioritized performance over security. Recognizing the widespread importance of the issue, we notified vendors immediately in January and February, and at considerable expense to us, worked with them for two months to explain the issues and solicit workarounds and fixes. Half the vendors could have protected customers, but did not, having shipped their firewalls with the protection off by default— leaving enterprise networks vulnerable out of the box. There are reasons, but no good ones in our opinion. An analogy is that of a car having the airbag disabled by default (but no warning). This is Job #1 for a firewall.

Bottom line: Your network firewall may not be protecting you, despite having multiple certifications from test labs. In fairness, these types of certifications were designed as minimum bars vendors must pass in order for the US government to purchase them, and not comprehensive assurance. Why? Because they don't test all the things we at NSS Labs do with the same rigor or as they're actually deployed at customer sites (like hackers do).

To be clear, claims that IPS or AV can stop a TCP split handshake attack are not accurate. Those are workarounds for trying to find malicious activity after the intruder has already gained access inside the firewall. It's like saying a metal detector will catch somebody who stole an employee ID card to get in the building. Only if they're carrying a gun or knife, but they could still roam freely and steal critical information if they stay under the radar.

Now, the only way for an enterprise to know it's firewall is blocking the attack is to check the configuration and/or test it against the specific attack. Given the market deployments of firewalls, millions of deployed firewalls need to be checked for this flaw in the field. As a public service, we've made a FAQ and remediation steps for the affected products that have TCP workarounds available on our site at no cost. We encourage anyone with a firewall to test their firewalls immediately for the issues described in detail in our full network firewall group test report. Also covered in the report: security effectiveness; evasion, performance, pricing and TCO test results. We utilized the BreakingPointSystems equipment for the testing, and special thanks to Tod Beardsley for the research (see paper).

Testing is not exactly straight forward, and many people are trying to come to terms with it, so if you have questions or need help, contact one of our security analysts. Given the number of firewalls out there, we all have a lot of work to do.

Why you need to Test it like a hacker!

Some tests you don't want to be too hard. Like those we take in school that we don't think will mean too much to us in life later on. Say, for some it's abstract poetry of the middle ages, basket weaving in the precambrian era, etc. For these you just want to get by, so when an easy test comes along, the tested party generally breathes a sigh of relief.

In contrast, some tests are hard for our own good. Physical endurance tests before summiting Mt. Whitney or K2. Crash tests of car safety equipment like seat belts, air bags and brakes. You really want to make sure those things work as advertised so they'll function when you need them.

So it is with enterprise security testing, and security product testing in particular. In a world where virtually every antivirus (antimalware) or endpoint security product is 'certified' by two or three different labs, one would think they're all equally good. And especially if they've got a certification from the government, right? Dead wrong. They've all been 'certified' because they've been able to figure out how to pass the test, or because the test is not hard enough, not necessarily survive the crash.

In our experience, there's rarely such a thing as 'too hard' of a test. In order to know how well a product will defend you, you've got to TEST IT LIKE A HACKER. You need to subject the products in your environment to the same stress and attacks that they will face against motivated, persistent adversaries sometimes even using advanced techniques. After all, fixing problems before a breach is always much less expensive than cleaning up the mess afterwards.

As more and more high-profile breaches are disclosed, securing our intellectual property and assets is no longer just a technical issue. NSS Labs makes a lot of its security research and educational content available for free. I encourage you to browse some of the results to find out more.

Network Intrusion Prevention Group Test Released

The security analysts at NSS Labs tested 13 different network IPS products, including stand-alone IPS and multi-function gateways, and one unified threat management product. If your organization is evaluating IPS solutions, or is looking to benchmark your current vendor, then this is the definitive report to read. Data and analysis are based on multiple man-years of complex, real-world testing that mimics how cyber-criminals are working to penetrate corporate defenses (see methodology). No surveys, interviews or soft trends. This is the hard test data upon which organizations base critical, big-dollar decisions.

The report includes valuable information not available anywhere else:

• Total cost of ownership analysis – are you getting the most security for your budget?

• Security effectiveness – how much effort is required to protect all your assets?
• Real-world performance benchmarks – can the device handle your traffic?
• Management and usability insights – how much time is really required to achieve results?

While the full breadth and depth of the research is available only to our subscribers, we are making a summary available to non-clients. Next week/year I will blog more about the key findings and what they mean for IT buyers in 2011.

Tested Products include (alphabetically):
1. CHECKPOINT POWER-1 11065
2. CISCO IPS 4260
3. ENDACE CORE-100 (IDS)
4. FORTINET FORTIGATE 3810A
5. IBM PROVENTIA NETWORK IPS GX6116
6. JUNIPER IDP-8200
7. JUNIPER SRX 3600
8. MCAFEE M-8000
9. NSFOCUS NIPS-1200
10. PALO ALTO NETWORKS PA-4020
11. SOURCEFIRE 3D 4500
12. STONESOFT IPS-1205
13. STONESOFT IPS 3205

Tales from the Trenches of IPS Testing

An update on our current 2010 IPS group test.


Those of you following network IPS know that the last NSS Labs IPS group test in Q4 2009 made quite an impact in the marketplace. The testing soundly destroyed the notion that an organization could buy one of the ‘leading products’ and rest easy
- Vendors with leading market share and analyst accolades were shown to have mixed to poor results on our robust exploit testing;
- Over half the vendors lacking coverage for well-known evasion techniques
- Many put performance ahead of security effectiveness

We were pleased to work collaboratively with many of the vendors in ensuing months to further identify and rectify issues in advance of our next group test. In the process, several vendors released new hardware and software. Excited as we were to test drive the new improved models, we were disappointed in more than a few situations to discover the latest and greatest versions had show-stopper level flaws that could cost their customers a great deal of money and time. A number of vendors requested additional time to remediate flaws, and it was clear that much would change very quickly.

Meanwhile, the size of the NSS Labs 2010 IPS group test has also grown in number of participants and complexity, making this by far the largest, most in-depth group test of its kind.
- We’ve added several new vendor products;
- We’ve refreshed the exploits used, changing and upgrading a third of the content;
- Added additional HTML evasions;

Determined to not let further delays keep our much awaited group test off the streets any longer, we decided to take a phased approach, releasing the test in two parts; Part I will contain five vendors, and Part II will contain the remaining vendors. While this phased release provides enterprises with much needed information on some of the top vendors, it arguably leaves them waiting to see the rest. This may be more or less relevant depending on one’s situation, and which products are under consideration. Certainly some vendor sales teams maybe playing catch up. In the meantime, NSS clients can utilize analyst advisory sessions to receive additional guidance and help fill in the gaps.

So, look for part one of the NSS Labs IPS group test to arrive middle of next week here, and part two in the middle of December. Clients will automatically be notified when it is posted via email alert. If you’re not a client, register for free here, or contact our advisory services group to learn more.

Happy Testers Day!

We have secretary's day, presidents day, mothers day, fathers day, etc. But I never suspected there would be a 'software developers day' or other professional recognition day in our field. Twitter educated me otherwise, so I'm going with it. Our profession could certainly use a little recognition and attention. The auspicious occasion as this blog explains is the discovery of the first (literal) bug in a computer system on Sept 9, 1947.
Since then, we've had a meager few decades to hone our software development skills, and our bug-finding skills are not far behind. To that end, a short homage to the hard-working testers and QA professionals who often hear these responses from developers.

24. "It works fine on MY computer"
23. "Who did you login as ?"
22. "It's a feature"
21. "It's WAD (Working As Designed)"
20. "That's weird..."
19. "It's never done that before."
18. "It worked yesterday."
17. "How is that possible?"
16. "It must be a hardware problem."
15. "What did you type in wrong to get it to crash?"
14. "There is something funky in your data."
13. "I haven't touched that module in weeks!"
12. "You must have the wrong version."
11. "It's just some unlucky coincidence."
10. "I can't test everything!"

For the top 9, see this site for a familiar chuckle...

Client-side expoit demonstration videos

NSS has been testing exploits and exploit-detection/protection products for the last 10 years or so, dating back to the early days of IDS and IPS . It's arguably one of our specialties. And we have written before about the differences between malware, exploits and vulnerabilities. Yet, as these pernicious threats are elevated into mainstream consumer and enterprise awareness, we are seeing quite a bit of terminology confusion. So, for our latest Host Intrusion Prevention System (HIPS) test of enterprise anti-malware products, we created some demonstration videos of client-side exploits used in the test.

Product	Vulnerability	Video Demonstration
Panda	CVE-2010-0249	http://www.youtube.com/watch?v=VPEzOZniuls
Panda	CVE-2010-0806	http://www.youtube.com/watch?v=_fstkScB7YM
F-Secure	CVE-2010-0249	http://www.youtube.com/watch?v=DdvtTuwfa5I
ESET	CVE-2006-0003	http://www.youtube.com/watch?v=U9WRkQyZvb8
Sophos	CVE-2006-4704	http://www.youtube.com/watch?v=F2gfNLzqBwg
Symantec	CVE-2010-0483	http://www.youtube.com/watch?v=RqQLmKI_R30

Passions of an assessor: Donde esta corazon?

Michelle is a passionate infosec pro and assessor. She gets some kudos today for expressing on a personal level the frustrations of many infosec practitioners whose job it is to audit, assess and help improve their clients' defenses. PCI DSS forces those who would do little or nothing for security to do something more. It also encourages those who would do more to do less because it is just enough to deal with a clear and present threat: the audit.

As Josh Corman at the 451 Group likes to say: “Why focus on compliance instead of security? I might be hacked, but I will be fined.” (if you handle cardholder data). Given the amount of client-side attacks and botnet infection data we see, the case could be made otherwise. Corporations are getting attacked daily. They might not be aware of it though, due to the holes in their security defenses, logs, and even alerting practices.

After all, security products can only alert and report on what they have detections for. Based on our testing, that leaves a significant gap with every vendor, between 12 and 83%. Do you know which holes matter on your network and where they are? Want to hear ideas on how to improve and not just pass?

I'm happy to echo Michelle's call for more heart and less check box.

Rick

Thanks for breaking it!

People hire us to break stuff (and lately we're pretty good at it). Well not just, but breaking is part of testing, as is validating You'd think folks wouldn't want to hire us for that, and a lot of times you'd be right. But, this week, we had a large networking vendor in the lab testing their product. On day 2 we discovered a significant vulnerability that we were able to exploit. We replicated it before their eyes. What did the vendor do? He gave our lead engineer a high five!

Why? because, after having visited several labs with this same product, we were the first to find something and not simply give it the 'rubber stamp.' This is why you test. Not just to validate features, but so you can find out what you still need to do to improve it. Good product developers like this one "get it." He just got tremendous value out of the engagement, and has already put in proposals on the spot for additional testing with us. And maybe his competitors have similar issues (which is often the case), so now this vendor is ahead of the game and will likely have it fixed very shortly.

IT Buyers: This is the attitude you want to see in your vendors.

Measuring Security

Vik Phatak, CTO participated on a panel discussion at SourceBoston conference titled "Measuring Security". This discussion explored the ins and outs of testing endpoint protection products, otherwise known as anti-virus/antimalware. Hosted by Andrew Jacquith of Forrester, and also with Peter Stelzhammer of AV-Comparatives, and Mario Vuksan of Reversing Labs. Watch the video link.

AV Testing double standards and independence

NSS Labs’ innovative tests are designed to inform end-users about how products truly perform against today’s motivated attackers. We perform a test or gap analysis on security products, so organizations can understand what is and isn’t being protected, and accurately assess the risk and take steps to mitigate it. While enterprises and government organizations appreciate this valuable, independent analysis, many of the AV vendors do not.

When NSS Labs published its uncensored, real-world results of endpoint protection products (AV), some vendors used the anti-malware testing standards organization (AMTSO) to try to discredit the test. One of their objections was that we recommend against buying products that scored on the bottom third of our test. Sorry, we unabashedly believe malware protection should indeed be the key purchasing criteria for an AV product. And for vendors who claim their anti-spam on the corporate desktop will improve their protection against socially-engineered malware hosted on web sites, that’s just stretching it.

Rather than shoot the messenger, vendors with their customer’s best interests in mind should seek to learn from tests like these in order to improve their products. Unfortunately, that’s usually not the case in the AV world after too many years of self-congratulatory testing and certification.

AMTSO is an AV vendor-driven consortium, and while it can be a useful information sharing organization for AV insiders, it has demonstrated its utter failure as a credible independent organization. Throughout the 3-year history of this organization, AMTSO has failed to evaluate the tests and certifications that most of its vendor members sponsor and fund; e.g. VB100% awards, ICSA Labs and West Coast Labs certifications. These validations are important sales material in the $9B market place, but they wouldn't pass the same AMTSO guidelines that were supposedly applied to the NSS Labs test.

Such market validations are a part of the industry, but can be dangerous when they convey a false sense of security to buyers as they do now. Meanwhile, end-users can stay well informed about what products do - and more importantly - what they DO NOT do, by reading our subscriber-funded research and test reports. If a vendor is complaining about our test, chances are they did poorly on an important metric. Learn what some vendors don’t want you to see by reading our independent anti-malware test reports or the Google Aurora protection analysis report in particular (free to non-clients).

caveat emptor

GroupThink and InfoSec - SecurityBSidesSF

At about 1:45h into the video, Vik Phatak begins his talk about group think in infosec. It is a well-reasoned argument for more aggressive testing in order to improve our defenses. We present methodologies and results from our testing of IPS and Anti-malware products to make the case that we have much work to do and must stay vigilant and innovative.

Network IPS Group Test Results Available

Over the past few years of working with clients and testing IPS products, we began to notice some troubling trends. Some vendors missed more attacks than seemed acceptable. A number of vendors refused to participate in our testing—even at no cost. Enterprise readers asked some tough questions about current products and issues. And breaches resulting in compromised data continued to increase. Meanwhile, we heard several opinions from our clients, industry analysts, and researchers. These beliefs can be summarized as follows:

· IPS is a mature market. There is relatively little difference between products; thus, management and price are the key purchasing factors

· Best-of-breed products are more effective than those from strategic vendors who provide a wider range of products

· The market leader (from an installed base perspective) provides the best protection

· Organizations are protected as long as they keep their IPS systems updated

This group test set out to determine if these beliefs were correct. In order to garner the greatest participation, and allay any potential concerns of bias, we invited all leading vendors to submit their products at no cost. Every vendor below brought and configured their best Network IPS products; all were generally available (GA), no Beta or otherwise unavailable products were included. The following is a current list of the products that were tested, sorted alphabetically:

1. Cisco IPS 4260 Sensor, version 444.0

2. IBM Proventia Network IPS GX4004, version 29.100

3. IBM Proventia Network IPS GX6116, version 29.100

4. Juniper Networks IDP-250, version 5.0.110090831

5. Juniper Networks IDP-600c, version 5.0.110090831

6. Juniper Networks IDP-800, version 5.0.110090831

7. McAfee M-1250, signature version 5.4.5.23

8. McAfee M-8000, signature version 4.1.59.23

9. Sourcefire 3D 4500, rules version 4.8.2.1

10. Stonesoft StoneGate IPS 1030, 5.0.1 build 5053 update package 261

11. Stonesoft StoneGate IPS 1060, 5.0.1 build 5053 update package 261

12. Stonesoft StoneGate IPS 6105, 5.0.1 build 5053 update package 261

13. TippingPoint (TP) 10 IPS, DV 2.5.2.7834

14. TippingPoint 660N IPS, DV 2.5.2.7834

15. TippingPoint 2500N IPS, DV 2.5.2.7834

Some vendors provided multiple test units of varying performance levels. Across the board, all vendors claimed identical protection across models.

The results will be shocking to most and have already generated plenty of buzz. 1,159 live exploits is the most ever tested. If you currently operate one of these products, or are considering investing in an IPS, the information in this exclusive test report is invaluable.

TippingPoint Tests

In response to inquiries about the blog posting made by TippingPoint President, Alan Kessler, we provide the following:

In August 2009, NSS Labs performed an independent test of the TippingPoint 10 product and determined it only blocked 39% of common exploits. Subsequently, TP came to our lab for private testing for further assistance, as they stated. TP customers can see a spike of hundreds of filters which appeared in October and November.

In early December, NSS Labs released its independent IPS group test of 15 different IPS products submitted by 7 vendors, including TP. The product improved marginally, but is rated ‘caution’ due to its subpar protection on our tests. Now TippingPoint has publicly complained in this tippingpointblog that the test must be inaccurate because it didn’t correspond with the results of their private testing with NSS, with 'customer experience', nor with their internal testing.

3 response points:
1st. These modern IPS products are so complex, that customers will rarely be in position to question or test a vendor properly. And they rarely do when it is a brand name. Very few enterprises have the sophisticated testing tools, expertise and access to exploits like the vendors and a professional security testing lab like NSS. So, having a lot of customers does not necessarily mean they are aware of the true protection they are receiving. In fact, not knowing is a liability in itself for all involved.

2nd. RE: Private testing results. At NSS we don’t use the same attack set in our private testing, as we do in public testing. That would be like getting a copy of the test and answers beforehand, and would give private clients an unfair advantage over other vendors. We do test the same vulnerabilities, but the specific exploits we use vary. This should underscore the integrity of NSS Labs testing principles and procedures. In general, differences in results could be attributable to signatures written too narrowly; e.g. for specific exploits vs vulnerabilities, or to signatures written for a test lab environment.

3rd. We certainly cannot account for any vendor's internal testing procedures. However, the findings of our two previous tests were ultimately corroborated.

As far as delaying the Network IPS Group Test Report. It would be unfair to enterprise readers all around not to disclose validated testing results that could help them mitigate threats that might not be stopped by their defenses as they expect (that could be considered irresponsible non-disclosure). A delay would also not be fair to the other 6 vendors who also participated. As with the previous tests, NSS took great care to validate the results. Deciding to act positively upon them to deliver the better customer protection is the next imperative.

Maintaining Test Integrity during Private Testing

NSS does not utilize the same exploits in the Private test as in Public tests or certifications. This is to maintain the integrity of the public tests and ensure that vendors engaged in private testing are not given an unfair advantage.Attacks used in private testing target the same range of vulnerabilities used in the public tests. However, different versions of the exploits are used.

· The same range of vulnerabilities are represented in both the private and public tests. However, different exploit variances are used between the two types of test to ensure vendors are writing vulnerability-based signatures in order to adequately protect their customers, and not simply writing exploit-specific signatures to perform well in testing. For example, private tests utilize a higher number of Proof Of Concept (POC) exploits and PCAPs, whereas public testing and certification relies exclusively on NSS’ unique and comprehensive live exploit test harness.

· Vendors who write vulnerability based signatures rather than exploit specific ones will achieve similar results in both private and public tests

· Vendors that write signatures to catch POC PCAPs, but not real exploits and variants, may experience different test results between private and public tests.

Of course, this policy means we have to keep investing in keeping things fresh, accurate and relevant. It's a never-ending job, and we have some of the best people in the industry doing it.

Raising the Bar in Testing

It is important for NSS Labs to periodically raise the bar as the industry advances, both offensively and defensively. In other words, we increase the difficulty of the test to match the needs (as driven by cyber criminals' innovative new attacks), and the capabilities of vendors who have developed new approaches to countering the threats.

This is what NSS Labs has been doing over the past 2 years in particular: raising the bar. As such, our reports and the scores vendors receive on them deserve to be put into context. We perform a number of different types of tests over the past few years:

1. Product Certification - a full review of the whole product; including security effectiveness, performance, manageability and stability.

2. Group Tests - comparative testing of a class of products from leading vendors. Due to the volume of vendors, the depth of the testing may be abbreviated and focused. The browser security, endpoint protection, and Network IPS tests are great examples of these.

3. Security Update Monitor (SUM) Tests - unique to the industry, NSS Labs has been testing IPS products on a monthly basis. Every quarter, we tally the average scores and bestow awards. The attacks in this test set are focused on vulnerability disclosures made the previous month. As such, the volume of new additions is generally in the couple dozen range. Overall, there are currently 300 entries.

4. Individual Product Reports - these are brand new detailed reports that were created during group testing. They capture the nitty gritty details that are rolled up into the group test.

5. Exposure Reports - An industry and NSS first. These reports list specific vulnerabilities that are NOT shielded by an IPS. This information is critical in helping organizations knowing where their protection is and is not. Contact us for confidential demonstration. (Knowing can help identify appropriate mitigations, such as writing custom signatures, implementing new rules, resegmenting the network, or ultimately switching or adding a security product.)

By the Numbers:
Exploit selection and validation is a serious matter. Our engineers take care to identify the greatest risks to enterprises, based on commonly deployed systems, utilizing a modified CVSS.


Certifications performed in 2008 and 2009 tested 600 vulnerabilities. Compare this with other tests, e.g. ICSA IPS tests only 120 vulnerabilities (all server side). Our 600 is more than 480 better, because vendors did not know what they were, thus preventing the 'gaming of the test'.

SUM testing generally adds 20-30 vulnerabilities per month. Relatively small, directly reflective of popular attacks.

Our latest group test utilized 1,159 live exploits. More than 2x the number we used previously for certifications (and nearly 10x more than the next lab).

NSS Labs Tests are Harder Than The Other Guys'...
Difficult, real-world tests are an important part of raising the bar. Marketers like to have big numbers, and when it comes to scores, 100% is the target. Unfortunately, that's not the reality of information security products. We at NSS do not expect any product to catch 100% of the attacks in any of our tests. If they do, we probably are not working hard enough (or the bad guys gave up and went home - unlikely). There are more threats than can be protected against, and depending on the vendor, the margin can be acceptable, or pretty significant.

Take the difficulty of the test into consideration when comparing products. The lower the bar, the easier it is to score 100%. And the less meaningful it is. A 70% score on an NSS Labs Group Test (1,159 exploits) is still 6.7 times more validation than a 100% of 120 exploits in an ICSA Labs(r) test. And a 95% score on an NSS certification is 4.9 times more.

When we at NSS Labs raised the bar on this Q4 2009 IPS Group Test, we really cranked it up. So, if you're wondering why a vendor who previously scored in the 90's is scoring lower on the group test, it's not necessarily because they are slacking off. In most cases it is quite the opposite. Thus, one should be sure to compare products within the test set and methodology. Ergo, the 80% product definitely bested the 17% product by a wide berth. Be very cautious of the latter. And reward those vendors who submitted to this rigorous testing in the first place.

NSS Labs Mission Revisited

NSS Labs' mission is to help raise the bar of information security capabilities and practices, for enterprises and vendors alike.

For enterprises, that means helping them choose and implement better defenses. We do this by performing rigorous testing of leading products in various configurations and publishing test reports for purchase as an information service, much like other analyst firms, like Gartner, IDC, Forrester, etc. (However, we are the only ones that actually perform hands on, comprehensive testing of security products). There are several types of reports, individual certifications (full 360 reviews), comparative group tests, Security Update Monitoring, and our new Exposure Reports. The products in this information service help subscribers understand what is protected and what is not. Nothing protects 100%, so knowing the specifics is important.

For security product vendors, this means testing them against standardized evaluation criteria to establish a baseline, and drawing attention to key issues and requirements. We then test according to best practices methodologies. Our reports also reward those vendors that perform well, and they can use those for marketing. When it comes to improving products, vendors have great resources, and some of the smartest teams around. In addition, they often turn to outside experts for help. NSS is well equipped to assist with this type of private testing and consulting. However, we are always careful to maintain integrity during the process.

CISOs - the Wild List isn't

In a Forbes article this week, Analyst Charlotte Dunlap outlines questions CISOs should be asking when evaluating anti-malware products. There's a common misperception about the 'wild list' and how meaningful it is. If you're buying antivirus products because they're certified by one of the organizations that uses the Wild List in its testing, you're not exactly referencing the most rigorous, meaningful standards.

The Wild List:
- contains a couple hundred virus samples (922 in August 2009 to be exact)
- contains only viruses, except a couple worms (Koobface and Confiker - and dozens of variants of the same), only added a couple months ago. There are no rootkits, trojans, downloaders, or spyware! Note: Trojans and downloaders are arguably the most prevalent initial infectors (exploits are another story)
- contains viruses that have been agreed upon by at least TWO antivirus researchers, who in almost all cases work for AV companies
- is generally 2-3 months behind emerging threats by the time folks agree
Now, this was a good idea back when there were a hundred viruses a month. But, the volume and complexity has outpaced the organization's ability to keep up, and has become less relevant.

In our opinion, the Wild List is NOT representative of threats on the Internet, and it is extremely biased based on sharing and narrow definition of scope. Should you be basing your purchasing decisions off of certifications that use it? (ICSA Labs, VB100, West Coast Labs)

Awareness Month

October has been national Breast cancer Awareness Month for the last 25 years. For the last 5 years, it has also been the national Cybersecurity Awareness Month. I don't know how these things get set, but it seemed to me there were a couple common threads going on.

Both are major problems for our society. One is a condition when cells replicate uncontrollably, the other a premeditated malicious digital attack. In 2009, there are 193,000 new breast cancer cases expected. My mother is a breast cancer survivor, thanks to early detection, great doctors and divine will. And we all likely know someone who is a cyber-security attack survivor: after all, there are 339 million victims of data loss and breaches (see: Data Loss DB and Privacyrights Clearinghouse).

When it comes to breast cancer, early detection is the key; there are even earlier technologies than the mammogram. But, what's the corollary for cybersecurity? Testing of course! Testing of our knowledge of threats and best practices. And testing of our defenses: whether individual products, or layered defense architectures and policies.

Unfortunately, there is far too little testing going on. Erecting defenses and not periodically evaluating their effectiveness is a far too common practice. Requirements of certain compliance regimes like PCI DSS are helping drive awareness and require at least some level of testing. However, there seems to be a common perception that you can 'set it and forget it.' For technologies like IPS and anti-malware that require constant updating, nothing can be further from the truth.

Are anti-malware products a commodity?

As any antivirus/security vendor will tell you: "No. we're not all the same. Mine is better." :-) Yet, in the Sept 2009 issue of Infosecurity Magazine, Forrester analyst Natalie Lambert suggests that all anti-malware products are essentially the same. "Generally speaking, antimalware is antimalware; what you get from one vendor is not much different than what you get from another."

It is perhaps understandable how one might believe this given all the marketing and the sheer difficulty in empirically discerning otherwise (but not really for an analyst). Much of the testing shows scores between 98 and 99%. And other long-standing organizations have essentially declared as much through their certifications. Dozens of products have achieved the Virus Bulletin VB100%(tm) award, and still others tout the Westcoast Labs Checkmark(tm) certification as a moniker of distinction. And ICSA Labs has certified 52 antivirus products to be up to snuff. So they must all be great, right?

Wrong. This is where real-world independent testing comes in that actually measures meaningful differences, like proactive protection (keeping malware off the machine), time to add protection, and protection over an extended period of time. In our recent Group Test of corporate and consumer endpoint protection products using our Live Testing methodology, we found a dramatic stratification of products' abilities to stop socially engineered malware (the kind that tricks users into clicking 'download and run'), currently the largest infection vector. Here are some key findings from the consumer report:

• Proactive 0-hour protection ranged from 26% to 70%
• Overall protection varied between 67% and 96% (over the course of 17 days of 24x7 testing)

Given such vast differences in real-world effectiveness, what value are these certifications anyways? In our opinion, not much... The threatscape has accelerated and some vendors are adapting faster than others. Unfortunately, testing was not adapting, resulting in a huge gap in trusted, real-world knowledge. The ensuing false sense of security creates greater risk for companies and consumers. We are filling that hole by delivering data based on our Live Testing methodology.

Since we performed these tests on our own, without any vendor funding, we are selling the group test of corporate endpoint protection products. See all the anti-malware product reports.

Which products we tested:

1. AVG Internet Security, version 8.5.364
2. Eset Smart Security 4, version 4.0.437
3. F-Secure Client Security version 8.01
4. Kaspersky Internet Security 2010, version 9.0.0.459
5. McAfee VirusScan Enterprise:8.7.0 + McAfee Site Advisor Enterprise:2.0.0
6. Norman Endpoint protection for Small Business and Enterprise
7. Sophos Endpoint Protection for Enterprise - Anti-Virus version 7.6.8
8. Symantec Endpoint Protection (for Enterprise), version 11
9. Panda Internet Security 2009, version 14.00.00
10. Trend Micro Office Scan Enterprise, version 10

What % of threats do you expect your anti-malware product to stop?

We are about to publish a new round of anti-malware testing data and would like to compare perceptions with reality. I'm expecting some interesting results to say the least.

7 simple questions here:
http://www.surveymonkey.com/s.aspx?sm=oiGBnkYL3i_2bBTEE4P24QNA_3d_3d

Thanks for your help