Moved

This blog has moved to www.nsslabs.com/blog
Please update your bookmarks.

Enterprise Network Firewalls Leak

NSS Labs released yet another hard-hitting test report, not on the latest security innovations, but rather on decades old technology: network firewalls. They've been around long enough to take for granted and are starting to be replaced by next generation firewalls.

In our testing, five out of six, or 83%, of the enterprise network firewalls we tested in January leaked traffic using the default settings that the vendor ships to customers, letting external attackers become trusted insiders. Yes, let that sink in for a minute as there is no way to understate the importance of this. Several currently deployed enterprise firewalls are leaking traffic. And half are also failing stability testing, which jeopardizes integrity and continuity of operations. Everything is well documented in our Enterprise Network Firewall Group Test Report (client access required), FAQ, and Remediation Brief (free to registered users). Tested firewalls include: Check Point, Cisco, Fortinet, Juniper, Palo Alto Networks, Sonicwall.

This is not some new vulnerability. Rather, it's a well-known problem for which papers have been written, and attack code is available on the net. The bad guys have the info already. But apparently most of the vendors, and probably nearly all their customers who rely on firewall protection either don't know or have prioritized performance over security. Recognizing the widespread importance of the issue, we notified vendors immediately in January and February, and at considerable expense to us, worked with them for two months to explain the issues and solicit workarounds and fixes. Half the vendors could have protected customers, but did not, having shipped their firewalls with the protection off by default— leaving enterprise networks vulnerable out of the box. There are reasons, but no good ones in our opinion. An analogy is that of a car having the airbag disabled by default (but no warning). This is Job #1 for a firewall.

Bottom line: Your network firewall may not be protecting you, despite having multiple certifications from test labs. In fairness, these types of certifications were designed as minimum bars vendors must pass in order for the US government to purchase them, and not comprehensive assurance. Why? Because they don't test all the things we at NSS Labs do with the same rigor or as they're actually deployed at customer sites (like hackers do).

To be clear, claims that IPS or AV can stop a TCP split handshake attack are not accurate. Those are workarounds for trying to find malicious activity after the intruder has already gained access inside the firewall. It's like saying a metal detector will catch somebody who stole an employee ID card to get in the building. Only if they're carrying a gun or knife, but they could still roam freely and steal critical information if they stay under the radar.

Now, the only way for an enterprise to know it's firewall is blocking the attack is to check the configuration and/or test it against the specific attack. Given the market deployments of firewalls, millions of deployed firewalls need to be checked for this flaw in the field. As a public service, we've made a FAQ and remediation steps for the affected products that have TCP workarounds available on our site at no cost. We encourage anyone with a firewall to test their firewalls immediately for the issues described in detail in our full network firewall group test report. Also covered in the report: security effectiveness; evasion, performance, pricing and TCO test results. We utilized the BreakingPointSystems equipment for the testing, and special thanks to Tod Beardsley for the research (see paper).

Testing is not exactly straight forward, and many people are trying to come to terms with it, so if you have questions or need help, contact one of our security analysts. Given the number of firewalls out there, we all have a lot of work to do.

What's your next browser?

Web browsers have become the new killer app - serving as the platform for accessing our favorite personal and business applications in the cloud. As we've discussed previously on this blog and in our research, web browsers, and more often their plug-ins, represent significant vulnerability risks to individuals and organizations. This week we find ourselves in a unique point in time, when several major browser upgrades have been released (or are imminently upon us): Chrome 10, Firefox 4, Internet Explorer 9, Opera 11, Safari 5.

Many of the key enhancements include:
- rendering and standards compliance
- security and privacy, including 'do not track' provisions

- javascript and graphics performance acceleration

- 'enhanced' user interfaces

Which one will you upgrade to?

Why you need to Test it like a hacker!

Some tests you don't want to be too hard. Like those we take in school that we don't think will mean too much to us in life later on. Say, for some it's abstract poetry of the middle ages, basket weaving in the precambrian era, etc. For these you just want to get by, so when an easy test comes along, the tested party generally breathes a sigh of relief.

In contrast, some tests are hard for our own good. Physical endurance tests before summiting Mt. Whitney or K2. Crash tests of car safety equipment like seat belts, air bags and brakes. You really want to make sure those things work as advertised so they'll function when you need them.

So it is with enterprise security testing, and security product testing in particular. In a world where virtually every antivirus (antimalware) or endpoint security product is 'certified' by two or three different labs, one would think they're all equally good. And especially if they've got a certification from the government, right? Dead wrong. They've all been 'certified' because they've been able to figure out how to pass the test, or because the test is not hard enough, not necessarily survive the crash.

In our experience, there's rarely such a thing as 'too hard' of a test. In order to know how well a product will defend you, you've got to TEST IT LIKE A HACKER. You need to subject the products in your environment to the same stress and attacks that they will face against motivated, persistent adversaries sometimes even using advanced techniques. After all, fixing problems before a breach is always much less expensive than cleaning up the mess afterwards.

As more and more high-profile breaches are disclosed, securing our intellectual property and assets is no longer just a technical issue. NSS Labs makes a lot of its security research and educational content available for free. I encourage you to browse some of the results to find out more.

Redefining the security gateway

This week I'm at the Pacific Crest Emerging Technologies Summit in San Francisco. And security is hot.

Apparently, enterprise IT buyers are not the only ones interested in information security products. Investors - institutional, hedge funds, private equity, etc - are all trying to read the tea leaves of the marketing soup being slung by security vendors. It's a stark contrast to the crowds at BlackHat and Defcon. These investors want to understand which companies will outperform or under-perform their competitors in the marketplace. While they clearly posses great knowledge about the financials of these companies, several are admittedly struggling to understand the technology table-stakes and differentiators required to compete. Increasingly, they're realizing they need to understand the security tech a little better in order to formulate and justify their investment thesis. I'm fielding questions like: Why do we need new security gateways? What is application control about? How are enterprises buying/using the technology? Can opensource security compete? Which approach will win? Which companies have products vs. platforms? With 20 to 40 competing companies in security market segments, surely not all of them can 'perform' and survive long term as stand-alone entities.

In a few hours I'll be tackling some of these questions on a panel with some of the leaders in network security - Barracuda, Fortinet and Sourcefire. This should be a good debate, and we'll have to follow up with some of those larger players who aren't represented, like Check Point, Cisco, HP/TippingPoint, IBM/ISS, Juniper.

RSA update

Like most, I'm recovering from the annual pilgrimage to the RSA conference in San Francisco two weeks ago. As usual, it was a great mecca in which to reconnect with friends, clients, business partners and new folks in the community. I'd especially like to thank all the supporters in our enterprise and vendor advisory boards. The NSS team is grateful for getting 60 of the busiest people in the biz to actively participate in discussions about how to improve information security through testing. It’s a topic that’s garnering momentum. I thank all of you for your input and suggestions on how we can improve and continue to deliver meaningful, actionable information services.

What are your priorities and concerns for 2011? Let us know and you could win a $100 AMEX card. Respondents will receive complementary access to the research results. Take the survey.

New Research

• We have a number of endpoint protection platform (EPP) and network security reports we are rolling out, including EPP evasion, multi-vector attack protection, next generation firewall (NGFW), and firewall (FW). There will be a subsequent post on each of these.
• We have been busy coordinating remediation of an important security issue with a number of firewall vendors. Stay tuned for the full report.
• As Anti-malware continues to fail to protect endpoints, we have been investigating alternatives such as application control (application whitelisting), and secure browsing. While much of this has been performed for private clients in the financial services industry, we are gearing up for a proper group test of these technologies in Q2. Vendors, submit your products. Enterprise buyers, let us know what challenges you’re facing, your criteria and experiences. Contact us. 

Also New from NSS Labs

• We rolled out a new video explaining the use cases for our services and how we can help organizations make informed infosec decisions.
• We also have new collateral which goes into greater detail. See the services overview, or dive into our
• Research calendar
• Consulting and custom testing services for proof of concept and bake-off testing
• Modeling defense-in-depth and tracking risk exposure using our VulnScope service
• Understanding the balance between security and cost with our Security Value Map
• Finally, NSS is actively expanding to meet the demands of our growing Fortune 2000 client base. If you’re a talented, hands-on infosec professional who understands the value of testing and ethical hacking, and is passionate about improving information security, we should talk. Contact us about career opportunities.

test

Google Blogger seems to be having issues, along with some other Google Apps, like mail. Check the dashboard.

Network Intrusion Prevention Group Test Released

The security analysts at NSS Labs tested 13 different network IPS products, including stand-alone IPS and multi-function gateways, and one unified threat management product. If your organization is evaluating IPS solutions, or is looking to benchmark your current vendor, then this is the definitive report to read. Data and analysis are based on multiple man-years of complex, real-world testing that mimics how cyber-criminals are working to penetrate corporate defenses (see methodology). No surveys, interviews or soft trends. This is the hard test data upon which organizations base critical, big-dollar decisions.

The report includes valuable information not available anywhere else:

• Total cost of ownership analysis – are you getting the most security for your budget?

• Security effectiveness – how much effort is required to protect all your assets?
• Real-world performance benchmarks – can the device handle your traffic?
• Management and usability insights – how much time is really required to achieve results?

While the full breadth and depth of the research is available only to our subscribers, we are making a summary available to non-clients. Next week/year I will blog more about the key findings and what they mean for IT buyers in 2011.

Tested Products include (alphabetically):
1. CHECKPOINT POWER-1 11065
2. CISCO IPS 4260
3. ENDACE CORE-100 (IDS)
4. FORTINET FORTIGATE 3810A
5. IBM PROVENTIA NETWORK IPS GX6116
6. JUNIPER IDP-8200
7. JUNIPER SRX 3600
8. MCAFEE M-8000
9. NSFOCUS NIPS-1200
10. PALO ALTO NETWORKS PA-4020
11. SOURCEFIRE 3D 4500
12. STONESOFT IPS-1205
13. STONESOFT IPS 3205

Stopping malware with a browser

This week we released another report on socially-engineered malware protections delivered by browsers. While most articles and blogs seemed to interpret the results properly, there were some inaccuracies that we wish to address. Security is a complex field and the terminology can sometimes be misinterpreted. This can be compounded when vendors who did poorly add their spin, or when the data challenges one’s own beliefs.

Stopping Malware vs. Security:
Some of the key security terms are clarified in a previous post, especially relating to “most secure” browser. Some articles incorrectly stated that we found IE to be the “most secure” browser. What we tested was browser effectiveness at stopping malware from reaching a user or their PCs, not the security of the browser itself or its plug-ins. Modern browsers are a wonderful, free additional layer of protection. They work well with your favorite antivirus software. Browsers however will not stop malware coming via email or USB drives.

The Malware threat:
Malware is arguably the largest security threat facing users today – with more than 60,000 unique, new samples entering circulation each day (source: McAfee). There’s a $14B industry addressing the problem of malware. These test results challenge the comfortable status quo of many of the vendors. The notion that a free product adds so much protection can easily upset the industry apple cart. The assertion that the focus of the test was narrow (made by Google and some others) flies in the face of all generally accepted data. To say one’s browser was “built with security in mind” is nice, but marketing speak. What we’ve offered is hard data about malware protection. The exploitability of the browser is also a very important topic. But, even in this case, data doesn’t support Chrome being ‘more secure’ (i.e. less vulnerable) than other browsers (see CVEs, Secunia disclosures, etc.). We do sincerely applaud the innovations and bug bounties though, and encourage all vendors to build more security in.

Versions tested:
The claim that we tested an “old” version of Chrome is patently false. As stated in the report, the test was run in late September, when version 6 was the current browser. Since then Google has released two other so-called “major” releases, none of which have claimed improvements to the tested SafeBrowsing functionality. Here is the Timeline for Chrome Version Release:
Stable Version Release Date
5.0.375      2010-05-25
6.0.472      2010-09-02
7.0.517      2010-10-21
8.0.552      2010-12-02
Furthermore, Chrome’s sandboxing is designed for exploits protection; it does not protect against socially-engineered malware. You click it, it runs.

At the time of the test, Opera’s website marketed protection from malware as a feature, yet our results showed no protection was available. AVG officials have separately acknowledged that the integration of its technology was not yet complete, confirming our results. Features should only be marketed after they are actually in the product.

Application Reputation:
In a world of dizzying information, there’s much rush to a sound bite of who “won”. The most significant technological message of this test may have been overlooked. This test benchmarked the world’s first implementation of an application reputation system within free web browsers that goes beyond simple black lists. Nascent stand-alone security products such as SolidCore (acquired by McAfee), Bit9 and CoreTrace utilize what is commonly referred to as white listing, and commercial endpoint security products are starting to include some form of this as well. Apple uses a “walled garden” approach to limit exposure to malware on its tightly controlled platforms by pre-approving apps.

In web browsers, so far we’ve seen just black listing. A URL or application is either known to be bad, or unknown. What’s unique about Microsoft’s approach in the IE9 browser is that applications have 3 states: known good (white), known bad (black) and unknown (grey). The combination of good and bad indicators is clearly powerful, stopping 99% of malware via the web download vector. The use of application reputation to identify good applications and bad ones is unique to IE, for now. Will other vendors follow Microsoft’s lead?

Methodology and Open Invitations:
No vendor has influence over what/how we test or where we get malware from. We constantly run our Live Test network with a variety of security products - antivirus, browsers, and other network devices.
Over the past 2 years we’ve been running these tests, NSS has discussed results and methodology (which is included in the report) with all of the browser vendors; even providing sample URLs for validation to them in past tests. Some had even privately acknowledged issues.

Our past invitations to become more involved in the ongoing testing and review of results still stands.

Threat Types and Terminology

Terminology used to describe attacks is often misunderstood by the broader public. Thus, we are providing this brief explanation of threat types and the terms we use in our reports.

End users and their computers face a number of different attack types. At a high level there are two: 1) Socially-engineered attacks target the user, and work only when the user is tricked into performing an action; running a malicious file or giving up personal data to a fraudulent site. 2) Other attacks target vulnerabilities in systems and applications. The following chart gives a rough breakdown of common threats against end user systems.

Layers of Security
These types of security threats can be mitigated by a range of security products; including IPS, UTM, SWG appliances, and on the endpoint: Internet security suites, most anti-malware products, and even web browsers. Modern browsers have implemented an additional layer of security to help users differentiate between good and bad web sites and downloads.

When selecting security products, either for home or business environments, it's often hard to tell from the marketing literature which products actually stop threats. And protection levels offered by products in the different categories can vary greatly. The above taxonomy should help you ask more specific questions of vendors. It also acts as a guide to terminology used in NSS Labs test reports.

Security products protecting users and their computers
When someone says “Product X stops more malware, exploits etc.” or “Product X offers better malware or exploit protection”, what they mean is that Product X inspects traffic passing through it and stops these attacks from reaching and/or affecting the end user or the operating system.

Security products themselves susceptible to threats
In addition, security suites and browsers (and their plug-ins) can be susceptible to exploits if the software has vulnerabilities in them. When someone says “browser X is more secure” what they are trying to say is that browser X has fewer vulnerabilities. Unfortunately, most software, and all browsers have vulnerabilities. For example in the first 9 months of 2010, Microsoft Internet Explorer had 43 new published vulnerabilities, while Google Chrome had 106, according to Secunia research.

For more exhaustive treatments on threat types including product test results, consult our research services at nsslabs.com.

Tales from the Trenches of IPS Testing

An update on our current 2010 IPS group test.


Those of you following network IPS know that the last NSS Labs IPS group test in Q4 2009 made quite an impact in the marketplace. The testing soundly destroyed the notion that an organization could buy one of the ‘leading products’ and rest easy
- Vendors with leading market share and analyst accolades were shown to have mixed to poor results on our robust exploit testing;
- Over half the vendors lacking coverage for well-known evasion techniques
- Many put performance ahead of security effectiveness

We were pleased to work collaboratively with many of the vendors in ensuing months to further identify and rectify issues in advance of our next group test. In the process, several vendors released new hardware and software. Excited as we were to test drive the new improved models, we were disappointed in more than a few situations to discover the latest and greatest versions had show-stopper level flaws that could cost their customers a great deal of money and time. A number of vendors requested additional time to remediate flaws, and it was clear that much would change very quickly.

Meanwhile, the size of the NSS Labs 2010 IPS group test has also grown in number of participants and complexity, making this by far the largest, most in-depth group test of its kind.
- We’ve added several new vendor products;
- We’ve refreshed the exploits used, changing and upgrading a third of the content;
- Added additional HTML evasions;

Determined to not let further delays keep our much awaited group test off the streets any longer, we decided to take a phased approach, releasing the test in two parts; Part I will contain five vendors, and Part II will contain the remaining vendors. While this phased release provides enterprises with much needed information on some of the top vendors, it arguably leaves them waiting to see the rest. This may be more or less relevant depending on one’s situation, and which products are under consideration. Certainly some vendor sales teams maybe playing catch up. In the meantime, NSS clients can utilize analyst advisory sessions to receive additional guidance and help fill in the gaps.

So, look for part one of the NSS Labs IPS group test to arrive middle of next week here, and part two in the middle of December. Clients will automatically be notified when it is posted via email alert. If you’re not a client, register for free here, or contact our advisory services group to learn more.

Happy Testers Day!

We have secretary's day, presidents day, mothers day, fathers day, etc. But I never suspected there would be a 'software developers day' or other professional recognition day in our field. Twitter educated me otherwise, so I'm going with it. Our profession could certainly use a little recognition and attention. The auspicious occasion as this blog explains is the discovery of the first (literal) bug in a computer system on Sept 9, 1947.
Since then, we've had a meager few decades to hone our software development skills, and our bug-finding skills are not far behind. To that end, a short homage to the hard-working testers and QA professionals who often hear these responses from developers.

24. "It works fine on MY computer"
23. "Who did you login as ?"
22. "It's a feature"
21. "It's WAD (Working As Designed)"
20. "That's weird..."
19. "It's never done that before."
18. "It worked yesterday."
17. "How is that possible?"
16. "It must be a hardware problem."
15. "What did you type in wrong to get it to crash?"
14. "There is something funky in your data."
13. "I haven't touched that module in weeks!"
12. "You must have the wrong version."
11. "It's just some unlucky coincidence."
10. "I can't test everything!"

For the top 9, see this site for a familiar chuckle...

Client-side expoit demonstration videos

NSS has been testing exploits and exploit-detection/protection products for the last 10 years or so, dating back to the early days of IDS and IPS . It's arguably one of our specialties. And we have written before about the differences between malware, exploits and vulnerabilities. Yet, as these pernicious threats are elevated into mainstream consumer and enterprise awareness, we are seeing quite a bit of terminology confusion. So, for our latest Host Intrusion Prevention System (HIPS) test of enterprise anti-malware products, we created some demonstration videos of client-side exploits used in the test.

Product	Vulnerability	Video Demonstration
Panda	CVE-2010-0249	http://www.youtube.com/watch?v=VPEzOZniuls
Panda	CVE-2010-0806	http://www.youtube.com/watch?v=_fstkScB7YM
F-Secure	CVE-2010-0249	http://www.youtube.com/watch?v=DdvtTuwfa5I
ESET	CVE-2006-0003	http://www.youtube.com/watch?v=U9WRkQyZvb8
Sophos	CVE-2006-4704	http://www.youtube.com/watch?v=F2gfNLzqBwg
Symantec	CVE-2010-0483	http://www.youtube.com/watch?v=RqQLmKI_R30

Client-side exploits

Client-side exploits are aggressive weapons used by cybercriminals that allow them to silently take control of computers that visit a web site. The infected widgets on nearly 5 million Network Solutions sites are a prime example (see Krebs' report on Armorize discovery). And NSS Labs' Q2 2010 test of 10 endpoint protection products for HIPS evaluates protection against client-side exploits.


In recent discussions with clients and journalists we've found the need to clarify some definitions of how these attacks work, and how they're different from typical socially-engineered malware campaigns. So, the following definitions and analogies are provided in an effort to provide clarification, as well as to bridge an ongoing communication gap between security vendors and their customers.

Vulnerability

Like a locked door that can be opened with the right key or combination, a vulnerability is a bug in software code that allows a product to be exploited. An example of a software vulnerability is improperly-defined memory usage within a function that enables content sent to a specific memory location to be run with privileged rights.

Exploit

An exploit is a specially crafted code sequence which can ”trigger” or ”unlock” a vulnerability within an application, such as a heap spray, buffer overflow attack, etc. An exploit can be hiding in an infected website (client-side attack) where it ambushes visiting computers or be launched from an another computer (remote attack).

Payload

The payload is the content that gets delivered once the vulnerable application has been exploited. Payloads are the actions that are performed on the compromised target computer, such as command execution, writing a file to disk, returning a reverse shell, etc. This may be malware, but does not have to be.

The Test

The test utilized 123 common and public vulnerabilities dating from 2006 to 2010. These vulnerabilities were exploited when a user visited an infected web page hosting the attack code. The attacks occurred in two stages:
1. The attacker caused a specially-crafted stream of data and code to be delivered to a precise location. This exploited the victim’s computer, gaining the attacker the ability to perform arbitrary code execution.
2. Malicious code was silently executed on the victim’s computer.

If the attack can be thwarted in Stage One (successful exploit), then it cannot progress to Stage Two, where a malicious payload can be delivered. As long as the exploit is not defeated, then installing malware is just one of many possible actions the attacker can take. And the choice of malicious code is nearly infinite. Since there are far fewer exploits than malware, it is imperative that attacks be defeated in the earliest possible stage. In other words, it is advantageous for AV suites to detect the exploit vs. chasing malware samples.

Results and Next Steps

Unfortunately, 75% of corporate users are under-protected, based on vendor market share and their respective scores, which ranged dramatically from 29% to 100%; and even worse for variants. Depending which product you have, you may have significant cause for concern. So, what's next?

For starters, if you're an organization with critical data behind any of these products, I suggest you buy and read the full report here. There has been a lot of press coverage of the report highlights, but not the details, generally reporting failure of AV suites in this area. While you may not be ripping out thousands of endpoint deployments, you may be asking your vendor some tough questions and setting expectations for improvement. Exploit detection & blocking is clearly an area that the security industry needs to focus on going forward. Got questions about other threat mitigation options, consult our other reports, or contact us.

Fragroute – Bug in 24-byte fragmentation

In our Q4 2009 IPS test, we tested Network Intrusion Prevention Systems on their ability to resist 60 different evasion techniques, IP Fragmentation (9), TCP Segmentation (11), RPC Fragmentation (16), URL Obfuscation (15), and FTP / Telnet evasions (9).

On May 19, 2010, TippingPoint brought to our attention a bug in fragroute, the de facto evasion tool for IP fragmentation. This flaw corrupts one of the evasion techniques – 24 byte fragments. Over the past few weeks, NSS Labs has retested all of the products from the Q4 2009 IPS Group Test with an alternate version of the tool that does not corrupt traffic when fragmenting into 24-byte segments.

We found that all of the devices that initially passed this test, continued to pass. In addition, we determined that TippingPoint does block the 24 byte fragmentation evasion with the non-corrupted attack. The findings for all other evasion tests remain unaltered.

The Q4 2009 IPS Group Test is being modified to reflect this update.

Approaches to Detecting Evasion

When an attacker uses an evasion technique, he is altering traffic so that it cannot be detected by a security product such as a Network IPS. To accomplish this, the traffic is run through a tool which manipulates the data stream and modifies it using a pre-determined pattern – similar to an encoder. Thus, to detect the attack, a network IPS needs to do the same in reverse – in essence, decode the data stream.

Alternatively, an IPS can drop traffic that appears to have been altered (e.g. fragmented or segmented) under the assumption that it is bad. Unfortunately, this is not the case much of the time since legitimate network traffic comes in all shapes and sizes. Thus, when vendors elect to drop such traffic instead of normalizing / decoding it and inspecting the content, they drop legitimate traffic. Knowing this, we have found that multiple vendors turn off those anti-evasion protections by default. This is a problem.

And this is why NSS Labs tests anti-evasion using vendor default settings.

Subsequent to our last IPS Group Test, NSS Labs found a loophole in our testing where multiple vendors enabled evasion detection that would block legitimate traffic. These evasion defenses would therefore never be deployed in the real world. We are therefore adding false positive testing for evasions to our upcoming IPS Group Test scheduled for Q3 2010.

Passions of an assessor: Donde esta corazon?

Michelle is a passionate infosec pro and assessor. She gets some kudos today for expressing on a personal level the frustrations of many infosec practitioners whose job it is to audit, assess and help improve their clients' defenses. PCI DSS forces those who would do little or nothing for security to do something more. It also encourages those who would do more to do less because it is just enough to deal with a clear and present threat: the audit.

As Josh Corman at the 451 Group likes to say: “Why focus on compliance instead of security? I might be hacked, but I will be fined.” (if you handle cardholder data). Given the amount of client-side attacks and botnet infection data we see, the case could be made otherwise. Corporations are getting attacked daily. They might not be aware of it though, due to the holes in their security defenses, logs, and even alerting practices.

After all, security products can only alert and report on what they have detections for. Based on our testing, that leaves a significant gap with every vendor, between 12 and 83%. Do you know which holes matter on your network and where they are? Want to hear ideas on how to improve and not just pass?

I'm happy to echo Michelle's call for more heart and less check box.

Rick

Thanks for breaking it!

People hire us to break stuff (and lately we're pretty good at it). Well not just, but breaking is part of testing, as is validating You'd think folks wouldn't want to hire us for that, and a lot of times you'd be right. But, this week, we had a large networking vendor in the lab testing their product. On day 2 we discovered a significant vulnerability that we were able to exploit. We replicated it before their eyes. What did the vendor do? He gave our lead engineer a high five!

Why? because, after having visited several labs with this same product, we were the first to find something and not simply give it the 'rubber stamp.' This is why you test. Not just to validate features, but so you can find out what you still need to do to improve it. Good product developers like this one "get it." He just got tremendous value out of the engagement, and has already put in proposals on the spot for additional testing with us. And maybe his competitors have similar issues (which is often the case), so now this vendor is ahead of the game and will likely have it fixed very shortly.

IT Buyers: This is the attitude you want to see in your vendors.

Measuring Security

Vik Phatak, CTO participated on a panel discussion at SourceBoston conference titled "Measuring Security". This discussion explored the ins and outs of testing endpoint protection products, otherwise known as anti-virus/antimalware. Hosted by Andrew Jacquith of Forrester, and also with Peter Stelzhammer of AV-Comparatives, and Mario Vuksan of Reversing Labs. Watch the video link.

AV Testing double standards and independence

NSS Labs’ innovative tests are designed to inform end-users about how products truly perform against today’s motivated attackers. We perform a test or gap analysis on security products, so organizations can understand what is and isn’t being protected, and accurately assess the risk and take steps to mitigate it. While enterprises and government organizations appreciate this valuable, independent analysis, many of the AV vendors do not.

When NSS Labs published its uncensored, real-world results of endpoint protection products (AV), some vendors used the anti-malware testing standards organization (AMTSO) to try to discredit the test. One of their objections was that we recommend against buying products that scored on the bottom third of our test. Sorry, we unabashedly believe malware protection should indeed be the key purchasing criteria for an AV product. And for vendors who claim their anti-spam on the corporate desktop will improve their protection against socially-engineered malware hosted on web sites, that’s just stretching it.

Rather than shoot the messenger, vendors with their customer’s best interests in mind should seek to learn from tests like these in order to improve their products. Unfortunately, that’s usually not the case in the AV world after too many years of self-congratulatory testing and certification.

AMTSO is an AV vendor-driven consortium, and while it can be a useful information sharing organization for AV insiders, it has demonstrated its utter failure as a credible independent organization. Throughout the 3-year history of this organization, AMTSO has failed to evaluate the tests and certifications that most of its vendor members sponsor and fund; e.g. VB100% awards, ICSA Labs and West Coast Labs certifications. These validations are important sales material in the $9B market place, but they wouldn't pass the same AMTSO guidelines that were supposedly applied to the NSS Labs test.

Such market validations are a part of the industry, but can be dangerous when they convey a false sense of security to buyers as they do now. Meanwhile, end-users can stay well informed about what products do - and more importantly - what they DO NOT do, by reading our subscriber-funded research and test reports. If a vendor is complaining about our test, chances are they did poorly on an important metric. Learn what some vendors don’t want you to see by reading our independent anti-malware test reports or the Google Aurora protection analysis report in particular (free to non-clients).

caveat emptor