I had the opportunity a few weeks ago to see advanced copies of the NSS Labs' reports on consumer and corporate endpoint protection (AV/anti-malware). It was rather interesting reading the corporate report (both available here), in particular, as it highlights how little progress we've made in this industry. No solution had a 100% success rate (how could a signature-based solution? answer: it can't). It was also interesting to find that the heavier the software install, the better it generally performed in detection. Products with a much smaller footprint (e.g. ESET) just don't hold up very well in comparison.
The report emphasizes some interesting points. First and foremost, the AV vendors are having to reinvent themselves, and dramatically at that. They realize the inadequacy of traditional approaches, and as such are trying to come up with new ideas. The increasingly relevant role of web 2.0 and social networking, in particular, is forcing some of these changes.
One of the key evolutions we're seeing is a move to so-called cloud-based solutions. That is, there is now an online component to the AV solution that is designed to keep signatures fresh and position products to more rapidly adapt to the changing threat landscape. Unfortunately, this online component also creates an opportunity for malware to further defeat these solutions. The cat-n-mouse game will continue, and I'd put my money on the attackers continuing to win.
The results of these tests also highlighted a back-shift in the state of the industry; namely, that there is now again a sizable performance gap between the best and worst products. NSS Labs reports a nearly 30% difference between the top and bottom, indicating that AV is no longer a commodity. What does this mean in practical terms? For one thing, it means you can't just go grab any old AV off the shelf and get the same level of protection. I doubt it will take long for a re-stablization and return to the old commodity status, but we'll probably measure it in terms of years, not days, weeks, or months.
Overall, my main criticism of the report is that they simply did not test enough products. NSS Labs only tested the following 10 products:
* AVG Internet Security, version 8.5.364
* Eset Smart Security 4, version 4.0.437
* F-Secure Client Security version 8.01
* Kaspersky Internet Security 2010, version 9.0.0.459
* McAfee VirusScan Enterprise:8.7.0 + McAfee Site Advisor Enterprise:2.0.0
* Norman Endpoint protection for Small Business and Enterprise
* Sophos Endpoint Protection for Enterprise - Anti-Virus version 7.6.8
* Symantec Endpoint Protection (for Enterprise), version 11
* Panda Internet Security 2009, version 14.00.00
* Trend Micro Office Scan Enterprise, version 10
In the future, I'd hope to see other products included in the testing, and not just traditional commercial solutions. For example, I'd love to know how Amavis and ClamAV stack up, as well as Free AVG. Moreover, I'd like to see AV solutions from Sunbelt Software and Immunet also included. These solutions tout themselves as next-generation, and so it would be great to know just how well they perform.
The other interesting note, to me anyway, is the disparity in results between the NSS Labs testing and the certification from ICSA Labs. If one were to compare solutions that were ICSA Labs certified, then one would think that all of the above solutions were equal. However, the NSS Labs testing clearly shows that this is simply not true. I would not be surprised to see ICSA Labs evolving their testing and certification program in the near future given the business threat from NSS Labs.
Kudos to NSS Labs for this useful and informative report!
I think it is important to understand that ICSA Labs performs certification testing, not comparative testing. There is a huge difference in the two. Please view our monthly test results over time and you will see that not all vendors products are equal.
Andy Hayter, Anti-Malcode Program Manager, ICSA Labs
@Andrew -
This begs the question, what value is ICSA Labs adding? If it's not a differentiator, and it's not proof of an adequate degree of performance, then I'm not sure that you guys are really all that relevant any more...
-ben