Our Product is Great! It’s 93 Percent Effective! Buy Our Product!
October 27, 2010
By Jack Walsh

A product developer proudly sent out an e-mail blast to all of their friends and family stating that their product had done well in a recent third-party network IPS test. The developer was excited that the product had better than 93 percent coverage protection during the test. Ninety-three percent may be great for a high school exam, but for a product designed to protect enterprises from attacks, the product may not be doing very well if 7 percent of relevant enterprise vulnerabilities go unprotected. It only takes one successful attack to exploit a vulnerable system. Thus, if there were 100 vulnerabilities in the test set that would mean the product isn’t providing protection for seven of them. Remember, it only takes one for a compromise to occur.

One could argue that the testing conducted on this product may or may not be targeting relevant vulnerabilities or testing them the right way. And, that should be a concern for enterprises.  Unless the vulnerability test set for this testing is published, there is no way to confirm whether the vulnerabilities tested are in enterprise-relevant software or not. The 7 percent missed could be irrelevant vulnerabilities in software not found in the enterprise. Perhaps the 7 percent missed are in relevant enterprise software but to exploit them, either authentication is first required or the software has to be in an unlikely configuration. Or, maybe some of the exploits missed aren’t actually working exploits. 

All of these things about the test may be true, but let’s assume everything was as relevant as possible and that the test was done as well as possible. Let’s pretend that the developer requested to have the product tested by ICSA Labs. What would have happened and how do enterprises benefit? 

ICSA Labs would have told the developer that its product is not providing 100 percent coverage protection for the vulnerabilities in the test set. The vendor would then have to go back and try to repair the coverage protection holes. It might conserve test lab resources and ultimately cost less to perform testing if testing ended after the initial coverage protection test as it did in the scenario above; however, no enterprise wants the IPS it deploys to offer 30, 60, or 90 percent coverage when 100 percent is possible. 

The resulting iterative, back-and-forth testing conducted by ICSA Labs can create many headaches, in that testing becomes increasingly labor intensive potentially affecting the bottom line.  However, ICSA Labs recognizes that vulnerability coverage protection testing should be done with the best interests of enterprise organizations in mind. Thus, ICSA Labs continues to do what it believes is best for enterprises when it comes to IPS testing. 

Note that ICSA Labs’ goal is not simply to test products.  Our goal is for products to improve as a result of testing we conduct. Enterprises should have multiple choices from products that go the extra mile to provide complete coverage protection. Also, enterprises should have some assurance that they are protected against a published, evolving set of relevant high severity vulnerabilities within enterprise software. In the end, ICSA Labs wants enterprises to be able to make a true comparison of products. 

Don’t be satisfied with pretty good network IPS protection based on tests where the vulnerabilities may or may not be relevant to enterprises.  Instead, consider certified network IPS products that provide 100 percent coverage protection for the enterprise-relevant test set against which they were most recently tested.



Post new comment

  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>

More information about formatting options

This question is for testing whether you are a human visitor to prevent automated spam submissions.
Enter the characters shown in the image.