When it comes to testing coverage protection for their network intrusion prevention system (IPS), enterprise end users may use a commercial tool that contains and replays many exploit packet captures.
While this type of product may work fine for an end user, ICSA Labs has to approach the question of whether or not to use such tools from a different perspective. As a test lab we have to worry about credibility as well as building and retaining end-user trust. It is precisely because we need to remain trustworthy that ICSA Labs does not use such tools in our network IPS testing - at least not to replay packaged exploit packet captures.
Wondering how trust and credibility are related to the use of commercial tools that contain exploit packet captures? Here's how the two things come together. If ICSA Labs were to use one or more exploit packet captures created elsewhere, then we would be effectively vouching for the quality and accuracy of these packet captures. But that is the problem; we cannot vouch for their quality and accuracy.
ICSA Labs does not know whether the code for each would-be exploit actually works as expected. Even if it did work, we cannot confirm that the would-be exploit was run against a vulnerable system when the capture was made. And assuming it was a working exploit that was run against a vulnerable system, we do not know whether the attack succeeded when the packet capture was made. Also, information in the commercial tool typically indicates at which vulnerability each exploit packet capture is aimed. But again, a test lab has no reasonable way to confirm that. To use the tool in this way ICSA Labs would have to make many assumptions and essentially trust an entity outside of our control.
Now let's give the commercial tool the benefit of the doubt. Maybe the exploit code worked. Maybe it was run against a vulnerable system when the packet capture was made. Maybe the vulnerable system was compromised as expected. Maybe the exploit does indeed target the vulnerability claimed by the commercial tool. But what happens if the vendor's IPS proxies traffic or alters the content of traffic as some IPS products do? Keep in mind that this is a replayed packet capture, not a live exploit. If the commercial tool with its packet capture of an exploit is run against an IPS that does one of these things and the IPS fails to block the attack, did the IPS really fail? Remember, the IPS modified the traffic on the line.
If the IPS vendor cannot reproduce the issue reported to them by the test lab, then the test lab should be able to confirm its findings in some way. But minus the real attack and actual vulnerable system, that is either a very tall order or impossible! A final consideration is this: in some cases, another company creates the exploit packet captures that the commercial tool vendor then incorporates into their product. In this case, not even the tool vendor can vouch for the accuracy of the exploit.
ICSA Labs is unwilling to risk its reputation and the trust of end users through the use of packaged exploit packet captures in its testing. All of the exploit packet captures we use in network IPS testing were captured here in the lab by our experts. And in ALL cases, we are in a position to verify our coverage protection test results by running the real, live attack against the actual vulnerable system.
 A packet capture is a snapshot of the network traffic on a network during a particular window of time. To create an exploit packet capture, the test lab uses an attack system with the compiled exploit code, a vulnerable system (the victim), and a "sniffer" to capture the network traffic (e.g., tcpdump). With the sniffer actively capturing all traffic back and forth between the attacker and victim, the exploit is run to completion by the test lab personnel. Once the victim has been compromised in the manner described by the vulnerability, the test lab analyst stops sniffing the traffic. The resulting snapshot of packets over that period is an exploit packet capture. Note that the resulting packet capture, called a pcap for short, may need some cleaning to remove stray frames such as any arp-related frames.