Anti-malware testing: results challenged

When we run anti-malware tests we sometimes find anomalies in how anti-virus products behave when faced with threats.

Sometimes a new beta product from a vendor appears to be less effective than a previous version. In other cases there are differences between corporate and consumer products from the same vendor.

Because we want to recommend good products we find these differences to be interesting, but the vendors themselves seem to find them even more so.

For example, if we discover that a beta version is not up to scratch, the vendor has a chance to fix the problem before releasing the finished product. That is a valuable result of testing in the way that Dennis Technology Labs does.

In one memorable case we found that the corporate version of a product failed to protect against a threat that the related consumer product managed to handle perfectly well. The vendor in question found this result to be unlikely.

When we find a result that surprises a vendor we are challenged to provide evidence. This makes sense because if we are right then a lot of expensive time is going to be spent by the vendor in fixing the problem. Of course, the end result is good for everyone.

In the example above, we provided a report that led the vendor to discover a significant problem with its back-end updates system. A redacted PDF version is available on Dennis Technology Labs' website.

We expect and welcome challenges to our results. They provide vendors with useful information, which means better products (one hopes!) and they ensure that our own procedures are up to scratch.

The end result is better testing, better software and better protection for the user, which is what we care about the most.