Three anti-virus testing questions/accusations

When we, at Dennis Technology Labs, publish results from our anti-malware tests internet users can be predicted to raise a number of questions, points of opinion and direct accusations.

Here are three of the most common, along with my responses:

1. The test is rigged because I don't believe that Product X would do well but it did. (Or Product Z performed badly but I know that it's good).

The test is not biased in favour of any vendor or vendor's product. Claims to the contrary should be accompanied by evidence.

We deal directly with all vendors involved in our tests and any corruption on our part would, I have no doubt, be discovered and publicised very quickly.

Our reputation is crucial and cheating in tests really makes no sense from a business perspective.

2. Testing on unpatched systems is pointless and produces worthless results.

Anti-malware tests by all well-known testing labs, including Dennis Technology Labs, AV Comparatives, AV Test and NSS Labs focus on testing the actual security software and not other elements. For this reason no tester runs what we call 'security endpoint' tests with the very latest Windows patches deployed.

To use a tired car analogy, if you were to test tyres you would use sub-optimal conditions, such as wet roads and sharp bends. Similarly, providing vulnerable software used by today's malware allows testers to stress the anti-malware software and determine which products are most effective.

In fact, what we see in terms of threats are far more likely to target third-party applications rather than Microsoft Windows components.

In an experimental test that we ran last year, we found that patching Windows with the very latest updates (on a daily basis) had a small effect in preventing the threats, but not enough to make much difference in a test such as we run.

That said, we're not saying that updating Windows is pointless. Far from it - it makes a lot of sense to fix known vulnerabilities.

3. Why doesn't this test include Product Y?

The way that we test is very detailed and labour-intensive. This means we are quite restricted in how many products we test.

Any anti-malware vendor is welcome to engage with us and discuss the inclusion of its products into the test suite.