by Henrich C. Pöhls during the course of my diploma thesis on "Risk Analysis of Mobile Devices with Special Concern of Malware Contamination" - August 2003
Index:
1. Introduction & Goals of the Tests
5. Evaluation of Products & Problems & Results
5.1 Problems
5.2 Results
5.3 Graphical Representation
5.4 Summary
More and more mobile devices are used in today’s computing, and even more will be used in the future.
Mobile devices shall be protected at least with the same safe guards against malware contamination and distribution as desktops and servers are today. Malware contamination and malware distribution are defined at the end of this document. A holistic approach to malware defence would have to avoid a weak link in the chain of protection. If malware can be distributed through mobile devices also the desktop and server environment is at risk.
The number of malware grows, and so also mobile device will be exposed to malware. The following graphic shows the increase of malware in the aVTC testbeds.
The Pocket PC 2002 test shall allow identifying the protection offered by the anti-malware products running on a mobile device in general and for individual products. This shall be done in a comparable way, so that it can be contrasted with the protection offered by desktop or server anti-malware products.
The goal of the test is to measure the detection rate and detection quality of malware with anti-malware products running on mobile devices.No malware for Pocket PC 2002 devices is known today, so this test uses test beds with desktop based script and macro malware, also from the currently running aVTC test (see testbeds).
The test shall mimic a real life environment as best as it can, therefore the scanning will take place on the mobile device, more on the process in procedures.
Distribution does not need to be an automatic task.
The result of an execution of malware is a contaminated mobile device.
The following conditions resulted in the selection of the products for the test.
This file lists the test conditions, they differ slightly from the test conditions for normal aVTC tests.
Compared to the normal aVTC test conditions, it shall be noted, that some conditions
are left out, as they do not
apply to mobile devices in general or to the tests conducted on the mobile device
for this work, as for example the test
conditions for boot viruses, or packed viruses.
Also some conditions have been made optional or their strictness is lowered
to suit the mobile device requirements
better. Conditions that are not essential are indicated by putting the condition
name (for example “AA”) in square brackets
(for example “[AA]”). If the condition is not essential a product
can be tested even the condition is not fulfilled
by the product. In some cases the strictness of a condition can only be lowered.
A less strict condition is then given
in square brackets and must be fulfilled instead of the stricter original condition.
If any condition is not met
this is listed in the detailed product description for each product in my diploma
thesis or the problems section.
A) Common conditions:
M) Conditions for tests against macro viruses:
Q) Conditions for other classes of viruses:
Additionally the following new test conditions must be met for the test on
the mobile devices running MS Pocket PC 2002 operating system:
C) Conditions for test on a mobile device
These two additional conditions ensure that the tested products run stand-alone on all Pocket PC 2002 devices.
This sections contains details on how the test is conducted, and how the results are aggregated.
The test will be performed on the test device: T-Mobile MDA.
This PocketPC 2002 device has 32 MB RAM, 32 MB ROM and the intel StrongARM
processor is running at 206 MHz.
The mobile device is connected to a desktop machine, running Windows 2000, via
the USB connection. The desktop is running ActiveSync Version 3.6 (Build 2148)
to facilitate the connection to the mobile device.
The test-beds come from a CD-ROM and no anti-malware is installed on the desktop.
To mimic a real life environment best the scanning will take place on the mobile device. As the complete test beds exceed the size of the memory, they are split into test sets of 16 MB. Information on the test beds is in section testbeds.
Each anti-malware product is installed on a clean mobile device, with the same setting for each installation.
The malicious samples of each test set are copied to the mobile device's RAM via the ActiveSync connection, set up in such a way that no conversion of exchanged files takes places.
The mobile device is then disconnected and the scan is started. When finished the log file is saved and transferred to the desktop by reconnecting it to the USB connection.
This is repeated for all test sets all test beds and all products.
See the following figure for a graphical representation.
The report file (or log-file) is used to calculate the measurements.
All tested products only log the path and name of infected objects, not the
path and name of all the objects scanned in their reports.
This partly violates test condition A2 (see conditions),
but most important the report file shall contain the path and filenames as this
is used to identify the malicious samples that are reported by the product.
All tested products do list the total number of objects scanned in their report, allowing calculating if all samples from the test set were touched.
For calculating the measures the following is assumed:
1. If the product reports a number equal to the number of samples in the test
set as scanned, it is assumed that the product has scanned them all. So only
those listed in the report have been identified as malicious.
2. If the product reports a lower number of objects as scanned than there are objects in the test set, it is assumed that the product has missed some objects. The product will then be retested on a sub test set. This sub test set will only contain the objects that have not been reported as obviously scanned by being identified as malicious in the reports. Thereby the number of objects that need to be scanned is reduced. This will be called a "re-test". Each product is allowed two such re-tests.
Any such problems and others are reported in the test results in evaluation section.
For Anti-malware products that are not able to scan a defined portion of the mobile device only, a pre-scan is initiated to reliably calculate the number of samples scanned. The pre-scan identifies the number of objects reported with the scanner options without the samples from the test bed on the mobile device.
The total number of objects scanned reported in the regular scans will then be decremented by the number of objects reported by the pre-scan.
Again any problems are reported in the test results in evaluation section.
For the following four products were tested during the course of my diploma thesis:
The four products were tested standalone on a Pocket PC device (T-Mobile MDA) according to the test procedures laid out in test procedures. To be tested they had to meet the conditions, as stated in conditions.
The problems occured during the course of this test are listed in this section gruped by products.
AVP: Kaspersky Anti-Virus for Windows CE
The user manual clearly states that AVP only detects Pocket PC malware, which
did not exist at the time of the test and which was therefore in none of the
testbeds used (see testbeds). Therefore it was not
tested, as in a pre-test (Pocket PC Scanner Test 2003-05) AVP had a detection
rate of zero. But AVP was also unable to detect the eicar testfile.
AVP is only able to scan the whole mobile device, it is not possible
to select a certain directory, so a pre-scan (see procedures)
was used to calculate the measurements from a full device scan.
FSE: F-Secure Antivirus For Pocket PC
According to the vendor FSE only detects Pocket PC malware, which did not exist at the time of the test and which was therefore in none of the testbeds used (see testbeds). Therefore it was not tested, as in a pre-test (Pocket PC Scanner Test 2003-05) FSE has shown a detection rate of zero.
INO: eTrust Antivirus for Pocket PC
No problems.
PCC: PC-cillin Wireless for Pocket PC
The manual states that PCC only detects Excel-type and Word VB script-type
viruses. Please see testbeds and the full PCC report
files to judge if that has affected the detection rate.
PCC does list the identification only on screen, it does not save the malware
identification in the report file, resulting in unreliable identification of
100%.
PCC can only scan the whole mobile device, so it was necessary to conduct pre-scans
(see procedures).
The test shows the detection rate of older ITW Script and Makro malware, as the one testbed included samples from viruses from November 2001 till December 2002, and most recent viruses from December 2002 only. I also conducted a pre-test using older testbeds, please see the evaluation text-file of the pre-test 2003-05. Please see testbeds for details.
The full details are available as text-files either for the Pocket PC 2002 Scanner Test 2003-07 evalutation or the pretest evaluation.
Malware contamination:
No malware known today is able to execute and so contaminate the Pocket PC
platform (see definitions), so no detailed test results
can be made. But even with no Pocket PC malware in existence there are already
four products that can combat upcoming malware. Malware always used to be there
first on different platforms in the past, on the Pocket PC platform the anti-virus
programs are there first, before the first malware arrives, hopefully this lead
will protect from future Pocket PC malware.
AVP, FSE, INO and PCC all run standalone on the device so the user can scan
incoming objects and detect Pocket PC malware, if the patterns include the malware.
To keep this lead the patterns must be kept up-to-date on all the mobile devices.
Please read more details on that in my diploma thesis "Risk Analysis of Mobile Devices with Special Concern on Malware Contamination".
Malware distribution:
Mobile devices can be used as gateways, storing and forwarding probably malicious code, even if the malware is not executed on the Pocket PC platform. To combat malware distribution, also malware from other platforms, such as desktop Windows operating systems (i.e. Windows 2000) needs to be detected.
Two out of four products: INO and PCC detect also non Pocket PC malware.
The following table shows the avarage detection rate over both testbeds splitted into Script and Macro for the two products INO and PCC in the Pocket PC scanner Testt 2003-07:
ITW Macro-Viruses |
ITW Script-Viruses |
|||
samples |
malware |
samples |
malware |
|
| INO | 100.~% |
100.0% |
93.4% |
100.0% |
| PCC | 96.7% |
97.0% |
55.4% |
56.7% |
This work has conducted the first test of the detection-rate of anti-malware products running stand-alone on Pocket PC 2002 devices.
The test yielded interesting results:
Two out of four products running stand-alone on the Pocket PC will not provide
any protection against the distribution of desktop malware through Pocket PCs.
The two Products AVP and FSE do not detect desktop-based malware. However, AVP
and FSE might be able to protect against malware contamination with future Pocket
PC malware, as they have proven their ability to scan all the files, though
not identifying any of the desktop based malware from the testbeds.
The remaining two products INO and PCC are able to provide also protection against
malware distribution, as they are able to detect desktop based malware from
the testbeds. So INO and PCC allow to protect mobile devices against future
malware contamination and also against malware distribution. From these remaining
two products only INO offers a sufficient level of protection and could compete
with the average desktop product in the detection of in-the-wild script and
macro viruses.
None of the products has on-access scanning capabilities. On-access
scanners are deployed to desktop computers to prevent the execution of malicious
code on the desktop computer, as they prevent the access to infected objects
before they are invoked. The protection offered by on-access scanners can not
be offered by the mechanisms included in the Pocket PC anti-malware products,
as a test of the “Automatic Scan on Removable Storage Media Insertion”
mechanism offered by two products has shown.
The tests have shown that it is possible to build sufficient anti-malware
products also on the Pocket PC platform. It must be further noted that although
no Pocket PC malware is known today, there are already four commercial products
waiting to use new pattern updates to detect the first Pocket PC malware.
This could be the greatest advantage that anti-malware products ever had. However,
they will only be able to keep this advantage, if a large number of Pocket PC
users install such anti-malware software and if they manage to keep their patterns
up-to-date.
The two products FSE and INO offer a pattern update function that can be used
from the mobile device, if it has an Internet connection. AVP also offers a
pattern update function, but the pattern is only updated when the mobile device
is in the cradled state. Only the remaining of the four products, PCC, needs
manual pattern updates. The user is advised to regularly update these patterns;
no matter how complicated it might be, as they are important for the detection
of malware.
Also the very different marketing concepts behind the four anti-malware products
will have an impact on how many mobile devices will be equipped with such anti-malware
software in the future. The four tested products show various marketing concepts:
One product can be downloaded free, two products can be bought as stand-alone
versions, and one product is only available as part of a complete product suite.
This work has not further investigated the costs of the different anti-malware
products, but this already shows that different vendors see the Pocket PC platform
with different eyes, when it comes to the prices and marketing.
None of the tested software showed adverse effects on the functionality, speed
or stability of the mobile device, so I suggest it is better to have a ready
and waiting anti-malware software installed, that regularly updates its pattern
than to foolishly wait until it is to late and the first malware is in-the-wild.
This section gives an overview and lists the contents of VTCs testbeds used.
6.1 Testbeds for "Pocket PC Scanner Test 2003-07"
1) Script In-The-Wild Testbeds
1.1) scr_itw.dec02 (From upcoming aVTC Test status December 31,2002)
1.2) scr_itw.304 (Longtime ITW Nov.01-Dec.02 status December 31,2002)
2) Macro In-The-Wild Testbeds
2.1) mac_itw.dec02 (From upcoming aVTC Test status December 31,2002)
2.2) mac_itw.304 (Longtime ITW Nov.01-Dec.02 status December 31,2002)
The full list of the testbeds of this test is available as a textfile.
Status indicates when the Testbeds were frozen. The Testbeds for testing the products on Pocket PC 2002 were taken from the upcoming running aVTC Test probably named Test 2003-09. Additionally longtime testbeds were compiled, they contain samples of viruses that were in-the-wild in the time between the testbeds from the pre-test (Pocket PC Test 2003-05) and the regular aVTC December 2002 testbed. These second testbeds contain samples from all script and macro viruses that were listed to be in-the-wild by wildlist.org from November 2001 to December 2002.
6.2) Testbeds used in the pre-test "Pocket PC Scanner Test 2003-05"
1) Script In-The-Wild Testbeds
1.1) itwskri.002 (From Test 2002-12 with status October 31,2001)
1.2) scr_itw.021 (From HeurekaIII with status January 31,2002)
1.3) scr_itw.024 (From HeurekaIII with status April 30,2002)
2) Macro In-The-Wild Testbeds
2.1) itwmac.002 (From Test 2002-12 with status October 31,2001)
2.2) mac_itw.021 (From HeurekaIII with status January 31,2002)
2.3) mac_itw.024 (From HeurekaIII with status April 30,2002)
The full list of the testbeds of the pre-test is available as a textfile.
The Testbeds for testing the products on Pocket PC 2002 were taken from older and currently running aVTC Tests. Status indicates when the Testbeds were frozen.
Henrich C. Pöhls - 2003/09
