========================================================================= Filename: eval.txt "Pocket PC Scanner Test 2003-07" Henrich C. Pöhls ========================================================================= Evaluation and detailed results for Macro and Script Virus/Malware detection under Pocket PC 2002 in "Pocket PC Scanner Test 2003-07": The test is part of my diploma thesis in computer science, with the title "Risk Analysis of mobile devices with special concern on malware contamination" carried out at the University of Hamburg. The following *4* products participated in "Pocket PC Scanner Test 2003-07": ---------------------------------------------------------------------- Products for Pocket PC 2002 ---------------------------------------------------------------------- AVP Kaspersky Anti-Virus for Windows CE, Version 4.0.0.4 (Kaspersky Labs, Moscow, Russia) v(def) date: January 27, 2003 FSE F-Secure Antivirus For Pocket PC, Release 1.5 (F-Secure Corporation, Helsinki, Finland) v(def) date: December 16, 2002 - 12:13 INO eTrust Antivirus for Pocket PC, Version 2.00.31 (Computer Associates, Islandia, USA) v(def): 23.59.36 date: February 10, 2003 Eng: 23.59.00 date: December 17, 2002 PCC PC-cillin Wireless for Pocket PC, Version 2.0 (Trendmicro, Tokyo, Japan) v(def): 345 Eng: 5.200-0522 ---------------------------------------------------------------------- The four products were tested standalone on a Pocket PC device (T-Mobile MDA) according to the test procedures laid out in PROCEDURES.TXT. To be tested they had to meet the conditions, as stated in CONDITIONS.TXT. The test shows the detection rate of older ITW Script and Makro malware, as the one testbed included samples from viruses from November 2001 till December 2002, and most recent viruses from December 2002 only.Please see TESTBEDS.TXT for details. Summary of results: ------------------------------------------------------------------------- All results may be influenced by problems experienced during tests, see PROBLEMS.TXT for details. Malware contamination: ---------------------- No malware known today is able to execute and so contaminate the Pocket PC platform (see GOALS.TXT for a definition), so no detailed test results can be made. But even with no Pocket PC malware in existence there are already four products that can combat upcoming malware. Malware always used to be there first on different platforms in the past, on the Pocket PC platform the anti-virus programs are there first, before the first malware arrives, hopefully this lead will protect from future Pocket PC malware. AVP, FSE, INO and PCC all run standalone on the device so the user can scan incoming objects and detect Pocket PC malware, if the patterns include the malware. To keep this lead the patterns must be kept up-to-date on all the mobile devices. Please read more details on that in my diploma thesis "Risk Analysis of Mobile Devices with Special Concern on Malware Contamination". Malware distribution: --------------------- Mobile device can be used as gateways, storing and forwarding probably malicious code, even if the malware is not executed on the Pocket PC platform. To combat malware distribution, also malware from other platforms, such as desktop Windows operating systems (i.e. Windows 2000) needs to be detected. Two out of four products: INO and PCC detect also non Pocket PC malware. The following table shows the avarage detection rate over both testbeds splitted into Script and Macro for the two products INO and PCC: ITW Macro-Viruses | ITW Script-Viruses samples malware | samples malware -----------------------+------------------- INO 100.~% 100.0% | 93.4% 100.0% | PCC 96.7% 97.0% | 55.4% 56.7% ------------------------------------------- INO's avarage in detected samples is good except for ITW-Script samples. And it is a constant 100% in detected malware, where as PCC is far lower in the detection rates. Also with PCC not reporting the identification, the identification quality is very poor. See also the graphic in DETECTIONRATES_2003-07.BMP. The following tables show the detection and identification quality concerning MACRO and SCRIPT viruses for each of the testbeds, under Pocket PC 2002. For more information on certain testbeds see TESTBEDS.TXT Index of tables: ---------------- SCR.S1: "ITW-Script Testbed scr_itw.304": Results of Test for script viruses on scr_itw.304 (Nov.01 till Dec.02) SCR.S2: "ITW-Script Testbed scr_itw.dec02": Results of Test for script viruses on scr_itw.dec02 (December 2002) MAC.S1: "ITW-Macro Testbed mac_itw.304": Results of Test for script viruses on mac_itw.304 (Nov.01 till Dec.02) MAC.S2: "ITW-Macro Testbed mac_itw.dec02": Results of Test for script viruses on mac_itw.dec02 (December 2002) TOTAL: Macro/Script ITW Virus Detection Rates December 2002 Table SCR.S1: "ITW-Script Testbed scr_itw.304": Results of Test for script viruses on scr_itw.304 (Nov.01 till Dec.02): =================================================================== This includes Viruses ---- unreliably ---- Files Scanner detected identified detected detected ------------------------------------------------------------ Testbed 30 100.0% 201 100.0% ------------------------------------------------------------ INO 30 100.0 14 46.7 4 13.3 188 93.5 PCC 19 63.3 10 33.3 19 63.3 112 55.7 ------------------------------------------------------------ Remark: decimal ~ indicates that result is rounded: (100.~ up to 100.0%, 0.~ down to 0.0%). Table SCR.S2: "ITW-Script Testbed scr_itw.dec02": Results of Test for script viruses on scr_itw.dec02 (December 2002): =================================================================== This includes Viruses ---- unreliably ---- Files Scanner detected identified detected detected ------------------------------------------------------------ Testbed 22 100.0% 178 100.0% ------------------------------------------------------------ INO 22 100.0 6 27.3 3 13.6 166 93.3 PCC 11 50.0 11 50.0 0 31.8 98 55.1 ------------------------------------------------------------ Remark: decimal ~ indicates that result is rounded: (100.~ up to 100.0%, 0.~ down to 0.0%). Table MAC.M1: "ITW-Macro Testbed mac_itw.304": Results of Test for script viruses on mac_itw.304 (Nov.01 till Dec.02): =================================================================== This includes Viruses ---- unreliably ---- Files Scanner detected identified detected detected ------------------------------------------------------------ Testbed 118 100.0% 277 100.0% ------------------------------------------------------------ INO 118 100.0 5 4.2 0 0.0 277 100.0 PCC 114 96.6 114 96.6 2 1.7 266 96.0 ------------------------------------------------------------ Remark: decimal ~ indicates that result is rounded: (100.~ up to 100.0%, 0.~ down to 0.0%). Table MAC.M2: "ITW-Macro Testbed mac_itw.dec02": Results of Test for script viruses on mac_itw.dec02 (December 2002): =================================================================== This includes Viruses ---- unreliably ---- Files Scanner detected identified detected detected ------------------------------------------------------------ Testbed 76 100.0% 976 100.0% ------------------------------------------------------------ INO 76 100.0 5 6.6 1 1.3 975 99.9 PCC 74 97.4 74 97.4 1 6.9 950 97.3 ------------------------------------------------------------ Remark: decimal ~ indicates that result is rounded: (100.~ up to 100.0%, 0.~ down to 0.0%). Table TOTAL: Macro/Script ITW Virus Detection Rates: =================================================================== Scan I == Macro Virus == + == Script Virus == ner I Detection I Detection -----+-------------------+------------------- Test I PPC 2003-07 I PPC 2003-07 -----+-------------------+------------------- Test I I bed I mac_itw.dec02 I scr_itw.dec02 -----+-------------------+------------------- INO I 100.0% I 100.0% PPC I 97.4% I 50.3% -----+-------------------+------------------- ============================================================ End of: eval.txt "Pocket PC Scanner Test 2003-07" Henrich C. Pöhls ============================================================