========================================================================= Filename: eval.txt "Pocket PC Scanner Test 2003-05" Henrich C. Pöhls ========================================================================= Evaluation and detailed results for Macro and Script Virus/Malware detection under Pocket PC 2002 in "Pocket PC Scanner Test 2003-05": The test is part of my diploma thesis in computer science, with the title "Risk Analysis of mobile devices with special concern on malware contamination" carried out at the University of Hamburg. The following *4* products participated in "Pocket PC Scanner Test 2003-05": ---------------------------------------------------------------------- Products for Pocket PC 2002 ---------------------------------------------------------------------- AVP Kaspersky Anti-Virus for Windows CE, Version 4.0.0.4 (Kaspersky Labs, Moscow, Russia) v(def) date: January 27, 2003 FSE F-Secure Antivirus For Pocket PC, Release 1.5 (F-Secure Corporation, Helsinki, Finland) v(def) date: December 16, 2002 - 12:13 INO eTrust Antivirus for Pocket PC, Version 2.00.31 (Computer Associates, Islandia, USA) v(def): 23.59.36 date: February 10, 2003 Eng: 23.59.00 date: December 17, 2002 PCC PC-cillin Wireless for Pocket PC, Version 2.0 (Trendmicro, Tokyo, Japan) v(def): 345 Eng: 5.200-0522 ---------------------------------------------------------------------- The four products were tested standalone on a Pocket PC device (T-Mobile MDA) according to the test procedures laid out in PROCEDURES.TXT. To be tested they had to meet the conditions, as stated in CONDITIONS.TXT. The test shows the detection rate of older ITW Script and Makro malware, as the testbeds were taken from previous (frozen October 2001) and current testbeds (frozen January and April 2002). Please see TESTBEDS.TXT for details. Summary of results: ------------------------------------------------------------------------- All results may be influenced by problems experienced during tests, see PROBLEMS.TXT for details. Malware contamination: ---------------------- No malware known today is able to execute and so contaminate the Pocket PC platform (see GOALS.TXT for a definition), so no detailed test results can be made. But even with no Pocket PC malware in existence there are already four products that can combat upcoming malware. Malware always used to be there first on different platforms in the past, on the Pocket PC platform the anti-virus programs are there first, before the first malware arrives, hopefully this lead will protect from future Pocket PC malware. AVP, FSE, INO and PCC all run standalone on the device so the user can scan incoming objects and detect Pocket PC malware, if the patterns include the malware. To keep this lead the patterns must be kept up-to-date on all the mobile devices. Please read more details on that in my diploma thesis "Risk Analysis of Mobile Devices with Special Concern on Malware Contamination". Malware distribution: --------------------- Mobile device can be used as gateways, storing and forwarding probably malicious code, even if the malware is not executed on the Pocket PC platform. To combat malware distribution, also malware from other platforms, such as desktop Windows operating systems (i.e. Windows 2000) needs to be detected. Two out of four products: INO and PCC detect also non Pocket PC malware. The following table shows the avarage detection rate over all testbeds splitted into Script and Macro for the two products INO and PCC: ITW Macro-Viruses | ITW Script-Viruses samples malware | samples malware -----------------------+------------------- INO 100.~% 100.0% | 98.9% 100.0% | PCC 96.4% 96.6% | 77.0% 75.3% ------------------------------------------- INO's avarage is above 98.9% in detected samples and a constant 100% in detected malware, where as PCC is far lower in detection rates. Also with PCC not reporting the identification, the identification quality is very poor. The following tables show the detection and identification quality concerning MACRO and SCRIPT viruses for each of the testbeds, under Pocket PC 2002. For more information on certain testbeds see TESTBEDS.TXT Index of tables: ---------------- SCR.S1: "ITW-Script Testbed itwskri.002": Results of Test for script viruses on itwskri.002 (status October 31,2001) SCR.S2: "ITW-Script Testbed scr_itw.021": Results of Test for script viruses on scr_itw.021 (status January 31,2002) SCR.S3: "ITW-Script Testbed scr_itw.024": Results of Test for script viruses on scr_itw.024 (status April 30,2002) MAC.M1: "ITW-Macro Testbed itwmac.002": Results of Test for macro viruses on itwmac.002 (status October 31,2001) MAC.M2: "ITW-Macro Testbed mac_itw.021": Results of Test for macro viruses on mac_itw.021 (status January 31,2002) MAC.M3: "ITW-Macro Testbed mac_itw.024": Results of Test for macro viruses on mac_itw.024 (status April 30,2002) TOTAL: Macro/Script ITW Virus Detection Rates Table SCR.S1: "ITW-Script Testbed itwskri.002": Results of Test for script viruses on itwskri.002 (status October 31,2001): =================================================================== This includes Viruses ---- unreliably ---- Files Scanner detected identified detected detected ------------------------------------------------------------ Testbed 20 100.0% 122 100.0% ------------------------------------------------------------ AVP 0 0.0 0 0.0 0 0.0 0 0.0 FSE 0 0.0 0 0.0 0 0.0 0 0.0 INO 20 100.0 5 25.0 1 5.0 118 96.7 PCC 15 75.0 15 75.0 5 25.0 89 73.0 ------------------------------------------------------------ Remark: decimal ~ indicates that result is rounded: (100.~ up to 100.0%, 0.~ down to 0.0%). Table SCR.S2: "ITW-Script Testbed scr_itw.021": Results of Test for script viruses on scr_itw.021 (status January 31,2002): =================================================================== This includes Viruses ---- unreliably ---- Files Scanner detected identified detected detected ------------------------------------------------------------ Testbed 25 100.0% 34 100.0% ------------------------------------------------------------ AVP 0 0.0 0 0.0 0 0.0 0 0.0 FSE 0 0.0 0 0.0 0 0.0 0 0.0 INO 25 100.0 1 4.0 0 0.0 34 100.0 PCC 19 76.0 19 76.0 0 0.0 27 79.4 ------------------------------------------------------------ Remark: decimal ~ indicates that result is rounded: (100.~ up to 100.0%, 0.~ down to 0.0%). Table SCR.S3: "ITW-Script Testbed scr_itw.024": Results of Test for script viruses on scr_itw.024 (status April 30,2002): =================================================================== This includes Viruses ---- unreliably ---- Files Scanner detected identified detected detected ------------------------------------------------------------ Testbed 24 100.0% 33 100.0% ------------------------------------------------------------ AVP 0 0.0 0 0.0 0 0.0 0 0.0 FSE 0 0.0 0 0.0 0 0.0 0 0.0 INO 24 100.0 1 4.2 0 0.0 33 100.0 PCC 18 75.0 18 75.0 0 0.0 26 78.8 ------------------------------------------------------------ Remark: decimal ~ indicates that result is rounded: (100.~ up to 100.0%, 0.~ down to 0.0%). Table MAC.M1: "ITW-Macro Testbed itwmac.002": Results of Test for macro viruses on itwmac.002 (status October 31,2001): =================================================================== This includes Viruses ---- unreliably ---- Files Scanner detected identified detected detected ------------------------------------------------------------ Testbed 124 100.0% 1337 100.0% ------------------------------------------------------------ AVP 0 0.0 0 0.0 0 0.0 0 0.0 FSE 0 0.0 0 0.0 0 0.0 0 0.0 INO 124 100.0 8 6.5 1 0.8 1336 99.9 PCC 118 95.2 118 95.2 6 4.8 1276 95.4 ------------------------------------------------------------ Remark: decimal ~ indicates that result is rounded: (100.~ up to 100.0%, 0.~ down to 0.0%). Table MAC.M2: "ITW-Macro Testbed mac_itw.021": Results of Test for macro viruses on mac_itw.021 (status January 31,2002): =================================================================== This includes Viruses ---- unreliably ---- Files Scanner detected identified detected detected ------------------------------------------------------------ Testbed 115 100.0% 224 100.0% ------------------------------------------------------------ AVP 0 0.0 0 0.0 0 0.0 0 0.0 FSE 0 0.0 0 0.0 0 0.0 0 0.0 INO 115 100.0 2 1.7 0 0.0 224 100.0 PCC 111 96.5 111 96.5 1 0.9 215 96.0 ------------------------------------------------------------ Remark: decimal ~ indicates that result is rounded: (100.~ up to 100.0%, 0.~ down to 0.0%). Table MAC.M3: "ITW-Macro Testbed mac_itw.024": Results of Test for macro viruses on mac_itw.024 (status April 30,2002): =================================================================== This includes Viruses ---- unreliably ---- Files Scanner detected identified detected detected ------------------------------------------------------------ Testbed 99 100.0% 189 100.0% ------------------------------------------------------------ AVP 0 0.0 0 0.0 0 0.0 0 0.0 FSE 0 0.0 0 0.0 0 0.0 0 0.0 INO 99 100.0 1 1.0 0 0.0 189 100.0 PCC 97 98.0 97 98.0 1 1.0 185 97.9 ------------------------------------------------------------ Remark: decimal ~ indicates that result is rounded: (100.~ up to 100.0%, 0.~ down to 0.0%). Table TOTAL: Macro/Script ITW Virus Detection Rates: =================================================================== Scan I == Macro Virus == + == Script Virus == ner I Detection I Detection -----+-------------------+------------------- Test I PPC 2003-05 I PPC 2003-05 -----+-------------------+------------------- AVP I 0.0% I 0.0% FSE I 0.0% I 0.0% INO I 100.0% I 100.0% PPC I 96.6% I 75.3% -----+-------------------+------------------- ============================================================ End of: eval.txt "Pocket PC Scanner Test 2003-05" Henrich C. Pöhls ============================================================