IFIP SEC 2018, Abstract of the talk

Anti-forensic = Suspicious: Detection of Stealthy Malware that Hides its Network Traffic

Mayank Agarwal¹, Rami Puzis¹ Jawad Haj-Yahya², Polina Zilberman¹, and Yuval Elovici¹
¹ Telekom Innovation Labs, Ben-Gurion University of the Negev, Israel ² School of Computer Science and Engineering, Nanyang Technological University, Singapore

Abstract. Stealthy malware hides its presence from the users of a sys- tem by hooking the relevant libraries, drivers, system calls or manipulating the services commonly used to monitor system behaviour. Tampering the network sensors of host-based intrusion detection systems (HIDS) may impair their ability to detect malware and significantly hinders subsequent forensic investigations. Nevertheless, the mere attempt to hide the traffic indicates malicious intentions. In this paper we show how comparison of the data collected by multiple sensors at different levels of resilience may reveal these intentions. At the lowest level of resilience, information from untrusted sensors such as netstat and process lists are used. At the highest resilience level, we analyse mirrored traffic using a secured hardware device. This technique can be considered as fully trusted. The detection of a discrepancy between what is reported by these common tools and what is observed on a trusted system operating at a different level is a good way to force a dilemma on malware writers: either apply hiding techniques, with the risk that the discrepancy is detected, or keep the status of network connections untouched, with a greater ability for the administrator to recognize the presence and to understand the behaviour of malware. The proposed method was implemented on an evaluation testbed and is able to detect stealthy malware that hides its communication from the HIDS. The false positive rate is 0.01% of the total traffic analysed, and barring a few exceptions that can easily be white-listed, there are no legitimate processes which raise false alerts.

Keywords: Stealthy, Malware, Command & Control, Trusted Network Monitor, Security

The paper published in the IFIP SEC 2018 confeence proceedings by Springer Verlag