Make Sense of Endpoint Malware Protection Technology

Published: 25 April 2017 ID: G00320339

Analyst(s):

 Ian McShane

Summary

The goal of endpoint malware protection is a solution that offers low administrative overhead, low end-user impact and the best available protection. Security and risk management leaders can make educated trade-offs within endpoint protection to achieve two of these three aims.

Overview

Key Challenges

• The marketing hype around "next-gen AV" and the IT industry's fascination with machine learning distracts from and creates confusion about the real value provided by different protection techniques.
• Unclear perceptions turn up constantly, as many techniques have similar names or umbrella terms like "application control," which can vary wildly in terms of actual capabilities.
• Blending technologies from multiple vendors risks agent bloat and software conflicts, resulting in disabled protection features and less-than-optimal configurations.
• Not all malware requires an exploit. Users can simply be tricked into downloading and running malware that does not require an exploit.

Recommendations

Security and risk management leaders overseeing endpoint and mobile security should:

• Design an endpoint protection strategy that consists of good security hygiene, layered protection and detection technologies, and end-user education.
• Avoid duplication of security capabilities across multiple solutions; instead, fully deploy existing protection and then begin to identify specific areas to augment.
• Avoid knee-jerk reaction purchases by mapping new purchases to gaps and taking the time to run a useful proof of concept to ensure the technology can fit or enhance existing workflows.
• Use a combination of internal testing and third-party effectiveness tests to verify vendor claims. Vendor-sponsored or -commissioned comparisons can be useful data points, but should not be given the same weight as impartial tests.

Introduction

Endpoint protection is not simple. Security and risk management leaders struggle to find the right balance between threat coverage, administrative overhead and end-user impact. Table 1 illustrates, at a high level, the impact that the most common anti-malware techniques can have for most organizations.

Table 1.   Common Anti-Malware Techniques

Technique	Threat Coverage	Admin. Requirement	End-User Impact
Signatures	Low	Low	Low
Machine Learning	Medium	Low	Low
Application Control	High	High	High
Application Isolation	Medium	High	High
Behavioral Analysis	High	Medium	Low
Exploit Mitigation	Medium	Low	Low

Source: Gartner (April 2017)

These technologies each carry different capabilities and, importantly, limitations. Although some technologies appear to offer similar functions, they are often marketed as the ideal solution for malware prevention. The hype around artificial intelligence and machine learning is adding more confusion to the matter.

In practice, a combination of technologies will provide the widest protection against malware attacks. Most attacks exploit well-known unpatched vulnerabilities, use social engineering to trick users to install malware, or use interpreted code such as Java to download and install malware. Fileless malware is becoming more and more prevalent in the threat landscape. To address such challenges, security and risk management leaders have a range of options from both established and emerging vendors. Most buyers continue to consider emerging solutions to be complementary, rather than outright endpoint protection platform (EPP) replacements. These options are covered from a technical perspective in "Comparing Endpoint Technologies for Malware Protection."

The expansion of malware protection technologies in EPPs over the past five years has delivered various advantages, including fewer updates and less administrative overhead, and provided for better protection at specific stages of the kill chain or for specific classes of malware.

It is important to consider education as a key part of the fight against malware. Users remain the weak links — they are impressionable, and subject to deception and coercion. Security awareness programming plays an important part in informing staff and partners of their responsibility in limiting vulnerable behavior.

Signature-based detection is the most well-known approach to malware detection. Because signatures and heuristics use pattern matching to identify malicious files — meaning the vendor must have seen the file to create the signature — it is also the most criticized. Of course, no modern malware protection solution relies solely on malware signatures. Modern endpoint protection platforms will also include one or more of the following technologies:

• Application control limits the applications and processes that may execute on an endpoint. The goal is to apply a "default deny" enforcement model, whereby everything that is not known or trusted is not executed.
• Isolation or containment solutions allow installed endpoint applications to process potentially malicious files (such as web pages or downloaded documents) safely by isolating the processing of those files from the rest of the system.
• Behavior analysis provides rule-based monitoring where applications and processes are observed for particular indicators of intrusions that may be blocked or detected.
• Endpoint detection and response (EDR) technologies monitor endpoint activities and aid in the detection, containment, investigation and remediation of malicious behavior.
• Exploit technique mitigation prevents software exploits by enforcing in-memory protection. It guards against memory overflow attacks and against other attack methods that take advantage of software vulnerabilities.

By themselves, none of these technologies are a panacea to the intricacies of malware intrusion. Some technologies carry their own weaknesses. Security and risk management leaders should assess new malware protection solutions by discerning what distinguishes these technologies and how the various solutions can combine to form a more formidable malware prevention plan.

Analysis

Include Signature Technology in a Layered Protection and Detection Strategy

The majority of anti-malware solutions, such as EPPs, secure web gateways (SWGs), secure email gateways and unified threat management (UTM) solutions, include some form of signature detection — a fundamental piece of endpoint protection. A purely signature-based detection method has low success rates against sophisticated malware because, by its nature, it can only match to known malware and minor variants. Signature detection is easy to evade and signatures may take a while to develop. They require every endpoint to update frequently or to use cloud-based signature look-ups. For these reasons, it is uncommon to find EPPs that solely rely on signatures.

Most solutions use the cloud to look up the latest reputation information for a previously unseen file; however, the cloud is not available to systems that aren't connected to the internet but are nonetheless vulnerable to malware.

Signature-based detection is strong at blocking common attacks without using more resource-intensive or end-user-impacting technology, but some security vendors incorrectly frame this method of detection as an indicator of outdated technology. Despite some marketing claims to the contrary, signatures and heuristics do have advantages:

• Proactive protection against known malware. Scanning a file prior to execution prevents infection, assuming a signature exists for that threat. There is no need to utilize more resource-intensive inspection techniques if a file is known to be bad.
• Very low false-positive rates (FPRs). False positives do occur, especially with more aggressive heuristics engines, but most solutions have a very low FPR. Having a low FPR is critical for EPP solutions that are expected to protect endpoints autonomously. Almost every traditional vendor has at one time incorrectly convicted critical Windows files as malicious, rendering operating systems unusable.
• Prevents false positives in other, more aggressive techniques. Signatures can be used to help mitigate false positives in more aggressive detection techniques. When used as a method to "protect" known good files instead of purely to detect known bad, signature-based detection is a strong addition to a solution's technology stack.

Use Machine Learning to Reduce the Reliance on the Distribution of Signature Updates

The technology community in general is thrilled by the potential of machine learning, and machine learning has the potential to play an even greater part in the malware prevention space than it does today. Vendors use supervised machine learning engines to process large numbers of malicious files and large numbers of prevalent but known good. The resulting algorithm can be run locally on the endpoint device or in the cloud, and it can test a file for similarities to good or malicious files.

The advantages of this form of detection include:

• No malicious code is run. The detection is usually made in the pre-execution phase, before running code.
• No signatures are used when run on the endpoint. A mathematical model is used instead of the traditional signature database, removing the dependence on large disk and memory footprint along with the struggles associated with updating endpoint devices.
• New malware can be detected by the same model. Predictive models can use the statistical scoring to detect malware that has not been analyzed before.
• No internet connection is required. All scanning is local, and no cloud-based look-ups are required.

However, security and risk management leaders should also recognize the limitations and current weaknesses of machine learning as a stand-alone anti-malware resource.

The use of packer and encryption technologies limits the inspection model's coverage of the actual malware. Solutions running a purely predictive machine learning model on the endpoints suffer the risk that malware authors will: (1) study the detection behavior of the model on the endpoint, (2) adapt their malware code, and (3) attempt to evade detection.

Solutions should be able to avoid false positives, but it is inevitable that there will be files that are very close to the good and the bad model, resulting in both false positives and false negatives. EPP solutions solely relying on machine-learning-based detection can carry a high false-positive rate. EPP solutions generally combat false positives by adding other techniques, such as whitelisting known good files or cloud lookups for files that are too close to call, or by using signature-based whitelisting. With mathematical models that are infrequently updated, organizations may find themselves building an extremely long and hard-to-manage whitelist.

Recommendations

• Ignore biased claims by endpoint security vendors that signatures are useless.
• Update to the latest version of the incumbent EPP, as newer releases are less dependent on signatures and supplemented by additional protection techniques.
• Ensure the vendor provides a solid workflow to manage false positives and false negatives — be wary of solutions relying on a manual whitelist and blacklist capability.

Improve Visibility With EDR or EPP Tools That Focus on Applications and Processes

Security analysts cannot truly begin to harden systems and infrastructure without a solid understanding of what is running in an environment. EDR and EPP tools that report on applications and processes will provide data points that can be used to strategize a plan to reduce the attack surface.

Application Control/Whitelisting

Application control and application whitelisting apply a default deny enforcement model, where an application or process that is not explicitly whitelisted is deemed to be untrusted. Untrusted processes can be blocked outright or, with solutions that provide for dynamic decision making, can run with extra protection or scrutiny.

As a malware protection technology, application control has various strengths:

• Provides strong default deny prevention. If tight policies are used, application control provides strong protection against malware, especially when used in concert with technology that prevents legitimate processes from acting maliciously.
• Incurs low machine overhead. Application control solutions do not have a significant impact on endpoint resources.
• Offers broad platform support. Application control can be used to keep unsupported and/or unpatched systems secure. Legacy systems that still run on Windows 2000 or Windows XP only, for example, can be locked down by using a restrictive application control policy, typically in combination with some form of memory protection.
• Requires no signature files/updates. Application control is independent of malware signature files that require frequent updates. However, more advanced use, such as relying on file reputation in a more dynamic environment, requires access to the latest file reputation databases, typically over the internet.
• Applies to all potentially unwanted programs. Application control catches categories of applications that are not technically malware but might compromise security. Such categories include consumer remote access control applications, and file sync and share agents.

There are several considerations that security and risk management leaders must take into account when exploring application control for wide endpoint deployment. There are notable impacts on users and operations.

Application control can be very successful for fixed-function devices such as servers, where their applications and workloads are predictable. Users with well-defined work styles (for example, call center employees) are also ideal candidates for a successful deployment. For other user types, such as mobile workers or developers, the default deny approach may not provide an acceptable experience, unless workflow procedures can minimize approval delays for unknown, untrusted software.

In terms of operations, managing exceptions introduced from untrusted sources can incur substantial overhead. Organizations should plan for such overhead and provide administrators with the proper tooling. Such tooling will allow administrators to streamline the exception management process and to make the right decisions in the least amount of time. Allowing trusted sources of change minimizes the number of exceptions necessary.

Managing fine-grained application control policies in a dynamic endpoint environment is operationally complex. Leading solutions solve this problem by allowing more lenient policies: Trusted publishers, locations, installers and users may be allowed to install new software, automatically updating the application control policy. However, lenient policies may compromise security.

The strength of application control, as a protection technology against malware, greatly depends on the policy and the additional technology deployed on the endpoint. Malware authors have been able to release digitally signed malware using stolen certificates, exploit legitimate applications in memory and launch fileless malware, thus lowering the effectiveness of application control against sophisticated attackers.

Security and risk management leaders should carefully consider vendor claims around application control features. Simply blacklisting executables by name or file path is not considered a strong application control capability.

Application Isolation

Application containment solutions, also known as isolation solutions, implement malware protection using a paradigm best expressed as: Run risky processes and content, but isolate them from the rest of the system.

Security and risk management leaders should consider several strengths of application isolation, beginning with the provision of unrestricted user access. Malware containment does not block users from accessing sites or from downloading and processing potentially harmful content. In the most extreme form of application containment, users, should they choose to do so, may run malware in the isolated environment.

Some solutions discard the isolated environment and reset it to a clean state at launch or at regular intervals. Others do so when malicious behavior is detected in the isolated environment.

Isolation is valuable as a safeguard against a malware author's evasion techniques. The actual suspicious code runs on the endpoint, but in a contained environment. Even though the code runs, its ability to cause damage is limited by the sandbox. Organizations interested in deploying application containment solutions must be aware of the following cautions:

• User impact. By design, containment solutions limit interaction between isolated and nonisolated environments, which may impact the user experience.
• Operational impact. Administrators must manage trusted sites, applications, file locations and policies for moving files between zones of different trust levels.
• Lack of application support. The isolated environment may not support all preferred applications and versions.
• Hardware support. Some solutions depend on specific CPUs and chipsets, and the RAM requirements for a successful isolation deployment can be larger than the amount of memory found in typical corporate endpoints.
• Large differences in implementation. Solutions differ greatly in terms of policy control options, technologies used to enforce isolation, support for multiple zones, supported applications, management and reporting, and malware behavior analysis in the sandbox.
• Limited protection. Applications that run outside of the contained environment are not protected by the containment solution. Some vendors have started to extend their solutions by offering EDR technologies both inside and outside of the contained environment.

Recommendations

• Prepare for increased help desk calls, and put a well-tested and well-documented exception workflow in place, as additional administrative overhead is inevitable with a default deny implementation.
• Enforce default deny only for a subset of devices that have predictable workloads. For other types of users who have a less rigid set of requirements, like developers, use the client in monitoring mode to identify suspicious-looking behavior.
• Verify the hardware requirements can be met with your devices, and that critical applications are fully supported.
• Plan to deploy isolation technology to the group of users that are most at risk, rather than attempting to deploy for every single user.

Reduce the Attack Surface With Technologies That Look for Signs of a Malicious Outcome

While there are a steady stream of new vulnerabilities and attack vectors, the outcome is almost always the same. Consider the case of ransomware, where the goal is to encrypt the data — if technologies can detect the behavioral intent behind malware, the method of compromise is less important. That said, mitigating known vulnerabilities should be near the top of all organizations' priority lists.

Behavioral Analysis

Behavioral analysis within endpoint protection has several strengths, even when used as an isolated technology. Such analysis can provide runtime protection against attack activity. The solutions not only provide point-in-time detection, but also monitor the behavior of all, or at least all suspicious, processes over time to generate a greater understand of the context of the behavior.

For example, an Outlook.exe process spawning a Word.exe process is typical behavior for an information worker that receives documents by email. However, when the Word.exe process begins to connect to the internet, or to spawn other processes, the behavior becomes more and more suspicious.

EPP solutions using behavior analysis can also detect and block previously unknown malware without the need for resource-intensive scanning or inspection. This detection is not dependent on the malware code, but rather on the behavior, which means that vendors with a focus on this type of detection do not require any signature databases or file scanning. Behavioral analysis can detect multiple stages of the kill chain, such as droppers, network-borne attacks and some exploit techniques.

Some cautions are associated with deploying behavior analysis as a malware protection technology:

• Potentially high FPR. There is a fine line between malicious and normal behavior, so any behavior-based blocking technology incurs a risk of false positives. What appears to be malicious behavior is not always malicious. Kernel hooks and OS API calls that seem malicious may be legitimate.
• Detection instead of prevention. Sophisticated malware that does not trigger clear malicious-behavior-blocking rules will, at best, be detected after it runs, instead of being prevented before execution.
• Requires tuning, expertise and updates. Behavior-based malware protection requires organizations to carefully select rules, specify actions to take after detection, and whitelist trusted applications or digital certificates.
• May impact users. Because behavior analysis continuously monitors all activity on the endpoint, it may incur a performance penalty to the endpoint device.

Exploit Technique Mitigation

Exploit technique mitigation aims to stop malicious code from running in memory and, thus, make it more difficult for attackers to exploit software vulnerabilities. It does so by protecting the memory allocated to a process or application. It does not necessarily block the attacker from putting the malicious code into memory; it can also use techniques to prevent the code from being executed. This technology enforces security mechanisms already supported by the operating system, and adds capabilities beyond basic protection.

Security and risk management leaders can expect several benefits for organizations, including low management overhead, as the focus is on a small number of exploit techniques and does not rely on signatures or updates. Solutions generally incur limited performance overhead and operate transparently to the user. Microsoft provides a free Enhanced Mitigation Experience Toolkit (EMET) for free. It is officially supported by Microsoft until mid-2018, can be managed through Group Policy and makes for a good baseline of exploit mitigations.

For more details and recommendations on exploit mitigation, see "Get Ready for 'Fileless' Malware Attacks."

Recommendations

• Use third-party effectiveness tests to verify vendor claims. Vendor-sponsored or -commissioned comparisons can be useful data points but should not be given the same weight as impartial tests.
• Ensure that incident response tools are adequate, as behavioral analysis is largely a detect-after-execution technology.

Evidence

This research is based on 1,505 client and vendor inquiries on endpoint security across Gartner for IT Leaders and Gartner for Technical Professionals analysts since January 2016.