Use Central Log Management for Security Event Monitoring Use Cases

Published: 12 October 2016 ID: G00299625

Analyst(s):

 Toby Bussa, Kelly M. Kavanagh

Summary

Security and risk management leaders seek to improve threat monitoring and detection capabilities, regardless if they are starting from scratch or already have capabilities in place. The use of central log management tools can address these cases, in addition to traditional compliance use cases.

Overview

Key Challenges

• Fragmented or incomplete log collection hampers security monitoring and incident response. Examination and analysis of log data from multiple sources is critical in quickly determining whether a suspected security incident has occurred (and is mandated by many compliance regimes).
• Midsize organizations often lack the staff and training to support the deployment and operation of a security information and event management (SIEM) product to support security log management and investigation activities.
• SIEM tool licensing approaches are causing friction and increasing frustration for organizations, as they must tailor log and event collection to maximize existing license capacity and budgetary constraints.
• Use cases beyond foundational security capabilities are driving increasing interest and use of central log management (CLM) tools, but many clients are not aware of these new use cases.

Recommendations

Security and risk management leaders responsible for security monitoring and operations should:

• Use a CLM tool to address security monitoring and compliance use cases where there are insufficient resources or budget for a SIEM or for managed security services.
• For midsize organizations, look to use existing IT and network operations log management tools to collect and manage security event logs.
• Consider a multitier approach using a CLM tool when planning a SIEM deployment to avoid overutilization, and overlicensing, from the start.
• Use a CLM tool to better manage your existing SIEM tool investment if your organization has an existing SIEM solution that cannot scale its collection and analysis capabilities due to budget constraints.

Introduction

The use of CLM — the collection, aggregation and retention of event data from security technologies and other related IT log sources — to support security use cases (i.e., threat detection and compliance) is not new. However, it fell out of favor as increasingly capable SIEM products offered real-time correlation, alerting and reporting capabilities (especially for compliance requirements), in addition to log collection and management. As midsize organizations and resource-constrained enterprises have discovered, SIEMs come with budget and expertise requirements that present challenges to successful deployment and ongoing operations. At the other end of the spectrum, enterprises that have started their SIEM journey usually end up in one of two places: underinvested in their initial implementation and having to find budget to increase capacity to meet their use cases, or overlicensed and being stuck paying higher maintenance costs to the SIEM vendor for years for that unused capacity.

Given these scenarios, centralized log management for security event monitoring is having a renaissance. Several factors have raised the visibility of CLM as a stand-alone capability across both enterprises and midsize organizations:

• Threat detection and response. Organizations are increasingly concerned with detecting and responding to threats, but lack the resources (budgets, dedicated security teams) that can take advantage of resource-intensive SIEM tools, or afford a managed security service provider (MSSP) or managed detection and response (MDR) provider.
• Compliance. Compliance mandates, such as PCI DSS, HIPAA, FISMA and Sarbanes-Oxley Act, have not disappeared. Other regions may see an increasing need for security log collection and monitoring in the future. 1 Many organizations are also finally acknowledging the need to address these regulations, whether or not they were ignored in the past, or entry into new lines of business, activities and markets is driving adoption.
• New log collection tools and services. There are new entrants offering log collection and analysis for IT operations, typically deployed on customer premises, but increasingly delivered "as a service" from public cloud service providers leveraging open-source big data tools. In many organizations, these tools are also being used to collect security events and logs, in addition to operational logs from servers, network devices and applications.
• SIEM costs. Licensing models for many SIEM tools often dissuade customers from collecting all relevant security events and logs. SIEM tools that are licensed according to a volumetric (for example, the number of events or amount of data collected per some time period [seconds, minutes or days]) result in greater costs as data volumes increase. As organizations with SIEM solutions identify additional use cases, this may require adding new log sources, which may require the organization to spend more money to add additional license capacity. This is creating friction for many organizations that may not have sufficient budgets to expand their SIEM tool licenses, leaving them to choose between deprioritizing existing use cases, forgoing adding new use cases or decreasing the scope of monitoring. All of that equates to increasing the chances of not detecting a breach and/or affecting the ability to respond to one.
• Managed service costs. MSSPs have traditionally focused their monitoring services on network-based security controls. These threat-facing controls offered a combination of factors that made use of an external service provider desirable: a limited number of devices, generating a large volume of alerts with a high percentage of false-positive indications. MSSPs could monitor those alerts and, based on their expertise and knowledge of the threat environment, assess them at lower costs than a customer could do. As customers seek to include more user context, servers and applications in the scope of security monitoring, the benefits of using an external provider may not be as compelling. The events involve activities of systems and users where an external provider has little insight, and the number and different types of event sources increase the costs to deliver services. This compounds some existing client perceptions that MSSPs provide alerts with little to no value due to this lack of adequate knowledge about the customer's environment.
• Open-source log management . Open-source tools, like Elasticsearch, typically combined with Logstash and Kibana to form the ELK Stack; 2 big data platforms like Hadoop; and NoSQL databases, are gaining visibility, acceptance and use by many forward-leaning organizations as low-cost alternatives to purchasing commercial tools. Increasing adoption by organizations is leading to better documentation and community support, in addition to expanding pay-for-support options provided by the vendors maintaining the tools, which reduces the cost and risk of implementing and maintaining these tools.

In conversations with Gartner clients, a variety of use cases for CLM have emerged. Two use cases are most frequently mentioned: improving foundational security capabilities in the absence of other means, and augmenting new or existing SIEM deployments or service engagements. Both approaches seek to leverage CLM, but from different starting points (see Figure 1). The pool of tools and services tends to overlap across the use cases, indicating that due diligence in selecting the appropriate approach and tools is important.

Figure 1. How Organizations Get to CLM

Source: Gartner (October 2016)

Analysis

Improve Basic Threat Detection and Incident Response Capabilities

Many organizations, especially midsize and small enterprises, have deficiencies in their threat detection and incident response capabilities. The decentralized approach to log management in their IT environments makes detecting threats nearly impossible. Gartner's adaptive security architecture (see "Top 10 Strategic Technology Trends for 2016: Adaptive Security Architecture" ) emphasizes the use detection and response capabilities, in addition to the use of preventative security controls. The best case an organization can hope for is when a system administrator, who is likely intimately knowledgeable about the events generated by his or her systems, notices anomalous activities and investigates. Once a suspected incident is identified then the harder part occurs — trying to validate the incident, and determine the scope and severity across an organization. Without CLM, incident response activities associated with reviewing events in system logs becomes a multiweek effort, as logs are manually reviewed by administrators and analysts, and manual correlation of events is attempted. During this time, attackers may continue to operate inside an organization, or may have already stolen critical information and are long gone.

Increasingly, security and risk management leaders who have not invested in security event monitoring tools and capabilities are seeking the minimum floor to have a basic capability in place, especially when they understand the benefits that can be achieved (e.g., when a potential security incident is being investigated, demonstrating basic security controls to third parties or auditors).

Recommendations and Cautions

• Before looking at technology, determine all the event sources you plan to consolidate into a CLM tool. These event sources should be prioritized into at least two buckets — mandatory and secondary. Mandatory sources should include security controls (firewalls, unified threat management [UTM], intrusion detection, endpoint protection), network security devices (VPNs, core routers and switches), identity and access tools, user directories (Active Directory), and critical platforms and applications (externally facing web applications, Windows domain controllers, business-critical internal apps). Secondary sources may include all other network devices (routers/switches), OS and platform logs, and less critical application logs.
• Use the buckets described above to understand the potential amount of storage required over the three implementation phases, and establish a plan to retain logs for at least one year (or 366 days). Some Gartner clients report that 90 days is commonly retained, but there may be specific log sources that require longer retention periods due to compliance and regulatory mandates (such as PCI DSS, which requires one year). Additionally, industry reports on time to detect a breach indicate that 90 days is not sufficient if the logs are needed for investigation purposes. 3One of the benefits of using a log management tool is that they all generally offer log compression and warm archiving, such that logs can be stored much more efficiently in a dedicated tool, compared to how they are stored on the hosts or devices where they are generated.
• Determine where most security logs are being generated, including those related to wider IT sources — on-premises or cloud (public and private). Increasingly, as organizations adopt and move workload to cloud environments, it may be preferable to use a cloud-based log management service rather than deploy an on-premises solution. Consideration has to be made for where the bulk of the data collection will occur in order to minimize the associated costs with sending data into or out of a cloud environment. For example, Amazon Web Services (AWS) does not charge for data sent in, but will charge for data sent out of the environment.
• Consider commercial log management solutions and services before looking at open-source solutions depending on the use cases. Commercial log management solutions are available from a variety of vendors, either as stand-alone products or as the logging tier of commercial SIEM tools. Commercial tools will be more expensive than open source, but they are typically quicker to install and configure, offer flexible deployment options (either hardware, virtual appliances or SaaS), come with event parsers for most common products and platforms, have reporting capabilities and default report templates for compliance purposes, and will have dedicated support options if issues arise.
• Organizations (typically midsize and smaller enterprises) with existing tools that collect logs across the IT environment should consider adding security events into these tools before pursuing a dedicated security log management tool. Many of these tools also support SIEM capabilities as an add-on, such as ManageEngine, Micro Focus, SolarWinds and Splunk.
• Consider open-source tools, like the Elastic Stack (also known as the ELK stack 2 ) when your organization has the resources and expertise to dedicate to implementing, customizing and administering these tools. Security teams that have available staff who are looking for development opportunities are strong candidates to assign to this type of activity. Interested security staff will view the opportunity as a skills and career development effort, and the security team will also benefit. Cloud-based options are also increasingly available, either directly from Elastic or through third parties using Elasticsearch and other open-source tools with their own UI and custom capabilities.
• Log management tools that offer reporting capabilities may need to parse and normalize log data, so data source content and structure must be understood by the CLM tool or the person implementing it. If custom apps or specialized systems exist in an organization, then custom parsers may have to be written (requiring internal expertise) or provided by the vendor at additional cost and time. If normalization is required, seek out vendors that can make this process as simple as possible (for example, automatic detection of formats or visual interfaces to aid in constructing a parser). Tools that collect raw machine logs and will be used for log retention, search and log forwarding capabilities may not require normalization activities.

Use Central Log Management to Maximize SIEM Tool Investments

An increasingly popular question from Gartner clients that are planning to or already have deployed a SIEM tool is how to maximize their existing investment. Many clients face the need to expand their security monitoring use cases, but lack the budgets to purchase additional licenses for their SIEM tools. Other clients investing in a SIEM tool raise concerns about how to achieve an optimal balance of purchasing enough capacity without overinvesting in license capacity that may not be used, thus leading to misallocation of existing budget and a future "tax" on that unused capacity. Clients in this position report frustration with their vendors due to being constrained by their licensing inflexibility.

A number of clients have reported successfully implementing a log collection and management tier leveraging commercial and open-source log tools, as well as big data platforms where organizations have already deployed those technologies and have internal expertise available to the security team. This is the proverbial security data lake (see "Three Architecture Styles for a Useful Data Lake" ) that many mature, lean-forward enterprises are implementing. Client interest has increased in the past year regarding industry acceptance and feasibility of this approach.

The goal is to gain better incident response capabilities by capturing all security-related event logs into a single log collection and management platform, with the ability to selectively forward the raw logs needed to achieve the use cases in SIEM tools. The approach does not replace the need for a SIEM, as it still provides the event normalization, real-time threat detection, workflow and incident response management, and reporting capabilities (see Figure 2 for a sample architecture).

Figure 2. Representative Log and Event Management Tier Use Case

Source: Gartner (October 2016)

Recommendations and Cautions

• Purchase a commercial solution if your organization, including the security team, does not have the internal capabilities and expertise with large data acquisition and management platforms, such as data lakes . Many commercial log and event management products can provide similar features as a data lake (for example, ingestion of raw machine data and horizontal scaling to grow as performance and storage needs increase).
• Ensure that there is a clear life cycle management plan in place for security event monitoring use cases to manage the flow of events and log data from the central log collection tier to the downstream SIEM tool (see "How to Develop and Maintain Security Monitoring Use Cases" ). Identify data that will not be sent to a SIEM tool initially, but may be useful to support investigation, and plan integrations that make that data available on demand.
• Organizations that do not yet have this capability and are budget-challenged should review open-source applications first before pursuing commercial tools. Commercial Hadoop vendors are moving into the security log and event management and analytics market. These developments should be monitored for future use opportunities by organizations already using these vendors. 4 However, many of these offerings are in development and new, so a cautious approach is warranted depending on the maturity of the offering.
• The addition of a central log collection and management tier needs to be considered as yet another platform to manage, either for the security team or IT. The addition of another technology requires evaluating the costs and risks to be weighed against the impact and benefits, appropriate resources allocated, and sufficient management support, or else it will be another failed security project.

Representative Vendors and Providers

• Open-Source Tools— Balabit (syslog-ng), Elastic, Graylog
• Commercial Tools — Balabit (syslog-ng Store Box), TIBCO LogLogic, Tripwire Log Center
• SIEM Log Tier Tools — see "Magic Quadrant for Security Information and Event Management" and "Critical Capabilities for Security Information and Event Management"
• Cloud-Based Services — Logentries, Loggly, Splunk Cloud, Sumo Logic
• On-Premises Big Data Platforms — Cloudera, Elastic, Hortonworks, MapR (for additional insights, see "Market Guide for Hadoop Distributions" )

Evidence

1 The EU Directive on Security of Network and Information Systems states: "Under the NIS Directive, identified operators of essential services will have to take appropriate security measures and to notify serious incidents to the relevant national authority."

2 See The Elastic Stack .

3 Mandiant's M-Trends 2016 report indicates the median time to detect a breach in 2015 was 142 days.

4 Hortonworks is providing support for Metron via the Apache incubation model .

Elastic has announced that it acquired Prelert , which offers security analytics capabilities.

Cloudera supports the Open Network Insight (ONI) project , which is applying big data analytics to threat detection use cases.

MapR offers functionality for security log analytics .

© 2016 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its affiliates. This publication may not be reproduced or distributed in any form without Gartner's prior written permission. If you are authorized to access this publication, your use of it is subject to the Usage Guidelines for Gartner Services posted on gartner.com. The information contained in this publication has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information and shall have no liability for errors, omissions or inadequacies in such information. This publication consists of the opinions of Gartner's research organization and should not be construed as statements of fact. The opinions expressed herein are subject to change without notice. Gartner provides information technology research and advisory services to a wide range of technology consumers, manufacturers and sellers, and may have client relationships with, and derive revenues from, companies discussed herein. Although Gartner research may include a discussion of related legal issues, Gartner does not provide legal advice or services and its research should not be construed or used as such. Gartner is a public company, and its shareholders may include firms and funds that have financial interests in entities covered in Gartner research. Gartner's Board of Directors may include senior managers of these firms or funds. Gartner research is produced independently by its research organization without input or influence from these firms, funds or their managers. For further information on the independence and integrity of Gartner research, see "Guiding Principles on Independence and Objectivity."