ManageEngine APM kembali muncul di Gartner sebagai Challenger.

Magic Quadrant for Application Performance Monitoring

28 October 2014 ID:G00262851

Analyst(s): Jonah Kowall, Will Cappelli

VIEW SUMMARY

Demand for APM products and services intensifies, with a focus on mobility and analytics, allowing IT operations, application support, nontechnical business users and users evolving toward DevOps to support continuous release. APM provides better insight into applications and business execution.

Market Definition/Description

Although the market definition remains unchanged from our 2013 research (see "Magic Quadrant for Application Performance Monitoring"), there have been changes in the definition of an application. Gartner defines an application as a software program or group of programs that interact with their environment via defined interfaces and which are designed to perform a specific range of functions. They may be end-user-facing, presenting a UI, or provide the interface between two applications themselves. Applications are not (normally) wholly independent, as they typically require an operating system or multiple operating system instances to manage their use of physical and logical computing, storage, and network resources within a data center, or provided by third parties.

Gartner defines APM as having five dimensions of functionality:

1. End-user experience monitoring (EUM) — The capture of data about how end-to-end latency, execution correctness and quality appear to the real user of the application. A secondary focus on application availability may be accomplished by synthetic transactions simulating the end user.
2. Application topology discovery and visualization — The discovery of the software and hardware infrastructure components involved in application execution, and the array of possible paths across which these components communicate to deliver the application.
3. User-defined transaction profiling — The tracing of user-grouped events, which comprise a transaction as they occur within the application as they interact with components discovered in the second dimension; this is generated in response to a user's request to the application.
4. Application component deep dive — The fine-grained monitoring of resources consumed and events occurring within the components discovered in application topology discovery and visualization dimension. This includes the server-side components of software being executed.
5. ITOA — The combination or usage of the following techniques:
   1. Complex operations event processing
   2. Statistical pattern discovery and recognition
   3. Unstructured text indexing, search and inference
   4. Topological analysis
   5. Multidimensional database search and analysis

These techniques are used to discover meaningful and actionable patterns in the typically large datasets generated by the first four dimensions of APM. Additionally, these datasets are increasingly being analyzed not only for operational information, but also for business and software analytics (see "Apply IT Operations Analytics to Broader Datasets for Greater Business Insight").

Magic Quadrant

Figure 1. Magic Quadrant for Application Performance Monitoring

Source: Gartner (October 2014)

Vendor Strengths and Cautions

AppDynamics

AppDynamics shipped AppDynamics Pro 3.8 in mid-2014, and continues to innovate and push APM technologies forward, growing over 150% in 2013 and evolving its offerings by releasing its mobile APM solution in version 3.8. The vendor completed integration of the acquisition of DBTuna monitoring into the single main core platform, and has added MongoDB support. Additionally, AppDynamics has integrated the acquired Nodetime's Node.js monitoring technology. End-user experience monitoring has been enhanced with new visualizations and depth in capture. AppDynamics uses self-tuning instrumentation to keep overhead to a minimum; additionally, the ability to reinstrument the application at runtime allows for capturing of essential data to report on performance and business transactions. AppDynamics supports Java, .NET, PHP and Node.js applications. The vendor also monitors server and operating system metrics with the machine agent. AppDynamics also released its mobile product, which supports native mobile applications on Android and iOS, inclusive of deep performance analysis and crash reporting. Transactions can be traced from the mobile device and application-side into the server-side components. AppDynamics also allows for the definition of business metrics data on the fly, enabling the collection of, monitoring and reporting on metrics and other software data without requiring code changes. This easy, autotuned instrumentation, along with an easy install process and intuitive UI, has allowed AppDynamics to grow considerably over the past several years. It has a self-service try-and-buy sales strategy, as well as traditional enterprise sales, which makes it appealing across small or midsize businesses (SMBs) up to large enterprises.

Strengths

• Acquisition and integration keeps the core platform unified and easy to deploy and expand — a key growth strategy.
• AppDynamics' leadership is soundly executing on a strategy to expand using its strength in application instrumentation and APM as the foundation of a broader IT operations management (ITOM) toolset.
• AppDynamics products are identical when deployed on-premises or SaaS, providing usability continuity. There is strong adoption for both models.

Cautions

• AppDynamics frequently makes feature announcements at beta release; the delta between announcement and general availability can be lengthy.
• While the vendor's strategy is broad, the current offering is focused on modern applications only.
• Packaged application support is limited; only modern applications are supported. Currently, there is no support for common applications, such as SAP ERP or those delivered via Citrix.

AppNeta

AppNeta delivers real EUM, application runtime architecture discovery and modeling, user-defined transaction profiling, most aspects of deep-dive application component monitoring, and ITOA via TraceView v.3.8. This technology supports critical Java and .NET applications, and also supports emerging languages, such as PHP, Python and Ruby. AppView Web v.7.8 provides synthetic transaction-based EUM and some elements of application component deep-dive monitoring. ITOA is delivered via FlowView v.7.8 and PathView v.7.8. AppView, FlowView, and PathView are available as SaaS or on-premises, and the network-based tools have seen enterprise adoption. AppNeta's SaaS-only TraceView has seen success targeting SMB users deploying applications on public cloud infrastructures, along with its support of less common languages (such as Python and Ruby). AppNeta has moved into enterprises recently and is being considered for more shortlists, as enterprises look for depth and ease of deployment when buying APM solutions.

Strengths

• AppNeta's technology smoothly integrates SaaS-based synthetic and real user experience monitoring, which allows for straightforward transitions between availability testing and performance monitoring.
• Common data models and data capture mechanisms enable the product portfolio to support network and application performance issues.
• AppNeta has demonstrated particular strength in the ability to monitor the performance of key NoSQL platforms such as HBase and MongoDB, establishing itself as an early favorite among enterprises seeking to push distributed file system technology into production.

Cautions

• To date, AppNeta lacks a mobile offering, and has yet to articulate a clear mobile strategy.
• While effectively collecting and giving access to performance data, AppNeta does not provide rich automated pattern discovery or other analytics-related functionality.
• Leadership changes and shifting corporate direction have undermined user confidence in the vendor's viability.

BMC Software

BMC Software delivers EUM, user-defined transaction profiling, application component deep-dive monitoring and ITOA through the TrueSight Operations Management Suite v.9.5. This new branding is a transition as BMC looks to rebrand and provide additional benefits to clients. Through this transition, the vendor is undergoing changes in leadership, strategy changes and product vision. These have yet to transpire, but the underpinning technologies in the APM offering still consist of many moving parts and complexity, which typically require services for implementation. The fragmentation is apparent when BMC requires application runtime architecture discovery and modeling functionality, primarily delivered via Atrium Discovery and Dependency Mapping v.10.0. BMC's focus is on large enterprise customers, with the SaaS capabilities mirroring more of a managed service. The vendor has a strong set of technologies and capabilities that, with the right usability and simplification improvements, may result in a broadly applicable APM solution.

Strengths

• BMC's TrueSight platform delivers highly granular, yet easily summarizable, real EUM across a broad range of application environments.
• The SaaS-based ability to monitor Akamai-enhanced Web-based applications that provide full visibility into how the Akamai infrastructure impacts application performance remains unique and in high demand.
• The BMC portfolio is very effective in its application of machine-learning-style algorithms to determine trends in application performance and to enhance root cause analysis.

Cautions

• Despite BMC's packaging of APM components as a single suite, the components remain distinct, and users find them difficult to integrate to obtain a coherent view of application performance.
• Uptake of the vendor's application component deep-dive monitoring functionality has been limited, based on Gartner's interactions and discussions with clients. Feedback indicates a lack of granularity as the primary reason.
• Since being taken private, BMC has not articulated a clear strategy and road map for APM, leading to user concerns about the vendor's long-term commitment to this market.

CA Technologies

CA Technologies has been undergoing significant changes, including the appointment of a new CEO, management reorganizations, and the creation of a new APM business unit and general manager position. The vendor offers CA Application Performance Management 9.6 for the bulk of the APM product line, complemented by CA APM Cloud Monitor 8.2 for SaaS synthetic transactions, CA Cross-Enterprise APM 9.6 for legacy support, CA Application Delivery Analysis 10.0 for network-based performance analysis and CA Executive Insight for Service Assurance 2.1 for dashboarding. With an incremental release since 2013, CA is focusing on large transformations to keep pace with the innovators in the industry. Its offerings are deep on mainframe, and support Java and .NET technologies. End-user experience monitoring is still basic in nature, normally not meeting demands of today's APM requirements. The new leadership understands that a larger product enhancement will be necessary, including offering a lightweight SaaS solution; hence, CA has removed its SaaS offering, which was based on legacy products. CA sells its products via its standard enterprise outside sales approach, making it more appealing to traditional enterprises.

Strengths

• CA’s APM technologies are well-understood, due to a significant installed base driven by the CA Introscope product being the underpinning of all modern APM technologies. CA's ability to support IBM mainframe, although complex, provides an end-to-end story for that small, but critical, installed base.
• Renewed focus and leadership change have resulted in a new strategy and increased visibility, which should help drive needed product evolution.
• CA has released a mobile APM product with a modern design that is competitive.

Cautions

• CA's early leadership in this market has subsided due to a lack of investment and focus; while plans are in motion to reverse the situation, there is no short-term fix.
• Although the vendor has a large, but dwindling, installed base, complexity in CA's offerings makes the tools challenging in the modern and complex Web applications being built today.
• CA delivers only one element of ITOA — statistical pattern discovery — which is provided through an OEM relationship with Prelert.

Compuware

Compuware's comprehensive offerings span multiple products. Most of Compuware's growth and customer buying focus revolved around Compuware dynaTrace 6 for full five-dimensional functionality for modern applications. Compuware Data Center Real-User Monitoring (DC RUM) v.12.2 is applied to network-based agentless monitoring primarily for non-Web based and legacy applications. APM as a service (APMaaS), formerly called Gomez, for synthetic SaaS-based EUM is revised monthly. APMaaS also has capabilities to offer dynaTrace as a service, and a stand-alone, free-of-charge mobile APM offering, but adoption of these offerings has been limited. Although these on-premises and SaaS products are integrated with PurePath Technology to tie the tools together and provide deep visibility, the products are not well-integrated from a UI perspective. Compuware has made strides to build dashboard views that incorporate cross-product data and visibility for troubleshooting across today's silos. The vendor plans to introduce Ruxit, an additional SaaS-delivered product. This offering is more simplistic to deploy, making it appealing to SMBs and those deploying on cloud environments. Ruxit will include more usable, advanced and well-integrated technology in a single product than do Compuware's current enterprise-focused offerings. dynaTrace's application life cycle support makes it well-suited for cross-team collaboration. The instrumentation and being able to adjust it at runtime, which is rare across APM technologies, allows for the definition and monitoring of business transactions, thus enabling increased business relevance to APM. Compuware's APM business unit execution has improved with sales skills, partner focus and the intention of selling off the mainframe business, allowing the vendor to become an APM specialist.

Strengths

• Compuware's comprehensive offerings address a diverse set of application types, spanning both legacy (IBM mainframe, Citrix, SAP and others) and modern (Web and mobile) applications. The SaaS-delivered APMaaS services measure global application availability.
• Compuware's dynaTrace technology is among the most favored solutions for sharing information between production teams and developers, providing the depth needed by developers. This is a good tie into the APM development life cycle, which helps improve the performance of software before it's deployed in production.
• Compuware provides extensive service offerings to help customers develop APM skills and best practices, including its unique APM Guardian Services, which offers on-site or remote staffing assistance as a managed service. This is often requested by Gartner clients, as applications that must be managed are increasingly complex and built with divergent technologies.

Cautions

• Portfolio fragmentation and complexity are common complaints from Compuware buyers. With the forthcoming introduction of another product line, this seems to be moving in the wrong direction.
• While the vendor has extensive analytics capabilities within its APM products, there is limited broader cross-product analytics strategy.
• Compuware lacks additional capabilities around infrastructure monitoring — specifically the need for greater network visibility, virtualization performance and storage performance monitoring. These data sources are increasingly critical for problem isolation, identification and remediation.

HP

HP's comprehensive ITOM platform includes extensive APM capabilities delivered via traditional enterprise software and SaaS. The on-premises products include HP Business Process Monitor v.9.24 for synthetic transactions, HP Real User Monitor v.9.24 for network-based EUM, HP Diagnostics v.9.23 for deep-dive and JavaScript-based EUM, HP TransactionVision v.9.24 for host-based network analysis, HP Service Health Analyzer v.9.24 and HP Operations Analytics v.2.1 for ITOA. HP's SaaS offering includes the enterprise hosting of HP Business Service Management (BSM) SaaS v.9.24. In December 2013, HP announced Pronq, a self-service SaaS platform providing the new channel needed by today's buyers. This offering is multitenant, including HP AppPulse v.2.1 for synthetic monitoring and deep dive, along with HP AppPulse Mobile v.1.0 for native mobile APM. HP's mobile offering supports both iOS and Android native applications via wrapper technology, focused on appealing to those who may not own source code for mobile applications. The vendor has also improved its ITOA offerings with a centralized data store and analysis based on its HAVEn technology. HP is shifting strategy to consolidate and normalize offerings built on this new analytics architecture.

Strengths

• HP's extensive portfolio and large installed base provide an upsell opportunity for the vendor to get APM solutions in place, and extend the use of SaaS in the installed base.
• Pronq provides a new avenue for HP to gain new customers, specifically those wishing to spend less money on software. This capability is ahead of other large ITOM vendors, but end-user adoption has yet to take hold.
• HP's offerings include extensive capabilities around ITOA for advanced users and use cases, along with simple embedded technology within the lightweight offerings.

Cautions

• Product complexity and disjointed UIs are issues across the portfolio, even within the APM-focused tools. Integrations exist between hosted SaaS and on-premises, but do not yet exist between Pronq-hosted and on-premises APM products.
• Most clients with HP solutions use synthetic transactions, with fewer customers using real-user monitoring via network packet analysis. This indicates stronger preference for HP's availability solutions, rather than its performance monitoring or diagnostics solutions often associated with APM.
• Management and account team churn has improved versus a year ago, but continues to be an issue, with some client references concerned about keeping strategic relationships healthy.

IBM

IBM's large APM portfolio includes several products and evolving delivery models, but must overcome challenges such as product complexity. IBM delivered a multitenant SaaS APM solution in the first quarter of 2014. This offering remains central to overall IBM initiatives, including the Bluemix PaaS cloud platform. Bluemix includes the IBM Bluemix Monitoring and Analytics service, which is SaaS and integrated with the PaaS. The vendor offers IBM Performance Management (SaaS), IBM SmartCloud Application Performance Management v.7.7.0.1 (on-premises), and analytics via IBM SmartCloud Analytics Log Analysis and IBM SmartCloud Analytics Predictive Insights for on-premises deployments. Mainframe visibility is delivered via the IBM Tivoli Omegamon XE v.5.1 family of products. Additional monitoring capability and integration layers are provided by IBM Tivoli Monitoring v.6.3 for on-premises deployments. IBM continues to address product and portfolio complexity in two ways: via a SaaS-delivered product, which removes much of the complexity for the user, and by reducing the number of components needed, thereby reducing complexity; however, more work is required. Some on-premises configuration is still done in multiple product UIs. Synthetic monitoring provided by IBM Tivoli Composite Application Manager (ITCAM) is dated and lacks synthetic monitoring SaaS capabilities. The vendor's offerings have improved with deep-dive code-level visibility, adding .NET and Ruby to Java support. IBM's mobile strategy continues to lag, even when corporate messaging is about mobile focus. Its try-and-buy model is available for SaaS products, and an outside sales model applied to enterprises is for on-premises products. Early signs indicate usage is changing among progressive IBM customers.

Strengths

• IBM's comprehensive and complex portfolio includes monitoring of just about every component in the application environment. The vendor has professional services and consulting to manage and maintain popular implementations.
• IBM is looking to expand outside its installed base by offering more open platforms, such as IBM Bluemix and IBM Service Engage.
• The vendor's leadership has increased the emphasis on the APM market and its importance in future environments and applications.

Cautions

• IBM continues to have one of the most complex portfolios in a market where simplicity is a clear driver and differentiator based on the growth of APM.
• The vendor needs to improve EUM, especially client-side visibility, which is a challenge; buyers find EUM a critical feature when selecting APM products.
• Portfolio fragmentation between SaaS and on-premises offerings still exists, but IBM is trying to move the APM products toward a SaaS-first strategy.

ManageEngine

ManageEngine's Applications Manager 11 supports all five dimensions of APM functionality as a single, integrated platform. The product not only handles APM use cases, but also monitors the health and availability of servers, virtualization and application instances. ManageEngine has made investments to better support .NET and Ruby, and has increased platform support for monitoring the health of VMware and Azure virtualization. SaaS functionality is delivered via Site24x7. The vendor has increased monitoring capabilities via the SaaS platform, and has strengthened the data integration between SaaS and on-premises deployments. Site24x7 was also enhanced with real-user monitoring capabilities and full APM functionality from the SaaS platform. ManageEngine offers the lowest-cost products, compared with the other vendors evaluated in this research, and has increased its market visibility with attention to the UI, as well as via online and direct marketing.

Strengths

• ManageEngine provides low-cost and easily deployed APM functionality across all five dimensions of APM, making it particularly attractive for SMBs.
• ManageEngine's Applications Manager is easy to integrate with the vendor's other monitoring and management products, allowing the user to put together a comprehensive, high-level view of the health of the IT stack.
• The vendor has built up a strong global presence, with its brand and acceptance being particularly strong in the Asia/Pacific region and Australia.

Cautions

• The lack of end-user monitoring functionality has limited the appeal of Applications Manager, as enterprises of all sizes increasingly regard synthetic-transaction-based technology as, at best, supplementary to end-user data.
• The SMB market has become extremely competitive and is a segment accustomed to a high rate of innovation. ManageEngine has historically been a comparatively noninnovative vendor, adopting features and functions only after they have been pioneered and proven by other vendors.
• ManageEngine provides little in the way of ITOA at a time when this dimension is becoming a central factor in buying decisions.

Microsoft

Microsoft System Center 2012 R2 provides management across multiple functional product areas; the monitoring offering includes Operations Manager (OpsMgr). This offering is used by IT operations teams that have a heavy Microsoft presence in their environments. While this product is focused on infrastructure monitoring, it also has APM functionality, which is now being leveraged within some standard management packs for monitoring Microsoft SharePoint and Microsoft Exchange. Microsoft, like other vendors, has made major changes to its APM strategy, as it realizes that APM is central to many parts of its overall strategy. In mid-2013, Microsoft moved the APM organization from within the System Center organization into the development tool organization. This created a new developer-centric approach to instrumentation, which facilitated the ability to get monitoring in earlier in the cycle, versus being a preproduction exercise as it is today. Microsoft launched a preview of Application Insights for Visual Studio Online, which provides a SaaS-only instrumentation capability, paired with the System Center Global Service Monitor (GSM) SaaS offering. This product allows for custom instrumentation with integration into Visual Studio, along with providing visibility into Java and .NET application code, and EUM for mobile and Web applications. Microsoft also introduced a preview of System Center Advisor, which collects and analyzes event log data in a SaaS manner by leveraging the collection technology within System Center OpsMgr. Microsoft's offering is available as part of a Software Assurance or Open License agreement subscription.

Strengths

• Like other vendors in this Magic Quadrant, Microsoft has developer reach, and is exploiting it effectively by pushing APM deep into the development organization for an application life cycle story.
• Innovation in APM via SaaS delivery allows the vendor to deliver features and functions much faster than previous offerings tied only to on-premises deployments of System Center.
• Microsoft's change in strategy has been large and visible to customers, providing them with early access to software and features. This transforms Microsoft from a traditional provider of solutions to an agile software company, pushing new features and functions via SaaS.

Cautions

• Microsoft's previously unified single offering System Center OpsMgr has evolved into a disconnected and disjointed solution, lacking integration from SaaS to on-premises. While the vendor has delivered a common agent powering both SaaS and on-premises technologies, major differences in product functionality and usability maintain a degree of uncertainty in customers' minds as Microsoft seeks to shift its strategy.
• Microsoft's APM solution within System Center is largely exception-based; hence, it doesn't meet the demands of today's APM buyers. The Application Insights product corrects this, but is offered only via SaaS.
• The vendor seems to be less focused on providing IT operations buyers with tools, and more focused on moving deeper into software analytics use cases, which drive developer buy-in.

New Relic

New Relic has transitioned from a single-product company to a multiproduct company during the past year. This SaaS-only offering continues to gain traction, delivering an easy-to-use product with fast implementations. The solution has been expanded to include New Relic APM, which focuses on code instrumentation of Java, .NET, Python, PHP, Ruby and Node.js. New Relic Browser provides deep EUM, along with browser-side JavaScript debugging, which is unique in the APM industry. New Relic Mobile has had two full product cycles since being launched in 2013, but still lacks crash reporting features, which the vendor is expected to deliver in the second half of 2014. New Relic Mobile is delivered from the same platform as the other products, and automatically detects when native mobile code makes HTTP requests to server-side application code, linking the mobile transaction and server-side transaction code. Better reporting, workflow and data integration is needed to tie together mobile and distributed APM technologies. New Relic Servers, a free offering, provides server and operating system monitoring for Linux, Windows and SmartOS. New Relic Platform provides a plug-in ecosystem for additional monitoring, but is less integrated into other modules. The most differentiated offering released this year is New Relic Insights, providing software usage and ad hoc analytics capabilities on top of the instrumented data. This platform is unique among shipping products due to its scale, sophisticated language and open-ended design. The vendor does most of its business via low-touch SaaS or inside selling, but that has changed considerably through 2014 with the build-out of a highly competent and well-run sales organization. With the majority of APM products traditionally fully housed on-premises, New Relic's SaaS-only delivery model excludes the vendor from many opportunities.

Strengths

• As a SaaS-only business, New Relic is a disrupter and innovator, delivering product before most APM players. Being first to market with mobile APM and software analytics is tied to a broader APM strategy.
• Ease of use and ease of implementation have been key differentiators that have led to widespread adoption and implementation.
• Marketing and positioning of a bigger picture story led to New Relic having a high degree of mind share and a unique business.

Cautions

• New Relic's SaaS delivery model and early focus on SMBs or departmental implementation had historically limited enterprise buyer adoption. The vendor has focused on expansion into Enterprise buyers, but this is a recent effort.
• New Relic's strategy has not placed infrastructure monitoring and IT operations buyers front and center, which could isolate those buyers as they expand into enterprises. The solution is well-adopted by developers and those operating in hybrid and public cloud environments.
• Although the capabilities across APM are broad and the products are integrated into a single UI, the features within the products are less integrated (for example, server monitoring, mobile and core APM have minimal amounts of data crossover).

Riverbed Technology

Riverbed Technology delivers the five dimensions of APM functionality via a suite of products delivered by the Riverbed Performance Management (RPM) business unit. End-user experience monitoring is made possible by SteelCentral AppResponse v.9.0 and SteelCentral Web Analyzer v.3.0; application runtime architecture discovery and modeling is primarily delivered by SteelCentral AppMapper v 2.0.1; user-defined transaction profiling is the task of SteelCentral Transaction Analyzer v.17.0; and application component deep-dive monitoring is made possible by SteelCentral AppInternals v.9.0 and SteelCentral NetSensor v.2.0, while ITOA is an integral element of all of the above-mentioned products and is brought together in the SteelCentral Dashboards v.2.3. Riverbed has released the first major versions of its core products in over 18 months, showing it is moving forward in the long integration of Opnet Technologies, which it acquired. AppInternals captures every transaction, and now includes EUM in the core offering. Current APM selling has been focused on network-centric technologies such as AppResponse, versus the software-based AppInternals offering; the lack of traction is primarily due to a sales and channel education gap. With management changes occurring recently, this may finally change. The technology offerings provide extensive capabilities, but implementations and comfort levels tend to focus on network-centric use cases versus software instrumentation.

Strengths

• Riverbed's SteelCentral AppResponse and SteelCentral NetSensor technologies deliver highly effective packet-capture-based performance monitoring that can support both APM and network performance monitoring (NPM) requirements. SteelCentral AppResponse is also one of the few platforms in the market equipped with algorithms capable of interpreting Independent Computing Architecture (ICA) and other hybrid virtual desktop (HVD)-oriented protocols.
• Riverbed's topology-based analytics provide particularly effective support for application performance problem root cause analysis, particularly when the root causes are network-related.
• Riverbed's market positioning has effectively grafted its powerful WAN optimization brand onto its APM portfolio, with the result that users will consider a Riverbed solution purely on the basis of their typically positive experiences with the vendor's WAN optimization technology.

Cautions

• Based on client inquiry and references, Riverbed's RPM portfolio can be difficult to implement and integrate. This is particularly true of its deep-dive component monitoring technology which, until the release in June 2014, had not been significantly enhanced since the Opnet acquisition in December 2012. Riverbed must accelerate the integration and simplify the portfolio, where appropriate.
• While the vendor has added some functionality for application development buyers, it needs to take additional steps to make its APM offerings attractive to the application development community — an increasingly important influencer in APM buying decisions. It does provide integrated development environment (IDE) integration, but more must be done from a workflow, UI and feature perspective.
• Riverbed's marketing efforts have yet to convince current and potential users that it is as focused on the application layer, which is the focus of AppInternals, as it is on the network layer. Riverbed's long-term commitment to non-network APM is questionable.

SmartBear

SmartBear is a developer-centric organization specializing in quality assurance (QA) and testing tools. The popular SmartBear SoapUI free and open-source functional testing tool has launched a full-featured APM product. The AlertSite synthetic monitoring SaaS solution was acquired by SmartBear in 2011, and this commoditized market has had pretty limited growth. In late 2013, SmartBear acquired Lucierna, which was based in Spain. SmartBear launched the AlertSite UXM SaaS-delivered platform in mid-2014, which includes synthetic monitoring, real-user monitoring and transaction tracing of Java, .NET and PHP applications. Additionally, SmartBear is still offering the on-premises Lucierna offering, which is a full-featured APM solution. AlertSite UXM supports Java, .NET and PHP; Lucierna supports Java, .NET, PHP and Android applications. SmartBear has to unify these platforms and build a cohesive strategy, which it is expected to deliver in 2015. The Lucierna and Alertsite UXM solutions have unique capture and decompilation capabilities, allowing for code viewing in context during diagnostics. Additionally, capture and compression of event data is differentiated in the Lucierna product, allowing for full transaction capture all the time.

Strengths

• Historical knowledge and participation in SaaS-delivered synthetic monitoring, along with a good-size installed base, provides market awareness of the AlertSite brand, but it's not associated with APM.
• SmartBear offers low-cost and open-source technologies, which appeal to developers and those seeking an open ecosystem. Further extension to the development toolsets will create cohesion between APM and other offerings.
• On-the-fly code visibility is accomplished by "hot decompilation" of code, and capture of all transactions all the time provides strength in deep dive, while keeping overhead low.

Cautions

• Marketing execution and clarity in messaging are issues with AlertSite; SmartBear has not sold enterprise software to date.
• AlertSite entered the APM market in mid-2014 — much later than other offerings in this Magic Quadrant— and has much catch-up work to do in order to have the technology to compete.
• Depth of analysis is not yet fully built out in AlertSite UXM, versus the on-premises Lucierna offering; therefore, parity is still elusive.

Vendors Added and Dropped

We review and adjust our inclusion criteria for Magic Quadrants and MarketScopes as markets change. As a result of these adjustments, the mix of vendors in any Magic Quadrant or MarketScope may change over time. A vendor's appearance in a Magic Quadrant or MarketScope one year and not the next does not necessarily indicate that we have changed our opinion of that vendor. It may be a reflection of a change in the market and, therefore, changed evaluation criteria, or of a change of focus by that vendor.

Added

• SmartBear — Added with the acquisition of Lucierna and the creation of the newly converged AlertSite UXM platform.

Dropped

• OpTier — Dropped due to the company ceasing operations, and the intellectual property (IP) being bought by SAP.
• Dell — Dropped due to removing SaaS product offerings. Dell is refocusing efforts on a new full-featured APM offering to be delivered as SaaS and on-premises.

Inclusion and Exclusion Criteria

Vendors were required to meet the following criteria to be considered for the 2014 APM Magic Quadrant. In comparison to 2013, we have adjusted numerical thresholds:

• The vendor's APM product must include all five dimensions of APM (EUM, application topology discovery and visualization, user-defined transaction profiling, application component deep dive, and ITOA). The deep-dive monitoring capabilities must include Java and .NET, but also may include one or more key application component types (e.g., database, application server). The solution must include user-defined transaction profiling, ITOA technologies applied to text and metrics collected by the other four dimensions.
• The APM product must provide compiled Java and .NET code instrumentation in a production environment.
• Customer references must be located in at least three of the following geographic locations: North America, South America, EMEA, the Asia/Pacific region and/or Japan.
• The vendor should have at least 50 customers that use its APM product actively in a production environment.
• The vendor references must confirm that they are monitoring at least 200 production application server instances in a production environment.
• A full five-dimensional APM offering, including real-user monitoring, must be available via a SaaS delivery model. This offering must be delivered directly from the vendor.
• The product must be shipping to end-user clients for production deployment and designated with general availability by 15 July 2014.
• Total revenue (including new licenses, updates, maintenance, subscriptions, SaaS, hosting and technical support) must have exceeded $5 million in 2013.

In addition to these criteria, we will be evaluating the vendor's ability to cross multiple buying centers, as well as its ability to target specific verticals, as validated by reference customers.

Evaluation Criteria

While a vendor may meet the inclusion criteria for the APM Magic Quadrant, placement within the finalized Magic Quadrant will depend on its scoring in a number of categories. Ratings in these categories will be used to determine final placement within the 2014 APM Magic Quadrant. The 2014 evaluation criteria are based on Ability to Execute and Completeness of Vision.

Ability to Execute

Product/Service: Gartner evaluates the capabilities, quality, usability, integration and feature set of the solution, including the following functions:

• Day-to-day maintenance of the product
• Ease and management of deploying new APM
• Ease of use and richness of functions within the product
• Product deployment options and usability
• Integration of an overall APM-related portfolio or unified APM offering

Overall Viability (Business Unit, Financial, Strategy and Organization): We consider the vendor's company size, market share and financial performance (such as revenue growth and profitability). We also consider the leadership in the company in terms of number of people, what employees think of the leadership and the ability to drive the company forward. We also investigate any investments and ownership, and any other data related to the health of the corporate entity. Our analysis reflects the vendor's capability to ensure the continued vitality of its APM offering.

Sales Execution/Pricing: We evaluate the vendor's capability to provide global sales support that aligns with its marketing messages; its market presence in terms of installed base, new customers and partnerships; and flexibility and pricing within licensing model options, including packaging that is specific to solution portability.

Market Responsiveness and Track Record: We evaluate the execution in delivering and upgrading products consistently, in a timely fashion, and meeting road map timelines. We also evaluate the vendor's agility in terms of meeting new market demands, and how well the vendor receives customer feedback and how quickly it builds it into the product.

Marketing Execution: This is a measure of brand and mind share through client, reference and channel partner feedback. We evaluate the degree to which customers and partners have positive identification with the product, and whether the vendor has credibility in this market.

Customer Experience: We evaluate the vendor's reputation in the market, based on customers' feedback regarding their experiences working with the vendor, whether they were glad they chose the vendor's product and whether they planned to continue working with the vendor. Additionally, we look at the various ways in which the vendor can be engaged, including social media, message boards and other support avenues.

Table 1. Ability to Execute Evaluation Criteria

Evaluation Criteria	Weighting
Product or Service	High
Overall Viability	High
Sales Execution/Pricing	Medium
Market Responsiveness/Record	Medium
Marketing Execution	High
Customer Experience	Medium
Operations	Not Rated

Source: Gartner (October 2014)

Completeness of Vision

Market Understanding: This criterion evaluates vendor capabilities against future market requirements. The market requirements map to the market overview discussion and look for the following functionality:

• EUM, including real and synthetic availability testing
• Runtime application architecture discovery
• User-defined transaction profiling
• Application component deep dive
• ITOA for problem isolation and resolution
• ITOA to answer questions about software or business execution
• Ability to address the mobile APM market

Marketing Strategy: We evaluate the vendor's capability to deliver a clear and differentiated message that maps to current and future market demands, and, most importantly, the vendor's commitment to the APM market through its website, advertising programs, social media, collaborative message boards, tradeshows, training and positioning statements.

Sales Strategy: We evaluate the vendor's approach to selling APM to multiple buying centers. We also evaluate the vendor's ability to sell in the appropriate distribution channels, including channel sales, inside sales and outside sales.

Offering (Product) Strategy: We evaluate product scalability, usability, functionality and delivery model innovation. We also evaluate the innovation related to the delivery of product and services.

Business Model: This is our evaluation of whether the vendor continuously manages a well-balanced business case that demonstrates appropriate funding and alignment of staffing resources to succeed in this market. Delivery methods will also be evaluated as business model decisions, including the strength and coherence of on-premises and SaaS solutions.

Vertical/Industry Strategy: We evaluate the targeted approaches in marketing and selling into specific vertical industries. Commonly, APM solutions are bought and targeted toward the financial services, healthcare, retail, manufacturing, media, education, government and technology verticals.

Innovation: This criterion includes product leadership and the ability to deliver APM features and functions that distinguish the vendor from its competitors. These include unique approaches to application instrumentation, mobile visibility and catering toward the increased demands of continuous release. Specific considerations include resources available for R&D, and the innovation process.

Geographic Strategy: This is our evaluation of the vendor's ability to meet the sales and support requirements of IT organizations worldwide. In this way, we assess the vendor's strategy to penetrate emerging markets.

Table 2. Completeness of Vision Evaluation Criteria

Evaluation Criteria	Weighting
Market Understanding	High
Marketing Strategy	Medium
Sales Strategy	Medium
Offering (Product) Strategy	High
Business Model	High
Vertical/Industry Strategy	Low
Innovation	High
Geographic Strategy	Low

Source: Gartner (October 2014)

Quadrant Descriptions

Leaders

Eight aspects characterize vendors that appear in the Leaders quadrant: (1) competitive offerings related to all five dimensions of APM, and best-of-breed functionality in two or more of the dimensions; (2) credibility in the monitoring of application domains assembled from heterogeneous sources; (3) deep integration across most of the dimensions; (4) the ability to deliver and support APM on a global basis; (5) a consistent track record of innovation; (6) a vision that places APM at the heart of operations, application support, development, and organizations performing agile release and philosophies (such as DevOps); (7) high levels of automation that ease the burden of technology deployment and maintenance; and (8) demonstrated APMaaS capabilities across all functionality dimensions.

Challengers

Five aspects characterize vendors appearing in the Challengers quadrant: (1) they have competitive offerings in all five dimensions of APM, but some of the offerings are restricted either in terms of functional depth in one or more of the dimensions, or with regard to the environments to which their technologies are applied, that keeps them from being considered by some large enterprises; (2) while staying abreast of market trends, Challengers rarely get out in front of them; (3) Challengers typically have a strong global support and services infrastructure; (4) they have a well-regarded brand, although that regard is not generated by APM; and (5) they recognize the importance of APM, if not its centrality to their overall software product portfolios.

Visionaries

Four aspects characterize vendors appearing in the Visionaries quadrant: (1) they have competitive offerings in all five dimensions of APM, but often support that broad reach with a complex, difficult-to-implement and weakly integrated portfolio of products; (2) they have demonstrated the ability to grow rapidly and maintain the position of their brand among demanding submarkets, such as financial services and telecommunications; (3) they have consistently innovated in at least two of APM's functional domains; and (4) have aggressively developed and delivered APMaaS offerings.

Niche Players

One of three aspects characterizes vendors appearing in the Niche Players quadrant: (1) they are explicitly focused on a limited number of application types, whether those types are defined in terms of vendor provenance or domain; or (2) they cannot demonstrate equal depth across all five dimensions of APM functionality; or (3) they keep abreast of market trends, but have not demonstrated innovation with regard to multiple APM functionalities.

Context

The effective deployment of APM technologies and services depends on the acceptance of three fundamental premises regarding IT operations that run counter to the grain of many traditional practices.

First, monitoring (and, indeed, all aspects of ITOM) must become primarily application-centric, and, within that application centricity, primary place must be given to ensuring a high quality of end-user or customer experience. That is not to say that monitoring infrastructure components (e.g., servers, storage, networks and the virtual fabric) becomes unimportant. Rather, processes must be reshaped to ensure that the data collected during the course of monitoring such components must always be analyzed for what it reveals about how the behavior of those components impacts the end-to-end performance of the applications that exercise those components. In other words, applications must become the windows through which the infrastructure and its performance are observed.

Second, monitoring's historical focus on past events signaling faults that must retroactively be analyzed and remedied has to give way to a focus on continuous performance monitoring, with the goal of anticipating problems before they make a palpable impact on the end user and the customer.

Third, the walls traditionally segregating application development from IT operations need to be knocked down, or at least perforated. This movement, known as DevOps, provides the foundational cultural changes needed to accomplish this change in operations, especially with regard to application understanding. Additionally, the data gathered and analyzed by APM technologies can greatly assist the tasks performed by application developers and testers; that community's expertise will be required to effectively interpret APM data.

Even if these premises are accepted in full, it is still not advisable to try to deploy all five dimensions of APM across the entire application portfolio. Given the cost and complexity of APM technologies, an enterprise should first focus on, at most, the top 10 most business-critical applications for those technologies, concentrating initially on EUM. Once that is mastered, other dimensions may be taken up. Finally, once a state of familiarity with APM has been obtained, the enterprise can contemplate extending EUM and application performance analytics to the top 40 or 50 most business-critical applications. It is unlikely that further expansion, either in terms of functional dimension or portfolio coverage, will deliver a positive ROI.

Based on discussions during the research process with 91 reference companies that responded to our survey, we have uncovered the following analysis for this Magic Quadrant:

• The cost of the average APM implementation is $828,000, with the largest being $5 million.
• Many legacy product installs have not been updated in more than two years, including those with known exploits.
• The average number of daily APM users is 3.2, showing APM is still not widely used across the organization.
• Java is twice as popular as .NET for APM uses. Alternate languages are under half of the popularity of .NET, and C++ is not far behind alternate languages.
• Mobile APM is almost as popular as .NET, but respondents said they were doing mobile with vendors that have no mobile solutions, so this might be a survey issue.
• Big data monitoring using APM has virtually no adoption, yet APM tools are best-suited to monitoring big data platforms.
• Packet capture is used among almost 40% of the references for EUM. We expect to see the use of packet data continue to decline for APM purposes.
• Synthetic monitoring is in use by under 33% of respondents.
• SaaS is in use by 40% of references.
• Forty percent of references are using log analytics, and half of those are using Splunk. Several companies are looking at open source to replace or augment their Splunk investments (often, an Elasticsearch, Logstash and Kibana [ELK] stack).

The installed base for the vendors in this Magic Quadrant shows the following criteria: There was a total of 373,448 free accounts reported, and 38,522 paid accounts (representing 10.32%). Keep in mind this was across vendors that had freemium strategies and those that did not.

Market Overview

The Changing APM Market

There are aspects of the APM market that have proven to be stable over the last three years, while others have changed dramatically. On one hand, a broad definition of the application performance problem space and accompanying high-level product taxonomy have remained more or less constant since 2007. On the other hand, vendors have come and gone, and have been coalesced and divested at a rate exceeding that of most other IT markets, new or old, while the technologies and delivery models addressing application performance problems and populating the high-level product taxonomy have evolved and multiplied more rapidly than the technologies that make up and circumscribe the applications being monitored.

Necessarily, then, any document that claims to describe the comprehensive state of the APM market at any point in time will date rapidly, even with regard to some of its core theses. Nonetheless, after reviewing the research that led to the 2014 APM Magic Quadrant results, the following conclusions about the state of the market are likely to shape user decisions and vendor strategies for the next 12 months.

A Year of Growth

The APM market will continue to grow at a rate greater than that of most other segments of operations management; it accounted for $2.6 billion in 2013, and we are expecting that to climb around 10% through 2014, to almost $2.9 billion by year-end (see "Market Share Analysis: Application Performance Monitoring, 2013"). This growth represents a slight acceleration over the 2013 growth rate, an acceleration accounted for by:

• A general upward spend on ITOM technology and services in North America
• The enfolding of more powerful analytics capabilities directly into APM product platforms and suites, thereby lessening the importance and influence of (although not eliminating) the ITOA pure-play vendors, such as Splunk, Sumo Logic and Netuitive.

The Importance of SaaS

The composition of APM spending will also continue to shift away from pure on-premises implementations toward pure SaaS or hybrid SaaS/on-premises solutions, eventually culminating in completely integrated and hybrid solutions available in the future. Users are growing ever more convinced that there is little or no functional or performance loss when consuming APM through a SaaS delivery mode. In fact, security and operations issues can often be reduced or eliminated by consuming SaaS technologies. At the same time, the advantages of a zero-management platform and reduced maintenance and continuous feature evolution are becoming ever more salient in a "do more for less" and DevOps-influenced IT environment. Vendors that can deliver a near-identical experience between the on-premises and SaaS versions of their offerings will be particularly favored by users who want the ability to switch among on-premises and SaaS, and ultimately to choose hybrid solutions based purely on the dictates of continually changing business needs. SaaS accounted for approximately 10% of the APM spend in 2013 and, given our considerations, is likely to account for nearly 20% of the spend in 2014. The only factor preventing that figure from being even higher is the recently intensified concern for data privacy felt particularly among European enterprises and government agencies.

The SMB Market Comes of Age

2014 will also see an increase in the overall percentage of spend on APM that originates from SMBs. Historically, SMBs have focused most of their operations management attention on infrastructure monitoring. However, over the past two years, they have belatedly followed in the footsteps of their global enterprise colleagues and have begun to shift their attention to monitoring applications and, largely, for the same reasons: increased reliance on applications for direct revenue generation and increased business awareness of that fact. Of course, SMBs have different priorities and requirements than do global enterprises. Simplicity and cost are fundamental drivers for SMBs, even more than for large organizations,. Functionality and scale come second. Also, particularly in North America, SMBs are enthusiastic about APM delivered in SaaS mode and, hence, are partly responsible for the growing SaaS demand mentioned above. In all, we expect that SMB share will grow from approximately 5% in 2013 to 9% in 2014.

Continued Vendor Turmoil

Turmoil in the vendor community will continue through 2014. IBM, HP, CA Technologies and BMC Software will continue to lose share to newer players (like AppDynamics and New Relic) as a consequence of the large legacy ITOM players' continuing collective inability to reduce the relative complexity of their product offerings. The term: "relative" is critical here, because IBM and HP, in particular, are investing heavily in product simplification efforts and are achieving some results. Unfortunately for them, however, the newer players are building on their existing advantage in simplicity and pushing it even further.

Another factor ensuring continued market volatility will be the role of the investor community in forcing the rationalization of business practices and product portfolios. BMC Software, Compuware and Riverbed Technology are undergoing rapid transformations as a consequence of investor community actions. Many of these actionist investors are ultimately aimed at requiring the businesses to retarget and focus more exclusively on the promise of APM capabilities. The success or failure of these APM-centered transformations will shape the structure and makeup of the APM vendor community for years to come.

Shifts in Functional Emphasis

With regard to functionality, 2014 will be marked by two key developments. First, driven by the increasing significance of mobile application endpoints and dynamic Web technology, EUM is becoming even more important than it currently is to enterprises. In fact, it is likely to shift its center of gravity away from monitoring the end-user experience of individual applications toward the monitoring and analysis of end-user behaviors across the entire portfolio of applications available to an end user at any given endpoint. Specifically, this entails understanding and tying together mobile, Web and other application platforms being consumed by the users, and specific interactions with the software and ultimately the business.

Second, the 2013 argument between an approach to application performance analytics that would couple ITOA functionality tightly to an APM portfolio and one that envisioned APM as one discipline that used a domain-independent ITOA platform, among others, will be decided in favor of the latter approach. This means that APM vendors and service providers will have to decide whether to generalize and deepen their existing analytics functionality or to focus on partnerships and technology-sharing arrangements with ITOA specialists, such as Splunk and AppFirst.

Third, the importance of integrating database performance monitoring with other aspects of APM will come to be universally recognized and, hence, vendors will find themselves having to either modernize existing database performance monitoring functionalities or acquire/develop such functionalities if they do not already exist. Vendors that have already taken steps in this direction, such as AppDynamics, Idera and SolarWinds, will find their positions in the market further enhanced.

Fourth, the link between APM and application development will solidify. Deep-dive component monitoring technologies from the APM side of things replacing static profiler technology on the development side of things, along with a concerted effort on the part of enterprises, will ensure that application models are shared and evolve consistently through the life cycle. In general, a strong APM strategy will come to be seen as a prerequisite for a strong DevOps strategy.