Magic Quadrant for Integrated Software Quality Suites
11 July 2013 ID:G00235234
Analyst(s): Thomas E. Murphy, Nathan Wilson
VIEW SUMMARY
Software testing is one of the key processes in the delivery of applications. The market has been steadily maturing to deliver improved user success, but technology changes such as agile practices, mobile computing and cloud platforms can disrupt vendor evaluation for testers and test managers.
Market Definition/Description
The automated software quality (ASQ) assurance market is a subsegment of the overall software development life cycle market. It is composed of three traditional areas:
- Test management — Tools to manage and plan testing activities and their results.
- Automated stress and load testing — Tools that simulate the load of multiple users against a server-based application to understand and tune performance.
- Automated functional and regression testing — Tests that mimic a single user to find defects in the application.
Software quality encompasses a much broader number of activities than these, and thought leaders are driving broader solutions and creating integration across the life cycle. These key software testing facilities are our core focus in this Magic Quadrant, but it is important to consider other areas, including test data selection and management, unit testing, security and compliance, cross-browser testing, and usability. We expect to see strong demand and growth in the market for testing mobile applications, and the leading tool providers are beginning to address this either by product expansion or by partnerships. Expect acquisitions during the next three years as mobile requires more than an extension of existing tools and represents an inflection point in the market. We did not include any of the mobile-only testing providers in this report, because they fall under our revenue guidelines and we believe most purchases at this point will remain tactical. The market is also evolving to better support packaged applications, as well as new technologies that are at the foundation of cloud and composite applications. Another driver in the market is the move toward agile practices, which have been spreading from developers to include the whole team. This is putting pressure on teams to improve automation, not only of testing but of the provisioning of test labs and the release of software into the lab. We expect this change to create a diversity of testing styles that are matched to business types as well as utilizing the pace-layered approach of portfolio management (see "Application Deployment Options Through the Pace Layer Lens").
The past few years have seen an improvement in the integration of ASQ tools with the rest of the application development life cycle management (ADLM) platform (which also includes requirements management, and software change and configuration management), to help automate the overall execution of software projects. A key value here is the integration between requirements and test cases driving traceability for compliance and also aiding in workload management by defining what work is ready and the impact of any changes. It is important to select testing tools not in isolation, but connected to the rest of the life cycle's tools and development practices. Organizations selecting tools should ensure that RFPs and proofs of concept (POCs) consider integration with current tools and understand the value delivered. Only a few of the players profiled in this market have full ADLM solutions (Borland, HP, IBM and Microsoft — all Leaders), and it should be noted that most organizations have a mixture of tools (see "Application Life Cycle Management Matters Where Diversity Persists"). This diversity won't just be in the selection of tools around the life cycle, but also in the testing itself: where different vendors may be chosen to support Web, mobile, package, and legacy solutions; and different tools selected to fit different processes or skills.
As the market matures, we believe the majority of Visionaries will either settle (over time) into niche roles or be acquired, because it will be hard to compete against the established Leaders that represent 90% of the total revenue currently spent on distributed application testing. In general, this market is not one in which the Leaders provide a lot of innovation, but at the same time, most of the innovative companies do not provide comprehensive solutions. Buyers must balance the desire for best-in-class functionality against a single-provider solution. Strong innovation has been present in the use of cloud technologies (for example, load testing, lab management, cross-browser/device testing), manual and exploratory testing, mobile applications, and improving the productivity and maintenance of test automation. An emerging area of test improvement is analytics-driven testing: the utilization of analytics to help target testing efforts, to create more realistic test scenarios, and to understand platform and feature use. In testing, the answer can't simply be more; but at the same time, risks must be effectively mitigated and positive user experiences delivered.
Most companies are happy with their selected tools though we continue to see users seek improved solutions. This trend is borne out in our surveys: This year, we found that companies have, on average, tools from 3.6 different companies. The drive for new tools comes from three primary vectors: support for new technology, support for new processes, and reducing costs. An area of diversity is where defects are managed, with a fairly even split between test management, defect management and ADLM tools. The products mentioned most often are HP's Quality Center and Atlassian's Jira, followed by Selenium, IBM and Microsoft. The greatest level of satisfaction is with support and service and the pace of innovation; the lowest levels come from price, integration and the upgrade experience. While users surveyed are generally satisfied, we still see plenty of product failures caused by technology support issues or the effectiveness of the solution. These issues drive client inquiries regarding reducing software license costs, vendors expanding SaaS-style offerings and other alternative pricing schemes. Cost savings in SaaS generally require a different relationship model and are not inherently cheaper. We note that price is where the highest variance in score is recorded. Many users note that they feel they are getting superior price/performance from their tools and that this is a key value of the product. Overall, complaints are more widespread, including cost and difficulty, but also including a number of technical shortcomings. We note that many of the vendors are facing challenges in keeping up with the pace of technology change, be it new browser releases, mobile devices or UI technologies.
The key trends in the ASQ market are:
- Drive for productivity. Organizations are struggling to keep up with the rapid changes in technology, with new client capabilities and the re-emerging need to test on multiple platforms (that is, a variety of browsers and devices). The shift to agile methods drives change in team structure and practices and requires high levels of automation. The majority of time spent on manual testing is still the largest portion of a typical project. Companies are, in general, spending on people as well as tools, with 79% allocating budget to tester training. In testing, test automation has often been the major focus, but there are many other places to improve efficiency, including test lab management, test data management (subset/mask), build and release automation (DevOps) and analytic-driven testing. This demands that vendors partner or expand to create more comprehensive approaches.
- Changing technology. Support for emerging technology continues to be an area of concern for users of test automation products. Generally, the larger and more established vendors tend to move more slowly as they wait for the market to settle; they also have to support a larger installed base. Current hot spots are devices and mobile computing platforms, the growing number of browsers — and validation that applications work across multiple browsers and devices, and a lack of support for Apple desktops and devices. Technology is running much faster than testing companies, making it difficult at best to test many cutting-edge technologies. In addition, bring-your-own-device initiatives create pressure to test against a wider variety of technologies — driving increased testing cost.
- Cloud. This nexus force affects testing with tools that leverage cloud platforms, such as Amazon EC2, to create on-demand load testing. There are also a number of SaaS offerings and complete on-demand platforms delivering portions of the tools — as well as the ability to dynamically create test labs utilizing cloud technologies. These test solutions can also help with functional testing, especially where applications need to be tested for multiple client configurations (that is, browsers, devices and others). Combined with the shift to service-oriented architecture (SOA) and composite applications, these represent strong potential for market disruption. This has led to market acquisitions of Green Hat and ITKO and rapid growth for vendors in the load testing services (see "What IT Leaders Need to Know About Cloud SDLC Services").
- Mobile. The mobile market will reshape the landscape for testing. At this point, most mobile testing solutions are provided by new companies that only offer device testing support. We are seeing partnerships, for either products or technology, as a common solution from vendors, but believe that this will shift to acquisitions and more aggressive product development. Devices add a number of new complexities that are important challenges to automation, including location, gestures (currently touch but also in video recognition), accelerometers and orientation, and variant complexity (devices, OS builds, carriers, resolutions and others). Over time, the move to an Internet of intelligent devices will lead to increased use of voice input (especially in automotive applications) and a challenging network topology. Note that many innovative solutions for mobile testing involve cloud delivery (see Note 1 for a description of mobile application testing, and the Recommended Reading section for relevant references).
- Distributed development. Enterprise software development projects tend to be complex and are often carried out in a very distributed fashion, whether entirely internally sourced or done in partnership with a system integrator (SI) or offshore outsourcing provider. The goal is increased productivity and more-flexible resources, but that means strong requirements practices and tools to enable collaboration and governance. This all increases the need for application life cycle management (ALM) support to provide the required collaboration, traceability and reporting.
- Agile techniques. The growth of agile development practices continues, and is spreading from developers to encompass the entire team. These practices put a premium value on collaboration and alter the development cycle, because requirements are mutable and are completed incrementally as the project is underway. This practice is a challenge to those currently utilized by many companies — for example, distributed development and testing centers of excellence (TCOEs). Agile methods focus on the drive to push quality upstream, or on achieving continuous quality. This includes utilizing techniques such as automated acceptance testing, static analysis, and code reviews — all tied into continuous integration environments. Extending continuous integration is the concept behind DevOps and the automation of the build/release pipeline. This is making dramatic changes to the way that companies approach testing, with a greater emphasis on automation at the unit and source level, exploratory testing, and the use of real user monitoring (RUM) to learn how applications work in the field. The majority of tools in the market are still built around the traditional models of development. This also puts a premium on smart, efficient tests and a strong partnership between developers and testers. A test matrix can't take hours to complete, and throwing more hardware at it won't solve the problem.
- Constant technology updates. All around the technology stack, providers have moved to more agile release cycles. Packaged applications have shifted from infrequent upgrades to regular updates. Browsers and mobile OSs are updated frequently, and standards are continuing to evolve. Additional challenges are faced by applications built on an SOA, including services that may be provided by third-party partners. This will demand a high degree of test automation.
- SOA. Testing for services introduces a great deal of complexity and requires organizations to increase minimum standards just to operate as well as they currently do. Services are supposed to provide business-level agility, yet companies have traditionally struggled with reuse. If services are to provide a dynamically adjustable business operating environment, then they must have a high level of quality and automation for quality assessment and change impact analysis. This has been a major catalyst for acquisitions (CA Technologies, IBM and SmartBear have all made acquisitions over the past 18 months), as well as investments in development by others. Expect to see more virtualization come into play in development and test labs, including services, databases and networks.
- Open source. Open-source testing tools continue to make progress, and Selenium in particular has high adoption. Open-source tools fit well in organizations that are working on smaller projects with a more limited technology scope, and where the automation is being performed by developers. Because open-source products focus on very specific tasks there are no single-vendor solutions that cover test management and automation for performance and/or functional testing. In addition, these tools have less documentation and little support, though the options are growing for supported offerings such as PushToTest, and there are commercial offerings layered on top of open-source components. The most widely used tools include Geb, Selenium (33% of surveyed users), soapUI (27% of users), Sahi, Watir, Bugzilla and JMeter. There are many other open-source tools used to drive quality early in the development process, including Jenkins, Puppet, Chef, JUnit, Maven, Sonar, TestLink and FindBugs. Agile teams are becoming a driver of open-source adoption with test-driven approaches to development and a focus on achieving a high degree of automation.
- ALM. This Magic Quadrant looks at software quality from a suite perspective, including quality management and test execution tools. Leading vendors have broader solutions encompassing requirements and change management and, in general, provide ALM solutions. Fundamentally, the drive for improved productivity demands solutions that automate workflow and enable collaboration among team members. Expect vendors not only to build partnerships, but also to expand product offerings — through development and acquisition — to compete in this market. ALM integration is especially important for teams moving to agile, because these help to break down the traditional silos between developers and testers. It is also creating an opening for several new tools to enter the market, and will force a decision between a fully integrated experience that may deliver high productivity but not support all technologies (thus requiring different toolkits for different platforms), and a less integrated generic set of tools or products that integrate with a variety of ALM hubs.
Magic Quadrant
Source: Gartner (July 2013)
Vendor Strengths and Cautions
Automation Anywhere
Automation Anywhere is relatively new to the market and is currently rated as a Niche Player. Having been in the market for several years with a solution enabling automation of repetitive tasks, it has now expanded — via Testing Anywhere — to the software test market. This product is a strong entry in functional test automation, with good cross-platform support and coverage of functional and performance tests for a wide variety of technologies. The company has a hands-on approach to enabling successful product trials and makes use of a purely online sales model.
Its goal at this point is to be very focused on making functional test automation easy and fast, which means there is no support for load/stress testing, test management or other areas of the testing life cycle. Testing Anywhere is focused on corporate IT projects tested by subject matter experts (SMEs), where the focus is on time to market and where testing may not be the everyday job. Testing Anywhere is delivered using a client/server price model. The server has a fixed fee and includes one client; additional test developers just need additional client licenses. The combination of ease of use and price model is creating good growth, but this tool deviates from traditional testing tools with different terminology and multiple recorder types that may present an initial learning curve. However, the company has a strong set of training materials and customer support to ensure initial customer success.
Testing is currently aimed at Web applications and the product does a good job of cross-browser testing, but there is no support for mobile application testing. At this point, the tools are not integrated into other life cycle tools, which makes sense as there is currently no push to developers. Over time, however, stronger facilities will be required to drive productivity and traceability. We believe the current solution is good for the smaller teams of SMEs, or technical teams wanting a single tool that can be leveraged for many test types.
Strengths
- Productivity and cost
- Coverage of a broad set of client technologies
- Customer support
Cautions
- No load/stress testing
- No test management
Borland
The Borland division of Micro Focus continues in the Leaders quadrant, having moved through a period of product line stabilization — to re-establish innovation and execution — to make positive strides over the past 18 months that are showing as good market growth. Recent extensions include support for SAP and the Silk Mobile product. A mobile testing product, Silk Mobile has support for both optical character recognition (OCR) and object recognition (supporting all major platforms) and was built on top of licensed technology from Experitest. Unlike most other vendors that have ventured into mobile testing, Borland has support for load and performance testing built into its Silk Performer product.
Borland recently released its Silk WebMeter site, which is a free basic APM site that uses seven different access points to collect information about availability and responsiveness. Borland has full support for the development life cycle, including tools for assessing code quality and integration between requirements and test management. The company has put a focus on how to deal with cross-browser/platform issues and with custom controls. The company has also added improved support for manual testing in its Silk Central product.
Overall, there is positive momentum with the company and its products, but it is still recovering from the downward trend prior to acquisition and the time-lag to execution postacquisition. The product line could use a stronger life cycle vision, better evangelism and connection to modern developments, and some selected strong partnerships that would help to move the message out to a broader audience and deliver enhanced customer value.
Strengths
- Continuous innovation including mobile and agile
- Integration to integrated development environments (IDEs) for developers
- Support for COBOL application testing to aid its Micro Focus core
Cautions
- Gap in SOA and virtualization
- Users of former Compuware tools should develop migration plans
CA Technologies
We rate CA, whose Lisa product (acquired with ITKO) established a visionary position for SOA testing, as a market Challenger. This product continues to be the strongest in the category and CA provides a strong global organization for sales and marketing. However, it doesn't have a surrounding cast of products, like HP and IBM, and its focused direction is less down the pure testing path and more in the application delivery and environment management direction. CA is focused on how to make the developer and tester experience more productive by having a more accurate and complete environment available throughout the development process. The company recently acquired Nolio and formed a new unit focused on application delivery.
While its product is category-leading and it continues to have strong vision, it currently can't compete in the broader market on its own. CA is more in a position to enhance the other products in this Magic Quadrant than to compete directly, except for the space surrounding SOA and virtualization. However, as applications shift toward cloud services and device clients delivered via agile development and DevOps practices, there are opportunities for the toolset to deliver a great deal of test-lab and functional test automation below the UI layer. An initial story around devices was delivered at CA World in April 2013. CA has also executed OEM agreements with Shunra and Grid-Tools, to provide network virtualization and test data management respectively. The next two years will be pivotal to the company's success and position.
Customer results with the toolset record improved productivity, by driving defect detection earlier in the life cycle as well as enabling more complete development and test environments. In addition, while not a focus of this document, it should be noted that CA has a broad set of tools for mainframe and batch process testing that is matched only by IBM.
Strengths
- Service virtualization
- Broad protocol support
- Enables test early, test often for early defect detection
Cautions
- Focus of the company is more toward acceleration of development than pure software testing
- Product is costly
- Limited UI testing facilities
HP
HP continues to be rated as a Leader in this Magic Quadrant and plays a dominant role in the software testing market by leading in market share and annual revenue, but has seen strong pressure from new market entrants and reinvigorated competition. The company has responded by increased investment into the product set and integrations, and by creating simplified packaging and licensing models. For example, Quick Test Professional (QTP) and Service Test have been combined into a single package called HP Unified Functional Testing (UFT). Also, HP Business Process Testing (BPT) and Sprinter are part of the HP Quality Center and Application Lifecycle Management (ALM) packages and these in turn are integrated with UFT to enable sharing of assets. There has been some confusion in the market while these changes have rolled out, but we expect that — in combination with HP's transition to support a broader set of tools available in competitive SaaS-based pricing schemes — these changes will lead to greater value for money. However, we expect the transition will continue to create competitive opportunities and may create some sales disruption.
The greater challenge for HP is the overall role of software in the HP portfolio, and the continued focus on expanding market position from just software quality to encompass the entire ADLM market. This expansion will be welcomed by those with strong investment in HP that may have gaps in the rest of the life cycle (primarily in management of requirements and work items) that can be filled by HP's ALM solution. However, it has created challenges during contract renewals with pressure to move from HP Quality Center to HP ALM. We believe HP will also create questions about investment balance as it seeks to clarify its position in this market.
HP has broad technology support, covering packaged, client/server and Web applications for both functional and performance testing. However, the company has lagged at times in supporting new technologies (such as rich browser client frameworks and Web browsers other than Internet Explorer) though it often partners to fill this gap. For example, in service virtualization, HP resold a partner product until it had built its own. HP is currently utilizing partners for functional testing on devices. Because of the dominant share HP enjoys, it is most often the target of competition, and younger companies have often been able to be more nimble in solving specific problems. A current gap is in supporting agile testing paradigms, which include a high degree of developer-driven and code-centered techniques: unit testing, peer code review, static analysis. The company has begun to make a push into agile teams with its Agile Manager service and the extension of Quality Center as HP ALM.
Strengths
- Market share leader
- Broad technology support
- Broad network of partners, including competitors
Cautions
- Lack of appeal and support for developer testing
- Carries a price premium with improving, but complex, pricing model
IBM
We rate IBM as a Leader, with one of the broadest portfolios in the testing tool market: covering not only the core of test management and test automation, but also static analysis (both code quality and security), unit testing, and a comprehensive solution for test data through its Optim line. This positions IBM clearly in the Leaders quadrant and accounts for its No. 2 position in overall market share. As Rational Quality Manager (RQM) has matured, it has become more competitive with HP, enabling IBM to gain wins. IBM has increased the amount of marketing for this product line with webinars and other activities and has also sought to increase customer satisfaction. However, users — while feeling that the products provide good value for the money — noted that integration between the products is incomplete, and there are gaps and lags in the feature set. Rational Functional Tester is also code-centric, increasing time to productivity. IBM also has a leadership position in ADLM for companies seeking a consolidated set of tools across requirements, development, and quality management and execution. IBM is one of the few vendors with support for mainframe and distributed computing technologies that also support both traditional IT and system-level development. In addition, it is one of the few vendors to have a robust test data management tool (Optim) including subsetting and data masking.
However, IBM's greatest strength is not the products themselves, but the combination of the products with IBM Global Services and the expertise the company can bring on metrics, testing center of excellence and agile transformation. For organizations seeking to make a transition and gain greater insight into metrics and maturity, IBM has unique abilities. However, the company has gaps in its coverage (filled by services via partners) in packages and device-based testing (a new offering has just been released here). Over the past 18 months, IBM has acquired Green Hat — to fill out its support for SOA testing and service virtualization, and UrbanCode — to build a compelling offering around test environment provisioning and continuous test.
IBM has long taken part in supporting open source and will continue to embrace a strategy that extends around open-source solutions — enabling Rational Functional Tester to run on different execution engines, for instance. Another major investment will be in continuing to incorporate analytics solutions to aid with driving good decision making around test execution utilizing heat maps, code churn and risk analysis.
Strengths
- Platform coverage: Web, legacy, embedded, SOA
- Services and support
- Partners and integrations
Cautions
- Consistency of toolsets due to the acquisition nature of the portfolio
- Reliance on third parties emphasizes multitool/UI/framework fragmentation
Microsoft
Microsoft has become a leader in the testing market via the adoption of its Visual Studio Team Foundation Server (TFS) and a solid focus on treating testing as a "team sport" involving product owners, developers and testers. The incidence of use puts Microsoft third in overall market adoption behind HP and IBM, but growing in use more rapidly. Microsoft has had a strong story for developer-driven quality, which has expanded over the past few years to create an overall suite. This background has resulted in the company also providing guidance on which tools to use when, and a core set of four defined patterns:
- Acceptance-Test Driven
- Early/Frequent Stake Holder Engagement
- QE Toolbox/Practices
- Metrics that Matter
Microsoft is focused on teams "shipping software" and how to remove impediments. The core toolset is best thought of in concert with the overall TFS platform and the strongest position for the product is in product-focused teams that have a strong integration between the developers and testers. The products feature strong tools for manual and exploratory testing that few other providers offer, and support for unit testing and code quality is only matched by IBM and Parasoft.
Because of the core integration to development teams and TFS, Microsoft provides solid support for integration of testing into the build process and for governing code quality processes. We also note that Microsoft has been extending its development and testing stack with support for open source, enabling support for a wide variety of technologies running on its Azure cloud offering and supporting cross-browser testing. However, while winning position in phones and tablets is important to Microsoft, it currently relies on integration with Perfecto Mobile to support native device testing via coded UI tests. The main lag for the tools is support for scriptless test automation, thus relegating SMEs to manual and exploratory test roles. Support for exploratory testing is exemplary.
Strengths
- Abundance of training and best-practice materials available through its Microsoft Developer Network
- Integration with the development life cycle
- Lab Center — test lab management
Cautions
- Lacks script-free approach to functional automation
- Little support for package application testing
- Less valuable for testers outside the Microsoft stack, including those not utilizing Team Foundation Server
Oracle
Oracle's strong footprint in the database, middleware and application stack, combined with continued product innovation, positions it as one of the market's Leaders. However, while the tools are solid, the execution into the market has lacked visibility outside of Oracle enterprise deals, and even within the Oracle customer base, there is often a lack of market presence. We are seeing improvement, however, with increased presence in competitive situations over the past six months and with solid revenue growth and a strong partner network (for example, Atos, Capgemini/Sogeti, Infosys). Because of its partnership with Sogeti, Oracle fits well for organizations seeking to work through the Test Management Approach (TMap) maturity model. Oracle is expanding its market presence with a testing track at Oracle OpenWorld in September 2013, as well as participation in other industry testing conferences.
A key differentiator for Oracle is its strength in database testing, including database virtualization, data masking, SQL performance and database replay. Recent product extensions include private cloud test services, lab management and tools to aid with capacity planning. Oracle's database virtualization is a unique offering called Snap Clone, which provides developers and testers with no need to go through the data cloning or subsetting process. This dedication to supporting the Oracle stack means these tools are targeted not at the broad development market (though Java and the Oracle Database are a common enterprise platform), but at organizations or teams focused on either Oracle's application stack or Web-facing Java applications. While the company does have SOA testing tools, it packages service virtualization as part of a WebLogic package rather than as part of an overall testing suite. It is this type of packaging that keeps Oracle out on the periphery of many testing tool conversations, but at the same time provides different points of attachment into users of its technology stack.
Unlike the other major providers, Oracle lacks a cohesive ALM platform or integration to other life cycle products. There is more of a connection to production operations, rather than development, and the company needs to make a stronger connection between its assets to deliver a stronger life cycle and DevOps story. Currently, the only mobile story is based on the use of desktop emulators, but we expect this to be addressed during this year.
Strengths
- Test data management and database virtualization
- Support for high-capacity load testing of database tier
- Acceleration for Oracle packages
Cautions
- Current mobile support is weak
- Lack of ties to ALM and development suites
- Currently lacks keyword-driven testing
Original Software
Original rates as a market Visionary, with a strong solution for functional test automation and a process that enables users to progress easily from manual to automated testing and tools that do not require scripting, thus reducing costs to create and maintain automation. While automation is good, customers note that time to proficiency is about six months. Although there is a learning curve to the approach, customers are reporting strong gains in the percentage of test automation, and resulting savings in the overall testing process.
Original has done a good job of staying on top of the ever-increasing number of presentation technologies, but supports mobile applications via partner integration. Original Software also supports both SAP and Oracle E-Business Suite (EBS) testing (including predefined test packs), and is one of few vendors to support IBM iSeries. Original's test planning tools have a unique workflow process with the ability to see and manage progress across multiple projects, as well as integration to Microsoft Outlook. This will be beneficial to organizations with large-scale testing efforts. Another key element of the completeness of the platform is its test data management, with tools to subset and cleanse data and manage it through the cycle of testing.
A challenge to partnering may be Original's emerging ALM story, which places it in a crowded field and in competition with potential partners. We note that full integration with Microsoft TFS is available. In addition, a key element of focus from a life cycle perspective has been on service desk support. Currently, too many tools get you through development and into production but then provide little support or integration for applications once in production. However, Original will need to fill in its partner chain and create additional integrations to other life cycle tools to fill this gap.
Strengths
- Test data management — extract and mask data, as well as rollback
- Ease of use in code-free approach and process to migrate from manual to automated testing
- Self-healing scripts
Cautions
- Overall global presence
- Limited technical partnerships
Parasoft
We rate Parasoft as a Niche Player, with a strong message around development productivity that comes from driving quality through the entire development process. However, it has less appeal to the broader testing market as a whole. Parasoft has a strong focus on overall quality and early detection of defects, but does not play in the load/stress or UI automation markets. Its Concerto solution is well suited to technical development groups and those creating APIs and services. Parasoft is well positioned to take advantage of the growth in use and expansion of roles involved in agile development.
The company's service virtualization facilities are driving business growth, and a key direction for the company lies in helping bridge gaps between development, test and business. This is facilitated by the ability to define and measure quality against expectations. Strong dashboarding functionality enables everyone connected with a project to gauge progress and quality levels. When used in conjunction with the company's Concerto ALM product, a full application development (AD) life cycle approach to traceability, and adherence to stated quality standards, is enabled.
Examples of the target audience for these tools are regulated processes and industries. We generally see Parasoft in an augmentation role in enterprise deployments (that is, used with other tools for UI functional test automation), or with agile teams that don't utilize a separate quality assurance (QA) function. A growing number of integrations, including open source and a reseller agreement with Grid-Tools for test data management, provide the potential to broaden the appeal of the product set. The company has good global coverage and has steadily executed, but needs to extend beyond its traditional customer base to drive broader adoption.
Strengths
- SOA testing and service virtualization
- Support for regulatory compliance
- Drives quality early in the development cycle
- Integration to common continuous integration tools
Cautions
- Lack of GUI test automation
- Not oriented around basic acceptance tests or SME-level users
SmartBear
SmartBear is rated as a Visionary, with a broad set of tools assembled through acquisition that cover developer-oriented code quality in addition to the core testing suite and the open-source SoapUI/LoadUI tools. With the breadth of its products (including the popular open-source SoapUI), SmartBear has a large user base. The company supports the ability to migrate users from HP's Quality Center product and also includes a strong set of tools in the Web monitoring market. SmartBear has a good set of partners for global sales and support, but like the majority of vendors, the products are only available in English and are not Section 508 compliant (U.S. Rehabilitation Act of 1973). The company has been built through a series of acquisitions that have yielded fast growth and good product breadth, but this also means that the tools are inconsistent in UI and don't always leverage each other well. The company is developing a good social presence and has a good connection to developers that may be less worried about whether the icons are all the same than whether the tools work and enable them to perform the task at hand in an efficient manner. This is where SmatBear does well, and the company has an effective range of products and price points with which to address the market.
While SmartBear is generally out in front of most new technologies, it is lacking in support for device testing. We expect this gap to be closed because SmartBear has established an excellent record of acquisition and execution and offers products with strong price points. Customer satisfaction is positive, and for its size, the company has produced a good level of training material and community support. The company needs to work on a stronger partnership model, which is currently an unclear path given the acquisition model of the company. It will be important to effectively compete in the market while partnering with the broad set of solutions available in it.
Strengths
- Support and service
- Pricing model
- Full life cycle view of software quality
Cautions
- Lack of technical partnerships and integration with other tools
- Lack of consistency in UI and frameworks
- Currently no support for mobile applications
SmarteSoft
SmarteSoft is a Niche Player that delivers a complete set of tools that users find to be flexible and easy to learn and make use of, along with a full coverage of the three primary areas of management, functional automation and performance. The company has developed a focus on the OEM and channels model, initially in healthcare software validation and insurance systems. The company's recent releases of its eProgressa, HCLL and Cerner editions emphasize this direction while also delivering the ability to license the technology for integration into other solutions. This licensing ability enables companies delivering solutions into regulated markets to include an extensible validation suite.
SmarteSoft has built upon the initial success in this market to deliver an extensible test platform. We believe that this provides a good foundation for inclusion into embedded systems, SaaS and platform as a service (PaaS) offerings, and vertical market solutions. However, the company will need to deliver strong support for devices and new interaction paradigms.
While there is a focus on specific markets, the company still produces a full set of tooling for test planning and management and the execution of both functional and load/performance tests. These tools are well suited to small or midsize businesses (SMBs) and smaller test teams made up of SMEs, rather than engineers, offering ease of use and productivity at a reasonable price. The extensibility of the platform means that engineering teams can utilize the tools for early testing but also extend scriptless testing readily for new custom controls, for example.
Moving forward, the company needs to create stronger partnerships both with service providers and in other areas of the development life cycle. Independent software vendors looking for tools with full extensibility and the backing of a commercial product will find value in this toolset.
Strengths
- Scriptless test automation platform enables embedding of validation suites and a high level of field extensibility
- Targeted solutions for healthcare, biomedical and insurance use
- Excellent customer support
Cautions
- On the smaller end of the vendors profiled, lacks resources of large-scale vendors for global enterprise support
- Limited partners and integration in the ADLM space
- No support for mobile testing
Soasta
Soasta has moved from being a strong leader positioned as an innovator in the testing market, to being one of the Leaders via strong product and sales execution. Its initial focus was on performance testing for Web applications, where the products were innovative and disruptive from a pricing and ease-of-use perspective. This focus has now expanded, with the TouchTest product, to provide market-leading functional automation for devices. The tools have been proven on several highly visible and highly concurrent user applications. TouchTest provides full object-level testing without modification to the device. This means that scripts are more robust, requiring less maintenance and providing more accurate capture and playback of gestures. Tests can be run on devices via Internet Protocol (IP) address rather than tethering or use of rack-based systems. This has also been followed by an OEM agreement with Appcelerator and a partnership with CloudBees to plug into Jenkins — bringing mobile cloud testing to a continuous integration environment.
During the past year Soasta acquired the mPulse Real User Monitoring (RUM) solution. While testing before delivery is important, we believe that there will be a continued convergence between load/stress testing and user monitoring tools (see "Leverage Your Application Performance Monitoring Through the Application Life Cycle"). The combination of Soasta's strong real-time analytics with a codeless automation environment and RUM solution provides a good foundation for driving improved test effectiveness.
The company has been expanding its presence globally and expanding its partnerships, and has continued to deliver innovation in a consistent fashion. The tools are well suited to modern applications and the shift to mobile; these are not tools for traditional client/server and other legacy technologies in much the same way that legacy-centered tools are not well suited to a modern application stack and development approach. However, Soasta needs to create stronger partnerships with agile AD life cycle tools and build or buy a test management solution, as well as filling out its overall test platform.
Strengths
- Ease of automation
- Native device testing solution
- Real-time analytics in cloud performance testing service
Cautions
- Don't look here for support for pre-Web technologies
- Needs additional partners in life cycle and application delivery
- Lack of test planning and management
Telerik
Telerik is a relatively recent entry to the testing market that lands as a Visionary with a complete set of testing tools that support modern UI frameworks, agile testing techniques and the testing of iOS apps. The product model is simple, a single tool — Test Studio — that is used for all test types and is priced at the low end of the market while still being backed by high-touch support. The license model is subscription-based, providing updates and support. Support is provided online via Citrix's GoToMeeting to provide interactive product assistance.
Telerik has had a long association with the Microsoft developer audience, and Test Studio continues this with integration to Visual Studio and use of C# or Visual Basic (VB).NET as the scripting language. The mobile support does not require jailbreaking of apps and supports native, Web and hybrid applications using object identification. Tests can be recorded either on the device or using an emulator; the ability for beta testers of applications to perform ad hoc testing and submit defects and feedback to the Telerik QA cloud is also available. The current weakness here is the iOS focus of the tool, particularly as Android gains strength and because of the strong overall support and focus of the company on Microsoft technologies.
Telerik also has an agile project management tool, TeamPulse, which integrates with Test Studio to provide requirement management, bug tracking and other planning tasks. The scope and target of the company's products is such that it has limited partnerships on both the services and technical fronts. While Test Studio is a stand-alone solution, we find the current tools to be best oriented to teams using Visual Studio or targeting iOS — those that are acting as tightly knit product teams, rather than traditional siloed-software testing organizations.
Strengths
- Ease of use and familiarity for Microsoft developers
- High-quality support
- Frequent updates adding new features
Cautions
- There is no direct phone support
- Outside of iOS, tools focus on the .NET developer/tester environment
Tricentis
As a Visionary, Tricentis continues to lead the market in successful functional automation with users achieving extremely high percentages in automation coverage. Its test planning tools are also market leading, with strong test case design providing support in driving effective test coverage. Test case design is the foundation of the toolset, driving not only the test cases and automation but also the creation of test data appropriate to the required validations.
The challenge for the company has been building an effective organization in the U.S. By utilizing venture capital it is taking steps to build a stronger U.S. presence. The company also needs to broaden partnerships to aid in expanding its market presence and is starting to make strides here through a relationship with Neotys, to fill a gap in performance testing, and a relationship with Polarion Software and Rally for the broader agile planning and development life cycle. The company has also partnered with agosense to provide a broader set of integrations.
Tricentis has formed a partnership with DeviceAnywhere to provide mobile testing and has broadened its solution set organically to support SAP and data warehouse testing. The company has combined a consistent pace of innovation with consistent strong financial growth and overall execution. While the focus of the company has been on test automation, support for manual testing has been delivered and ties to the same planning method, known as Linear Q, to create a smooth transition for teams new to automation and to support ad hoc testing efforts. Linear Q optimizes test coverage, driving high coverage rates with fewer test cases.
Strengths
- Market-leading, code-free automation
- Market-leading test case design and planning
- Integrated synthetic test data management
Cautions
- Still building global presence
- Potential philosophical challenges with agile developers
- No General Services Administration (GSA) listing and slightly complex pricing model
Worksoft
Worksoft is a market Visionary with a leading test automation platform that shines in the SAP market. It also has a strong partnership with IBM Global Business Services, which provides a strong extended sales force. The company is seeing increased interest in moving SAP testing in-house due to dissatisfaction with outsourcing efforts. This is positive for Worksoft's business since it requires highly efficient tools that can be effectively utilized by SMEs to drive acceptance and validation testing.
The company has partnered with DeviceAnywhere to deliver mobile application testing support and continues to fill out its overall tool suite. Growth in Europe has been especially strong over the past 12 months. While the toolset may initially appear relatively expensive, the ROI seen by customers seems to justify the expense — with more than 50% of their functional tests being automated, on average. As SAP users often struggle to keep up with upgrades and enhancements and adapt to more agile business practices, Worksoft's Certify Capture simplifies the capture of business processes by SMEs to produce test automation, while Certify Impact aids the identification of test cases affected by system changes. Users report seeing, on average, a 40% decrease in upgrade time.
Worksoft has been expanding automation support for Siebel, PeopleSoft, Syclo, Red Prairie, salesforce.com, Promax, Workday, Siemens TeamCenter, Maximo, Oracle Financials and EBS. In addition, with a number of successful implementations and a strong partner network, the company has also developed a set of best practices for TCOEs, and approaches to automation and upgrades.
In general, users rate Worksoft well, liking the breadth of functionality and the feeling of partnership around solving problems and attention to detail. The company is aware that it is important to recognize there is still a need for test engineers to aid in the definition of test cases, and that there have been some inconsistencies in the UI. Large-scale test systems also require a fair amount of server horsepower to manage large repositories effectively.
Strengths
- Full range of tools for script-free testing of SAP and other enterprise solutions
- High productivity and reduction of maintenance costs
- Reporting and documentation generation for compliance support
Cautions
- Consistency of delivery
- Must consider value of a single tool for all testing versus targeted tools
- Growing number of competitive solutions in the market
Vendors Added or Dropped
We review and adjust our inclusion criteria for Magic Quadrants and MarketScopes as markets change. As a result of these adjustments, the mix of vendors in any Magic Quadrant or MarketScope may change over time. A vendor appearing in a Magic Quadrant or MarketScope one year and not the next does not necessarily indicate that we have changed our opinion of that vendor. This may be a reflection of a change in the market and, therefore, changed evaluation criteria, or a change of focus by a vendor.
Added
Automation Anywhere, Telerik
Dropped
ITKO (acquired by CA Technologies), Green Hat (acquired by IBM), Seapine Software
Inclusion and Exclusion Criteria
Vendors in this Magic Quadrant must provide the ability to create, manage and execute functional test automation. Their tools must support the creation of software tests on the Windows platform to test Web applications (additional platform test execution support is desirable, and the most complete vendors cover a wide set of technologies). The vendors in this market must have a basic global presence, which means that they are actively selling their products in multiple geographies, and have at least $10 million in annual revenue. (Note: This market includes a set of mainstream or traditional players that cover the testing of desktop and Web applications, plus a set of companies from emerging or niche areas, such as SOA/external service bus, cloud and code-free solutions.) This Magic Quadrant looks at the entire market (traditional and new niches) as a complete market; however, that is not to say that the vendors in the Leaders quadrant are the only solutions that should be explored, or that looking at the market through more specific lenses would not change the shape of the market. It is important to keep in mind that a Magic Quadrant also emphasizes criteria that help assess where companies are with respect to being able to support an enterprise deployment to a Global 2000 company.
Evaluation Criteria
Ability to Execute
Because of the general maturity of the market, the Ability to Execute in a consistent fashion is critical and has been the defining attribute of the Leaders. This will continue with those that gain in the market providing a clean combination of technology with very clear market positioning. It is not enough just to have "better technology" than the incumbents. Better technology is still important, but generally opens short-term windows until the incumbents acquire or catch up. However, mobile and cloud are creating a great opportunity for market disruption. Another challenge will be the Ability to Execute in a market that is increasingly aligned to agile practices. It will be very difficult for companies to execute well with solutions that satisfy any use, which opens up the opportunity for focused solutions to move into leadership positions. A number of factors, including regulatory compliance, distributed and agile teams, are driving the need for strong partnerships across the life cycle and to service providers. Market pressure to reduce costs will continue to create openings for new tools and open-source solutions, and will force vendors to deliver clear ROI; but, this will result mainly in market expansion rather than replacement. At the enterprise level, company stability is also critical.
Source: Gartner (July 2013)
Completeness of Vision
While innovation is, in our view, the most important criterion, Completeness of Vision for enterprise use looks not only at innovation but at how complete the functionality and the company as a whole are. Direct marketing capability is becoming less important than community feedback and competitive pricing. The ability for a vendor to build community, show results and enable customers to drive positive change is critical. Vendors have one of three options: create strong innovation, target a specific market segment or build a comprehensive portfolio. Strong innovation is the most difficult to position and hold as the market has a general distrust of marketing claims and there is a strong need to be able to prove results. Innovation is hard to hold on to because the larger vendors are in a position to acquire or imitate. This means that innovators need to continue to find new areas of innovation or become relegated to niche roles over time. Fortunately, the Nexus of Forces (social, mobile, cloud and information) is providing a tremendous number of opportunities to innovate, and this will last for the foreseeable future. It will still be a challenge for new companies to sustain high levels of innovation and at the same time build strong execution. (During the next three years, mobile — of all the nexus forces — will have the greatest impact on the market.) But vision isn't purely about innovation, Leaders have a complete vision or an ability to support everything that a quality team will need. Vendors that support leading technologies and architectures with a clear vision of the shift this introduces to applications, and the complexity of testing those applications, are key here. In addition, vendors with a complete vision either provide a robust view of the product life cycle or have strong partnerships to help fill in those areas of the life cycle in which they are not involved. Overall, vendors still have gaps in their product lines, with most still focused on the core of automation of functional/regression and load/stress testing, as well as elements of quality management. Common gaps are in unit testing, integration to other areas of the life cycle, test data management and lab management facilities. Many of these gaps are filled through partnerships, but the Leaders are also increasingly filling these gaps. Thus, while the Visionaries and Niche Players will scramble to grab market share, the Leaders and more established players will consolidate through acquisitions as markets mature. These players are generally positioning to become Leaders not only in software QA but also in the overall ALM market, or they are strongly partnered to do so.
Source: Gartner (July 2013)
Quadrant Descriptions
Leaders
The market leaders all have broad portfolios covering test management, functional and performance test automation. Each also has support for additional quality measures such as code quality, database testing, and either deep platform or broad technology support. At its core, the testing market is quite mature with a strong stratification of vendors by annual revenue. These companies are well suited as the backbone of a testing organization's tool suite, and this is proven via a global presence and significant market shares. These vendors have significant partnerships and show consistency in execution of programs including acquisitions. However, note that, because of the breadth of customer bases and pace of change in the market, these vendors may at times lag in technology adoption which opens up space for competitors. In alphabetical order, the vendors in this quadrant are: Borland, HP, IBM, Microsoft, Oracle and Soasta.
Challengers
Challengers have a strong capability to execute on product delivery and sales and deliver strong but less complete functional coverage than the Leaders. We expect over time that companies in the Visionary quadrant will often settle into the Niche Player or Challenger quadrants, by focusing either on a specific category of testing functionality or on a market segment. CA Technologies is in this quadrant.
Visionaries
Visionary companies generally have a focus on a particular part of the software quality process (for example, cloud, load/stress, functional automation). Look to this segment for the best chance to find disruptive innovation or support for new technologies, but recognize that the products may not be as polished and that the companies at times will be stressed to support global enterprises as their sales teams and partnerships are less mature. In alphabetical order, the vendors in this quadrant are: Original Software, SmartBear, Telerik, Tricentis and Worksoft.
Niche Players
The vendors in this category either provide unique function to a specific market or use case or are early in establishing a position in what we note is a mature market. In alphabetical order, the vendors in this quadrant are: Automation Anywhere, Parasoft and SmarteSoft.
Context
The traditional focus in the application quality management market has been for specific testing activities (for example, load/stress and functional/regression). Economic conditions and pressure to improve time to market are placing increased emphasis on productivity. Agile development continues to make inroads into QA and testing teams, and this is reflected in the growing support of test planning tools for agile processes, such as the Scrum framework. Devices, cross-browser testing support and DevOps are placing stress on the testing organization to adequately test software and increase automation. On another front, packages, SaaS and business process management are driving greater business analyst involvement in overall quality efforts, along with a shift in focus from finding defects in validation to ensuring that business objectives are met. Testing software can be an expensive process, but poor software quality leads to user dissatisfaction as well as increased development and maintenance. Therefore, having a well-defined set of tools and practices to drive software quality will positively affect the bottom line of the business.
Market Overview
The software testing market is made up of several markets tracked by Gartner. The focus of this document is the distributed testing tool market, which is approximately $1.5 billion in size and growing at 6.5% (compound annual growth rate). However, we find that a number of providers are growing at 50% to 100% (annually), and we expect to see very strong growth in the following areas: mobile testing, and testing SaaS and PaaS solutions. Open source and tools supporting agile testing techniques will also do well, but will be a challenge to measure in terms of impact on the market.
The market is being driven by time-to-market issues, while seeing an increased need for governance and regulatory compliance. A surprising number of companies still are relatively immature in overall testing practices, with low rates of automation and relatively ad hoc test management practices. These factors create pressure around driving automation, improved workflow and traceability, and reporting and metrics capabilities. Some of the vendors in this Magic Quadrant play in the broader AD life cycle management market, providing tools for requirements, change and work item management. It is important to consider the other tools in the AD portfolio and how they fit together and support the development practices used by an organization.
While tools are important and necessary, they don't solve fundamental flaws in practices and skills, and many organizations need to improve organizationally before tools will provide maximum benefit. This can drive the importance of service partner relationships (outsourcing, SIs, consultants), which can aid in process transformation. This, in turn, means that vendors with good partner programs have an advantage, and also emphasizes the need for the improvements provided by the tools to guide decisions and practices. Many of the newer players are winning because they can provide hands-on support, improved ease of use or targeted solutions.
It is crucial to consider who will be using the testing tools and their skill set, as well as the types of applications or services to be tested, the development methodology, and the types of testing activities to undertake. Is the focus acceptance testing done by SMEs? This is very different from an agile team trying to find defects in a new mobile application. A focus on quality (rather than risk) may also drive a more expansive view of the activities as well as the tools utilized: static analysis, code review, unit tests, code-level automation, globalization and localization testing, and others are all required when either seeking specific compliance (such as with Section 508) or working to support standards such as International Organization for Standardization (ISO) 9126. Many organizations are more focused on risk management and maintaining a tight control on testing costs.
The Leaders have broad portfolios covering all aspects of software quality, and match this with a global presence and significant market shares. Their solutions are often not the most innovative products in the space, having time to wait for markets to mature or adding needed components via partnerships and acquisitions, but we are beginning to see visionary companies get near and cross into the Leaders quadrant. Visionary companies generally have a focus on a particular part of the software quality process (for example, cloud, load/stress, functional automation), but we often find that many of these companies and products are purchased as additions to tools from the market leaders. There is also strong innovation in the Niche Players space at this time, but these vendors either have limited geographic appeal or approach the market from a new direction. There is a lot of change going on in the market as an increasing number of development groups adopt agile practices. Note that, while application testing as a whole is a mature market, the continued entry of new technologies means that there has also been continued entry of new providers. We believe that during the next five years the market will continue to go through a consolidation cycle, with the potential for two or three more companies to move into the Leaders quadrant — driven by the shift to mobile and cloud applications — and there will also be acquisitions to accelerate growth and shore up aging toolsets.
The past 18 months have seen acquisitions by CA (ITKO), IBM (Green Hat) and SmartBear (eviWare), a number of partnerships for mobile technology support, and many new entrants to the market (there continue to be new entrants to the market that cover specific problems). There are many strong technologies that don't qualify for the Magic Quadrant, because of either breadth of functionality or revenue.
Some notable areas and tools are:
- Cross-Browser Testing: CrossBrowserTesting, BrowserStack
- Test Planning and Management: QMetry, SQS, Zephyr, Hexawise, Testuff
- Load and Performance Testing: Neotys, Sauce Labs, BlazeMeter, TestPlant (Facilita)
- Automation: Ranorex, TestPlant, Kalistick, Seapine Software (full ALM and test platform), PushToTest, eXept Software AG
- Mobile: Mobile Labs, Perfecto Mobile, TestPlant, Jama Software, Keynote's DeviceAnywhere, Experitest, Gorilla Logic
- Test Data and Test Lab: Informatica, Grid-Tools, Delphix, Skytap, Dell, VMware, Citrix (VMLogix)
- Package Software Testing: Panaya
Unlike the other major providers, Oracle lacks a cohesive ALM platform or integration to other life cycle products. There is more of a connection to production operations, rather than development, and the company needs to make a stronger connection between its assets to deliver a stronger life cycle and DevOps story. Currently, the only mobile story is based on the use of desktop emulators, but we expect this to be addressed during this year.
Strengths
- Test data management and database virtualization
- Support for high-capacity load testing of database tier
- Acceleration for Oracle packages
Cautions
- Current mobile support is weak
- Lack of ties to ALM and development suites
- Currently lacks keyword-driven testing
Original Software
Original rates as a market Visionary, with a strong solution for functional test automation and a process that enables users to progress easily from manual to automated testing and tools that do not require scripting, thus reducing costs to create and maintain automation. While automation is good, customers note that time to proficiency is about six months. Although there is a learning curve to the approach, customers are reporting strong gains in the percentage of test automation, and resulting savings in the overall testing process.
Original has done a good job of staying on top of the ever-increasing number of presentation technologies, but supports mobile applications via partner integration. Original Software also supports both SAP and Oracle E-Business Suite (EBS) testing (including predefined test packs), and is one of few vendors to support IBM iSeries. Original's test planning tools have a unique workflow process with the ability to see and manage progress across multiple projects, as well as integration to Microsoft Outlook. This will be beneficial to organizations with large-scale testing efforts. Another key element of the completeness of the platform is its test data management, with tools to subset and cleanse data and manage it through the cycle of testing.
A challenge to partnering may be Original's emerging ALM story, which places it in a crowded field and in competition with potential partners. We note that full integration with Microsoft TFS is available. In addition, a key element of focus from a life cycle perspective has been on service desk support. Currently, too many tools get you through development and into production but then provide little support or integration for applications once in production. However, Original will need to fill in its partner chain and create additional integrations to other life cycle tools to fill this gap.
Strengths
- Test data management — extract and mask data, as well as rollback
- Ease of use in code-free approach and process to migrate from manual to automated testing
- Self-healing scripts
Cautions
- Overall global presence
- Limited technical partnerships
Parasoft
We rate Parasoft as a Niche Player, with a strong message around development productivity that comes from driving quality through the entire development process. However, it has less appeal to the broader testing market as a whole. Parasoft has a strong focus on overall quality and early detection of defects, but does not play in the load/stress or UI automation markets. Its Concerto solution is well suited to technical development groups and those creating APIs and services. Parasoft is well positioned to take advantage of the growth in use and expansion of roles involved in agile development.
The company's service virtualization facilities are driving business growth, and a key direction for the company lies in helping bridge gaps between development, test and business. This is facilitated by the ability to define and measure quality against expectations. Strong dashboarding functionality enables everyone connected with a project to gauge progress and quality levels. When used in conjunction with the company's Concerto ALM product, a full application development (AD) life cycle approach to traceability, and adherence to stated quality standards, is enabled.
Examples of the target audience for these tools are regulated processes and industries. We generally see Parasoft in an augmentation role in enterprise deployments (that is, used with other tools for UI functional test automation), or with agile teams that don't utilize a separate quality assurance (QA) function. A growing number of integrations, including open source and a reseller agreement with Grid-Tools for test data management, provide the potential to broaden the appeal of the product set. The company has good global coverage and has steadily executed, but needs to extend beyond its traditional customer base to drive broader adoption.
Strengths
- SOA testing and service virtualization
- Support for regulatory compliance
- Drives quality early in the development cycle
- Integration to common continuous integration tools
Cautions
- Lack of GUI test automation
- Not oriented around basic acceptance tests or SME-level users
SmartBear
SmartBear is rated as a Visionary, with a broad set of tools assembled through acquisition that cover developer-oriented code quality in addition to the core testing suite and the open-source SoapUI/LoadUI tools. With the breadth of its products (including the popular open-source SoapUI), SmartBear has a large user base. The company supports the ability to migrate users from HP's Quality Center product and also includes a strong set of tools in the Web monitoring market. SmartBear has a good set of partners for global sales and support, but like the majority of vendors, the products are only available in English and are not Section 508 compliant (U.S. Rehabilitation Act of 1973). The company has been built through a series of acquisitions that have yielded fast growth and good product breadth, but this also means that the tools are inconsistent in UI and don't always leverage each other well. The company is developing a good social presence and has a good connection to developers that may be less worried about whether the icons are all the same than whether the tools work and enable them to perform the task at hand in an efficient manner. This is where SmatBear does well, and the company has an effective range of products and price points with which to address the market.
While SmartBear is generally out in front of most new technologies, it is lacking in support for device testing. We expect this gap to be closed because SmartBear has established an excellent record of acquisition and execution and offers products with strong price points. Customer satisfaction is positive, and for its size, the company has produced a good level of training material and community support. The company needs to work on a stronger partnership model, which is currently an unclear path given the acquisition model of the company. It will be important to effectively compete in the market while partnering with the broad set of solutions available in it.
Strengths
- Support and service
- Pricing model
- Full life cycle view of software quality
Cautions
- Lack of technical partnerships and integration with other tools
- Lack of consistency in UI and frameworks
- Currently no support for mobile applications
SmarteSoft
SmarteSoft is a Niche Player that delivers a complete set of tools that users find to be flexible and easy to learn and make use of, along with a full coverage of the three primary areas of management, functional automation and performance. The company has developed a focus on the OEM and channels model, initially in healthcare software validation and insurance systems. The company's recent releases of its eProgressa, HCLL and Cerner editions emphasize this direction while also delivering the ability to license the technology for integration into other solutions. This licensing ability enables companies delivering solutions into regulated markets to include an extensible validation suite.
SmarteSoft has built upon the initial success in this market to deliver an extensible test platform. We believe that this provides a good foundation for inclusion into embedded systems, SaaS and platform as a service (PaaS) offerings, and vertical market solutions. However, the company will need to deliver strong support for devices and new interaction paradigms.
While there is a focus on specific markets, the company still produces a full set of tooling for test planning and management and the execution of both functional and load/performance tests. These tools are well suited to small or midsize businesses (SMBs) and smaller test teams made up of SMEs, rather than engineers, offering ease of use and productivity at a reasonable price. The extensibility of the platform means that engineering teams can utilize the tools for early testing but also extend scriptless testing readily for new custom controls, for example.
Moving forward, the company needs to create stronger partnerships both with service providers and in other areas of the development life cycle. Independent software vendors looking for tools with full extensibility and the backing of a commercial product will find value in this toolset.
Strengths
- Scriptless test automation platform enables embedding of validation suites and a high level of field extensibility
- Targeted solutions for healthcare, biomedical and insurance use
- Excellent customer support
Cautions
- On the smaller end of the vendors profiled, lacks resources of large-scale vendors for global enterprise support
- Limited partners and integration in the ADLM space
- No support for mobile testing
Soasta
Soasta has moved from being a strong leader positioned as an innovator in the testing market, to being one of the Leaders via strong product and sales execution. Its initial focus was on performance testing for Web applications, where the products were innovative and disruptive from a pricing and ease-of-use perspective. This focus has now expanded, with the TouchTest product, to provide market-leading functional automation for devices. The tools have been proven on several highly visible and highly concurrent user applications. TouchTest provides full object-level testing without modification to the device. This means that scripts are more robust, requiring less maintenance and providing more accurate capture and playback of gestures. Tests can be run on devices via Internet Protocol (IP) address rather than tethering or use of rack-based systems. This has also been followed by an OEM agreement with Appcelerator and a partnership with CloudBees to plug into Jenkins — bringing mobile cloud testing to a continuous integration environment.
During the past year Soasta acquired the mPulse Real User Monitoring (RUM) solution. While testing before delivery is important, we believe that there will be a continued convergence between load/stress testing and user monitoring tools (see "Leverage Your Application Performance Monitoring Through the Application Life Cycle"). The combination of Soasta's strong real-time analytics with a codeless automation environment and RUM solution provides a good foundation for driving improved test effectiveness.
The company has been expanding its presence globally and expanding its partnerships, and has continued to deliver innovation in a consistent fashion. The tools are well suited to modern applications and the shift to mobile; these are not tools for traditional client/server and other legacy technologies in much the same way that legacy-centered tools are not well suited to a modern application stack and development approach. However, Soasta needs to create stronger partnerships with agile AD life cycle tools and build or buy a test management solution, as well as filling out its overall test platform.
Strengths
- Ease of automation
- Native device testing solution
- Real-time analytics in cloud performance testing service
Cautions
- Don't look here for support for pre-Web technologies
- Needs additional partners in life cycle and application delivery
- Lack of test planning and management
Telerik
Telerik is a relatively recent entry to the testing market that lands as a Visionary with a complete set of testing tools that support modern UI frameworks, agile testing techniques and the testing of iOS apps. The product model is simple, a single tool — Test Studio — that is used for all test types and is priced at the low end of the market while still being backed by high-touch support. The license model is subscription-based, providing updates and support. Support is provided online via Citrix's GoToMeeting to provide interactive product assistance.
Telerik has had a long association with the Microsoft developer audience, and Test Studio continues this with integration to Visual Studio and use of C# or Visual Basic (VB).NET as the scripting language. The mobile support does not require jailbreaking of apps and supports native, Web and hybrid applications using object identification. Tests can be recorded either on the device or using an emulator; the ability for beta testers of applications to perform ad hoc testing and submit defects and feedback to the Telerik QA cloud is also available. The current weakness here is the iOS focus of the tool, particularly as Android gains strength and because of the strong overall support and focus of the company on Microsoft technologies.
Telerik also has an agile project management tool, TeamPulse, which integrates with Test Studio to provide requirement management, bug tracking and other planning tasks. The scope and target of the company's products is such that it has limited partnerships on both the services and technical fronts. While Test Studio is a stand-alone solution, we find the current tools to be best oriented to teams using Visual Studio or targeting iOS — those that are acting as tightly knit product teams, rather than traditional siloed-software testing organizations.
Strengths
- Ease of use and familiarity for Microsoft developers
- High-quality support
- Frequent updates adding new features
Cautions
- There is no direct phone support
- Outside of iOS, tools focus on the .NET developer/tester environment
Tricentis
As a Visionary, Tricentis continues to lead the market in successful functional automation with users achieving extremely high percentages in automation coverage. Its test planning tools are also market leading, with strong test case design providing support in driving effective test coverage. Test case design is the foundation of the toolset, driving not only the test cases and automation but also the creation of test data appropriate to the required validations.
The challenge for the company has been building an effective organization in the U.S. By utilizing venture capital it is taking steps to build a stronger U.S. presence. The company also needs to broaden partnerships to aid in expanding its market presence and is starting to make strides here through a relationship with Neotys, to fill a gap in performance testing, and a relationship with Polarion Software and Rally for the broader agile planning and development life cycle. The company has also partnered with agosense to provide a broader set of integrations.
Tricentis has formed a partnership with DeviceAnywhere to provide mobile testing and has broadened its solution set organically to support SAP and data warehouse testing. The company has combined a consistent pace of innovation with consistent strong financial growth and overall execution. While the focus of the company has been on test automation, support for manual testing has been delivered and ties to the same planning method, known as Linear Q, to create a smooth transition for teams new to automation and to support ad hoc testing efforts. Linear Q optimizes test coverage, driving high coverage rates with fewer test cases.
Strengths
- Market-leading, code-free automation
- Market-leading test case design and planning
- Integrated synthetic test data management
Cautions
- Still building global presence
- Potential philosophical challenges with agile developers
- No General Services Administration (GSA) listing and slightly complex pricing model
Worksoft
Worksoft is a market Visionary with a leading test automation platform that shines in the SAP market. It also has a strong partnership with IBM Global Business Services, which provides a strong extended sales force. The company is seeing increased interest in moving SAP testing in-house due to dissatisfaction with outsourcing efforts. This is positive for Worksoft's business since it requires highly efficient tools that can be effectively utilized by SMEs to drive acceptance and validation testing.
The company has partnered with DeviceAnywhere to deliver mobile application testing support and continues to fill out its overall tool suite. Growth in Europe has been especially strong over the past 12 months. While the toolset may initially appear relatively expensive, the ROI seen by customers seems to justify the expense — with more than 50% of their functional tests being automated, on average. As SAP users often struggle to keep up with upgrades and enhancements and adapt to more agile business practices, Worksoft's Certify Capture simplifies the capture of business processes by SMEs to produce test automation, while Certify Impact aids the identification of test cases affected by system changes. Users report seeing, on average, a 40% decrease in upgrade time.
Worksoft has been expanding automation support for Siebel, PeopleSoft, Syclo, Red Prairie, salesforce.com, Promax, Workday, Siemens TeamCenter, Maximo, Oracle Financials and EBS. In addition, with a number of successful implementations and a strong partner network, the company has also developed a set of best practices for TCOEs, and approaches to automation and upgrades.
In general, users rate Worksoft well, liking the breadth of functionality and the feeling of partnership around solving problems and attention to detail. The company is aware that it is important to recognize there is still a need for test engineers to aid in the definition of test cases, and that there have been some inconsistencies in the UI. Large-scale test systems also require a fair amount of server horsepower to manage large repositories effectively.
Strengths
- Full range of tools for script-free testing of SAP and other enterprise solutions
- High productivity and reduction of maintenance costs
- Reporting and documentation generation for compliance support
Cautions
- Consistency of delivery
- Must consider value of a single tool for all testing versus targeted tools
- Growing number of competitive solutions in the market
Vendors Added or Dropped
We review and adjust our inclusion criteria for Magic Quadrants and MarketScopes as markets change. As a result of these adjustments, the mix of vendors in any Magic Quadrant or MarketScope may change over time. A vendor appearing in a Magic Quadrant or MarketScope one year and not the next does not necessarily indicate that we have changed our opinion of that vendor. This may be a reflection of a change in the market and, therefore, changed evaluation criteria, or a change of focus by a vendor.
Added
Automation Anywhere, Telerik
Dropped
ITKO (acquired by CA Technologies), Green Hat (acquired by IBM), Seapine Software
Inclusion and Exclusion Criteria
Vendors in this Magic Quadrant must provide the ability to create, manage and execute functional test automation. Their tools must support the creation of software tests on the Windows platform to test Web applications (additional platform test execution support is desirable, and the most complete vendors cover a wide set of technologies). The vendors in this market must have a basic global presence, which means that they are actively selling their products in multiple geographies, and have at least $10 million in annual revenue. (Note: This market includes a set of mainstream or traditional players that cover the testing of desktop and Web applications, plus a set of companies from emerging or niche areas, such as SOA/external service bus, cloud and code-free solutions.) This Magic Quadrant looks at the entire market (traditional and new niches) as a complete market; however, that is not to say that the vendors in the Leaders quadrant are the only solutions that should be explored, or that looking at the market through more specific lenses would not change the shape of the market. It is important to keep in mind that a Magic Quadrant also emphasizes criteria that help assess where companies are with respect to being able to support an enterprise deployment to a Global 2000 company.
Evaluation Criteria
Ability to Execute
Because of the general maturity of the market, the Ability to Execute in a consistent fashion is critical and has been the defining attribute of the Leaders. This will continue with those that gain in the market providing a clean combination of technology with very clear market positioning. It is not enough just to have "better technology" than the incumbents. Better technology is still important, but generally opens short-term windows until the incumbents acquire or catch up. However, mobile and cloud are creating a great opportunity for market disruption. Another challenge will be the Ability to Execute in a market that is increasingly aligned to agile practices. It will be very difficult for companies to execute well with solutions that satisfy any use, which opens up the opportunity for focused solutions to move into leadership positions. A number of factors, including regulatory compliance, distributed and agile teams, are driving the need for strong partnerships across the life cycle and to service providers. Market pressure to reduce costs will continue to create openings for new tools and open-source solutions, and will force vendors to deliver clear ROI; but, this will result mainly in market expansion rather than replacement. At the enterprise level, company stability is also critical.
Source: Gartner (July 2013)
Completeness of Vision
While innovation is, in our view, the most important criterion, Completeness of Vision for enterprise use looks not only at innovation but at how complete the functionality and the company as a whole are. Direct marketing capability is becoming less important than community feedback and competitive pricing. The ability for a vendor to build community, show results and enable customers to drive positive change is critical. Vendors have one of three options: create strong innovation, target a specific market segment or build a comprehensive portfolio. Strong innovation is the most difficult to position and hold as the market has a general distrust of marketing claims and there is a strong need to be able to prove results. Innovation is hard to hold on to because the larger vendors are in a position to acquire or imitate. This means that innovators need to continue to find new areas of innovation or become relegated to niche roles over time. Fortunately, the Nexus of Forces (social, mobile, cloud and information) is providing a tremendous number of opportunities to innovate, and this will last for the foreseeable future. It will still be a challenge for new companies to sustain high levels of innovation and at the same time build strong execution. (During the next three years, mobile — of all the nexus forces — will have the greatest impact on the market.) But vision isn't purely about innovation, Leaders have a complete vision or an ability to support everything that a quality team will need. Vendors that support leading technologies and architectures with a clear vision of the shift this introduces to applications, and the complexity of testing those applications, are key here. In addition, vendors with a complete vision either provide a robust view of the product life cycle or have strong partnerships to help fill in those areas of the life cycle in which they are not involved. Overall, vendors still have gaps in their product lines, with most still focused on the core of automation of functional/regression and load/stress testing, as well as elements of quality management. Common gaps are in unit testing, integration to other areas of the life cycle, test data management and lab management facilities. Many of these gaps are filled through partnerships, but the Leaders are also increasingly filling these gaps. Thus, while the Visionaries and Niche Players will scramble to grab market share, the Leaders and more established players will consolidate through acquisitions as markets mature. These players are generally positioning to become Leaders not only in software QA but also in the overall ALM market, or they are strongly partnered to do so.
Source: Gartner (July 2013)
Quadrant Descriptions
Leaders
The market leaders all have broad portfolios covering test management, functional and performance test automation. Each also has support for additional quality measures such as code quality, database testing, and either deep platform or broad technology support. At its core, the testing market is quite mature with a strong stratification of vendors by annual revenue. These companies are well suited as the backbone of a testing organization's tool suite, and this is proven via a global presence and significant market shares. These vendors have significant partnerships and show consistency in execution of programs including acquisitions. However, note that, because of the breadth of customer bases and pace of change in the market, these vendors may at times lag in technology adoption which opens up space for competitors. In alphabetical order, the vendors in this quadrant are: Borland, HP, IBM, Microsoft, Oracle and Soasta.
Challengers
Challengers have a strong capability to execute on product delivery and sales and deliver strong but less complete functional coverage than the Leaders. We expect over time that companies in the Visionary quadrant will often settle into the Niche Player or Challenger quadrants, by focusing either on a specific category of testing functionality or on a market segment. CA Technologies is in this quadrant.
Visionaries
Visionary companies generally have a focus on a particular part of the software quality process (for example, cloud, load/stress, functional automation). Look to this segment for the best chance to find disruptive innovation or support for new technologies, but recognize that the products may not be as polished and that the companies at times will be stressed to support global enterprises as their sales teams and partnerships are less mature. In alphabetical order, the vendors in this quadrant are: Original Software, SmartBear, Telerik, Tricentis and Worksoft.
Niche Players
The vendors in this category either provide unique function to a specific market or use case or are early in establishing a position in what we note is a mature market. In alphabetical order, the vendors in this quadrant are: Automation Anywhere, Parasoft and SmarteSoft.
Context
The traditional focus in the application quality management market has been for specific testing activities (for example, load/stress and functional/regression). Economic conditions and pressure to improve time to market are placing increased emphasis on productivity. Agile development continues to make inroads into QA and testing teams, and this is reflected in the growing support of test planning tools for agile processes, such as the Scrum framework. Devices, cross-browser testing support and DevOps are placing stress on the testing organization to adequately test software and increase automation. On another front, packages, SaaS and business process management are driving greater business analyst involvement in overall quality efforts, along with a shift in focus from finding defects in validation to ensuring that business objectives are met. Testing software can be an expensive process, but poor software quality leads to user dissatisfaction as well as increased development and maintenance. Therefore, having a well-defined set of tools and practices to drive software quality will positively affect the bottom line of the business.
Market Overview
The software testing market is made up of several markets tracked by Gartner. The focus of this document is the distributed testing tool market, which is approximately $1.5 billion in size and growing at 6.5% (compound annual growth rate). However, we find that a number of providers are growing at 50% to 100% (annually), and we expect to see very strong growth in the following areas: mobile testing, and testing SaaS and PaaS solutions. Open source and tools supporting agile testing techniques will also do well, but will be a challenge to measure in terms of impact on the market.
The market is being driven by time-to-market issues, while seeing an increased need for governance and regulatory compliance. A surprising number of companies still are relatively immature in overall testing practices, with low rates of automation and relatively ad hoc test management practices. These factors create pressure around driving automation, improved workflow and traceability, and reporting and metrics capabilities. Some of the vendors in this Magic Quadrant play in the broader AD life cycle management market, providing tools for requirements, change and work item management. It is important to consider the other tools in the AD portfolio and how they fit together and support the development practices used by an organization.
While tools are important and necessary, they don't solve fundamental flaws in practices and skills, and many organizations need to improve organizationally before tools will provide maximum benefit. This can drive the importance of service partner relationships (outsourcing, SIs, consultants), which can aid in process transformation. This, in turn, means that vendors with good partner programs have an advantage, and also emphasizes the need for the improvements provided by the tools to guide decisions and practices. Many of the newer players are winning because they can provide hands-on support, improved ease of use or targeted solutions.
It is crucial to consider who will be using the testing tools and their skill set, as well as the types of applications or services to be tested, the development methodology, and the types of testing activities to undertake. Is the focus acceptance testing done by SMEs? This is very different from an agile team trying to find defects in a new mobile application. A focus on quality (rather than risk) may also drive a more expansive view of the activities as well as the tools utilized: static analysis, code review, unit tests, code-level automation, globalization and localization testing, and others are all required when either seeking specific compliance (such as with Section 508) or working to support standards such as International Organization for Standardization (ISO) 9126. Many organizations are more focused on risk management and maintaining a tight control on testing costs.
The Leaders have broad portfolios covering all aspects of software quality, and match this with a global presence and significant market shares. Their solutions are often not the most innovative products in the space, having time to wait for markets to mature or adding needed components via partnerships and acquisitions, but we are beginning to see visionary companies get near and cross into the Leaders quadrant. Visionary companies generally have a focus on a particular part of the software quality process (for example, cloud, load/stress, functional automation), but we often find that many of these companies and products are purchased as additions to tools from the market leaders. There is also strong innovation in the Niche Players space at this time, but these vendors either have limited geographic appeal or approach the market from a new direction. There is a lot of change going on in the market as an increasing number of development groups adopt agile practices. Note that, while application testing as a whole is a mature market, the continued entry of new technologies means that there has also been continued entry of new providers. We believe that during the next five years the market will continue to go through a consolidation cycle, with the potential for two or three more companies to move into the Leaders quadrant — driven by the shift to mobile and cloud applications — and there will also be acquisitions to accelerate growth and shore up aging toolsets.
The past 18 months have seen acquisitions by CA (ITKO), IBM (Green Hat) and SmartBear (eviWare), a number of partnerships for mobile technology support, and many new entrants to the market (there continue to be new entrants to the market that cover specific problems). There are many strong technologies that don't qualify for the Magic Quadrant, because of either breadth of functionality or revenue.
Some notable areas and tools are:
- Cross-Browser Testing: CrossBrowserTesting, BrowserStack
- Test Planning and Management: QMetry, SQS, Zephyr, Hexawise, Testuff
- Load and Performance Testing: Neotys, Sauce Labs, BlazeMeter, TestPlant (Facilita)
- Automation: Ranorex, TestPlant, Kalistick, Seapine Software (full ALM and test platform), PushToTest, eXept Software AG
- Mobile: Mobile Labs, Perfecto Mobile, TestPlant, Jama Software, Keynote's DeviceAnywhere, Experitest, Gorilla Logic
- Test Data and Test Lab: Informatica, Grid-Tools, Delphix, Skytap, Dell, VMware, Citrix (VMLogix)
- Package Software Testing: Panaya
