Why Your Business Needs On-Site Testing
Written by Chris CooperJune 23, 2014
Most recovery plans include off-site IT or data tests which involve recovering key systems and data from an off-site location. These tests protect critical information but overlook vital resources that keep your business going after a disruption has occurred such as people, locations, communications, and supplies. Disruptions impact every part of your organization. To truly be prepared, you have to test all of your company’s resources onsite. This allows you to address critical issues that go beyond data restoration such as how to resume operations if your physical structure or phone system has been destroyed.
On-site testing enables you to test recovery plans using real-life scenarios, which is far more effective. During an onsite test, mobile recovery stations and equipment are delivered to your business. Employees implement solutions to serve customers as if a disaster had actually occurred. This more realistic approach to testing benefits your overall business continuity strategy in three key areas.
Engaging senior ex
ecutives
ecutives
Onsite testing allows C-level executives to partner with continuity professionals and IT managers to reduce risk. This collaboration helps upper management see the value of onsite testing, evaluate company operations within a controlled environment, and identify and implement ways to improve the efficiency of day-to-day functions.
Lowering compliance risk
A multitude of laws and regulations require businesses to test their plans, particularly in the financial, healthcare, and utilities industries. Testing has become a key focus for examiners during audits, and regulators require companies to provide test results that demonstrate viable, executable plans. Companies that don’t provide proof of testing risk incurring penalties, failing audits, and having to scramble to meet examiners’ expectations.
Even if testing isn’t required by law or regulation, suppliers, partners, investors, and customers often have concerns over data integrity, availability, and internal controls, which can place additional demands on your organization. The ability to show that you can ensure the continuous availability of information will put your stakeholders at ease.
Data security and business continuity are separate, but they present a shared concern. Security breaches can cause an organization to execute its recovery plan. Even if regulations don’t require plans to be tested, an organization remains accountable for its data-related systems and processes.
Being better prepared
According to research firm Janco Associates, nearly 67 percent of organizations reported that errors in planning accounted for disaster recovery failures. Testing at your facility provides a more complete assessment of your organization’s risks and recovery capabilities.
During an on-site test, a mobile recovery facility, satellite, equipment, and desktops are brought to your building. Continuity professionals and IT managers create solutions to potential problems such as where customers will park, how to reconnect systems, and where to station the mobile recovery unit.
Without onsite testing, you risk overlooking gaps in your disaster recovery plans. Attempting to address these flaws in the midst of a disruption can lead to failure.
CHALLENGES
Lack of support from C-level execs
One of the most difficult issues for continuity professionals and IT managers is obtaining and maintaining support from C-level executives who are evaluated primarily on their ability to generate profits and grow the business. Many senior executives view business continuity as a low priority and onsite testing as an unnecessary expense. To convince C-level executives to pay for an onsite test, you’ll need to do the following:
- Show that benefits outweigh cost. To secure executive buy-in, you need to demonstrate how onsite testing achieves business goals. Onsite tests enable businesses to satisfy supply chain demands, regulatory requirements, service-level agreements, and fiduciary responsibilities to shareholders.
- Position onsite testing as a sales tool. Suggest publicizing your test efforts. Prospective and current customers who know that your company is prepared for business interruptions will feel more confident doing business with you.
- Speak their language. Most senior executives understand risk management, so present onsite testing as a way to reduce risk to your organization and its stakeholders. Cite specific examples of situations that could prevent your business from serving customers (e.g., a database failure or power outage) and then offer onsite testing as the solution to preparing for those scenarios.
- Stop saying “disaster recovery testing.” When senior executives hear the word “disaster,” they think of natural catastrophes that rarely occur. These unlikely events are a low priority for CEOs. Instead, use terms such as “onsite testing” or “testing the business continuity plan” and emphasize the need to prepare for common man-made disasters like fiber cuts, telephone outages, and cracked water mains.
Too disruptive
Some organizations assume onsite tests disrupt business operations for several days. They believe operations must be shut down to bring up data and applications for the test. Experienced vendors can conduct onsite tests within 24 hours and do so with minimal disruptions to the business. The first onsite test might take more time as you identify the parts of your plan that don’t work and develop solutions to these challenges, but future tests take less time.
Too expensive
Many companies assume onsite testing is too costly. Testing doesn’t have to be pricey if you use the right vendor. When vendors don’t own their resources, they outsource much of the testing process and pass down that additional cost to you, the customer. On the other hand, vendors that manage onsite testing in-house have less overhead. As a result, they can offer a more complete testing solution at a lower cost and easily estimate those costs so you can budget accordingly.
THE FIVE STEPS OF ONSITE TESTING
There are five key steps you should follow to help your business realize the full benefits of testing.
Set clear objectives
Deciding what constitutes a successful test is an important first step. Your goals can be specific or more abstract. Either way, they should be clearly defined and written down. Sample objectives include these:
- Building confidence throughout the organization that resilience and recovery strategies will satisfy business requirements;
- Demonstrating that critical IT processes can be maintained and recovered within agreed-upon service levels or recovery objectives regardless of the incident;
- proving that critical services can be restored to pre-test states in the event of a disruption;
- providing staff members with an opportunity to familiarize themselves with the recovery process;
- training staff and ensuring they have adequate knowledge of disaster recovery plans and procedures;
- verifying that disaster recovery plans are synchronized with business requirements;
- identifying opportunities to improve business continuity strategies and processes
- and providing auditors with evidence of testing.
Plan the test
Developing a plan is the only way to ensure a successful test. You should create and follow plans step-by-step to help ensure accurate testing. These steps include three things:
- Identifying the IT systems and business processes to be restored as well as the personnel who will execute recovery plans;
- Working with the IT department to determine how to connect hardware such as desktops and communications equipment; and
- Determining where the test will take place.
Hire a disaster recovery coordinator
A disaster recovery coordinator oversees the onsite testing process and can help make the process easier and more efficient. They manage the process like an orchestra conductor manages his musicians. And like any good conductor, they understand how all the elements work together to make beautiful music. DR coordinators know all the requirements of an onsite test, including data and voice connectivity, desktop images, and essential software. They make important decisions like where to park the mobile unit and how to route calls.
Some businesses appoint an employee to be the project manager during a test. Staff members often make great project managers but often lack knowledge about key aspects of testing such as moving operations to a different facility, replicating data, imaging desktops, and restoring network and electrical and voice circuits.
Analyze test results
After conducting the test, review the results to determine what worked correctly, what went wrong or not as expected, what areas can be improved, and what adjustments need to be made to your disaster recovery plan. Test results could show a missed recovery time objective or reveal that employees need further training in order to carry out tasks within the recovery plan.
Don’t expect the test to be error-free. It’s better to make mistakes during a test than in a real disaster. In fact, if the test doesn’t reveal flaws in recovery plans, it wasn’t thorough enough. Testing enables you to fix problems within a controlled environment rather than during an emergency situation.
It also enables you to increase your company’s overall efficiency by applying the lessons learned during the test to your everyday operations.
Retest regularly
An organization’s infrastructure is constantly changing. Servers, applications, and systems are added, modified, and removed. These changes can render your disaster recovery plan outdated.
Changes in hardware can be overlooked. When installed, new hardware is tested once. If the primary software has not changed, regular testing after the new hardware is installed might seem unnecessary. But if any new hardware has been installed or upgraded and the software hasn’t been tested on the new platform, cracks can emerge.
Finally, without consistent retesting, employees can lose interest in or forget important parts of your recovery plan. The more you retest, the better your team performs in the face of a business interruption and during routine activities.
Chris Cooper is the senior director of operations for College Station, Texas-based Rentsys Recovery Services, a provider of comprehensive disaster recovery services for banks, credit unions, mortgage lenders, and other organizations. Cooper has more than 20 years of experience in the information technology industry and was involved in the development stages of the company in 1995 to offer organizations a complete range of disaster recovery solutions. For more information, please visit rentsys.com.
