Data center security concerns with contractors: Advisory Board Q&A
It's one thing to train and keep track of your own IT staff, but watching contractors can be a challenging problem for businesses and IT managers. Sooner or later, you're going to need new servers rolled into the data center; heating, ventilating and air conditioning repairs; electrical and mechanical system repairs; and upgrades. All of these tasks require you to allow third parties within touching distance of your company's most valued IT assets.
But IT professionals are already spread thin with various projects. There's hardly time to fight the day's fires, much less watch contractors. In this month's column, we asked our Advisory Board members about the real data center security risks posed by third-party contractors, the value of background checks and references, holding contractors accountable for their actions and the practical steps to maintaining data center security and preventing damage to equipment and data.
Bill Bradford, senior systems administrator, SUNHELP.orgThe risks are very real for us. For example, we have source code for our primary software product offerings stored on systems in the data center. If someone knew what he or she were doing, a storage device could be smuggled in or out.
Generally speaking, we have not had problems with background checks and references. So far, our vendors' internal processes have been sufficient. But any damage or data loss caused by a contractor or service technician is the direct responsibility of their employer, period. If they can't fix it, they're responsible for contacting someone who can. Fortunately, the only problem we've had like this so far in my experience was a contractor hitting the “big red off switch,” thinking it would open the door.
The best way to prevent damage and maintain data center security is vigilance, even when time and staffing is short. All third-party contractors, vendor representatives and service technicians are escorted by an employee when in the computer room or data center. Nobody is allowed into the facility without proper authorization or an escort.
Generally speaking, we have not had problems with background checks and references. So far, our vendors' internal processes have been sufficient. But any damage or data loss caused by a contractor or service technician is the direct responsibility of their employer, period. If they can't fix it, they're responsible for contacting someone who can. Fortunately, the only problem we've had like this so far in my experience was a contractor hitting the “big red off switch,” thinking it would open the door.
The best way to prevent damage and maintain data center security is vigilance, even when time and staffing is short. All third-party contractors, vendor representatives and service technicians are escorted by an employee when in the computer room or data center. Nobody is allowed into the facility without proper authorization or an escort.
Robert Rosen, CIO, mainframe user group leaderWhen it comes to security risks from third-party contractors or others working in the data center, it all depends on whether you do a proper background check–you could have anyone from a terrorist, to a hacker, to the world's best operator. It is important to either do the check yourself or have it as a contractual requirement with significant failure penalties so the contractor is responsible. You also have to decide how to handle pure accidents as well as deliberate actions.
In terms of practical steps to maintain security and prevent damage to equipment and data, there are numerous tools, such as closed-circuit television, supervision by IT staff and so on. Really, none of this is rocket science, just good practices, but I'm constantly amazed at how many people ignore even the simplest of precautions. Think of it as though you were bringing a contractor into your house. What precautions would you take? It's no different just because it's your employer's property.
In terms of practical steps to maintain security and prevent damage to equipment and data, there are numerous tools, such as closed-circuit television, supervision by IT staff and so on. Really, none of this is rocket science, just good practices, but I'm constantly amazed at how many people ignore even the simplest of precautions. Think of it as though you were bringing a contractor into your house. What precautions would you take? It's no different just because it's your employer's property.
Robert Crawford, lead systems programmer and mainframe columnistThe security risks are the same for both contractors and employees. Both types of personnel must be trustworthy, competent and conscientious. The difference is a company has more leverage and direct access to an employee. If a contractor causes problems, the customer has to work with the company that sent him.
Background checks for employees are certainly appropriate. For contractors, a company might do better to ask the vendor for a copy of the background checks they should have already done. If the vendor hasn’t checked its employees, it should be removed from the bidding process. This also points out that customers may do better to stick to larger, well-established maintenance contractors who are likely to get and keep good employees.
Indemnity for damages, physical or logical, will have to be spelled out in a well-written contract. However, no matter how good the contract, to get redress, a customer will have to deal through intermediaries and possibly lawyers.
Normal data center security [procedures] should apply. Everyone has to have the proper credentials to get into the machine room. Cameras and intrusion alarms would be useful too. If a company has enough data center real estate, they might think about segregating the hardware types into separately controlled rooms. For example, keep the server hardware guy out of the direct-access storage device farm. Maintenance should be done during off-peak hours.
Bill Kleyman, director of technology, World Wide Fittings Inc.About three years ago I had hired an outside consulting firm to arrive on-site and install a physical virtualization platform, since all of my other resources had been tied up. The project was simple–roll out a new piece of hardware, patch it, update the service packs and deploy into the environment. From there, the project required a hypervisor to be installed on the machine, and a SQL Server virtual machine to be deployed. The project was detailed by our engineers, and a scope of work was drawn up by our staff and handed over to the consulting firm. The consulting firm we contracted is a major, national service provider that, on normal occasions, has been more than able to provide talented staff.
The engineer arrived and began setting up his laptop. He asked for the physical server, some network information and the installation media. About three hours into the project, the engineer was still looking at the primary installation screen for the hypervisor. It takes about 10 minutes to complete the entire process. When I asked him if there was a problem, he replied, "I've never installed this in a live environment. I've only installed this hypervisor in a test environment, and it had local storage and not a storage area network."
Security concerns aside, the engineer may not know what they are doing at all. I've learned to question the consulting engineer prior to them starting work on my environment. Yes, there are security risks, but if the engineer doesn't know the technology, they may do more damage accidentally instead of on purpose. Sometimes it's worth it to spend the extra time monitoring or working directly with the engineer rather than let them go on their own.
It is absolutely worth making background checks and checking references. I've learned my lesson and will interview every consultant coming into my environment. Even if it's a small project, I don't need any residual issues that may arise because of their work. I would rather take the time now to ask questions than have to answer for the mistakes of the engineer later. Talk to a reference and make sure the consultant's background fits into what you are trying to accomplish in the data center.
The engineer arrived and began setting up his laptop. He asked for the physical server, some network information and the installation media. About three hours into the project, the engineer was still looking at the primary installation screen for the hypervisor. It takes about 10 minutes to complete the entire process. When I asked him if there was a problem, he replied, "I've never installed this in a live environment. I've only installed this hypervisor in a test environment, and it had local storage and not a storage area network."
Security concerns aside, the engineer may not know what they are doing at all. I've learned to question the consulting engineer prior to them starting work on my environment. Yes, there are security risks, but if the engineer doesn't know the technology, they may do more damage accidentally instead of on purpose. Sometimes it's worth it to spend the extra time monitoring or working directly with the engineer rather than let them go on their own.
It is absolutely worth making background checks and checking references. I've learned my lesson and will interview every consultant coming into my environment. Even if it's a small project, I don't need any residual issues that may arise because of their work. I would rather take the time now to ask questions than have to answer for the mistakes of the engineer later. Talk to a reference and make sure the consultant's background fits into what you are trying to accomplish in the data center.
In terms of accountability, always have a contract ready whenever you use an outside consultant. I will always have a non-disclosure agreement, as well as a service-level agreement, prepared for work done on-site. Most of those documents will clearly state that if we are not satisfied with the work, we don't have to pay for it. Fortunately, that is precisely what happened in our situation with the engineer who was less than competent in completing our virtualization project. I called his contracting company and explained that it would be unfair to bill us for work that was never done, even though the engineer had spent close to seven hours on-site.
There are a few simple steps IT managers can take to make sure their environments are prepared for outside consultants. First, make sure security permissions, such as file/folder permissions, shares and network resources, are set. Next, set up locked accounts for consultants and make sure the account created for these outside consultants is monitored and restricted to only what the engineer needs access to. Keep an eye on them--every once in a while, ask them how they're doing and see what they are doing as well. And finally, record their session. If they're working on a Citrix platform or any other type, IT managers can record screenshots and keystrokes for every session.
Robert McFarlane, principal and data center design expert, Shen Milsom Wilke Inc.For years, I've heard people express concerns about vendors (and common carriers, in particular) intentionally sabotaging competitors' equipment, but I've never actually seen or heard of it happening. What I have heard of are internal personnel–primarily executives–feeling entitled to go anywhere they want, and while taking people on tours through the data center, touching something they shouldn't and turning off a device or pulling a cable. Or worse yet, hitting the dreaded emergency power-off switch, thinking it’s the door exit button. The biggest potential problem is people accidentally doing bad things, while doing what they're supposed to be doing, or people who shouldn't be in there at all doing stupid things. The intentional [disruption] is much harder to guard against, because it's usually a disgruntled employee who hides his or her intentions until it's too late.
For people who will be in the data center regularly, background and reference checks certainly couldn't hurt, but it's impractical for every person who may need to make a one-time entry. In those cases, if there's a concern, there's no choice except direct supervision and the authority to remove someone from the site if they appear to be bungling, incompetent or otherwise of concern to the operation. "Authority" is often the missing link here–by the time someone goes up the chain of command, it is usually too late.
Unfortunately, it's pretty difficult to hold third-party contractors accountable for accidental damage while they’re on-site. Most of the problems that occur involve outages that cannot be quantified in dollars, and most contracts preclude liquidated damages anyway. You can certainly insist on maintenance agreements that require a vendor to reimburse the costs of personnel time needed to restore service or correct a problem caused by a technician. And, of course, a tech that causes a problem can be banned from the site and the vendor may be deleted from future consideration, But I know of no situation in which they have been held accountable for the "real" cost of damages beyond equipment replacement and staff costs.
There are steps you can take to maintain security and prevent damage to equipment and data. For example, video is important. Everyone should know there is a record being made that will show that they put their fingers where they shouldn't have been, or that they were sloppy in doing their own work. But that's "after the fact."
For people who will be in the data center regularly, background and reference checks certainly couldn't hurt, but it's impractical for every person who may need to make a one-time entry. In those cases, if there's a concern, there's no choice except direct supervision and the authority to remove someone from the site if they appear to be bungling, incompetent or otherwise of concern to the operation. "Authority" is often the missing link here–by the time someone goes up the chain of command, it is usually too late.
Unfortunately, it's pretty difficult to hold third-party contractors accountable for accidental damage while they’re on-site. Most of the problems that occur involve outages that cannot be quantified in dollars, and most contracts preclude liquidated damages anyway. You can certainly insist on maintenance agreements that require a vendor to reimburse the costs of personnel time needed to restore service or correct a problem caused by a technician. And, of course, a tech that causes a problem can be banned from the site and the vendor may be deleted from future consideration, But I know of no situation in which they have been held accountable for the "real" cost of damages beyond equipment replacement and staff costs.
There are steps you can take to maintain security and prevent damage to equipment and data. For example, video is important. Everyone should know there is a record being made that will show that they put their fingers where they shouldn't have been, or that they were sloppy in doing their own work. But that's "after the fact."
There is no way to fully prevent human error, but some best practices can help. First, restrict access to only those who legitimately need to be in the data center. Conduct training and orientation for each new person who needs to be inside, and update the training as changes are made in the facility. Let everyone know that they are being video recorded when inside. Watch even the most regular and experienced people (vendors as well as employees) often enough and long enough to see if they're getting careless or are developing a poor attitude. Put escorts with all new technicians, and give them the authority to remove people from the data center if they see fit. In addition, if you use a sign-in procedure, it must be 100% enforced for everyone, at all times, upon entering and exiting. Nobody should be above it, and it should be monitored so there is a counter-signature proving that an authorized person knew the individual entered.
Michael Coté, analyst, RedMonk
There’s no excuse for being lax about your data and your data center. In the same way that backup tapes left on the back seat were an almost weekly occurrence years ago, data breaches are now a common occurrence. Some possible areas of concern include installing sniffers on networks, or otherwise preparing the space for future attacks, hacking into machines while on-site, getting the lay of the land for hacking from the outside (e.g., "They use these older routers, storage and servers that we know have these vulnerabilities ... ,"). There are other concerns, such as someone becoming friends with [an employee] and then exploiting that for future attacks, along with industrial espionage or sabotage. The list goes on.
Background and reference checks are always a good idea. You want to check on the people who'll have access to your servers. Having someone in the same space as your servers and hardware isn't like inviting a plumber over to fix your toilet. They need to be aware of things to avoid so that they don't screw things up, like unplugging cords. Second, they must be trusted when it comes to all of the paranoid, anti-hacking stuff you're worried about. The contractor's accountability is also an important consideration, but that's typically handled through the use of contracts and lawyers.
When maintaining data center security and preventing damage, I've noticed that simple things help, such as drawing the lanes it's safe to walk in on the raised floors. Don't be afraid to use signs, and even cones, to mark off areas for people to avoid when they're wandering around those flashing lights. Establishing policies about who can go where is key. You need to log people who come in and out of your data center and never be lax about the access policy and procedure. So much of this stuff results from inside jobs or people who've been compromised, so you have to remember to never skip policy just because that person is "one of us."
Add quality assurance to the data center security protocolMcFarlane also pointed out that quality assurance should also play a role in gauging the integrity and performance of your contractors. For example, all work performed by a contractor should be inspected by an IT employee knowledgeable in that area. Sign off before anyone is allowed to enter or leave the data center. This allows the organization to exercise more oversight in the project or task, and also benefits the contractor by ensuring the work was completed in a satisfactory manner before exiting the premises.
