Data Center is our focus

We help to build, access and manage your datacenter and server rooms

Structure Cabling

We help structure your cabling, Fiber Optic, UTP, STP and Electrical.

Get ready to the #Cloud

Start your Hyper Converged Infrastructure.

Monitor your infrastructures

Monitor your hardware, software, network (ITOM), maintain your ITSM service .

Our Great People

Great team to support happy customers.

Friday, August 10, 2012

Network traffic analysis in a virtualized environment

David Davis

Over the years, network administrators have come up with tried-and-true methods for analyzing and troubleshooting the physical network, using SNMP and NetFlow for data collection or protocol analyzers to look at raw network frames and packets. But what happens now that we've moved into the era of virtual networks?  The good news is that existing network traffic analysis strategies can be used in virtual networks with just a few small differences.

Virtual networks are not so different

Virtual networks work very much the same as physical networks. In many cases, only the names of network devices have changed. For example:
  • A network interface card (NIC) is now a "virtual NIC" (vNIC).
  • A switch is now a "virtual switch" (vSwitch). vSwitches work very similarly to physical switches but don't have the config ability commonly found in traditional switches (such as showing the MAC addresses).
  • Multiple vSwitches can be created on each host, and ports on a vSwitch are usually broken down into port groups for specific purposes, such as production or management.
  • VLANs are fully supported, and a switch port can be either an access port or a trunk port, just like the in world of physical switches.
  • Physical hosts that house the virtual switches are connected to the physical network with real physical server NICs and cables, which are called "uplinks" in a virtual infrastructure.
  • Features like promiscuous mode, NIC teaming and load balancing all exist in a virtual environment.
These features have changed:
  • Spanning-tree protocol is not needed.
  • Network traffic cannot flow from one switch to another on the same host.
  • Port groups exist in the virtual network but not in the physical network (they may be similar to Cisco SmartPort).
  • You can't see the virtual switches or physical cables connecting to the vNICS (and there are no more flashing lights to look at in the wiring closet for most of the servers).

Virtual network traffic analysis using SNMP and NetFlow

Just as in the physical network infrastructure, to analyze network traffic in the virtual world you'll use SNMP or NetFlow to collect data across multiple points of the infrastructure and then analyze it with a network performance management and monitoring tool. Examples of more general network performance monitoring tools include What's Up Gold and Solarwinds Orion. More specific NetFlow collectors and analyzers include Plixer Scrutinizer and Solarwinds NetFlow Traffic Analyzer.
Of course you can still do a generic Internet Control Message Protocol (ICMP) monitoring with an element manager like HP OpenView, but it's preferable to do that in addition to some level of utilization and error checking.
Figure 1. Enabling NetFlow in vSphere
Figure 1. Enabling NetFlow in vSphere
Prior to vSphere5, utilizing NetFlow to monitor the virtual infrastructure was not an option. However, once you implement vSphere 5 (assuming you have the edition with the vSphere Distributed Switch) you can enable NetFlow v5 at the port-group level on an individual dvPort or on an uplink.
By doing so, you'll be able to monitor the following:
  • IntRA-host virtual machine traffic (virtual machine-to-virtual machine traffic on the same host)
  • IntER-host virtual machine traffic (virtual machine-to-virtual machine traffic on different hosts)
  • Virtual machine-to-physical infrastructure traffic
While SNMP will only give you basic statistics about the network traffic sent, traffic received and errors, NetFlow goes much further by providing IP pairings and protocols. In other words, you can see who the "top talkers" are and who is talking to whom. For example, with SNMP you might see that a network interface has reached its throughput capacity, but that's all. With NetFlow, you would see that HTTP is taking up 95% of the interface utilization, and that a specific user's PC (looked up by DNS) is talking to an Internet website that streams rock concerts, for example. None of these options will show you inside the packets or allow you to decode any data. In this blog, you'll find more detailed information on VMware's vSphere 5 NetFlow implementation.
Figure 2. Xangati for vSphere
Figure 2. Xangati for vSphere
One of the best vSphere network performance monitoring and troubleshooting tools  is Xangati for vSphere (which is free) and the Xangati management dashboard. Both versions use NetFlow to collect data about the virtual infrastructure but combine it with other traditional performance metrics from vCenter to provide a very powerful performance monitoring and troubleshooting tool for vSphere infrastructures. The free version monitors a single host, while the management dashboard allows you to monitor many hosts and virtual networks from a single interface.
Note that if you are using Hyper-V instead of vSphere, Microsoft has announced that in Windows Server 2012 Hyper-V, the extensible virtual switch will support the addition of the open source Hyper-V sFlow agent that could then be monitored by sFlow collectors, such as the InMon sFlowTrend Tool.

Virtual network traffic analysis with packet decode

What if you want to do a packet decode from the virtual network? In order to do Deep Packet Inspection (DPI) on a physical network, you would connect your protocol analyzer (which would be running on a laptop, for example) to a switch port and then configure SPAN (or RSPAN if the traffic is on a different switch) to mirror traffic from a single switch port, multiple ports or an entire VLAN.

More on networking for virtualization

Using vSphere networking tools to gain control of virtual environments
vSphere vSwitch  design concerns
The storage and network hypervisor are the new virtualization
Now that most of our data centers' servers are virtualized, much of the traffic does not even hit the physical network -- so the traditional packet capture method is only useful in certain instances, such as analyzing your Internet connection or a connection to an iSCSI SAN.
Prior to vSphere 5, to use a protocol analyzer on the virtual infrastructure you took a VM running a protocol analyzer, created a new port group, configured it for promiscuous mode (so that all packets are sent to all ports), and then moved the VM that you wanted to analyze to that port group (for security reasons you don't want to enable promiscuous mode on a production port group). For details on that, see my post titled "Using a Network Packet Analyzer on a VMware vSphere Virtual Network" (which you would follow if you are still using vSphere 4.x or if you have vSphere 5 but don't have the distributed virtual switch running).
In the vSphere 5 Enterprise Plus, however, the port-mirroring functionality allows you to quickly and easily mirror any dvPort to another port, or you can choose a VLAN to encapsulate these mirrored packets by selecting the "Encapsulations VLAN" box when configuring distributed virtual switch port mirroring.
Once enabled, port mirroring provides visibility into:
  • IntRA-host virtual machine traffic (virtual machine-to-virtual machine traffic on the same host)
  • IntER-host virtual machine traffic (virtual machine-to-virtual machine traffic on different hosts)
Figure 3. Configuring vSphere 5 port mirroring
Figure 3. Configuring vSphere 5 port mirroring
To see how to configure vSphere port mirroring, read this VMware blog post on port mirroring features.
If you are a Hyper-V user, note that in Hyper-V 3 port mirroring is a new feature of the extensible switch.
Analyzing and troubleshooting the network once your servers are virtualized really isn't that different from performing these same tasks with physical servers in the physical network. You have two different paths to get this done depending on the level of detail you need. Using NetFlow is the best choice for high-level traffic analysis and bottleneck identification, where port mirroring with a protocol analyzer is what you would do to perform deep-packet analysis in the virtual infrastructure.
About the Author: David Davis is the author of the best-selling VMware vSphere video training library from Train Signal. He has written hundreds of virtualization articles published on the Web, is a vExpert, VCP, VCAP-DCA, and CCIE #9369, with more than 18 years of enterprise IT experience. His personal Website is

Wednesday, August 08, 2012

IT Management: Moving from Plan-Build-Run to Demand-Supply-Execute

I want to outline briefly some ideas I’ve had recently about our fundamental models for understanding IT management.
In 1968, Melvin Conway proposed an insightful law, basically stating that our systems are copies of our communication structure (how we interact as human beings). And while this law is often applied in discussions of computer program structure, his intent was broader, including any human created system.

In the broad system of IT management, we often hear of the basic three stages, “Plan-Build-Run.” We make plans for a new IT system, we construct it, and we run it. IT organizations structure themselves and communicate to a large degree along these lines. I’ve used them myself in any number of writings. However, increasingly I have come to believe that that as a basic structure for understanding IT management, “Plan-Build-Run” (which I’ll abbreviate PBR) has seen its day.
While it pervades IT management thinking, and can clearly be seen in the structures of frameworks like ITIL and COBIT, PBR is ill suited for the demands of 21st century IT management. Here are some of the problems I believe that the plan/build/run paradigm causes:
  • PBR encourages waterfall thinking, the idea that a complex system just needs to be designed, constructed, and operated. Agile trends turn this from a one time effort into an ongoing cycle, and this cycle is accelerating, challenging traditional IT organizations and methods.
  • Demand is split across the PBR phases. Because we don’t focus on integrated demand our customers have to knock on multiple doors to get the services they need.
  • PBR results in systems being built in isolation from one another. We thus lose opportunities to implement shared services and ensure architectural consistency across the IT estate. The result is complexity that is difficult to move to Cloud sourcing as an IT supply option.
  • Because we don’t see execution as one problem we are unprepared for DevOps, our people are multitasking, and our failure to prioritize across queues is resulting in poor service to the business.
  • Because we don’t view supply holistically we manage IT staff and their talents poorly, with little overall understanding of the dynamics between technical talent, technology products, and IT services.
What should replace PBR? I propose a new triad: Demand, supply, and execution (DSE).
These are not novel concepts. Supply and demand stem from fundamental economics and operational management. Execution is a widely accepted industrial term that covers the translation of supply and demand into value, via detailed resource and capacity management, dispatching, process monitoring, and performance analysis. The Wikipedia article on Manufacturing Execution Systems isn’t bad. I’ve also gained some insight from this article on ANSI/ISA-95.
The DSE concepts are distinct from (some might say orthogonal to) PBR.
Demand management starts from the premise that regardless of the size of the implied work, all demand on IT resources should be understood in a unified manner. From a new mobile device to the day’s incident reports and change requests, to a strategic initiative implying a $10 million projects, it’s all “just demand.” Different techniques come into play depending on value, scope, risk, and other parameters, but if we understand demand as a unified entity we position ourselves to provide much better service to our stakeholders while at the same time giving our IT staff a saner existence.
Supply boils down to the fundamentals of “atoms, bits, and cells”: hardware, software, and people, under various ownership and sourcing models. The CIO is responsible for increasingly complex IT sourcing and contract management strategies, and understanding one’s baseline supply is key to evaluating new supplier options for technology products and people with the skills to exploit them.
Finally, next generation IT execution management starts with demand and supply generally, and looks for optimal (or at least satisfactory) means of delivering value. “Projects” and “tickets” are seen as part of a unified management structure, not as occasional passersby. The availability of resources is always considered before releasing work, and ongoing scenario-based forecasting is employed to identify emergent constraints. And time tracking is completely transparent, relying on intelligent automation to determine what people have actually been working on. No Friday afternoon time reconstruction!
The DSE model is not intended not as a criticism or replacement of ITIL or COBIT or any other framework, but rather as an overlay, or perhaps an underlay. Well established IT process areas such as project, release, incident, change, and so forth are important and will continue, but a DSE approach can counteract the tendency to form functional silos around each, and instead promote a whole-systems approach to IT management.
To summarize Keynes, “even the most practical man of affairs is usually in the thrall of the ideas of some long-dead economist.” Basic conceptual structures like plan/build/run and demand/supply/execute have consequences. When widely adopted to the point where they are just “common sense,” they define our social relationships, operational thinking, problem solving, and more. And therefore, while we may think that “plan/build/run” is some form of IT natural law, it is a human construct that can be adapted or even discarded if we no longer find it useful.
Next week: The results of the EMA Unified IT Demand Management survey
Charlie Betz is Research Director at Enterprise Management Associates. His responsibilities include IT portfolio management, IT financial management, asset management, ITSM suites, and integrated IT management.

He has 20 years of experience across all aspects of enterprise IT practice, including 6 years at Wells Fargo as Senior Enterprise Architect and VP for IT Portfolio and Systems Management. He has also held architect and application manager positions for Best Buy, Target, and Accenture. He is author of the recent 2nd edition of Architecture and Patterns for IT: Service Management, Resource Planning, and Governance (Making Shoes for the Cobbler’s Children). Follow Charlie online via Twitter and Linkedin.
For more information, download Nimsoft’s free eBook: The Definitive Guide to Unified IT Management.