Defining the cloud Part 2: Self Service

In Defining the cloud Part 1: Applications, we discussed how applications are the reason for the cloud, and how we move abstraction from the servers to the applications.  Moving forward we now look at how the cloud should enable users to provision their own systems within given parameters.

Self Service

In the early days of virtualization, we were very excited because we were making IT departments more efficient. I remember IT managers actually telling young server admins to stall when creating virtual servers to prevent users from realizing how quickly it could be done. What took the IT department hours, weeks, or months previously, now was done with the press of a button, and a few minutes, assuming proper capacity planning.

IT is often seen as a cost center. For years now we have been preaching the gospel of IT As A Service, basically the concept that technology becomes a utility to the business. Nicholas Carr championed this concept in his book, The Big Switch. Basically he popularized the concept that much like electricity, technology was becoming something that should just work. IT is no longer just for those of us who understand it, but rather it becomes a tool that anyone can use just like flipping a switch to turn on a light, or turning on the TV.

So how do we make this happen? It is as simple as looking at the smart phone you have in front of you or in your pocket.  The thing that makes your phone so great is not the brand, not the operating system, not even the interface, the most important thing is the application ecosystem.  I can go on my phone and grab an app to do just about anything.  I don’t need to know how the system works, I just go grab apps and don’t really think about how they interact with the phone.

Imagine giving this to to our end users, simply give them an catalog to say what they need, a user wants to build an application, so they go to a catalog select from a pre-defined template, and the rest is handled by the system.  No IT intervention, no human interaction required, just a few simple clicks, almost like grabbing an app on a phone.  Their entire virtual infrastructure is built out for them and they are notified when it is complete.

So what does this all have to do with HP?  Stick with me on this, this is the future, this is HP Helion, and this is amazing.

Defining the cloud Part 2: Self Service

Defining the cloud Part 1: Applications

With the recent launch of HP Helion, and with HP Discover coming in a few weeks, it is a good time to talk about the difference between private cloud and virtualization.

Interestingly enough most companies assume that because they have virtualized 70-80% of their applications they have deployed a cloud environment.  This is the unfortunate result of marketing and sales people trying to move products or ideas without having an understanding of where we are headed, and what is happening in the market.  I confess I am guilty of this to some extent, I have written about private cloud being virtualizaiton, which is correct but incomplete.  So just what is the difference?  Well that largely depends on who you ask, but here is my take.

Application Centric

Virtualization came about to fill a gap in technology.  We were at a point where servers were more powerful than the applications, and so there was a great deal of wasted space.  When VMware started their server virtualization line, the big value was consolidation.  There was little to do with applications, it was about server consolidation, datacenter efficiency, and moving the status quo to the next level.  The application were only impacted in that they had to be supported in a virtual environment, but high availability, performance, everything was managed at the virtual server level similar to how it was managed at the physical server level previously.

In the cloud, abstraction is done at the application level rather than the server level.  the cloud rides on server virtualizaiton, but ideally applications should scale out, using multiple servers in a cluster each doing the same function with manager nodes to direct traffic.  While this may seem less efficient, since there needs to be multiple nodes to operate a single application, it actually frees the application from needing to reside in a specific datacenter or on a specific host, and indeed it should span multiple of each.  It also enables rapid scaling of applications since rather than adding additional physical CPU or Memory to the virtualized systems, you simply spin up additional resource nodes, and then when the peak demand is gone, you can tear them down.

So the first difference between virtualization and private cloud is the focus on applications rather than infrastructure. As we continue to explore this, I hope to tie this together and help to explain some of these differences.

Defining the cloud Part 1: Applications

Lead, follow, or stop trying to sell technology

“Leadership is a choice. It is not a rank.” —

If you haven’t seen the Ted Talk on leadership by Simon Sinek, this is a must see. I do enjoy watching these, but usually they are not this impactful. “Leadership is a choice”. So why would anyone choose to lead when it is so much easier to just follow the rules, get promoted, and become a manager, and what does all this have to do with technology and things that interest me, and hopefully you.

Sitting at breakfast with two friends, one in sales, one in pre-sales, we were talking about what it would take to be really great at our jobs. We came to the conclusion that the great tech sales companies generate excitement. If you remember the 80’s, Pontiac had a successful ad campaign around We Build Excitement They did not build excitement BTW.

What we need in the IT industry are motivational speakers. No one likes when someone comes in and does a powerpoint product pitch. I have actually seen people fall asleep in such sessions. Not ideal. In order to lead, we have to get in and generate some excitement, show people why the technology is exciting for them. Instead of trying to make products that fit our customers environments, we need to build the products that our customers don’t know they need and get them excited about following us on the journey. We are changing an industry here, imagine if Steve Jobs had made the phones most people wanted. Instead he came out with a product no one wanted and created a culture around it and an entire industry was born.

So my challange to you, look at your vendors if you are a customer, look at your products if you are a vendor. Are you excited about what you are selling or what you are buying? If not, maybe you are not talking to the right people. Don’t buy a product because it is safe, don’t buy based on what you have always done, take some interest, get excited, and buy ino a strategy.

Lead, follow, or stop trying to sell technology

Software Defined Storage Replication – vSphere Replication

When VMware introduced some of the storage API’s, some of the more important ones were around storage. I don’t say this because I am a storage guy, but more because this is an area that frequently causes problems for virtualization. When VMware released their backup API, features such as Change Block Tracking, CBT, became particularly compelling. Now vSphere could tell the backup software what had changed since the last backup, rather than relying on the backup catalog to look at everything. This created less reads on the storage, and more effecient.

It was not a huge leap then when vSphere replication was released as a standalone product separate from Site Recovery Manager, SRM, as well as a function of SRM. Prior to this, vSphere would rely on the underlying storage to do the replication, but now replication could be handled by vSphere itself.

One of the challenges with replication has traditionally been bandwith. To handle this we have used compression, caching, deduplication, and sending only the changed data to the remote site. When VMware introduced CBT for backups, this enabled them to later release replication using the CBT technology since they were already tracking changes and could use those for replication as well as backups. This would like like the diagram below.



In the previous post, Software Defined Storage Replication, the discussion was around abstraction through a third party product, in our case the HP StoreVirtual VSA. In this case, we are looking at a built in product. Much like VSAN, this is a solid way of doing things, but the challange comes in because it only replicates vSphere, unlike 3rd party products.

The other consideration here is the efficiency of doing things in software versus hardware. Of course the abstraction does have efficiencies from an operational sense, management is built in to the vSphere console, and you are not tied to a specific storage vendor. On the other side, we do need to look at the inherent performance benefits of replicating between two block storage arrays. Anytime we do anything in hardware, it is going to naturally be faster.

One thing VMware has done very well is providing solutions based in software to compete with hardware vendors. This does not mean that VMware does not partner with hardware vendors, but for some customers the price is right for this. VMware almost always tells us to use array based replication for maximum performance, but this is a good solution for abstraction or a smaller environment where cost is a larger factor over performance.

Software Defined Storage Replication – vSphere Replication

Software Defined Storage Replication

In a conversation recently with a colleague, we were discussing storage replication in a VMware environment.  Basically the customer in question had bought a competitors array, brand X, and wanted to see if there was a way to replicate to one of our arrays at a lower price point.

This is a fairly common question coming from customers, more so in the SMB space, but with the increasing popularity of Software Defined Storage, customers want options, they don’t want to be locked into a single vendor solution.  In an openstack environment, high availability is handled at the application level, and I have to say I strongly recommend this as a policy for all new applications, however how do we handle legacy apps in the interim?

In a traditional storage array, we typically do replication at the storage level.  VMware Site Recovery Manager allows us to automate the replication and recovery process integrating with the storage replication, and in smaller environments, can even handle replication at the vSphere host level. Array based replication is generally considered the most effecient, and the most recoverable. This does require similar arrays from the same vendor, with replication licensing. In a virtual environment this looks something like the picture below.


This works well, but can be costly and leads to storage vendor lockin, not a bad thing if you are a storage vendor, but not always the best solution from a consumer perspective. So how do we abstract the replication from the storage? Remember, one of the purposes of virtualization and openstack is to abstract as much as possible from the hardware layer. That is not to mean hardware is not important, quite the contrary, but it does enable us to become more flexible.

So to provide this abstraction there are a couple options. We can always rewrite the application, but that takes time, we can do replication at the file system level or similarly using a 3rd party software to move data, but in order to really abstract the replication from the hardware/software we need to insert something in the middle.

In the conversation I was having at the begining, the goal was to replicate from the production datacenter running brand X storage to a remote location using an HP storage product. To accomplish this, we discussed using vSphere replication, something I will discuss in a future post, we discussed host based replication, but that is not as seamless, and what we settled on is below. Not the most elegant solution, but something that helps us abstract the replication layer. Essentially using the HP StoreVirtual VSA, since it has replication built in, we put that in front of the brand X storage, and then on the other side we can put another VSA on a server with some large hard drives, and voila, replication and DR storage handled.

Storage_Replication_VSA - Edited

Not the most elegant solution, but it is a way to abstract the replication from the storage, and to do so at a reasonable cost. The advantage to this solution is that we have also given ourselves DR storage. Next I will explore vSphere replication, but as always I want to point out, this solution minimized vendor lock in on the hypervisor and storage levels.

Software Defined Storage Replication

Reference Architectures, Hardware Compatibility Lists, and you.

Recently I was giving a presentation on designing storage for VMware Horizon.  I was referencing the HP Client Virtualization SMB Reference Architecture for VMware View, based on an earlier version, but still valid.  The conversation kept coming back to well can’t I do more than that, or why wouldn’t I just do it this way.

One of the better hardware compatibility lists is actually the VMware Compatibility Guide.  The best feature is that it is simple to understand, searchable, and matrixed.  This is a critical tool because it enables us to know what has been tested and what works, but more importantly what can be supported.  Of course it is often more expensive to go with supported configurations, but if we are looking at cost as the primary criteria, it would make more sense to use open source technologies.  While I am a big fan of open source for labs and various projects, the cost of supporting these in a production environment is often far more than simply using supported configuration and paying for support.  This is also true for using commodity hardware which is not supported.

The same can be said of reference architectures.  HP does an excellent job of creating these, especially because they have hardware in all major categories.  In the example I started with, the major issue was that the questions were around cost.  The person creating the design wanted to know why the can’t remove parts or replace them for cheaper ones.  The short answer is simply that the reference architecture is tested with all the components it contains.  It is a known quantity so it will work, and if it doesn’t the support teams can fix it since they know all the pieces.

So to sum up, doing things the way the manufacturer recommends will save a great deal of heartache.  To answer the question, you can do things your own way, but you may find that it is more trouble to support than it is worth.

Reference Architectures, Hardware Compatibility Lists, and you.

The changing role of shared storage in the Software Defined Datacenter: Part 3

As we have discussed, the role of shared storage is changing.  VMware has supported vMotion without shared storage for a while now, software defined storage is enabling shared compute and storage virtualization, and for the past year or so, we have been hearing more about the concept of vVols.  I am certainly not the first to talk about this, there are a number of blogs on this, my personal favorite being The future of VMware storage – vVol demo by @hpstorageguy.

As always, in the interests of full disclosure, I do work for HP, but this is my personal blog, and I write about things I think are interesting.  I am not going into great detail on how vVol’s work, but I do want to show a few diagrams to differentiate current architecture from what we MAY see in the future.

So looking at the current and legacy architecture of VMware storage, we typically present storage to all hosts in the cluster in the form of a shared LUN or Volume.  This is very simple, the VMware admin asks the storage admin for a number of volumes of a specific size, in our example below, let’s say they are 2TB volumes and they request 2 of them.  The VMware administrator then creates datastores, which formats them with the VMFS file system and allows virtual machines to be created within it.  Of course this whole process can be done through the VMware GUI using the vSphere storage API’s, but the net effect is the same.  We still create another layer in the storage stack which is not the most efficient way of handling this.



vVols are VMwares new way of handling storage which resolves this problem in a rather unique way.  Currently we can bypass the datastore concept and do a raw disk map or RDM, which allows us to present a raw disk device to the virtual machine itself.  Unfortunately this does not give us a measurable difference in performance, and can become tedious to manage.  vVols on the other hand, appear to be datastores, but really pass through the individual volumes to the individual VM’s.  In the drawing below, the individual volumes appear to the VM administrator as Datastores, but they are broken out on the storage array.  This removes the performance layer, and enables a more policy based storage interface for the VMware administrator.  This is critical to note, policy based storage at a VMware level.  This brings us closer to self service in a virtualized environment.  I don’t yet have a full handle on how this will be managed, but I think it is safe to say the storage administrator will create a container giving the VMware admin a specific amount of storage with specific characteristics.  In the case of our example, 2TB containers.




Note above the volumes are of varying sizes, but what is not shown is the volumes or luns are individual disks presented directly to the virtual machine itself.  This is important to remember since we are offloading the performance of each individual disk presented to the virtual machine to the storage array, but we are still able to manage it as a datastore or a container on the VMware side.

Coming back to the policy based storage thought, this is not dissimilar to how the HP 3Par storage operates, volumes within common provisioning groups which are containers.  The policy is set on the container in both cases, so it isn’t a stretch to see how this will work well together.  Again I don’t have any inside information, but if you look at the post from referenced above, Calvin does an excellent job of showing us  what is coming.  This, combined with VMware’s VSAN announcements recently, seem to show that there is going to be a role for the traditional storage array in addition to software defined storage in the software defined datacenter at least for now.

The changing role of shared storage in the Software Defined Datacenter: Part 3