Head in the Clouds

Efficient use of CloudStack really demands configuration management (among other things). I've been a puppet user for many years predating my involvement with CloudStack, and thus I have a slight bias for puppet.

Thus it thrilled me to no end when I saw folks like Jason Hancock doing work around automating configuration of CloudStack instances. Jason really knows Puppet, and even operates a new Hosted Puppetmaster startup. Jason showed this off a few times at both PuppetCamp LA 2012, and at the CloudStack Collaboration Conference in 2012.

It's awesome work, and you should take the time to watch both of his videos and check out his blog.

The gist of what he was presenting is configuring instance metadata within CloudStack at deployment, having the instance read that metadata and set it as a fact, and then using case statements to apply different roles to the instances.

And then there was a knife.....plugin

Next I learned that the good folks at Edmunds.com had written a CloudStack plugin for knife. That was exciting in and of itself, especially as a CloudStack person. It wasn't just knife, which is a command-line tool for Chef, another configuration management tool. knife is commonly used to provision machines, and the folks at Edmunds.com had baked in some extra awesomeness. They had the ability to define an application stack based on a JSON definition of what the stack looked like.

So one could define a Hadoop Cluster like this in JSON, complete with network and firewall configuration:

"name": "hadoop_cluster_a",
"description": "A small hadoop cluster with hbase",
"version": "1.0",
"environment": "production",
"servers": [
  {
    "name": "zookeeper-a, zookeeper-b, zookeeper-c",
    "description": "Zookeeper nodes",
    "template": "rhel-5.6-base",
    "service": "small",
    "port_rules": "2181",
    "run_list": "role[cluster_a], role[zookeeper_server]",
    "actions": [
      { "knife_ssh": ["role:zookeeper_server", "sudo chef-client"] }
    ]
  },
  {
    "name": "hadoop-master",
    "description": "Hadoop master node",
    "template": "rhel-5.6-base",
    "service": "large",
    "networks": "app-net, storage-net",
    "port_rules": "50070, 50030, 60010",
    "run_list": "role[cluster_a], role[hadoop_master], role[hbase_master]"
  },
  {
    "name": "hadoop-worker-a hadoop-worker-b hadoop-worker-c",
    "description": "Hadoop worker nodes",
    "template": "rhel-5.6-base",
    "service": "medium",
    "port_rules": "50075, 50060, 60030",
    "run_list": "role[cluster_a], role[hadoop_worker], role[hbase_regionserver]",
    "actions": [
      { "knife_ssh": ["role:hadoop_master", "sudo chef-client"] },
      { "http_request": "http://${hadoop-master}:50070/index.jsp" }
    ]
  }

And then deploying a Hadoop Cluster is as simple as:

knife cs stack create hadoop_cluster_a

As a CloudStack guy I thought this was awesome, complex applications were suddenly deployable with ease, this is exactly the kind of automation that CloudStack is supposed to enable.

JEALOUSY

As a puppet afficionado though, it made me a bit sad, nothing existed like this for folks using puppet, and I was jealous.

An author, whose opinion I respect, wrote recently about how cloud portability 'is a bit of science fiction' at this point. While he is technically right - the idea of moving running machines from a CloudStack cloud to a vCD cloud, CloudStack, or AWS is still at best 'a bit of black magic'[1], and in other cases simply isn't feasible at all. And doing it at scale??? that's just crazy talk. That said, I'd argue (yes, this is largely just my opinion) that in most cases if you are trying for portability at the Infrastructure-as-a-Service layer, you are doing it wrong.

You see, this 'problem' of portability isn't a new one, and really isn't even cloud related, I think it's far more fundamental than that. I remember, in my days as a Pimply-Faced-Youth, of keeping exact copies of machines as cold spares - the thought being that we might need to migrate the services living on one piece of hardware, and particularly for a certain non-Unix operating system, restoring a copy of a disk, or even the disks themselves to different hardware often meant things just wouldn't work. (sound familiar?) Sometimes you got lucky and it would, and sometimes you could coerce things into a working state by installing new drivers, or updating initrd, or even adding kernel modules. So when buying hardware, we'd buy two of everything (or at least N+1) for those services that were truly business critical. Yes it was incredibly expensive, and wasteful, but it mitigated risk.

A bit later in my ops lifetime, virtualization became mainstream. Finally - we had this psuedo-hardware that would look the same regardless of the underlying physical hardware - hypervisors like VMware and Xen did wonderful things for us. Admitedly we couldn't easily migrate between hypervisor types, but life was still much better off. Not only were we able to stop buying so many 'spares', but we were a bit less concerned about machine lifecycle and had much better utilization to boot.

Folks came a bit further along with tools to convert these virt machines from VMware to KVM or Xen. As a matter of fact, on a previous blog I used to maintain, one of my most-viewed articles was one that detailed the use of qemu-img and other tools to convert VMDK files to raw disk images. I did more of this than I care to admit - and honestly when I did it, I was striving for portability, even if in a kludgy, barely automted way. Portability was the wrong thing to seek after - and I should have already known it at that point in my career.

Portability is the equivalent of trying to preserve a snowflake. It's difficult, usually messy, and often ineffective. What I didn't realize is that I really shouldn't have any snowflakes - that what I was really after was a way to consistently reproduce, not a system to save the only copy the world would ever see. Of course configuration management already existed, why I didn't understand the problem and the solution that was already out there I don't know. Perhaps I was a latent server-hugger, or feared obsolence by a few lines of ruby - but configuration management (coupled with automated provisioning) meant I didn't need portability - didn't even want portability.

One of the things we have found about running clouds at scale is the need for automation. To that end we often conduct training events called Build a Cloud Days with our open source partners. We like PuppetLabs puppet for automating configuration of CloudStack as well as Zenoss for automating the discover of cloud infrastructure to bring them under monitoring.  

One of the best ways to get up to speed on using puppet is to attend a Puppet Camp. The next one will be hosted in Atlanta on February 3rd.  

"Puppet Camp is a community oriented gathering of Puppet users and developers. You’ll have the opportunity to network with a diverse group of Puppet users, benefit from insightful lectures delivered by prominent community members, and be able to share experiences and discuss potential implementations of Puppet during our attendee generated breakout sessions."

If you can't attend this session there are many other Puppet Camps being held worldwide listed on the PuppetLabs website. 

At last week's USENIX LISA  we had the opportunity to teach users how to build and manage open source cloud computing environments using open source software at Build a Cloud Day Boston.  Our program featured presentations from Chef, CloudStack, Gluster, Puppet and Zenoss.  Our next Build an Open Source Cloud Day will be at the Southern California Linux Expo on January 20th, 2011 in Los Angeles. Here are the slides from some of the presentations.

CloudStack Community Manager, David Nalley gave an introduction to CloudStack for delivering infrastructure-as-a-service.

Opscode Technical Evangelist, Sean OMeara gave an introduction to Chef for Server Provisioning and Automation

Zenoss engineer and monitoring ninja, Simon Jakesch, discussed how to use Zenoss Core to manage "cloud infrastructure".

Here are the slides from today's Delivering Infrastructure-as-a-Service(Iaas) Open Source Software that will run from 1:00 p.m. - 2:00 p.m.. EST. 

Abstract:

The web was built using open source software like Linux, Apache, MySQL, PHP, Python and Perl. Just as with the web, open source is one of the core foundations of cloud computing as early cloud pioneers used the freely available, freely distributable model to power their web-scale deployments and achieving an unprecedented level of scale at a bare-bones cost. However, for businesses today, the attraction of open source is about the ability to develop a more flexible infrastructure and avoid vendor lock-in that often results from proprietary systems.