Head in the Clouds

A new version of the knife-cloudstack plugin has appeared (v0.0.14), and seems pretty much on time - averaging about 6 months now between releases.

This release, Sander Botman from Schuberg Philis really stepped up and lead the way in the 15 newly added capabilities, along with handling tons of pull requests for bugfixes and updating functionality. You can check out the complete list of new awesomeness here.

If you are a chef user, you should really check out knife-cloudstack, you can download the source from:
https://github.com/CloudStack-extras/knife-cloudstack

Or just install it with:

gem install knife-cloudstack

Alex Bederov from Nokia wrote up a post over a year ago about analyzing his cloud usage stats with Excel. I was going to refer to that, and when I went looking for it, I realized that it was posted as a KB article on the now defunct docs.cloudstack.org site. Since I still had the write up, and it's still relevant, I thought I'd repost that content here, so it is accessible again.
CloudStack comes with very well thought out and reasonably well implemented record keeping system. All information is there, however “batteries are not included”, you have to pull information and massage it yourself to create meaningful representation like billing records or usage reports. Fortunately, CloudStack Usage Server makes this task very easy to accomplish and Microsoft Excel pivot tables makes reporting fun and easy.
In my setup Usage Server runs every 24 hours. Usage database called “cloud_usage” is on the same MySQL server machine as main database called “cloud”. Resource usage is stored in table called cloud_usage (same name as database). I use following query to pull de-normalized usage records into comma delimited file ready to be processed in Excel:

(SELECT dc.name as "Datacenter", ac.account_name as "Account", u.domain_id
as "Domain ID", u.raw_usage,
s.cpu as "CPU count", s.speed as "CPU, MHz", s.ram_size as "RAM, MB",
8589934592/1024/1024/1024 as "Storage, GB", u.start_date,
"-", u.offering_id, u.usage_id, u.usage_type, u.description
FROM cloud_usage.cloud_usage u, cloud.account ac,
cloud.data_center dc, cloud.service_offering s
WHERE u.account_id=ac.id
AND u.zone_id=dc.id
AND u.offering_id=s.id
AND u.usage_type=1)
UNION
(SELECT dc.name as "Datacenter", ac.account_name as "Account", u.domain_id as
"Domain ID", u.raw_usage,
NULL as "CPU count", NULL as "CPU, MHz", NULL as "RAM MB",
u.size/1024/1024/1024 as "Storage, GB", u.start_date,
"-", u.offering_id, u.usage_id, u.usage_type, u.description
FROM cloud_usage.cloud_usage u, cloud.account ac,
cloud.data_center dc, cloud.disk_offering dsk
WHERE u.account_id=ac.id
AND u.zone_id=dc.id
AND u.offering_id=dsk.id
AND u.usage_type=6)

This query combines VMs and data volumes usage into the same table and de-referencing id fields. For now I am not including network usage data since I am using direct attached VLAN based networking model. Please note that usage_type “1” corresponds to VM, “6” – to data volume.
In order to import data into Excel we need to install MySQL ODBC connector which will allow us to connect to MySQL server directly. The MySQL ODBC connector is available free of charge from Oracle.
When you install the connector open a new Excel workbook, click Data, then select “From Other Sources” then select “From Microsoft Query”. Select “New Data Source”
Name the source and select MySQL ODBC driver:

I am using SSH tunnel and read-only account with read privileges to cloud and cloud_usage database. When you enter credentials click Test button and select cloud_usage database.

The new data source is available now.

Select the newly created data source and click OK button. Welcome to Windows 3.1 interface! Do not use Microsoft Query Wizard to create query. Cancel all the prompts and click SQL button go to SQL statement window.

Paste your SQL into ridiculously small window and save your query into a file with meaningful name. Excel will prompt you where to put the resulting data, accept the defaults.
The result of this exercise will be our query results imported into Excel spreadsheet. Here is how it looks like for me:

Please note that all my templates have 8GB HDD so I hardcode root disk size into SQL query. Now let’s analyze some data. Click top right corner to select the whole sheet and click Insert in Excel main menu, then Pivot Table, and then Pivot Chart. Make following selection in Pivot Table Field List:

Today was a great milestone for Apache CloudStack. If you haven't seen the news (but yet somehow have come across this blog post???) the Apache Software Foundation announced that CloudStack had graduated from the Incubator as a top-level project. While in many ways it marks the end of a number of personal and project goals, it's also just another milepost along the journey. I've been working on CloudStack since most folks knew it as 'Cloud.com', and it's amazing to see the difference over the space of a couple of years.
I've been involved with several open source projects for a number of years, and I knew of the ASF by reputation, but had no personal experience. I spent weeks reading the documentation on the website when we first began discussing the potential for a move to the ASF, and I rapidly became both impressed and afraid. Impressed, because I saw codified in front of me the most transparent and open expectations of anything I'd been involved in. I suddenly appreciated why the ASF had the reputation it did. Afraid because the magnitude of change was incredible. The transformation hasn't been perfectly smooth, I even questioned if the dramatic change would be so much as to be overly disruptive. Many folks have written of how dramatic it is to open source a project - but to take a project that was open source but still heavily commercially governed and move that to the Apache Software Foundation, is both extremes of the open source spectrum.
Many people will write about the tremendous growth in community numbers, the more interesting story to me is the tremendous growth of community responsibility, over 50% of the Project Management Committee don't work for Citrix. Of course committers and PMC members are expected to behave as individuals, and in the best of interest of the project, but that amount of diversity in a short time is impressive. I also find it fascinating just how many folks who are participating are CloudStack users; they have truly taken ownership and responsibility for their IaaS platform.
What does graduation mean from my perspective? A number of different things, but most poignantly, it means that we have met the expectations of our mentors, the Incubator, and the Apache Software Foundation.
I am excited about the future. I have no delusions that for many, or even most of our users, our graduation has little or no immediate impact. It does have an impact for the project, as we shift our focus forward, and I think that will tremendously benefit our users. I think that you'll continue to see impressive things from Apache CloudStack, we're really only getting started.

A couple of words of thanks to folks

I can't express how wonderful our mentors were. They understood the process, they saw the challenges, they stepped in where appropriate and let us find the solution when we needed to. I've walked away thoroughly impressed at both the individuals as well as the incubation process in general, I am sure it can be improved, but struggle to think of anyone doing it better.
The folks at Apache Infrastructure - they are doing an incredible and impossible job - supporting well over 100 top level projects including such behemoths as Hadoop, Maven, and now CloudStack; dealing with plenty of inbound incubator projects, some of which, like CloudStack, have years of history, and thus plenty of baggage to bring with them. Thanks for the immense amount of help you folks have provided.

I've written a bit about the CloudStack resources written by Dan Bode and how awesome they are. And while cheerleading and telling people it is awesome should be enough - it doesn't really tell you how to actually use it. I will try and remedy that in the next few posts.

CloudStack Resources

Grab the resources from here: https://github.com/bodepd/cloudstack_resources Copy that into your modules directory in your puppet repository.

FOG

You need to ensure that you have fog installed. And by installed, I mean on the machine that will be talking with CloudStack. In our example we'll be using the puppetmaster machine. In an enlightened distribution like Fedora you can install by:

yum install rubygem-fog

In less englightened distributions you can use:

gem install fog

Configuration

You'll need to add a transport.yaml file to /etc/puppet Exercise caution with this file, it will contain API credentials Mine looks similar to:

cloudstack: api_key: 'WgOdxLuGFA2h2g-Ln2Hg9yWwEN1D3W4aqdXNXKTt_4xIOu234sdfs32NfDRM43JK3j1szDJzZf6CIgNAA' secret_access_key: 'S9Bd2JnuKComulSaqqyaudfwsdfsfesnM6ss7aydWof9QUT9zQ9Jhtja1GhbKcjrquSv7eV11ZpL9bybHxS0xg' host: 'cloud.cloudstack.org' port: '8080' path: '/client/api' scheme: 'https'

Defining some infrastructure

We'll set up a simple class with a single machine defined, and install as it: modules/infra/manifests/init.pp

class infra { cloudstack_instance { 'foo': ensure => present, flavor => 'Small Instance', zone => 'SanJose', image => 'CentOS63KVMPW', group => 'role=db', } }

Applying the class

Now that we have a class defined it's time for us to apply that - I am going to assume that you have a site.pp - if you don't, I figure you must be using something like and ENC and are smart enough to figure out how to apply the infra class. in site.pp you likely some lines like:

node 'puppetserver.cloudy.loc' { include puppet::master }

You want that to look like:

node 'puppetserver.cloudy.loc' { include puppet::master include infra }

The next time that puppet runs on the puppetmaster you should see log messages that say: Notice: /Stage[main]//Cloudstack_instance[foo]/ensure: created This has us using Puppet to define our infrastructure. In the next part we'll hook up puppet to reading CloudStack metadata.

So I am working on writing an SELinux policy for the CloudStack KVM agent so that SELinux can be left enabled. Why you ask? Well I really dislike advocating for people to turn off a security mechanism to get software to work. Additionally I really want some of the advantages of sVirt. But here is where I'd like to solicit some help. If you are running KVM with CloudStack you naturally have SELinux set to disabled or permissive. If you have it set to permissive, and would consider installing my current policy definition - it would be greatly appreciated. What's the impact of you testing my policy? Well nothing right now - you're in permissive mode, and we won't change that during testing, so all you'd be doing is hopefully cutting down on AVC denials in /var/log/messages or in /var/log/audit/audit.log

So how do you help:

First install the new policy:

You can get the current version here: cloudstack-agent.pp once you have that on the hypervisor - run:

semodule -i cloudstack-agent.pp

Make sure you have auditd installed:

rpm -q audit

The above should show you whether or not you have audit installed. If not you can install and start auditd with the following commands:

yum -y install audit service auditd start chkconfig auditd on

The audit package ensures that all AVCs are logged to a dedicated file (/var/log/audit/audit.log) rather than /var/log/messages.

If you already had auditd up and running, lets rotate the logs

This will make it much easier to diagnose any missing policy items:

service auditd stop mv /var/log/audit/audit.log /var/log/audit/oldaudit.log service auditd start

Now go about your business, deploy machines, destroy machines, do weird and wacky things, we are essentially looking for new entries in audit.log to see what we have missed. If your audit log shows up with items in your audit log, please upload them to this bug: CLOUDSTACK-337

You have questions??

Do they match these below? If not ask on the list.

Wait, are you testing this yourself?

Of course I am - I've long since (by which I mean I applied it while writing this) applied this to all of my KVM nodes, however, I have only a small percentage of potential configuration options. Specifically, I am running CloudStack 4.0.1, with KVM on EL6.3, with NFS and local storage and VLANs for isolation.

Wait - are you making my KVM hypervisor less secure?

Probably not. I mean to begin with you are currently running with SELinux in permissive mode. This is an effort to allow us to turn on SELinux, use sVirt, and have a more secure hypervisor. Still not assuaged? Want to see the source? It's here.