Head in the Clouds

What do you do when you go to LinuxTag the premier Open Source conference in Berlin Germany ? You give a talk, you hand out tee-shirts at the CloudStack booth, you explain Cloud computing and you hack a CloudStack driver for SaltStack while patching libcloud.

The talk: Talking about Clouds is nice and all, but after many years and many talks, I shamelessly admit that it gets a little old. So lately I have been working on BigData, both as a backend to CloudStack (think Ceph, Riack CS, Gluster) and as a workload to a cloud. I am talking about using Apache Whirr or Apache Provisionr incubating to deploy "one-click" hadoop clusters on public clouds. It is a long story that I will keep for another post as I am trying to write this before going to bed. but check out the slides and keep an eye on pallet and exoscale.

The booth: An open source booth is ..well a booth. I came with my pop-up banner, table cloth, tee-shirts, post-cards, USB stick/bottle openers, it feels a little bit like a traveling sales man, not that I would know but I imagine it like this. I have to explain that the 2 and 3 XL shirts will shrink quite a bit and will fit perfectly people's M or L frames. Then I point at the banner to showcase the magnificient CloudStack UI, I explain that there is an API server behind it. Sometimes I launch devcloud and do a live demo to bring them to their knees, sometimes I have to ask for help on IRC to answer a question, and sometimes a german developer wants to trade a tee-shirt against illegal substances not to be named. Life in the fastlane let me tell you. But that's what it's like to build a community, it is very much an evangelization process.

...

Hi everyone!

I'm Mike Tutkowski. I work at an exciting data-storage company in Boulder, CO called SolidFire (http://solidfire.com).

The SolidFire storage area network (SAN) was designed from the ground up to support hard Quality of Service. On a volume-by-volume basis, administrators can decide on a minimum, maximum, and burst number of IOPS - eliminating the Noisy Neighbor effect and allowing Cloud Service Providers to confidently host all sorts of application workloads in their clouds that were previously unpractical.

I am a dedicated resource to the CloudStack project. I integrate advanced SolidFire features into CloudStack while also performing general-purpose CloudStack development activities.

We've received several inquiries from our customer base regarding how one goes about configuring storage in CloudStack. To help answer these questions, I've prepared a CloudStack Storage Configuration Guide and included it in this post.

...

I wish I could write like Shakespeare but since I don't you are left with this blog about Representational State Transfer (REST) and specifically a discussion on whether the CloudStack API is a REST API or not. The short answer is that the CloudStack API is RESTlike but not RESTfull since it is only based on the GET method. Being an http based API that can return JSON does not make it a RESTfull API. This should not be seen as negative criticism but just a clarification.

A few words first about the CloudStack API and a few pointers. It is very extensive, from user creation, vm management to firewall configuration and more advanced networking features. Github has lots of clients written by the community and I am sure you can find your favorite language in there. The Developer Guide explains in details how to make requests. With CloudMonkey interacting with the API has become even easier. It is a terrific way to learn the API and check the required parameters of each call. Coupled with devcloud you can have a fully functional local CloudStack testbed.

To explore the API, I often also look at the GUI and see how the API calls are made. To do this, I access the UI in Firefox and launch Firebug console. I can then see the calls that I am interested in and specifically check the parameters/headers etc or the http call being made.

REST is known as an architectural style and to put it in french-english it is basically a way to design your API to create a web service that only uses the HTTP methods to manipulate the state of web resources. In my opinion it was really a response to the complexity of the SOAP based web services and the many standards that came out. REST really took off with Web 2.0 and was seen as a way to create easy to use web services. I recently found a few articles by Luis Rei and they clearly explain how to design a REST web service as well as implement on with Flask a lightweight python based web framework.

...

The problem with middleware/backend work is that nobody sees what you do and since I am terrible at graphics/design and know nothing of User Interface principles I am pretty much stuck in the dark. So today I invested couple hours on the Tuesday Silly CloudStack Hack. It is made of Flask, a good read, Twitter Bootstrap and some stolen code from CloudMonkey

Flask is a terrific web microframework for Python. Of course I like Python so I think Flask is terrific. It is also great because you can use it to design clean REST services. Bootstrap is en vogue these days, and CloudMonkey, also written in Python is the new CloudStack command line interface boasting some cool features, like auto-completion, interactive shell and so on.

I will keep it short, grab requester.py from the CloudMonkey source tree, it will help you make API calls to a CloudStack instance. Create a simple script with flask, and set your CloudStack endpoint variables, plus the keys (I am using DevCloud):

import requester

from flask import Flask, url_for, render_template, request
app = Flask(__name__)

apikey='plgWJfZK4gyS3mOMTVmjUVg-X-jlWlnfaUJ9GAbBbf9EdM-kAYMmAiLqzzq1ElZLYq_u38zCm0bewzGUdP66mg'
secretkey='VDaACYb0LV9eNjTetIOElcVQkvJck_J_QljX_FcHRj87ZKiy0z0ty0ZsYBkoXkY9b7eq1EhwJaw7FF3akA3KBQ'
path='/client/api'
host='localhost'
port='8080'
protocol='http'

Setup a route that you will use to trigger a call to CloudStack. Something like this:

@app.route('/users')
def listusers():
    response, error = requester.make_request('listUsers',{},None,host,port,apikey,secretkey,protocol,path)
    resp=json.loads(str(response))
    return render_template('users.html',users=resp['listusersresponse'])

Now Download Bootstrap and stick it in a static directory in your Flask application. Then create html template files using the jinja2 syntax, something like this:

...

Alex Bederov from Nokia wrote up a post over a year ago about analyzing his cloud usage stats with Excel. I was going to refer to that, and when I went looking for it, I realized that it was posted as a KB article on the now defunct docs.cloudstack.org site. Since I still had the write up, and it's still relevant, I thought I'd repost that content here, so it is accessible again.
CloudStack comes with very well thought out and reasonably well implemented record keeping system. All information is there, however “batteries are not included”, you have to pull information and massage it yourself to create meaningful representation like billing records or usage reports. Fortunately, CloudStack Usage Server makes this task very easy to accomplish and Microsoft Excel pivot tables makes reporting fun and easy.
In my setup Usage Server runs every 24 hours. Usage database called “cloud_usage” is on the same MySQL server machine as main database called “cloud”. Resource usage is stored in table called cloud_usage (same name as database). I use following query to pull de-normalized usage records into comma delimited file ready to be processed in Excel:

(SELECT dc.name as "Datacenter", ac.account_name as "Account", u.domain_id
as "Domain ID", u.raw_usage,
s.cpu as "CPU count", s.speed as "CPU, MHz", s.ram_size as "RAM, MB",
8589934592/1024/1024/1024 as "Storage, GB", u.start_date,
"-", u.offering_id, u.usage_id, u.usage_type, u.description
FROM cloud_usage.cloud_usage u, cloud.account ac,
cloud.data_center dc, cloud.service_offering s
WHERE u.account_id=ac.id
AND u.zone_id=dc.id
AND u.offering_id=s.id
AND u.usage_type=1)
UNION
(SELECT dc.name as "Datacenter", ac.account_name as "Account", u.domain_id as
"Domain ID", u.raw_usage,
NULL as "CPU count", NULL as "CPU, MHz", NULL as "RAM MB",
u.size/1024/1024/1024 as "Storage, GB", u.start_date,
"-", u.offering_id, u.usage_id, u.usage_type, u.description
FROM cloud_usage.cloud_usage u, cloud.account ac,
cloud.data_center dc, cloud.disk_offering dsk
WHERE u.account_id=ac.id
AND u.zone_id=dc.id
AND u.offering_id=dsk.id
AND u.usage_type=6)

This query combines VMs and data volumes usage into the same table and de-referencing id fields. For now I am not including network usage data since I am using direct attached VLAN based networking model. Please note that usage_type “1” corresponds to VM, “6” – to data volume.
In order to import data into Excel we need to install MySQL ODBC connector which will allow us to connect to MySQL server directly. The MySQL ODBC connector is available free of charge from Oracle.
When you install the connector open a new Excel workbook, click Data, then select “From Other Sources” then select “From Microsoft Query”. Select “New Data Source”
Name the source and select MySQL ODBC driver:

I am using SSH tunnel and read-only account with read privileges to cloud and cloud_usage database. When you enter credentials click Test button and select cloud_usage database.

The new data source is available now.

Select the newly created data source and click OK button. Welcome to Windows 3.1 interface! Do not use Microsoft Query Wizard to create query. Cancel all the prompts and click SQL button go to SQL statement window.

Paste your SQL into ridiculously small window and save your query into a file with meaningful name. Excel will prompt you where to put the resulting data, accept the defaults.
The result of this exercise will be our query results imported into Excel spreadsheet. Here is how it looks like for me:

Please note that all my templates have 8GB HDD so I hardcode root disk size into SQL query. Now let’s analyze some data. Click top right corner to select the whole sheet and click Insert in Excel main menu, then Pivot Table, and then Pivot Chart. Make following selection in Pivot Table Field List: