satis egitimisatis egitimitengda.pro

Open@Blog

Discussion on the state of cloud computing and open source software that helps build, manage, and deliver everything-as-a-service.

  • Home
    Home This is where you can find all the blog posts throughout the site.
  • Categories
    Categories Displays a list of categories from this blog.
  • Tags
    Tags Displays a list of tags that has been used in the blog.
  • Bloggers
    Bloggers Search for your favorite blogger from this site.
  • Login

Coming back from CloudStack conference the feeling that this is not about building clouds got stronger. This is really about what to do with them and how they bring you agility, faster-time to market and allow you to focus on innovation in your core business. A large component of this is Culture and a change of how we do IT. The DevOps movement is the embodiment of this change. Over in Amsterdam I was stoked to meet with folks that I had seen at other locations throughout Europe in the last 18 months. Folks from PaddyPower, SchubergPhilis, Inuits who all embrace DevOps. I also met new folks, including Hugo Correia from Klarna (CloudStack users) who came by to talk about Vagrant-cloudstack plugin. His talk and a demo by Roland Kuipers from Schuberg was enough to kick my butt and get me to finally check out Vagrant. I sprinkled a bit of Veewee and of course some CloudStack on top of it all. Have fun reading.

Automation is key to a reproducible, failure-tolerant infrastructure. Cloud administrators should aim to automate all steps of building their infrastructure and be able to re-provision everything with a single click. This is possible through a combination of configuration management, monitoring and provisioning tools. To get started in created appliances that will be automatically configured and provisioned, two tools stand out in the arsenal: Veewee and Vagrant.

Veewee: Veewee is a tool to easily create appliances for different hypervisors. It fetches the .iso of the distribution you want and build the machine with a kickstart file. It integrates with providers like VirtualBox so that you can build these appliances on your local machine. It supports most commonly used OS templates. Coupled with virtual box it allows admins and devs to create reproducible base appliances. Getting started with veewee is a 10 minutes exericse. The README is great and there is also a very nice post that guides you through your first box building.

Most folks will have no issues cloning Veewee from github and building it, you will need ruby 1.9.2 or above. You can get it via `rvm` or your favorite ruby version manager.

git clone https://github.com/jedi4ever/veewee
gem install bundler
bundle install

Setting up an alias is handy at this point `alias veewee="bundle exec veewee"`. You will need a virtual machine provider (e.g VirtualBox, VMware Fusion, Parallels, KVM). I personnaly use VirtualBox but pick one and install it if you don't have it already. You will then be able to start using `veewee` on your local machine. Check the sub-commands available (for virtualbox):

...
Hits: 6353
Rate this blog entry:
0
Continue reading Comments

Fluentd recipe for CloudStack

Posted by on in CloudStack Tips

When it rains, it pours...Here is a quick write up to use Fluentd to log CloudStack events and usage. Fluentd is an open source software to collect events and logs in JSON format. It has hundreds of plugins that allows you to store the logs/events in your favorite data store like AWS S3, MongoDB and even elasticsearch. It is an equivalent to logstash. The source is available on Github but can also be installed via your favorite package manager (e.g brew, yum, apt, gem). A CloudStack plugin has been written to be able to listen to CloudStack events and store these events in a chosen storage backend. In this blog I will show you how to store CloudStack logs in MongoDB using Fluent. Note that the same thing can be done with logstash, just ask @pyr. The documentation is quite straightforward, but here are the basic steps.

You will need a working `fluentd` installed on your machine. Pick your package manager of choice and install `fluentd`, for instance with `gem` we would do:

    sudo gem install fluentd

`fluentd` will now be in your path, you need to create a configuration file and start `fluentd` using this config. For additional options with `fluentd` just enter `fluentd -h`. The `s` option will create a sample configuration file in the working directory. The `-c` option will start `fluentd` using the specific configuration file. You can then send a test log/event message to the running process with `fluent-cat`.

    $ fluentd -s conf
    $ fluentd -c conf/fluent.conf &
    $ echo '{"json":"message"}' | fluent-cat debug.test

The CloudStack plugin:
CloudStack has a `listEvents` API which does what is says :) it lists events happening within a CloudStack deployment. Such events as the start and stop of a virtual machine, creation of security groups, life cycles events of storage elements, snapshots etc. The `listEvents` API is well documented. Based mostly on this API and the fog ruby library, a CloudStack plugin for `fluentd` was written by Yuichi UEMURA. It is slightly different from using `logstash`, as with `logstash` you can format the log4j logs of the CloudStack management server and directly collect those. Here we rely mostly on the `listEvents` API.

You can install it from source:

...
Hits: 16749
Rate this blog entry:
0
Continue reading Comments

Data Driven Tests in Marvin

Posted by on in CloudStack Tips

Data driven tests (DDT) is useful for generating more tests with less code. If you have lot of assertions to be repeated with related data sets or parameters, Data driven tests are a way to go - saves a lot of redundant code. The data sets provided as input will be used iteratively in the test scripts in effect, running different tests cases. I’ve recently tried to implement this with Marvin, CloudStack’s Integration Test framework which is written in Python. 

Before beginning to code test suites in data driven fashion, it’s best to understand the entities which drive the tests.  For example, we may want to test the same set of VM life cycle tests for different network offerings such as a simple isolated offering, an offering with persistent network feature and an offering with Netscaler as a service provider for LB. These offerings then become the data sets which are supplied as input to the test script. And the assertions for VM life cycle are repeated for each of these network offerings.

DDT in Python

Python provides a powerful and efficient way to achieve this with the ddt library. It comes with a class decorator @ddt and method decorators @data and @file_data. Use the class decorator with the TestCase class and specify the data sets with which the tests need to be run with the @data or @file_data method decorator. While @data takes in the arguments directly to be passed to the test, @file_data will load the data set from a JSON file.

CloudStack Test Case Example

...
Hits: 9255
Rate this blog entry:
Continue reading Comments

I think that we can all admit that deploying a cloud infrastructure can be a complex endeavor. With so many moving parts, each with its own configuration not to mention limitations, so called fragility can become a central hurdle within the architectural blue prints of a cloud infrastructure. 

Many in the DevOps space (infrastructure as software) prefer to use the tactic of loosely coupled software components when building out clouds. This is an architecture defined by an endless number of components that come together to create a unique cloud environment. I’m not saying this is wrong, in many cases this type of approach can lead to differentiation between you and your competition. But it doesn’t necessarily make things easier, at least from the stand point of an easy to deploy and mange cloud infrastructure.

In my early exploration of CloudStack several analogies have been used to describe it to me. CloudStack has taken a much more structured approach to the packaging and deployment of a cloud platform where others in the industry have taken a much looser tact. One of my colleagues uses a shoe analogy to describe the differences saying “A loosely coupled architecture offers a seemingly endless variety of colors, sizes, and styles.  When you order it, you receive a box with soles, fabric, scissors, and instructions how to measure your foot, how to choose which blueprint to follow, how to cut the fabric, how to stitch it so it looks good, etc.  Most folks will need to engage a professional cobbler to assemble the shoe.”

He points out that CloudStack offers fewer colors and styles, but when you receive the box, you spend a few moments lacing up the shoes and you’re out running in short order. 

Another colleague describes it a bit differently saying its much more like a F1 car. “You get to choose every single piece that goes into an F1 car, but it's inherently more temperamental and requires an entire team to keep it running smoothly. That said if you need the absolute best performance of flexibility that model is the way to go. The problem in a loosely coupled stack is the differing levels of maturity of mandatory pieces of the infrastructure. Some of those pieces are time/production tested, others not so much, yet. It's the equivalent of having awesome race tires that grip really well and a fuel pump that's a bit dodgy.”

...
Hits: 10815
Rate this blog entry:
0
Continue reading Comments

A vote has started on the libcloud dev list for the 0.13 release. The release notes detail all the new features and fixes. I am stoked about this because a few patches that I submitted are included in this release candidate. I patched the CloudStack driver quite a bit to improve the support for Clouds with a basic zone like Exoscale. There is more work to do on this driver including better support for Advanced zone especially for port forwarding, firewall rules and more unit tests. A the CloudStack hackathon last sunday @pst418 submitted a few patches for unit tests and they made it as well into 0.13 RC, terrific.

If you don't know libcloud, it's a python based API wrapper to abstract the various cloud APIs. With libcloud you can create connections to multiple clouds, potentially using different API. At a high level, it is similar to jclouds for JAVA or deltacloud written in ruby. There was already a CloudStack driver but its functionality was limited. If you grab my quickie libcloud shell, you can try to follow this walkthrough of what you can do with libcloud and a CloudStack basic zone. Of course you will need a CloudStack endpoint.

Start the libshell and check what zone you are on:

$ python ./libshell.py 
Hello LibCloud Shell !!
You are running at: https://api.exoscale.ch/compute
>>> conn.list_locations()
[<NodeLocation: id=1128bd56-b4d9-4ac6-a7b9-c715b187ce11, name=CH-GV2, country=AU, driver=CloudStack>]

You might notice a wrong country code, it is hard-coded in libcloud, I need to file a bug for this. Get the list of templates (or images in libcloud speak):

...
Hits: 7100
Rate this blog entry:
0
Continue reading Comments

Open@Citrix

Citrix supports the open source community via developer support and evangeslism. We have a number of developers and evangelists that participate actively in the open source community in Apache Cloudstack, OpenDaylight, Xen Project and XenServer. We also conduct educational activities via the Build A Cloud events held all over the world. 

Connect