Head in the Clouds

In the spring of 2011 we conducted a survey of trends in cloud computing and published the results under a Creative Commons license so others could reuse the results to help in their own reports to their coworkers. This year we are doing another survey to identify the trends in cloud computing by comparing past results to the preferences and intentions of IT professionals going forward. The intention of the survey is to better inform the open source developers of Apache CloudStack, BitNami, Puppet, Xen.org and Zenoss Core how to improve their software as well as to help others understand the direction of the cloud computing industry.

This year’s survey can be taken via SurveyMonkey and all participants who choose to share their information with the sponsors will be entered to win one of the latest generation iPads for a total of five possible winners.

The 2011 State of Cloud Computing Survey includes the following findings:

This year’s survey will cover those questions asked in the previous survey as well as address emerging trends in cloud computing. The final deliverable will be a professional report with the findings that can be distributed to your peers and to help inform research in cloud computing adoption.

You can take the 2013 Cloud Computing Outlook Survey here:

https://www.surveymonkey.com/s/BACD

Sponsors

Participants who choose to share their contact information will be entered to win one iPad, each sponsor will be providing one iPad to be randomly awarded to a single survey participant.

Build a Cloud – www.buildacloud.org

BuildaCloud.org is a community resource for educating users on how to build their own cloud computing environments. The BuildaCloud.org community includes over 30,000 users and is an emerging resource for cloud computing education through their Build A Cloud Program. The Buildacloud.org community is sponsored by Citrix.

BitNami – www.bitnami.org

BitNami simplifies the process of deploying web applications natively, virtually and in the cloud. BitNami provides a growing library of the most popular open source applications, ready to be launched with the click of a button. With BitNami, solution providers and businesses of all sizes can deploy and manage business applications such as Jasperserver, SugarCRM, Alfresco, Drupal and many more. BitNami is developed by BitRock, a leader in cross-platform application deployment.

Citrix – www.citrix.com

Citrix Systems, Inc. (NASDAQ:CTXS) transforms how businesses and IT work and people collaborate in the cloud era. With market-leading cloud, collaboration, networking and virtualization technologies, Citrix powers mobile workstyles and cloud services, making complex enterprise IT simpler and more accessible for 260,000 organizations. Citrix products touch 75 percent of Internet users each day and it partners with more than 10,000 companies in 100 countries. Annual revenue in 2011 was $2.21 billion.

About Puppet Labs – www.puppetlabs.com

Puppet Labs, Inc. is the leader in IT automation and software-defined infrastructure. Puppet Labs’ software provides system administrators the operational agility, efficiency and insight they need to proactively manage dynamic infrastructure, scaling from tens of servers to thousands, on-premise or in the cloud. Thousands of the world’s leading organizations are using Puppet Labs’ software to configure and manage their IT infrastructure, including Citrix, eBay, NYSE, Match.com, Oracle/Sun, Shopzilla, and Zynga. Now numbering more than 100 employees and based in Portland, Oregon, Puppet Labs is backed by investors Kleiner Perkins Caufield &Byers, Google Ventures, VMware, Cisco, True Ventures, Radar Partners, and Emerson Street Partners.

Xen Cloud Platform (XCP) – www.xen.org

Xen Cloud Platform (XCP) is an open source project, providing a bare-metal installable ISO with a full Xen hypervisor virtualization stack. It was initially an open source version of XenServer, but has slowly developed into it's own system designed for use in cloud computing stacks. XCP is a full binary distribution of Xen, whereas the Xen hypervisor needs to be built from source. For more information visit - http://wiki.xen.org/wiki/Category:XCP.

About Zenoss Inc. – www.zenoss.com

Zenoss is a leading provider of management software for physical, virtual, and cloud-based IT infrastructures. Over 35,000 organizations worldwide have deployed Zenoss to manage their networks, servers, virtual devices, storage, and cloud infrastructure, gaining complete visibility and predictability into their IT operations. Customers include Rackspace, VMware, Hosting.com, LinkedIn, Motorola and SunGard.

I was asked the other day what was the connection between High Performance Computing (HPC) and Clouds, so I thought I would try to post an answer here. Let's first talk a little bit about HPC.

High Performance Computing is about finding every single flops and every single iops on the largest machine you can get your hands on, in order to run your code as fast as possible. It is about batch processing on as many cores as you can get, so you can solve the largest problem you are facing. For a while, supercomputers, were large shared memory machines but in the late nineties distributed memory systems appeared, they were cheaper and you could assemble lots of nodes to get hundreds of cpus. Today the Top500 supercomputers are ranked every 6 months, this ranking is the theater of great technological battle between countries, vendors, research labs and programmers. In the latest ranking, Sequoia the supercomputer from Lawrence Livermore National lab topped the ranking at 16.32 PetaFlop/s and 1,572,864 cores. Weather modeling, atomic weapons simulation, molecular dynamics, genomics and high energy physics are among those that benefit from HPC.

There is big difference however within HPC itself. It is the difference between applications that rely heavily on inter-process communication and need a low latency network for message passing, and applications where each process runs an independent task, the so-called embarrassingly parallel applications. (e.g Map-reduce is an example of how to express an embarrassingly parallel problem ). High Throughput Computing (HTC) defines the type of application where access to a large number of cores over a specific amount of time is needed. Protein folding popularized by the Folding@home project running on PS3 as well as desktops is a good example. Financial simulation such as stock price forecasting and portfolio analysis also tend to fall in that category due to their statistical nature. Graphics rendering for animated movies also falls under HTC. HTC cares less about performance -as measured by FLOPS- and more about productivity -as in processing lots of jobs-.

The HPC quest for performance seems totally antagonist with the IaaS layer of clouds, at least when one thinks of true HPC workload that consumes every flop. Virtualization, the key enabler of IaaS, introduces overhead, both in cpus and network latency, and thus has been deemed "evil" for true HPC. Despite directed I/O, pass thrus, VM pinning and other tuning possibilities to reduce the overhead of virtualization, you might think that this would be it, no connection between HPC and Clouds. However according to a recent academic study of hypervisor performance from a group at Indiana University, this may not be entirely true and it would also be forgetting the users and their specific workloads.

In november 2010 a new player in the Top 500 arrived: Amazon EC2. Amazon submitted a benchmark result which placed an EC2 cluster 233rd on the top 500 list. By June 2011, this cluster was down to rank 451. Yet it proved a point: that a Cloud based cluster could do High Performance computing, raking up 82.5 TFlops peak using VM instances and 10GigE network. In november 2011, Amazon followed with a new EC2 cluster ranked 42nd with 17,023 cores and 354 TFlops peak. This cluster is made of "Cluster Compute Eight Extra Large" instances with 16 cores, 60 GB of RAM and 10 GigE interconnect and now ranked 72nd. For $1000 per hour this allows users to get an on-demand HPC cluster that itself ranks in the top500. This is done on-demand and provides users with their personal cluster.

Cloud computing has been held up as a way for companies to improve their carbon footprint, though where they’re hosted really has a lot to do with it. In and of itself, migrating your workloads to the cloud isn’t a magic bullet for greener computing. But moving to a hosting provider that uses 100% renewable energy sources, like GreenQloud? Now you’re cooking with… geothermal and hydro power. Also? CloudStack.

GreenQloud was founded in 2010 and launched its beta in the summer of 2011. Eirikur Hrafnsson, co-founder and chief global strategist for the company, says that the idea was to “create a compute cloud that would start a new industry in Iceland, clean up the data industry, be competitively priced, efficent, and easy to use. We figured, there’s an abundance of renewable energy” in Iceland that would help solve the problem of dirty energy use in the industry.

Originally, the prototype for GreenQloud was built on a different platform. But Tryggvi Lárusson, co-founder and chief technology officer of GreenQloud, says that Dave Nielsen of CloudCamp “pointed us in the direction” of VMOps – the startup that became Cloud.com, and ultimately sold to Citrix. At the time, VMOps was in the process of open sourcing CloudStack, and GreenQloud started using the first open source version of that.

Says Hrafnsson, “we’re only using the open source version, we’ve done so since the beginning… the decision to use CloudStack was a great decision. Adoption of CloudStack has been much more prominent with the partners we want to work with. Utilizing CloudStack has opened a lot of doors for us and enabled us to move towards building an easy to use public compute cloud.”

In addition to being open source, Hrafnsson says that CloudStack had a few other winning points. First, “the GUI part was really important.” It’s not that the GUI is overly hard to implement – and GreenQloud hasn’t shied away from implementing features they find necessary – but that it showed “we were on the same path in terms of features. It accelerated our go-to-market by a factor of two compared to continuing what we had been doing.”

If you didn't see the news last week from Cloud Expo, Zenoss announced the release of a monitoring package for CloudStack. The initial version permits a cascading view of a CloudStack, and measuring a number of performance issues over time. There are plenty of future plans as well, including the monitoring of the CloudStack management server and performance for a number of additional items including speed of deployment and other factors.

Kudos to Zenossian Chet Luther for pushing out this awesome work.

Want to know more?

Check out Zenoss at http://community.Zenoss.org

CloudBridge is a free, open-source, downloadable add-on to CloudStack that translates Amazon EC2 API calls to native CloudStack API calls so that clients can continue using existing EC2-compatible tools with CloudStack.

CloudBridge 1.0.2 contains a significant refactoring of much of the internal code as well as improvements to the overal architecture. As a result of these changes, every supported command in CloudBridge has been refactored and in many cases actually work now.

Beyond just the code, the documentation has been improved and made more accessible.

You can find the installation package here:
http://sourceforge.net/projects/cloudstack/files/CloudBridge/

You can find the latest revision of the documentation here:
http://docs.cloud.com/CloudBridge_Documentation

Please let us know if you have any questions or problems using this release candidate for 1.0.2 by communicating on the cloudstack-devel mailing list, or filing bugs at http://bugs.cloud.com