Nati Shalom's Blog

The Disruption Cycle: A Dime a Dozen

 

Before the dust even settled around the whopping Dell/EMC deal, another bombshell was dropped with the Red Hat acquisition of Ansible over the weekend.  While I was still processing my thoughts on what I think the Dell deal means for the enterprise IT industry - I had to make an abrupt about-face and think about this poster-child of the community and bottom-up startup approach of Ansible means for the IT industry as a whole, and how this may be an even more substantial win. 

The EMC/Dell deal seems to be centered more on the merits of a financial transaction, i.e. grabbing billions of dollars from the existing customer base, refunding debts, latching onto VMware as the only publicly traded company and so forth.  The following diagram summarizes this quite nicely:

Dell Acquisition of EMC (and associated Federation assets) – blue text is new [Source: Wikibon, 2015]

What seems to be clear is that while the EMC/Dell deal is good for its shareholders its equally bad news for their customers as noted by the David Vellante report on Wikibon as both companies will have to squeeze their customer base to justify the ROI for the deal.

“Dell is promising within 18-24 months it will de-leverage (i.e. pay down its debt). How do companies manage cash cows? Answer – they leverage their lockin: 1) Longer product cycles; 2) Increase maintenance/support costs; 3) Cut salaries; 4) Narrow the R&D focus; 5) Charge a premium – We expect this to be the likely Dell playbook for VMAX, VNX, Data Domain, Compellent, etc”

It's hard not to notice that there's nothing in this deal that tells a clear story on how the new company will transform into the new cloud/web-scale business model.

So if anything I think that the Dell/EMC deal marks an end of an era more than a new beginning as Cade Metz puts this nicely on wired.com

“HP. Cisco. Dell. EMC. IBM. Oracle. Think of them as the walking dead. Oh, sure, they'll shuffle along for some time. They'll sell some stuff. They'll make some money. They'll command some headlines. They may even do some new things. But as tech giants, they're dead...When someone asked what we should call that IBM-HP-EMC-Dell-Cisco merger, his response was wonderfully descriptive. He suggested we call the company Fucked By The Cloud.”

It's also interesting to read the reaction to this post on twitter :100:

The case for Red Hat/Ansible is the exact antipode of the EMC/Dell deal. The financial factor didn't really play any real factor at all, what really mattered is the developer mindshare, adoption rate and popularity of Ansible in the DevOps community.

The Ansible stats tell the story - plain & simple - high adoption rate, low revenue, and I would add, in essence a high adoption rate at the expense of revenue.

• Founded in 2013

• 40 people

• $6M investment round

• High & rapid adoption

• Low sales - Estimated to be ~$1M

• Acquired at $150M

The ROI on this deal is a no brainer for Ansible.  The rationale for the Red Hat acquisition isn't so much on the direct revenue from Ansible as a product, but rather the potential monetization from its  impressive adoption.  This is interesting as it clearly put the benchmark for a successful startup these days back on eyeballs and popularity, and less on having a sustainable business model. But this seems to be the new game, and technology companies are all scrambling to get in line with this new model.

What's also interesting is that we’re starting to see new companies emerging to disrupt not just the dinosaur of the world, but also the early cloud companies such as Puppet and Chef that are now being disrupted...and there's a long list of such companies. Salesforce is a good example for one that is also being threatened by this rapid disruption.

We can see a clear vicious cycle that applies to all companies old and new. New generation companies rise on the wave of adoption by providing free and open source tooling, as they start to monetize and build a sustainable business model they start to lose popularity, and a call for a new disruptive tool is heard.

That explains the timing behind this acquisition, Ansible had reached a ceiling when it comes to adoption, and failed to build a sustainable business model out of their product. Containers/Docker have disrupted the entire configuration management market, and to a large degree render it obsolete (the adoption curve shows a clear declining trend on all the other configuration management tools).

With this in mind Ansible is clearly at its peak right now, so their best option was to be acquired by a bigger company like Red Hat. Red Hat is probably the only company with a sustainable business model around open source, so for them the direct revenue from Ansible isn't that interesting but more of the combined revenue that could be generated by bundling Ansible into a bigger DevOps tool set. I'm sure that the fact that Ansible’s founders are also formerly Red Hat played a not small factor in the overall decision.

Final words

What have we learned from all this?

• Cisco, EMC, Dell, HP, IBM are “walking dead”

• Adoption/popularity is the new king

• The only constant is change!

Disrupt yourself before your competitors do --> Embrace change

Steve Jobs’ main mantra was "disrupt yourself before your competitors do" and indeed what's clear in both deals is that change is inevitable, and it can and will happen to big companies just as it can happen to a smaller company. Therefore, the assurance that we used to think we had by relying on big companies isn't a safe bet anymore, similarly the risk of using new technologies from startups isn't as great as it used to be.

In this world those who would survive are only those who are set to adopt new technologies fast. There’s a direct correlation between the speed of adoption and the speed of innovation. The challenge is actually even bigger for companies who grow fast. Chef and Puppet are a good example in this regard. They created a new disruption, grew fairly fast, but stagnated for a long time, and failed to respond to the new disruption in their domain. 

This is where I see the biggest shift in the value chain specifically in the new data center. The value is less on how we build the tools, but more on how we put them together, and how we can adopt new tools or replace existing ones fast in way that will allow us to always be ready for the next disruption. This explains the rise in orchestration tools, as I pointed out at the last OpenStack Summit, and in my previous post orchestration tool roundup. that plays an important role as a "gluing" mechanism in the area of DevOps and automation.

Do I need OpenStack if I use Docker?

 

Docker has broken a record in the speed in which it moved from being a disruptive technology to a commodity. The speed of adoption and popularity of Docker brings with it lots of confusion.

In this post I wanted to focus on a trend of commentary that has been gaining popularity, that I’ve started to hear more often recently from users who just started using Docker.  Whether it makes sense to use OpenStack if they’ve already chosen to use Docker.

Before I give my take on this, I wanted to start with a short background for the rationale behind this question.

Background

In its simplest form, Docker provides a container for managing software workloads on shared infrastructure, all while keeping them isolated from one another. Virtual machines such as KVM do a similar job by creating a complete operating system stack of all the OS devices (through a Hypervisor). However, unlike the virtual machine approach, Docker relies on a built-in feature of the Linux operating system named LXC (Linux Containers). LXC utilizes the built-in operating system features of process isolation for memory, and to a lesser degree, CPU and networking resources. Docker images do not require a complete boot of a new operating system, and as a result, provide a much lighter alternative for packaging and running applications on shared compute resources. In addition, it allows direct access to the device drivers which makes I/O operations faster than with a hypervisor approach. The latter makes it possible to use Docker directly on bare metal which, often times causes people to ask whether the use of a cloud such as OpenStack is really necessary if they’re already using Docker.

This performance difference between Docker and a hypervisor such as KVM is backed by a recent benchmark done by Boden Russell and presented during the recent DockerCon event.

The benchmark is fairly detailed, and as expected, it shows a significant difference between the time it takes to boot a KVM hypervisor to a Docker container. It also indicates a fairly big difference in the memory and CPU utilization between the two, as can be seen in the diagram below.

This difference in performance can map to a difference in density and overall utilization difference between the two in similar proportion. This difference can easily map into a big difference in cost that is directly affected by the number of resources needed to run a given workload.

My Take:

• This question has nothing specific to do with OpenStack and can be applied similarly to any other cloud infrastructure. The reason it is often brought up in the context of OpenStack, in my opinion, is due to the fact that OpenStack is fairly popular in private cloud environments which is the only environment in which we can even consider a pure Docker alternative.

• It's about the hypervisor stupid:

Many of the performance benchmarks compare Docker versus KVM, and have little to do with OpenStack. In fact, this specific benchmark of both KVM images and Docker containers was running through OpenStack which shows that the two technologies works nicely together. In that context, most of the utilization  arguments become irrelevant when I choose to run OpenStack on top of a Docker based Nova stack as illustrated in the diagram below taken from the OpenStack documentation.

• Cloud infrastructure provides a complete data center management solution in which containers or hypervisors, for that matter, are only part of a much bigger system.  Cloud infrastructure such as OpenStack includes multi-tenant security and isolation, management and monitoring, storage and networking and more. All of those services are needed for any cloud / data center management and have little dependency on whether Docker or KVM are being used.

• Docker isn't (yet) a full-featured VM and has some serious limitations like security, Windows support (as indicated in the following email thread in the Red Hat mailing list), and therefore cannot be considered a complete alternative to KVM just yet. While there’s ongoing work to bridge those gaps, it is safe to assume that adding the missing functionality may come at an additional  performance cost.

• There’s a big difference between raw hypervisor performance/containerization and application performance, as indicated in the following graphs from the benchmark results. A possible explanation for that is that applications often use the caching technique to reduce the I/O overhead.

• If we package Docker containers within a KVM image the difference can become negligible. This architecture often uses hypervisors for managing the cloud compute resources, and an orchestration layer such as Heat, Cloudify, or Kubernetes on top to manage containers within the hypervisor resources.

Conclusion

This brings me to the conclusion that the right way to look at OpenStack, KVM and Docker, is as a complementary stack in which OpenStack plays the role of the overall data center management. KVM as the multi-tenant compute resource management, and Docker containers as the application deployment package. 

In this context a common model would be to use Docker for the following roles:

• Docker provides deterministic software packaging and fits nicely with the immutable infrastructure model.

• Docker for excellent great containerization of microservices pods

• Using Docker on top of OpenStack as well as a bare metal environment

Having said all of the above, I do see cases mostly for well defined workloads, where the use of cloud infrastructure isn't mandatory. For example, if I were to consider the automation of a small shop development and testing environment for DevOps purposes, I would consider using Docker directly on a bare metal environment.

Orchestration can be a great abstraction tool between the two environments

One of the benefits of using an orchestration framework with Docker, is that it can allow us to switch between OpenStack or bare metal environments at any given point in time. In this way, we can choose either options just by pointing our orchestration engine to the target environment of choice. OpenStack Orchestration (Heat) declared support for orchestrating Docker starting from the Icehouse release. Cloudify is an open source TOSCA based orchestration that works on OpenStack and other clouds such VMware, AWS and bare metal, and recently included Docker orchestration .  Google Kubernetes is associated mostly with GCE but could be customized to work with other cloud or environments.

References:

• Using Docker as and OpenStack compute resource (Nova)

• Orchestrating Docker containers through OpenStack Heat

• Orchestrating Docker containers on OpenStack, VMware, AWS, bare metal and Other clouds through TOSCA/Cloudify

• Docker limitations as indicated in Red Hat mailing list

• The boring truth: full virtualization and containerization both have their place

• Using Docker as immutable infrastructure

• Docker drops LXC as default execution environment

Our Open Source Cloud (Re) Architecture with OpenStack + TOSCA

Cloudify OpenStack Native approach - The Story behind the Scenes

In the previous post I outlined what I mean by native-approach to the OpenStack cloud. In a nutshell, a native approach represents out of the box tighter integration with OpenStack, all without being limited only to OpenStack.

In this post I want to share some of the stories behind the design of Cloudify 3.0 and the approach that we have taken to make Cloudify *native* to OpenStack.  I would first start by outlining our motivation behind this approach.

OpenStack-only or Not-only-OpenStack?

It is our core belief that for the foreseeable future supporting hybrid cloud is going to be important for many of the OpenStack users for the following reasons:

1. Transitioning to OpenStack is a long journey - Many enterprises have VMware-based stacks as their core infrastructure. Where many of these organizations have often started their journey toward OpenStack by creating a dev environment, and then gradually moving more workloads into their OpenStack environment as they get more comfortable with it. This process usually spans over months and possibly even years. It became apparent to us, that during this time these customers would still be using VSphere and VCloud in parallel to OpenStack. For such users, having a common platform to manage their app deployment across two environments can smoothen the transition and also reduce the risk of getting stacked if one of the environments doesn’t meet their needs.
2. The public cloud market is going to be dominated by non-OpenStack clouds - while OpenStack is on a trajectory to dominate the private cloud market, the public cloud market looks quite different with AWS as a strong and long-standing leader and other major clouds emerging such as GCE and Azure - hoping to close in. Many organizations wouldn’t want to host their entire data center on a private cloud, and would therefore need to take a hybrid cloud approach. In order to support hybrid cloud, we had to include an abstraction layer that will allow us to integrate with clouds other than OpenStack.

At the same time we wanted to have a more intimate integration with OpenStack in order to leverage the fact that it is open source and allows for a deeper integration with its core services. Through that we were able to enable smooth and simple integration of Cloudify into OpenStack.

Extending or Rewriting?

Cloudify 2.x already took a step toward deeper integration with OpenStack and we were one of the first to announce our support for OpenStack. Recently, we also added support for OpenStack Neutron.

We realized that if we want to make Cloudify fit natively into the OpenStack infrastructure it is not enough to have integration with the OpenStack components.  We also need to ensure that the design behind Cloudify will be consistent with the way other projects and services of OpenStack are implemented. This led us to the conclusion that we wouldn’t be able to be a first class citizen with OpenStack unless we underwent a major redesign of Cloudify.

Going Python?

Our first step in this direction was to move from Java to Python. Moving to Python wasn’t as painful as you would think for us, because many of our developers already knew the language. This decision quickly proved itself, as it became much simpler not just to develop the new version of Cloudify faster, but it also enabled us to join other OpenStack projects as well as reuse some of the same components and frameworks that are already used by OpenStack.

Integrating with Core OpenStack Services

Many of the products that declare support for OpenStack mostly integrate with the compute API (Nova), and quite often come with their own security, messaging, configuration, and other such considerations. We felt that this is not enough, and doesn’t fit in with the way other services are built within OpenStack. Hence, we decided to deepen our integration with other services such as Keystone, Neutron, Heat and later with Ceilometer, Mistral and others.

Going TOSCA?

Cloudify 2.x was designed around a proprietary Groovy-based domain specific language (DSL) for defining application topology and configuration. It became apparent that the DSL had become a central piece of Cloudify architecture. As a result, maintaining a proprietary DSL would only serve as a barrier going forward. TOSCA (Topology Orchestration Specification for Cloud Applications led by the Oasis Foundation) defines a standard templating language. TOSCA was similar and richer in concept to our previous 2.x specification. Because of this, using TOSCA seemed like a natural evolution.

At the time when we decided to use TOSCA, the specification was provided in a fairly complex XML. Our first step for adopting TOSCA was to come with a simplified YAML based version which better fits into the spirit of OpenStack. After the OpenStack summit in Hong Kong, when we first introduced the idea, it became apparent that members from the Oasis group were already thinking along this line.We decided to join forces and join the Oasis organization. We started working with the team to incorporate TOSCA as an official part of OpenStack. During the Atlanta summit, it was decided to integrate the TOSCA project into the Heat project and now there is an official definition of the TOSCA 2.0 specification based on YAML. Our goal is to use the TOSCA 2.0 specification as the official templating language for Cloudify instead of our current TOSCA like DSL.

Integrating with OpenStack Heat

OpenStack is growing, and with this the services that are included as part of the core OpenStack project continue to grow up the stack. One of these services is OpenStack Heat, which started as the equivalent of Amazon Cloud Formation, and is becoming more of a general purpose infrastructure orchestration these days.

We realized that since Heat is closely integrated with OpenStack it will follow the OpenStack release cycles and API, and therefore would provide a useful tool for setting up the OpenStack infrastructure. The approach that we have taken with Heat is an “overlay approach”. This means that users can use Heat as they do today to set up their infrastructure. Then, Cloudify integrates with Heat in a way that will allow it to discover any resource that was provisioned through Heat, and ultimately add the monitoring, logging and software stack on top of that environment. 

Integrating with Neutron - Network Orchestration

Networking becomes a core service in any cloud deployment. Networking refers to an element such as security-group, private IP, floating IP as well as routers, DNS, vLans, load balancers, and such.

Cloudify 2.7 included basic integration with Neutron that allows users to attach floating IPs and set the availability zone configuration.

With Cloduify 3.0, we included support for all the networking elements and can now create vLans, security groups and more, as part of the application deployment.

Design for Scale

Application orchestration often requires intimate and continuous integration with the application so that it can detect failure in sub-seconds and take corrective actions as needed. This often imposes a more complex scalability challenge.

Cloudify 2.x uses a hierarchy of managers, where each manager controls 100+ nodes as a scaling architecture. One of the things that we have experienced over the last year is that there is a growing class of services such as  Big Data and NFV. With these services, a single service can be comprised out of more than 100s and potentially even 1000s of instances. This is why the manager of managers approach didn’t fit well to these use cases.

To enable scaling of 1000s of nodes per manager, we decided to use a message broker approach based on AMQP as well as to separate the provisioning, logging and real-time monitoring tasks into separate services that can scale independently. This allows us to also control the level of intimacy between the Cloudify manager and the application services. As an example, we can tune Cloudify to handle the provisioning and logging only, and use the real-time information that has already been gathered by the infrastructure. This is especially important in the context of OpenStack, as we expect that a large part of the metrics can be gathered through the infrastructure itself.

Integrating your DevOps toolchain of choice

One thing that has become clear through the various OpenStack statistics based on their annual survey, is that many of the OpenStack users are using a toolchain that is comprised of tools available through OpenStack infrastructure, as well as tools that are not specific to OpenStack such as Docker, Chef, Puppet, Ansible, JClouds and others.

Integrating all these tools as part of a deployment system becomes a fairly complex task that can takes weeks and even months.

We realized that if we want to provide a full lifecycle management of the application, it wouldn’t be right to force a particular stack or toolchain on a user, but rather provide an open pluggable architecture that could enable you to easily integrate your choice of tools into the same deployment.

With Cloudify 2.x we included support for Chef, Puppet and also had a cloud driver that allows us to integrate with various cloud infrastructures, as well as with tools like JClouds.

With Cloudify 3.x we created a more generic plugin architecture that allows users to plug in almost any element of their application, starting from the cloud plugin to configuration management, orchestration, monitoring etc.

To make this simple we included a Bash plugin that provides a simple framework for integration with external tools, just by pointing the plugin to the relevant Bash script.

Final Notes

In this post I thought sharing the story behind the scenes that we went through to make Cloudify OpenStack native, and the rationale and decision process behind them, would be relevant to other users that are considering their OpenStack strategy.  Especially for those still deciding how deep they should integrate with OpenStack.

The good news is that, it is quite possible to integrate natively with OpenStack without limiting yourself only to OpenStack. Having said that, the initial investment required to get there is quite big. I hope that this lesson from our own experience will save others time.

Open Cloud Deconstructed. What does it mean to be OpenStack Native?

OpenStack is just not like any other cloud.

OpenStack is an open source cloud and that makes it quite unique in comparison to many of the closed source public clouds generally available today, the most common being AWS, Azure GCE, and others.

The fact that it is open source, allows us to integrate with the lower level of the stack and know exactly how each element of the infrastructure is designed and implemented.

As with any open source project we have the option to contribute and influence how the OpenStack cloud ultimately shapes out.

Given the above background, when we're designing an integration with OpenStack, we should approach it differently than we approach other clouds. A more native integration with OpenStack involves two main angles:

• Integration with OpenStack core and infrastructure services, which usually includes this full OpenStack component list - as well as those that are continually being developed and added with every release.

• Follow the OpenStack design principles - OpenStack is written in  Python and uses a certain stack for messaging, database and such . To be OpenStack native you should align yourself with the OpenStack design principles as closely as possible.

Current options to be a part of the OpenStack ecosystem

In addition to the general guidelines outlined above, there are couple of additional options to be part of OpenStack. These options mostly fit in with those looking to take an OpenStack *only* approach, and require heavy involvement in the OpenStack community project.

• OpenStack Core - be a part of the OpenStack distribution

• OpenStack StackForge is an incubation framework that allows you to add new projects into the OpenStack ecosystem. StackForge uses the same set of tools and frameworks that are used to develop the core OpenStack services, and as such it allows smoother transition from an incubation project into the core services. That makes this option more relevant for projects that are likely to be included as an OpenStack core service once they've successfully passed the incubation stages.

• OpenStack Marketplace - The marketplace is a web directory that lists the various ecosystem players and their OpenStack solutions. Currently the marketplace is focused on OpenStack distros, training, consulting and drivers, and doesn’t cover solutions on top of OpenStack.

What’s missing?

While there is a fairly clear process, and tool chain for handling core and incubated services on OpenStack, these options fit mostly with services that are primarily designed to only run on OpenStack. What’s missing is a clear certification process that will allow users to find solutions in the ecosystem, outside of OpenStack, and find out quickly what their level of integration and support for OpenStack is. This should include information such as version support, and platform the solution was certified on, which of the OpenStack services are integrated, how much of the stack of the solution is compatible with that of OpenStack’s, and other such important information. A good and simple starting point could be the OpenStack marketplace, however this is just the first step. What’s needed is a set of frameworks similar to StackForge but much more permissive that will provide a means for validating the compatibility with OpenStack.

Why should you care?

As with any popular framework there are many tools that claim support for OpenStack, however in many cases the integration mostly includes basic integration with the OpenStack compute API (Nova). A good example for this is Cloud Foundry, or other tools such as Chef, Puppet, Docker, just to name a few. While these tools do have support for OpenStack, the level of integration can vary quite dramatically. Quite often, each of these  solutions comes with a parallel stack for handling security, monitoring and other issues, which isn’t compatible with the OpenStack infrastructure. As an end user, it is important that to be able to find out quickly which of these products comes with basic support for OpenStack, and which one takes a more native approach to OpenStack. There are many implications to the level of complexity and also to the level vulnerability of those services based on the support level.

OpenStack should attract more ecosystem projects rather than competing with them

In my opinion, there has been too much focus developing more alternative projects for the OpenStack project in comparison to the focus that has been placed on projects that exist within the broader ecosystem. The main reason for this, is the lack of native support for OpenStack. A good example of this, is project Solum and its ecosystem equivalents Cloud Foundry and OpenShift. 

I think that it’s time that OpenStack realign its focus from growing all around the stack, to supporting and attracting more ecosystem projects, that focus on adding native support to OpenStack. In order to do this, there needs to be more emphasis on creating certification, compatibility and testing projects. Additionally, the marketplace initiative needs to receive more attention and investment in order to make it more than a web-based directory service, similar to the AWS marketplace.

I would like to propose a discussion on that subject during the next summit in Paris. Do you think this is a good or bad idea? I’d appreciate it if you could share your thoughts.

OpenStack by the Numbers

One thing that I like about working with an open source community like OpenStack is that you get direct access not just to the source code itself, but to its users and ecosystem as well.  This allows you to be less exposed to marketing spins, and actually analyze our actions based on real numbers.

The OpenStack Foundation conducts a very insightful survey twice a year which helps to measure how users are using the technology. This is an important and useful feedback loop that goes to the PTL of the various projects, and can help product companies to analyze their go to market plans and such. You can find the source

I found the original version a bit too verbose, so I took the liberty to put together an executive summary of the main points from the survey, mostly targeted at decision makers. I hope that you’ll find it useful, any feedback would be highly appreciated.

Main Reasons for Choosing OpenStack

The main reason cited for choosing OpenStack is Open Technology and Avoiding Lock-in.

Interestingly enough Cost Saving comes only 4th!

 

Size of Deployments

The large majority of the current deployments are under 100 nodes.

Around 30% of the deployments run relatively large scale deployments between 500-1000.

QA/DEV	Production

Production seems to follow a similar trend but lower in numbers to QA, which is reasonable as I believe that some of the QA deployments represent a transition stage to production.

 

User Type

The majority of the users seem to be service providers and cloud operators.

There is also a good number of ISVs and ecosystem partners, which is a strong indication that OpenStack is indeed an ecosystem.

There's a very strong presence of OpenStack in the Telco industry. Finance seem to be lagging behind the adoption curve with only three financial organizations declaring their use of OpenStack.

The majority of the deployments run on private or hosted private clouds. About 20% run on public or hybrid clouds. OpenStack also seems to attract mostly small to mid-size companies.

 

Private/Public Clouds	Size of Organizations

 

Deployment Tools

The survey includes two categories for deployment tools, one for those that are used for deploying the OpenStack infrastructure, and the other that are used for application configuration (covered in the application survey).

According to the two surveys users tend to use different sets of tools for the infrastructure deployment and for application configuration management. Puppet seems to be the most popular choice for OpenStack users in both categories. 

One of the interesting results is to see that Bare Metal and Docker are starting to appear in more deployments. This is fairly impressive given that the two are not yet officially supported by OpenStack.  VMware is around 10%-13% of KVM of Virtualization similar to XEN.

 

OpenStack Deployment Tools

 

Applications on top of OpenStack

As noted above, Puppet is popular for application configuration, as well as infrastructure deployment. Heat and Docker take the 2nd and 3rd place respectively,  putting Chef behind in the 4th place.

Given the popularity of Heat during the last summit I wouldn’t be surprised if Heat and Docker will surpass the use of Puppet in this category.

 

 

Popular Frameworks

Interestingly enough, JClouds seems to be gaining popularity amongst OpenStack users. This may be an indication that ~ ¼ of the users are interested in frameworks that are not tied only OpenStack. Not surprisingly, Python is far more popular than any other language, however Java seems to have a good and dominant presence more than other frameworks. One possible explanation is that the type of users that are attracted to OpenStack are more of the traditional enterprise organizations, and less of the new web-facing companies.

Operating Systems

Ubuntu and CentOS are the most popular choice. Windows adoption amongst OpenStack users is between 10-20%.

<

Application Survey	Production Survey

 

Compatibility / Portability

EC2 is a relatively popular choice which indicates that there is an increasing desire among the OpenStack community to maintain portability with EC2. This also correlates with the popularity of portability frameworks such as JClouds.

 

QA/DevTest	Production Survey

 

References

• OpenStack user survey presentation
• OpenStack user survey online session from OSS Atlanta 2014

Making Chef as Simple as Ansible

In his book Taste Test, Matt Jaynes compares a few popular configuration management tools such as Chef, Puppet, SaltStack and Ansible.

Jaynes points to some of the areas of complexity within Chef in comparison to Ansible. According to Jaynes, Chef’s areas of complexity can be summarized by the following:

• Ease of Installation: Chef is not easy to install

• Agent Installation: Chef doesn’t provide a remote agent installation 

• Remote Execution and Update: Chef lacks remote execution and push update

• Ordering: Controlling the execution order of tasks is not intuitive

Making Chef as Simple as Ansible using Cloudify

One of the things that has kept me busy lately is how to make the experience of using Cloudify and Chef a smooth and simple experience. We wanted to get to the point where users can run a Chef Server or Agent, install cookbooks and wrap it with Cloudify Application Service in just a few minutes without having to download or install any software.

We've launched a new online service that does just that – making Chef on the cloud simple.

The way this works is that once you click on the launch button a new server is created on the cloud (HP OpenStack cloud in this specific case). Next we install the Chef-Server using Cloudify on that machine. After 2-3 minutes once the server is up a new interactive shell window is opened. The interactive shell will allow you to interact with the service by invoking a set of custom commands that are listed just below the interactive shell screen.

If you want to view the progress and process of deployment you can click on the “Manage Your App” button. You can also click on the “Use Chef” to open the Chef web admin.

While we wanted to make the experience simple it was equally important that we wouldn’t take shortcuts in that process for the sake of a demo. To that end, we kept the demo identical to the way you would run things in production. That means that all that is needed to build a similar demo yourself is download Cloudify and follow the same steps in your own environment.

Learn more about how Cloudify makes the Chef experience simple

Here’s how Cloudify maps specifically to the challenges outlined in Matt’s book.

• Ease of Installation: Cloudify comes with built-in server provisioning support that works with a wide range of clouds out of the box, namely OpenStack, AWS, Azure, CloudStack, among others. Cloudify also comes with built-in integration with Chef. Cloudify allows you get your Chef Server installed and running on any of the supported clouds. 

• Agent Installation: Cloudify will provision your application machines, install a Chef agent and wire them up with the Chef server. Or, if you choose to run solo, it will allow you to install the agent with the appropriate cookbooks for that service.

• Ordering: Cloudify allows you to define the dependencies between services and will launch them based on that order.

• Remote Execution and Updates: Cloudify uses a custom-command abstraction that allows you to execute remote shell scripts on a group of instances that belong to a particular service or to a specific instance in a group. This feature is useful to run maintenance operations, such as snapshot in the case of a database or code push in a continuous deployment model.

 

Remote Execution with Cloudify

 

Beyond Deployment Automation

While improving your deployment experience with Chef is reason enough to use Cloudify, there are other benefits to gain as well. The real long-term benefit of using Cloudify is to manage post deployment through built-in management, logging and monitoring capabilities, as well as the built-in workflow for automating failover and auto-scaling processes.

Cloudify 3.0 Monitoring

 

Technical Overview of the Cloudify and Chef integration

To learn more on how Cloudify works and how you can build your own Cloudify/Chef service see the Cloudify Chef integration documentation.

 

What’s next with Cloudify 3.0

The upcoming release of Cloudify (currently in Beta) includes further enhancements - specifically the support of Yaml/TOSCA-based configuration. Using Python to reduce the agent and server footprint, allows custom workflow setup to allow users to add their own set of rules to automate upgrades, fail-over and scaling processes, enhanced monitoring and many other goodies, so stay tuned for further updates shortly..

Final Notes

If you're a Chef or Cloudify user, please share your experience  with me with this new service and, more importantly, let me know if there are areas where we can do even better. I would be most appreciative of your feedback.