Nati Shalom's Blog

OpenStack by the Numbers

One thing that I like about working with an open source community like OpenStack is that you get direct access not just to the source code itself, but to its users and ecosystem as well.  This allows you to be less exposed to marketing spins, and actually analyze our actions based on real numbers.

The OpenStack Foundation conducts a very insightful survey twice a year which helps to measure how users are using the technology. This is an important and useful feedback loop that goes to the PTL of the various projects, and can help product companies to analyze their go to market plans and such. You can find the source

I found the original version a bit too verbose, so I took the liberty to put together an executive summary of the main points from the survey, mostly targeted at decision makers. I hope that you’ll find it useful, any feedback would be highly appreciated.

Main Reasons for Choosing OpenStack

The main reason cited for choosing OpenStack is Open Technology and Avoiding Lock-in.

Interestingly enough Cost Saving comes only 4th!

 

Size of Deployments

The large majority of the current deployments are under 100 nodes.

Around 30% of the deployments run relatively large scale deployments between 500-1000.

QA/DEV	Production

Production seems to follow a similar trend but lower in numbers to QA, which is reasonable as I believe that some of the QA deployments represent a transition stage to production.

 

User Type

The majority of the users seem to be service providers and cloud operators.

There is also a good number of ISVs and ecosystem partners, which is a strong indication that OpenStack is indeed an ecosystem.

There's a very strong presence of OpenStack in the Telco industry. Finance seem to be lagging behind the adoption curve with only three financial organizations declaring their use of OpenStack.

The majority of the deployments run on private or hosted private clouds. About 20% run on public or hybrid clouds. OpenStack also seems to attract mostly small to mid-size companies.

 

Private/Public Clouds	Size of Organizations

 

Deployment Tools

The survey includes two categories for deployment tools, one for those that are used for deploying the OpenStack infrastructure, and the other that are used for application configuration (covered in the application survey).

According to the two surveys users tend to use different sets of tools for the infrastructure deployment and for application configuration management. Puppet seems to be the most popular choice for OpenStack users in both categories. 

One of the interesting results is to see that Bare Metal and Docker are starting to appear in more deployments. This is fairly impressive given that the two are not yet officially supported by OpenStack.  VMware is around 10%-13% of KVM of Virtualization similar to XEN.

 

OpenStack Deployment Tools

 

Applications on top of OpenStack

As noted above, Puppet is popular for application configuration, as well as infrastructure deployment. Heat and Docker take the 2nd and 3rd place respectively,  putting Chef behind in the 4th place.

Given the popularity of Heat during the last summit I wouldn’t be surprised if Heat and Docker will surpass the use of Puppet in this category.

 

 

Popular Frameworks

Interestingly enough, JClouds seems to be gaining popularity amongst OpenStack users. This may be an indication that ~ ¼ of the users are interested in frameworks that are not tied only OpenStack. Not surprisingly, Python is far more popular than any other language, however Java seems to have a good and dominant presence more than other frameworks. One possible explanation is that the type of users that are attracted to OpenStack are more of the traditional enterprise organizations, and less of the new web-facing companies.

Operating Systems

Ubuntu and CentOS are the most popular choice. Windows adoption amongst OpenStack users is between 10-20%.

<

Application Survey	Production Survey

 

Compatibility / Portability

EC2 is a relatively popular choice which indicates that there is an increasing desire among the OpenStack community to maintain portability with EC2. This also correlates with the popularity of portability frameworks such as JClouds.

 

QA/DevTest	Production Survey

 

References

• OpenStack user survey presentation
• OpenStack user survey online session from OSS Atlanta 2014

The Heat is on with OpenStack Icehouse

OpenStack Orchestration Moves up the Stack

OpenStack Icehouse, the Ninth OpenStack Release, was announced at at the last summit and was named for a street in Hong Kong. The Icehouse release is primarily focused on stability and maturity. The release marks another milestone in the growth of the project.  Just a few stats.

Icehouse had:

• 1,202 contributors, a 32 percent increase from the Havana release six months ago
• Approximately 350 new features added.
• 2,902 bug fixes were added in the Icehouse release cycle.

In this post  I’m going to focus on the updates in the OpenStack Heat project, which is a.k.a OpenStack Orchestration.  For a general overview on the Icehouse release I would recommend checking out the OpenStack Icehouse release notes or this article by InfoQ.

Heat Project Icehouse - Main Updates

The Heat project started as a clone of Amazon CloudFormation which is basically a declarative approach to automate the infrastructure set up on Amazon environments.  The Icehouse release marks an evolution of the project up the stack. If in previous releases Heat was mainly used to orchestrate the OpenStack infrastructure, with Icehouse it has added many more capabilities to handle orchestration of the software stack as part of the application boot process.

This shift is reflected in the new HOT template format  which allows you to describe a broader aspect of an application configuration including the new OS::Heat::ScalingPolicy. With the Icehouse release these HOT Templates have become the recommended approach. In addition, there is an ongoing effort to introduce TOSCA (Topology and Orchestration for Cloud Applications) into Heat, which is an OASIS-led standard. A new TOSCA translator project for Heat is now part of StackForge. A preview of the TOSCA simple profile can be found here supported by IBM, GigaSpaces, Huawei, Vnomic a preview version of a TOSCA-based Orchestration project for OpenStack is available with the new Cloudify 3.0 project.

Other updates include new the Trust-based security model that will allow the delegation of user roles and permissions without passing Keystone tokens or credentials. For more information on the new Heat Trust Model see the Steve Hardy post Heat auth model updates - part 1 Trust. In addition, the Icehouse release includes improvements to scalability by allowing the distribution of Heat deployments tasks between multiple Heat engines. 

To learn more about the Heat Project Icehouse updates I’d recommend looking at Steve Baker’sonline presentation

Try Out the Heat Icehouse Release on DevStack

Yoram Weinreb a Lead OpenStack Architect from GigaSpaces wrote a detailed example on how you can setup a Heat Icehouse environment on DevStack. What’s interesting with this example is the addition of Neutron orchestration as part of the simple WordPress example.

To try out the example with Heat Icehouse follow the steps from Yoram’s post.

I will be demonstrating this example at the upcoming OpenStack Design Summit in Atlanta next week in the talk - When Networks Meet Apps.

Making Chef as Simple as Ansible

In his book Taste Test, Matt Jaynes compares a few popular configuration management tools such as Chef, Puppet, SaltStack and Ansible.

Jaynes points to some of the areas of complexity within Chef in comparison to Ansible. According to Jaynes, Chef’s areas of complexity can be summarized by the following:

• Ease of Installation: Chef is not easy to install

• Agent Installation: Chef doesn’t provide a remote agent installation 

• Remote Execution and Update: Chef lacks remote execution and push update

• Ordering: Controlling the execution order of tasks is not intuitive

Making Chef as Simple as Ansible using Cloudify

One of the things that has kept me busy lately is how to make the experience of using Cloudify and Chef a smooth and simple experience. We wanted to get to the point where users can run a Chef Server or Agent, install cookbooks and wrap it with Cloudify Application Service in just a few minutes without having to download or install any software.

We've launched a new online service that does just that – making Chef on the cloud simple.

The way this works is that once you click on the launch button a new server is created on the cloud (HP OpenStack cloud in this specific case). Next we install the Chef-Server using Cloudify on that machine. After 2-3 minutes once the server is up a new interactive shell window is opened. The interactive shell will allow you to interact with the service by invoking a set of custom commands that are listed just below the interactive shell screen.

If you want to view the progress and process of deployment you can click on the “Manage Your App” button. You can also click on the “Use Chef” to open the Chef web admin.

While we wanted to make the experience simple it was equally important that we wouldn’t take shortcuts in that process for the sake of a demo. To that end, we kept the demo identical to the way you would run things in production. That means that all that is needed to build a similar demo yourself is download Cloudify and follow the same steps in your own environment.

Learn more about how Cloudify makes the Chef experience simple

Here’s how Cloudify maps specifically to the challenges outlined in Matt’s book.

• Ease of Installation: Cloudify comes with built-in server provisioning support that works with a wide range of clouds out of the box, namely OpenStack, AWS, Azure, CloudStack, among others. Cloudify also comes with built-in integration with Chef. Cloudify allows you get your Chef Server installed and running on any of the supported clouds. 

• Agent Installation: Cloudify will provision your application machines, install a Chef agent and wire them up with the Chef server. Or, if you choose to run solo, it will allow you to install the agent with the appropriate cookbooks for that service.

• Ordering: Cloudify allows you to define the dependencies between services and will launch them based on that order.

• Remote Execution and Updates: Cloudify uses a custom-command abstraction that allows you to execute remote shell scripts on a group of instances that belong to a particular service or to a specific instance in a group. This feature is useful to run maintenance operations, such as snapshot in the case of a database or code push in a continuous deployment model.

 

Remote Execution with Cloudify

 

Beyond Deployment Automation

While improving your deployment experience with Chef is reason enough to use Cloudify, there are other benefits to gain as well. The real long-term benefit of using Cloudify is to manage post deployment through built-in management, logging and monitoring capabilities, as well as the built-in workflow for automating failover and auto-scaling processes.

Cloudify 3.0 Monitoring

 

Technical Overview of the Cloudify and Chef integration

To learn more on how Cloudify works and how you can build your own Cloudify/Chef service see the Cloudify Chef integration documentation.

 

What’s next with Cloudify 3.0

The upcoming release of Cloudify (currently in Beta) includes further enhancements - specifically the support of Yaml/TOSCA-based configuration. Using Python to reduce the agent and server footprint, allows custom workflow setup to allow users to add their own set of rules to automate upgrades, fail-over and scaling processes, enhanced monitoring and many other goodies, so stay tuned for further updates shortly..

Final Notes

If you're a Chef or Cloudify user, please share your experience  with me with this new service and, more importantly, let me know if there are areas where we can do even better. I would be most appreciative of your feedback.