Nati Shalom's Blog

Ramping Up for the 3rd OpenStack Israel - the Biggest One Yet

OpenStack is one of the fastest growing open source projects in the history and is quickly becoming the de-facto standard for Open Cloud Data Center. The project is gaining maturity quickly with major companies such as PayPal, eBay, BestBuy, Intel, Samsung, Bloomberg and HubSpot using it in production, with major vendors such as HP, Rackspace, IBM, Redhat, Ubuntu and GigaSpaces behind the technology.

On May 27th, we will open the doors for OpenStack Israel for what will our biggest OpenStack Israel yet with over 350 attendees. We welcome top industry leaders to Israel to discuss the latest news in OpenStack, as well as showcase real life case studies from Intel, LivePerson, Alcatel , Mirantis and eNovance. This event would not be possible without support from the OpenStack Foundation and the IGTCloud as well as from our sponsors: GigaSpaces, HP, IBM, Rackspace, Cloudsoft, eNovance and CICC. The conference will be held in the Herzilya Arts Center in Herzilya, Israel on May 27th, 2013.

As a member of the OpenStack Israel Committee, I am excited to see that Israel is one of the three locations leading the adoption of OpenStack in Europe (with Germany and Hungary), with more startups, ISVs and organizations joining this year. Unlike in previous years, this year we have had tremendous support for the event, bringing in new committee members, building an agenda with a record number of speakers from all over the world as well as an unprecedented number of attendee registrations. The event will exhibit great content with fresh case studies as well as technical sessions covering the core aspects of OpenStack. For those who want to get their hands "dirty" with OpenStack, we have also arranged a three day training session following the event through collaboration with StackOps.  For details on the schedule see the event agenda.

With a focus on OpenStack technology, innovation and implementations, attendees will hear actual case studies as speakers from Intel, LivePerson, Alcatel and others will share their user experiences with OpenStack. Guests will learn how leading OpenStack cloud providers, software vendors and system integrators  are now delivering OpenStack-based clouds and solutions to the market. The session tracks will take guests on a technological deep-dive into the inner workings and frameworks around OpenStack technology.

This is sure to be a big event, both in size and in success. I encourage everyone to come out and see how OpenStack has had an impact on countless organizations, both here in Israel and abroad. Register today for your spot at OpenStack Israel. I hope to see you all there.

References:

• Podcast

• OpenStack Grizzly

• OpenStack Portland

Using the Cloudify Player as an Open Source Framework for Building Your Own Cloud Application Marketplace on OpenStack

In the previous post, I outlined the main lessons learned from the Amazon Cloud Marketplace and suggested an alternative approach for building an open marketplace.

In this post I'll refer to a specific implementation of that alternative approach through a new Cloudify Service called the Cloudify Player.

Introduction to the Cloudify Player

Cloudify Player is a new patent-based service that makes the experience of trying and deploying even the most complex, multi-tier big data applications as simple as playing a video on YouTube. The new service currently supports OpenStack-based clouds and is available as a free online service from HP Cloud Services and GigaSpaces’ Cloudify. The source code for the service is also available as an open source project on Github. The Player was developed in collaboration with HP Cloud Services, GigaSpaces’ cloud partner and leading provider of OpenStack-based public cloud services.

Cloudify Player an End User Perspective

The Cloudify Player provides a user-friendly and hassle-free user experience in which applications are "played" in similar fashion to the way you play a video online. Users can choose and browse applications from public catalogs that are based on Github. Similarly to YouTube, users can share the applications or embed them easily on their blog or website.

Users can choose to play any application as a free trial or for full production under their account.

Technical Overview

The Cloudify Player consists of the following components as outlined below:

Web Front End

The Web Front End is based on two main interfaces: a widget interface which allows users to browse through a list of applications and deploy them, and a widget management form is provided to allow users to easily publish new applications.

Users can use one of the two deployment options - a trial deployment provides a free non-gated experience for running the selected application on a single machine. Registered users can provide their cloud credentials and route the application into their cloud environment. In this way, they can also monitor the deployed instances through their cloud management console.

Multi-Tenant Orchestrator and Monitoring

The multi-tenant orchestration and monitoring service is responsible for parsing the selected application recipe and for executing the deployment of that application based on the description provided in the recipe. The web-based management allows users to monitor and manage the deployed application.

Application Analytics

Application activities are sent into an analytics service that produce real-time usage reports on the most-active application, as well as the user experience activities.

Service Catalogue Repository

The Service Catalogue is based on a Github clone.

Users use the repository to store their application recipes, track versions, manage history of updates, etc.

Cloud Pools

Cloudify can be integrated with various public and private clouds using out-of-the-box plug-ins.

There are basically two main cloud pools that are used in this architecture:

1. Free Trial Pool - The free trial pool is used to speed up the deployment experience during the trial stage. This pool is limited for time and is initially set for a single VM per deployment to avoid quick exhaustion of resources from the free pool.

2. Production User Cloud - Users that pass their credentials could use the same interface for deploying application under their account. In this case they are not limited in time nor the number of nodes per application.

Product Overview for IT Managers and DevOps

The Cloudify Player was designed to fit into a DevOps environment in which applications are expected to be modified on a continuous basis after they have been deployed. Users can add new applications easily by wrapping them with a simple script, called a recipe. Recipes can be based on either Chef or Puppet.  Recipes can be modified after they have been published by each end user before the deployment takes place.

Users can monitor and manage their applications using a single management dashboard. The management dashboard allows users to monitor the application topology and behavior as well as view real time logging and even open SSH console if needed.

Summary of the Main Features

• Out-of-the-box support for 100s of products (Leveraging the existing Chef/Puppet support)
• Supports full life cycle of user experience from simple free trial to full production
• Specific support for Big Data clusters, including Database as a Service, Hadoop as a Service, MongoDB as a Service, etc.
• Can run clusters rather than just individual VMs
• Application can run outside of the marketplace using the same tools
• Users can easily customize the services from the catalog for their own purposes, including the possibility to run their own version of each item in the catalog. Advanced users can leverage the built-in Chef & Puppet support for advanced configuration management.
• Built in multi-tenant application management so that each user can monitor and manage all of the products under his/her account using a single application-centric dashboard
• Built-in integration with analytics tools to support tracking of user activity

Using Cloudify to Build Your Own Marketplace

The Cloudify Player was designed to simplify the deployment of applications in the cloud. The open nature of the Cloudify Player provides many of the features required for building a marketplace in your own environment under your brand and with the application of your choice. In addition, we’ve put a lot of emphasise on making the user experience simple while ensuring a smooth transition from a hassle-free trial to full production usage.

You can try out the OpenStack version that we launched in collaboration with HP at www.cloudifysoure.org.

For more information about the Cloudify Player and how it works, please see Uri Cohen’s blog post Introducing the Cloudify Player.

Amazon Marketplace - Not a Fair Play, Thoughts on an Open Marketplace Alternative with OpenStack

The Promise of a Cloud Application Marketplace

The cloud has clearly emerged as the new world. Today, we find ourselves in an era of land-grabbing in which everyone wants the bigger piece of this new world.

Amazon has the advantage of being the first to grab. Others would like to follow in Amazon’s footsteps, but are met with the challenge of not being able to out-pace Amazon.

For those of us who aren’t Amazon, forming a coalition of some sort could be a good strategy.

One way of forming a coalition is through a marketplace, a system that enables faster growth through ecosystem partners.

A good example of how such a dynamic will work can be seen in today's world of mobile applications. Both Apple and Google were able to create strong marketplaces that grow much faster than they would have organically and add significant stickiness to their platforms.

Challenges of Building a Marketplace - Lessons from the Mobile Application Marketplace

A marketplace can be an excellent strategy for forming a coalition of partners and growing your offering significantly faster than you would if you were to try to grow organically.

Having said that, there is a significant advantage to being the first. As we have seen in the mobile application world, others who have tried to build a similar marketplace to that of Google and Apple are facing significant difficulties in attracting partners to their marketplaces (See Microsoft Surface as an example).

The main reason for these challenges is the Chicken vs. the Egg problem. In other words, for any marketplace to work, two essential factors must be in place: suppliers and consumers. Apple and Google already had their consumer-base and, therefore, it was easier for them to attract suppliers. Microsoft, on the other hand, does not yet have the mobile consumer-base on its side and is therefore facing a hard battle to bring in the suppliers.

A Cloud Application Marketplace

A Cloud Application Marketplace is similar in concept to that of mobile applications, but aims at cloud data-center services such as Database services, Application Containers, Messaging, CMS, CRM etc which are deployed on a cloud.

Amazon Cloud Marketplace

The marketplace concept is nothing new to Amazon and is already running a successful marketplace for its eCommerce business.

It only made sense for Amazon to latch on that model and use it as a basis for building its Cloud Application Marketplace, as described here.

The Limitations of Amazon Marketplace

While the promise of the a Cloud Marketplace as an ecosystem play is a no-brainer, an over-simplified implementation of it can yield a diminishing return as noted in Janakiram MSV article AWS Marketplace – An Oversimplification of the Cloud? Janakiram points specifically to the following limitations:

· Persistent Services Limitation:

By default, the database and other persistence-dependent services are confined to the ephemeral storage or instance storage of the EC2 instance. This is a very risky proposition as the data will be lost completely in the case of an instance failure. I wouldn’t be comfortable even running a non-mission-critical application like a WordPress blog on EC2 without attaching an EBS volume.

· Not Designed for Real Production:

Launching a LAMP or Tomcat stack with 1-Click is great for a small proof-of-concept or a demo, but no one would attempt this for a production application. Moving to the Cloud is not just about forklifting an app from on-premise server or a host to a VM. It is definitely much more than that! AWS Marketplace squarely focuses just on compute by launching an EC2 instance with pre-configured stack without considering any of the essential building block services like storage, networking and databases.

· Loadbalancer Limitations:

For example, the Amazon EC2 instances launched through the Marketplace cannot be added to an Elastic Loadbalancer. I attempted to launch 2 instances of the free Tomcat 6.0/MySQL Stack provided by OpenLogic and adding them to an Elastic Loadbalancer. ELB threw an error and refused to add the instances. This is a huge limitation for any real-world web application that intends to run on a Tomcat instance launched via the Marketplace. I wouldn’t be comfortable running my web application in just one instance without any possibility of scaling out in the future.

The Limitations of the VM-Centric Approach

Cloud applications are often packaged and run on VMs. It seems only natural that if applications are packaged on VMs, a marketplace would be a repository of pre-packed VMs, as in the case of the AWS marketplace.

Having said that, there are some basic flaws in this assumption - cloud applications and services are much more complex. Many of the services and applications need to run on high-availability mode, which often means that they need to run in more than one availability zone with inter-connectivity between them. In addition, many of the services, such as database services or web-services, need to run in a cluster deployment. In order to do so, multiple instances of VMs must be deployed. As real life applications tend to be multi-tier, there needs to be a way to deploy a set of VMs with some level of interdependency between them. When running an application on the cloud, we often need to think of it as a living organism that continues to change and evolve, rather than one-off deployment.

All these factors make the VM-centric approach a good trial and demo tool, but as Jankiram pointed out, a significant over-simplification of what cloud applications tend to be.

AWS Marketplace - Not a Fair Play

In order for a marketplace to be successful, it is essential that all players will get a fair shot to win consumers.

A marketplace owner in this context needs to take the role of "honest broker."

Amazon offers a long list of its own set of services as products outside the marketplace that don't face the limitations I listed above, in contrast to the applications that do run on the marketplace and are met with these challenges. This gives Amazon an unfair advantage over its own partners.

 

 

Building an Open Marketplace on OpenStack

The creation of an open marketplace would serve as an alternative to that of Amazon. A good substrate for that alternative approach would be to use open cloud frameworks like OpenStack. OpenStack already has a growing community of users and a growing support of cloud providers which provide an excellent starting point for the marketplace.

Below are some of the principles that I believe should serve as a foundation for building an open marketplace.

A Recipe-Based Marketplace

While the basic unit of deployment in many clouds tends to be based on VMs in many *real* deployments, the VM is used mostly as a generic container of the operating-system. The actual application services are deployed through a DevOps automation tool such as Chef, Puppet or Cloud Formation (Heat is the OpenStack alternative for CloudFormation on OpenStack) after the VM has been created. In this way, the deployment unit is the recipe rather than the VM. The recipe is used to describe the application services that will be deployed into the VM after it has been created. Unlike a VM, a recipe can be extended beyond the boundary of a VM and can describe fairly complex, multi-tier, clustered applications. It can also deal with the automation of post-deployment processes such as upgrades, monitoring, fail-over etc.

Based on that description, it only makes sense that we will use recipes as the unit of deployment instead of VMs. In this way, we can overcome many of the AWS marketplace limitations that are outlined above. As there is no standard recipe model as of yet, we'll need to be able to integrate with existing recipes such as Chef, Puppet and CloudFormation.

Other Considerations

Having a recipe-based marketplace is a good starting point. However, there is much more to building a marketplace than to choosing the recipe-based engine upon which to run an application. The main challenge is around the user experience and mastering the balancing act of keeping the marketplace attractive for both the consumers and suppliers.

Below are some of the key principles that I believe should be taken into consideration when building an open marketplace:

• Open source - An open source marketplace would enable different cloud providers to offer and share similar applications and services across different cloud providers.
• Popular frameworks – The marketplace should be based on popular open frameworks such as Chef, Puppet or CloudFormation/Heat templates.
• Reduce the barrier to entry - Make the investment of porting the application to the marketplace worth the effort beyond the marketplace. To reduce the barrier of porting applications into the marketplace we need to design it in such a way that we could use the same elements for deploying our application outside the marketplace.
• Support both multi-tier and cluster deployments
• Extensible - Users can pick a service from the marketplace and easily customize it to their needs.
• Fair play - Applications that are running through the marketplace should have the exact same capabilities as any application that is running outside of the marketplace.

If You Build It, They Will Come - Doesn't Work

These days I'm seeing many cloud providers trying to mimic Amazon’s success and launch their own marketplace. As in the case of Amazon, they also start with a VM-centric approach and thus, inherit many of its limitations. Furthermore, the main problem that I see with many of those attempts is that they seem to be taking the "if you build it, they will come" approach, not realizing that they don't have the advantage of being first and, more importantly, are not at the same point where Amazon was when it launched its marketplace ( i.e. with a huge consumer-base running on their platform).

As a result, I personally believe that many of those attempts are doomed to fail.

The real opportunity IMHO is to take a more open approach that could work outside the boundaries of a specific cloud provider. OpenStack could serve as a great starting point for forming this ecosystem.

To illustrate the points from this post on what an Open Marketplace could look like, in the next post I'll refer to how we use Cloudify as an open source framework for building your own cloud application marketplace on OpenStack.