Nati Shalom's Blog

Ramping Up for the 3rd OpenStack Israel - the Biggest One Yet

OpenStack is one of the fastest growing open source projects in the history and is quickly becoming the de-facto standard for Open Cloud Data Center. The project is gaining maturity quickly with major companies such as PayPal, eBay, BestBuy, Intel, Samsung, Bloomberg and HubSpot using it in production, with major vendors such as HP, Rackspace, IBM, Redhat, Ubuntu and GigaSpaces behind the technology.

On May 27th, we will open the doors for OpenStack Israel for what will our biggest OpenStack Israel yet with over 350 attendees. We welcome top industry leaders to Israel to discuss the latest news in OpenStack, as well as showcase real life case studies from Intel, LivePerson, Alcatel , Mirantis and eNovance. This event would not be possible without support from the OpenStack Foundation and the IGTCloud as well as from our sponsors: GigaSpaces, HP, IBM, Rackspace, Cloudsoft, eNovance and CICC. The conference will be held in the Herzilya Arts Center in Herzilya, Israel on May 27th, 2013.

As a member of the OpenStack Israel Committee, I am excited to see that Israel is one of the three locations leading the adoption of OpenStack in Europe (with Germany and Hungary), with more startups, ISVs and organizations joining this year. Unlike in previous years, this year we have had tremendous support for the event, bringing in new committee members, building an agenda with a record number of speakers from all over the world as well as an unprecedented number of attendee registrations. The event will exhibit great content with fresh case studies as well as technical sessions covering the core aspects of OpenStack. For those who want to get their hands "dirty" with OpenStack, we have also arranged a three day training session following the event through collaboration with StackOps.  For details on the schedule see the event agenda.

With a focus on OpenStack technology, innovation and implementations, attendees will hear actual case studies as speakers from Intel, LivePerson, Alcatel and others will share their user experiences with OpenStack. Guests will learn how leading OpenStack cloud providers, software vendors and system integrators  are now delivering OpenStack-based clouds and solutions to the market. The session tracks will take guests on a technological deep-dive into the inner workings and frameworks around OpenStack technology.

This is sure to be a big event, both in size and in success. I encourage everyone to come out and see how OpenStack has had an impact on countless organizations, both here in Israel and abroad. Register today for your spot at OpenStack Israel. I hope to see you all there.

References:

• Podcast

• OpenStack Grizzly

• OpenStack Portland

Using the Cloudify Player as an Open Source Framework for Building Your Own Cloud Application Marketplace on OpenStack

In the previous post, I outlined the main lessons learned from the Amazon Cloud Marketplace and suggested an alternative approach for building an open marketplace.

In this post I'll refer to a specific implementation of that alternative approach through a new Cloudify Service called the Cloudify Player.

Introduction to the Cloudify Player

Cloudify Player is a new patent-based service that makes the experience of trying and deploying even the most complex, multi-tier big data applications as simple as playing a video on YouTube. The new service currently supports OpenStack-based clouds and is available as a free online service from HP Cloud Services and GigaSpaces’ Cloudify. The source code for the service is also available as an open source project on Github. The Player was developed in collaboration with HP Cloud Services, GigaSpaces’ cloud partner and leading provider of OpenStack-based public cloud services.

Cloudify Player an End User Perspective

The Cloudify Player provides a user-friendly and hassle-free user experience in which applications are "played" in similar fashion to the way you play a video online. Users can choose and browse applications from public catalogs that are based on Github. Similarly to YouTube, users can share the applications or embed them easily on their blog or website.

Users can choose to play any application as a free trial or for full production under their account.

Technical Overview

The Cloudify Player consists of the following components as outlined below:

Web Front End

The Web Front End is based on two main interfaces: a widget interface which allows users to browse through a list of applications and deploy them, and a widget management form is provided to allow users to easily publish new applications.

Users can use one of the two deployment options - a trial deployment provides a free non-gated experience for running the selected application on a single machine. Registered users can provide their cloud credentials and route the application into their cloud environment. In this way, they can also monitor the deployed instances through their cloud management console.

Multi-Tenant Orchestrator and Monitoring

The multi-tenant orchestration and monitoring service is responsible for parsing the selected application recipe and for executing the deployment of that application based on the description provided in the recipe. The web-based management allows users to monitor and manage the deployed application.

Application Analytics

Application activities are sent into an analytics service that produce real-time usage reports on the most-active application, as well as the user experience activities.

Service Catalogue Repository

The Service Catalogue is based on a Github clone.

Users use the repository to store their application recipes, track versions, manage history of updates, etc.

Cloud Pools

Cloudify can be integrated with various public and private clouds using out-of-the-box plug-ins.

There are basically two main cloud pools that are used in this architecture:

1. Free Trial Pool - The free trial pool is used to speed up the deployment experience during the trial stage. This pool is limited for time and is initially set for a single VM per deployment to avoid quick exhaustion of resources from the free pool.

2. Production User Cloud - Users that pass their credentials could use the same interface for deploying application under their account. In this case they are not limited in time nor the number of nodes per application.

Product Overview for IT Managers and DevOps

The Cloudify Player was designed to fit into a DevOps environment in which applications are expected to be modified on a continuous basis after they have been deployed. Users can add new applications easily by wrapping them with a simple script, called a recipe. Recipes can be based on either Chef or Puppet.  Recipes can be modified after they have been published by each end user before the deployment takes place.

Users can monitor and manage their applications using a single management dashboard. The management dashboard allows users to monitor the application topology and behavior as well as view real time logging and even open SSH console if needed.

Summary of the Main Features

• Out-of-the-box support for 100s of products (Leveraging the existing Chef/Puppet support)
• Supports full life cycle of user experience from simple free trial to full production
• Specific support for Big Data clusters, including Database as a Service, Hadoop as a Service, MongoDB as a Service, etc.
• Can run clusters rather than just individual VMs
• Application can run outside of the marketplace using the same tools
• Users can easily customize the services from the catalog for their own purposes, including the possibility to run their own version of each item in the catalog. Advanced users can leverage the built-in Chef & Puppet support for advanced configuration management.
• Built in multi-tenant application management so that each user can monitor and manage all of the products under his/her account using a single application-centric dashboard
• Built-in integration with analytics tools to support tracking of user activity

Using Cloudify to Build Your Own Marketplace

The Cloudify Player was designed to simplify the deployment of applications in the cloud. The open nature of the Cloudify Player provides many of the features required for building a marketplace in your own environment under your brand and with the application of your choice. In addition, we’ve put a lot of emphasise on making the user experience simple while ensuring a smooth transition from a hassle-free trial to full production usage.

You can try out the OpenStack version that we launched in collaboration with HP at www.cloudifysoure.org.

For more information about the Cloudify Player and how it works, please see Uri Cohen’s blog post Introducing the Cloudify Player.

Amazon Marketplace - Not a Fair Play, Thoughts on an Open Marketplace Alternative with OpenStack

The Promise of a Cloud Application Marketplace

The cloud has clearly emerged as the new world. Today, we find ourselves in an era of land-grabbing in which everyone wants the bigger piece of this new world.

Amazon has the advantage of being the first to grab. Others would like to follow in Amazon’s footsteps, but are met with the challenge of not being able to out-pace Amazon.

For those of us who aren’t Amazon, forming a coalition of some sort could be a good strategy.

One way of forming a coalition is through a marketplace, a system that enables faster growth through ecosystem partners.

A good example of how such a dynamic will work can be seen in today's world of mobile applications. Both Apple and Google were able to create strong marketplaces that grow much faster than they would have organically and add significant stickiness to their platforms.

Challenges of Building a Marketplace - Lessons from the Mobile Application Marketplace

A marketplace can be an excellent strategy for forming a coalition of partners and growing your offering significantly faster than you would if you were to try to grow organically.

Having said that, there is a significant advantage to being the first. As we have seen in the mobile application world, others who have tried to build a similar marketplace to that of Google and Apple are facing significant difficulties in attracting partners to their marketplaces (See Microsoft Surface as an example).

The main reason for these challenges is the Chicken vs. the Egg problem. In other words, for any marketplace to work, two essential factors must be in place: suppliers and consumers. Apple and Google already had their consumer-base and, therefore, it was easier for them to attract suppliers. Microsoft, on the other hand, does not yet have the mobile consumer-base on its side and is therefore facing a hard battle to bring in the suppliers.

A Cloud Application Marketplace

A Cloud Application Marketplace is similar in concept to that of mobile applications, but aims at cloud data-center services such as Database services, Application Containers, Messaging, CMS, CRM etc which are deployed on a cloud.

Amazon Cloud Marketplace

The marketplace concept is nothing new to Amazon and is already running a successful marketplace for its eCommerce business.

It only made sense for Amazon to latch on that model and use it as a basis for building its Cloud Application Marketplace, as described here.

The Limitations of Amazon Marketplace

While the promise of the a Cloud Marketplace as an ecosystem play is a no-brainer, an over-simplified implementation of it can yield a diminishing return as noted in Janakiram MSV article AWS Marketplace – An Oversimplification of the Cloud? Janakiram points specifically to the following limitations:

· Persistent Services Limitation:

By default, the database and other persistence-dependent services are confined to the ephemeral storage or instance storage of the EC2 instance. This is a very risky proposition as the data will be lost completely in the case of an instance failure. I wouldn’t be comfortable even running a non-mission-critical application like a WordPress blog on EC2 without attaching an EBS volume.

· Not Designed for Real Production:

Launching a LAMP or Tomcat stack with 1-Click is great for a small proof-of-concept or a demo, but no one would attempt this for a production application. Moving to the Cloud is not just about forklifting an app from on-premise server or a host to a VM. It is definitely much more than that! AWS Marketplace squarely focuses just on compute by launching an EC2 instance with pre-configured stack without considering any of the essential building block services like storage, networking and databases.

· Loadbalancer Limitations:

For example, the Amazon EC2 instances launched through the Marketplace cannot be added to an Elastic Loadbalancer. I attempted to launch 2 instances of the free Tomcat 6.0/MySQL Stack provided by OpenLogic and adding them to an Elastic Loadbalancer. ELB threw an error and refused to add the instances. This is a huge limitation for any real-world web application that intends to run on a Tomcat instance launched via the Marketplace. I wouldn’t be comfortable running my web application in just one instance without any possibility of scaling out in the future.

The Limitations of the VM-Centric Approach

Cloud applications are often packaged and run on VMs. It seems only natural that if applications are packaged on VMs, a marketplace would be a repository of pre-packed VMs, as in the case of the AWS marketplace.

Having said that, there are some basic flaws in this assumption - cloud applications and services are much more complex. Many of the services and applications need to run on high-availability mode, which often means that they need to run in more than one availability zone with inter-connectivity between them. In addition, many of the services, such as database services or web-services, need to run in a cluster deployment. In order to do so, multiple instances of VMs must be deployed. As real life applications tend to be multi-tier, there needs to be a way to deploy a set of VMs with some level of interdependency between them. When running an application on the cloud, we often need to think of it as a living organism that continues to change and evolve, rather than one-off deployment.

All these factors make the VM-centric approach a good trial and demo tool, but as Jankiram pointed out, a significant over-simplification of what cloud applications tend to be.

AWS Marketplace - Not a Fair Play

In order for a marketplace to be successful, it is essential that all players will get a fair shot to win consumers.

A marketplace owner in this context needs to take the role of "honest broker."

Amazon offers a long list of its own set of services as products outside the marketplace that don't face the limitations I listed above, in contrast to the applications that do run on the marketplace and are met with these challenges. This gives Amazon an unfair advantage over its own partners.

 

 

Building an Open Marketplace on OpenStack

The creation of an open marketplace would serve as an alternative to that of Amazon. A good substrate for that alternative approach would be to use open cloud frameworks like OpenStack. OpenStack already has a growing community of users and a growing support of cloud providers which provide an excellent starting point for the marketplace.

Below are some of the principles that I believe should serve as a foundation for building an open marketplace.

A Recipe-Based Marketplace

While the basic unit of deployment in many clouds tends to be based on VMs in many *real* deployments, the VM is used mostly as a generic container of the operating-system. The actual application services are deployed through a DevOps automation tool such as Chef, Puppet or Cloud Formation (Heat is the OpenStack alternative for CloudFormation on OpenStack) after the VM has been created. In this way, the deployment unit is the recipe rather than the VM. The recipe is used to describe the application services that will be deployed into the VM after it has been created. Unlike a VM, a recipe can be extended beyond the boundary of a VM and can describe fairly complex, multi-tier, clustered applications. It can also deal with the automation of post-deployment processes such as upgrades, monitoring, fail-over etc.

Based on that description, it only makes sense that we will use recipes as the unit of deployment instead of VMs. In this way, we can overcome many of the AWS marketplace limitations that are outlined above. As there is no standard recipe model as of yet, we'll need to be able to integrate with existing recipes such as Chef, Puppet and CloudFormation.

Other Considerations

Having a recipe-based marketplace is a good starting point. However, there is much more to building a marketplace than to choosing the recipe-based engine upon which to run an application. The main challenge is around the user experience and mastering the balancing act of keeping the marketplace attractive for both the consumers and suppliers.

Below are some of the key principles that I believe should be taken into consideration when building an open marketplace:

• Open source - An open source marketplace would enable different cloud providers to offer and share similar applications and services across different cloud providers.
• Popular frameworks – The marketplace should be based on popular open frameworks such as Chef, Puppet or CloudFormation/Heat templates.
• Reduce the barrier to entry - Make the investment of porting the application to the marketplace worth the effort beyond the marketplace. To reduce the barrier of porting applications into the marketplace we need to design it in such a way that we could use the same elements for deploying our application outside the marketplace.
• Support both multi-tier and cluster deployments
• Extensible - Users can pick a service from the marketplace and easily customize it to their needs.
• Fair play - Applications that are running through the marketplace should have the exact same capabilities as any application that is running outside of the marketplace.

If You Build It, They Will Come - Doesn't Work

These days I'm seeing many cloud providers trying to mimic Amazon’s success and launch their own marketplace. As in the case of Amazon, they also start with a VM-centric approach and thus, inherit many of its limitations. Furthermore, the main problem that I see with many of those attempts is that they seem to be taking the "if you build it, they will come" approach, not realizing that they don't have the advantage of being first and, more importantly, are not at the same point where Amazon was when it launched its marketplace ( i.e. with a huge consumer-base running on their platform).

As a result, I personally believe that many of those attempts are doomed to fail.

The real opportunity IMHO is to take a more open approach that could work outside the boundaries of a specific cloud provider. OpenStack could serve as a great starting point for forming this ecosystem.

To illustrate the points from this post on what an Open Marketplace could look like, in the next post I'll refer to how we use Cloudify as an open source framework for building your own cloud application marketplace on OpenStack.

Real-Time Big Data with Storm, Cassandra, and XAP

Real-time Big Data is becoming more and more common in Big Data systems.

One of the most common frameworks used for running real-time Big Data system is Twitter Storm backed by a NoSQL database, as shown in the diagram below:

 

In this architecture, Storm is used for processing data as it comes in and the NoSQL data store is used as the output for that processing as well as for reference data storage.

Challenges

This architecture presents a couple of challenges:

• Performance - Storm runs in-memory, and is therefore set to process large volumes of data at in-memory speed. However, a typical storm architecture needs to interface with a data source for its input and a data source for its output. In this context, the overall performance is now determined not by Storm itself but by the data input and output sources. Quite often these interfaces rely on file-based message queues for streaming, and NoSQL for data storage. These interfaces are at least an order of magnitude slower than Storm itself, and therefore become the limiting factor.
• Complexity - Storm itself consists of several moving parts, such as a coordinator  (ZooKeeper), state manager (Nimbus), and processing nodes (Supervisor). The NoSQL data store also comes with its own cluster management. In addition, a typical Big Data system comes with more components such as the application front end, a reporting database, and more. This makes the process of managing the deployment, configuration, fail-over and scaling of such a system quite complex.

Meeting the Performance Challenge

Given that Storm itself runs in memory, it only makes sense that in order for Storm to run at maximum capacity the streaming and data store interfaces should be implemented in-memory as well, as shown below:

As you can see in the above architecture, we added two interfaces that rely on a built-in Storm plug-in - one for the data inputs and the second for data output. In both cases, the underlying implementation is memory-based, and thus removing the impedance that the previous architecture included.

You can read the full details on how this integration works, including a code example, in DeWayne Filppi's post on integration of in-memory computing with Cassandra.

The second optimization that can be added is to the NoSQL data store.

As most analytics applications tend to be read-mostly, we can speed up access to the NoSQL data store using an in-memory local cache. This architecture speeds up read performance between 10 to 1000 times as outlined by Shay Hassidim in his post on real-time Big Data performance.

Meeting the Complexity Challenge

To meet the complexity challenge, we will use a DevOps automation approach using Cloudify in conjunction with Chef or Puppet.

With this approach, we wrap every service with a deployment recipe that abstracts the underlying details on how to manage Storm, Cassandra, and our in-memory data store. Cloudify uses these recipes to automate the deployment of the entire stack. In this way, you only need to interact with Cloudify for the deployment, configuration, and also scaling and fail-over of your stack rather than separately manage each individual component. In addition, we use Cloudify to abstract the underlying infrastructure. This enables us to use the same deployment recipe across different environments. One for testing, another for production, etc. We can also use the same approach to deploy our apps based on the type of workload. We can use a public cloud for running sporadic workload, and thus leverage the elasticity of the cloud and also enable us to create and scale the environment as well as rip it off completely when done. At the same time, we can choose bare-metal machines for I/O intensive workloads, and so on.

You can read more on how you can set Storm using Cloudify on DeWayne's post on Storm and the cloud.

Cloudify comes with a rich set of built-in recipes for other Big Data services as well, making the integration process an out of the box experience. The main ones are listed below:

Final Notes - Optimizing without Code Change

Many real-time Big Data systems are now based on Twitter Storm and a NoSQL data store such as Cassandra.

In this post I tried to outline how we can optimize this architecture by addressing two areas: performance, and management.

The good news is that all this is possible to achieve seamlessly, without any code change. Let me explain:

In the case of Storm we used built-in plug-ins - Spout for streaming and Trident as the data store interface. In this way we can simply plug our memory-based plug-ins under these two integration points. All our existing Storm business logic would work pretty much the same.

We use the same plug-in approach to integrate our in-memory data store with our NoSQL data. This integration makes the data flow between our real time streaming to our NoSQL storage fairly transparently. In addition, it allows us to plug in different NoSQL data stores such as MongoDB, CouchBase, etc., giving us another degree of flexibility.

The same applies to our management layer. Existing Storm and NoSQL deployments are wrapped into plugged in recipes, and don't require any specific code changes.

Not Mutually Exclusive

We don't have to implement all of the optimizations to gain the benefit of this architecture. Each of the optimization points can be plugged in  independently of the others. For  example, if your main pain point is complexity, you can start by only by adding the DevOps automation first. At the same time if your main pain point is performance, you can use the memory-based plug-ins to speed up processing.

The other advantage of the combined architecture that I haven't discussed in this post is that it provides more flexibility in the degrees of consistency and availability in which you set your system, as I outlined in one of my recent posts on the subject In Memory Computing (Data Grid) for Big Data.

References:

• Big Data Real-Time Performance
• GigaSpaces and Storm Part 2: XAP Integration
• GigaSpaces And Storm Part 1: Storm Clouds
• In Memory Computing (Data Grid) for Big Data
• Check out how GigaSpaces XAP and Cassandra integration works

Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack - My Take

Ruven Cohen published an article on Forbes pointing to Paypal's plan to move a large part of its servers from VMware to OpenStack.

This is aligned also with what seems to be a general trend where enterprises use OpenStack to reduce their VMware cost. They often start with a hybrid approach in which they move the less mission critical apps to OpenStack and keep the rest for VMware. I assume that this is only a transitional stage until we will see almost all the workload moving to OpenStack as soon as the organization will feel confident with the technology. 

Paypal is not the only one who's moving away from VMware; Koby Holzer from Live Person will be speaking on how Live Person uses OpenStack internally to reduce their VMware cost during the upcoming OpenStack Israel event. I'm sure that there are many others who have already taken that path, but were not as public about it.

While the article focuses on the general trend, I thought that there is a bigger opportunity here worth noting.

The Opportunity

Clearly a failure of one player (VMware in this specific case) is an opportunity for others.

The opportunity is for OpenStack providers, such as HP, Redhat, Rackspace and IBM, to take-on VMware directly rather than focus on winning on Amazon.

Why?

• VMware has a huge enterprise footprint
• Private and hybrid clouds are at the heart of enterprise cloud strategy
• OpenStack is well positioned for the private cloud, given its open source foundation
• Having organizations such as HP, Rackspace, Redhat, Mirantis and recently IBM behind OpenStack should also provide enough comfort that the gaps that still exist will be bridged shortly in a way that will fit the enterprise requirements.

Timing is Everything..

OpenStack was created initially by Rackspace and NASA as a response to the strong dominance of Amazon as the leader in the cloud space. At that time, Amazon was the benchmark and the reference for the OpenStack project.

Many things have changed since OpenStack was first born that, IMHO, need to change some of the fundamental strategies for many of the OpenStack players.

• Amazon was quick to realize the threat and moved remarkably fast up the stack, leaving enough of a gap to preserve its leadership position.
• VMware's response, on the other hand, was fairly poor, despite various attempts to move up the stack, including major acquisition of SpringSource in 400M$, followed by a few other failed attempts, such as VMforce which was later forked into CloudFondry which recently moved out to a new pivotal fork under EMC. Now VMware seems to be taking another turn with its continuously changing cloud strategy, but without real substance. This creates lots of skeptisim in the market over their ability to execute their plans, as noted in Bernard Golden's report: VMware Tantrum Shows It's Not Connecting With Cloud Buyers or Sellers

Like VMware, many of the traditional enterprise players are now threatened by Amazon and need to fight back to defend their current position in the enterprise space.

From a technology perspective, the gap between OpenStack and VMware's offering is much smaller than with AWS. Most of those gaps can be easily bridged by the likes of HP, Redhat, Ubuntu, IBM and Rackspace.

All this makes VMware the right target for OpenStack providers rather than Amazon, at least at this point in time. With that in mind, the right strategy for OpenStack providers is to offer a VMware alternative based on OpenStack and have both a private and public offering of OpenStack that can be easily plugged together as a single compute cloud.

References

• Comparing VMware and OpenStack
• Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack (Updated)
• Moving from VMware to OpenStack talk - Live Person 
• VMware Tantrum Shows It's Not Connecting With Cloud Buyers or Sellers
• VMWARE'S GROWTH STALLS – LAYOFFS COMING, BUT SO ARE NEW HIRES

DevOps, PaaS and Everything in Between

This month Amazon announced AWS OpsWorks, which is the result of their acquisition of Peritor and the embedding of its Scalarium platform within AWS. OpsWorks is a new framework for simplifying the deployment of applications on the cloud.

Given that Amazon already has a few frameworks automating application deployment, such as CloudFormation and Elastic Beanstalk, I find the addition of OpsWorks to the mix to be an important recognition of the void that exists between the existing PaaS and Automation frameworks.

 

In this post I'll try to zoom in to that void between PaaS and DevOps.

I'll start with a post that I wrote a few months back - PaaS Jailbreaker, in which I pointed out the two models for delivering applications in the cloud, from PaaS to DevOps. In this post I suggested that rather than thinking of the two models as separate entities, it would make sense to combine the two.

James Urquhart wrote an excellent article on GigaOM - Is your PaaS composable or contextual? In his article, Urquhart refers to the differences between the two approaches to PaaS as composable vs contextual. A good example for a contextual approach is the plugin-based approach implemented by Heroku or Force.com. Plugins provide a flexible way to extend the platform, but the extension needs to fit into a certain model. For example, in the case of Heroku you don't control the actual VM or underlying cloud infrastructure, nor the network that is provided for that infrastructure. A composable approach, on the other hand, is focused on the composition of the different application components and, as such, doesn't need to assume anything about the underlying infrastructure that runs the components of our application. Amazon OpsWorks fits into that second category.

Urquhart specifically pointed out Dietzler’s Law for Access as a way to illustrate the limitation of the contextual approach.  While the contextual approach (customization via plugins) provides some degree of flexibility, it is the 10% of the things that you can't control that matters most, as they define whether you'll be able to be deliver your application the way you want to:

Every Access project will eventually fail because, while 80% of what the user wants is fast and easy to create, and the next 10% is possible with difficulty, ultimately the last 10% is impossible because you can’t get far enough underneath the built-in abstractions, and users always want 100% of what they want.

Interestingly enough, Amazon launched OpsWorks a week later.

The introduction of OpsWorks was quite interesting as it came two years after Amazon launched its Heroku/GAE equivalent - Elastic Beanstalk. I found the announcement by Amazon quite interesting. The fact that they had a PaaS offering already and still found it necessary to launch a new platform for delivering applications is:

1. Acknowledgement that the current PaaS offering isn't a one-size-fits-all solution, and despite the various extensions that Amazon delivered to support other languages within the platform, there are still many applications, and consequently organizations, that do not fit into the Elastic Beanstalk model.
2. Recognition of the fact that within the realm of the application delivery, there is a need for a more open framework for delivering applications, and that realm needs to take a more "composable" approach. In the case of OpsWorks, the composable approach happens to rely on Chef and the OpsWorks orchestration layer on top of it.

Werner Vogels uses the control vs convenience tradeoff as a way to position the various AWS offerings, as he pointed out in his post: Expanding the Cloud - Introducing AWS OpsWorks, a Powerful Application Management Solution:

With the launch of AWS OpsWorks customers now have a very powerful solution that allows them to manage their application easily without giving up control. 

In other words, OpsWorks is aimed at filling a void between the Beanstalk model and the DIY (do it yourself) model.

Interestingly enough, I used to describe the various solutions in the cloud stack in very similar terms to the way Vogels does, with the slight difference that I used the term Control vs Productivity and looked at a broader scope between IaaS and PaaS.

OpsWorks vs Other Kinds of PaaS

In a discussion I had on that subject last week with Ran Tavory and Ori Lahav as part of the Reversim podcast, we seemed to agree that currently, there is space for both models of PaaS - the OpsWorks model (composable) and the Beanstalk model (contextual). Having said that, when I framed the question "which of the frameworks/approaches will run 80% of the cloud workload in the future?" we all seemed to agree that the answer would more likely be OpsWorks. The last point led me to believe that while there is space for the two models right now, this may be only a transition stage, while over time we’ll see that first generation PaaS and the likes of Beanstalk will be replaced with second generation PaaS offerings, as in the case of OpsWorks.

OpsWorks and the blurring line between PaaS and IaaS

The fact that OpsWorks runs on Chef, as do many of the IaaS services in many cloud infrastructures, also raises the question whether it is right to draw a line between our IaaS services and applications, as I pointed out here: The blurring line between PaaS and IaaS.

OpsWorks and Cloudify

OpsWorks and Cloudify follow the same foundational concepts from an architecture perspective - both frameworks integrate with DevOps tools like Chef and add an orchestration layer on top of it, making it easier to compose various parts of the application stack. In many ways you can think of Cloudify as a framework for building your own OpsWorks.

PaaS, DevOps and Everything in Between

Vogels seems to avoid labeling either Beanstalk or OpsWorks as belonging to the PaaS category. Urquhart, on the other hand, looked at the two as two kinds of PaaS solutions - Contextual (Beanstalk) vs Composable (OpsWorks).

One thing that is clear to me is that the current PaaS category is too narrow to fit all the different kinds of PaaS.

Looking at the definition of PaaS in a broader context, I would frame it as a "Platform for running and managing applications on the Cloud." In that context, the way we choose to run those apps becomes an implementation detail. Having that context in mind, I tend to look at both Beanstalk and OpsWorks as two kinds of PaaS offerings. What we’re missing at this point is a simple way to label those different categories.

One approach would be to use Urquhart’s definition (“Contextual vs Composable) which categorizes the different PaaS solutions based on the approach through which they choose to deliver the apps. We can also look at CloudFoundry vs GAE as open vs closed. Another possible category is PaaS frameworks (i.e. frameworks that provide middleware to build your own PaaS), vs PaaS as a self-service solution.

RightScale, on the other hand, uses the term Cloud Application Management to define a similar category and looks at PaaS as a private use-case for using the Cloud Application Management framework. We also had internal debate on that subject at GigaSpaces and couldn't come to a final agreement on where to draw the line between the different categories.

What do you think? Is OpsWorks a new kind of (Composable) PaaS or Cloud Application Management Platform?

Does it really matter?

References:

• GigaSpaces Cloudify & VMware Cloud Foundry the new PaaS Jailbreaker
• Is your PaaS composable or contextual? (Hint: the answer matters)
• Mapping the Cloud/PaaS Stack
• The blurring line between PaaS and IaaS
• Expanding the Cloud - Introducing AWS OpsWorks, a Powerful Application Management Solution
• Amazon adds OpsWorks application life cycle management to AWS cloud
  

 

 

 

 

 

 

 

Carrier Grade Cloud/PaaS

Cloud is becoming the next-generation backbone for running the web.

As the technology for building and running cloud infrastructure matures, it is starting to spread out into more industries and revolutionizing how even the most conservative organizations are running their entire operations.

One of the areas undergoing a transformation is the carrier backbone services. For those who are not familiar, carrier backbone services includes services like cell and network services (DHCP, DNS, …), content serving (SMS, MMS, …), activation services, CRM, call centers, etc. Moving these critical carrier services from the existing environment tends to be labor intensive and proprietary. Selecting to move applications into a more open and virtualized environment such as cloud could yield a significant cost saving. An open cloud environment also enables Carrier organizations to reduce their time to market for delivering new services.

At time where the Telco/Carrier market is increasingly competitive, moving to a cloud-based carrier backbone can be more than a cost saving initiative. It can be a differentiator from the competition and is critical for survivability and success of the business.

That said, running carrier grade services requires special care to meet the required SLAs in terms of  latency, deterministic behavior, performance, location awareness, etc. These challenges are unique enough not to fit into your mom-and-pop cloud.

The purpose of the Carrier Grade Cloud and Carrier Grade PaaS is to address these gaps and challenges.

In this post I'll try to provide a more detailed overview on what Carrier Grade Cloud and PaaS actually means. I will use examples based on GigaSpaces’ joint work with Alcatel. Alcatel-Lucent recently launched a new product in this space named CloudBand which is using Cloudify for its Carrier PaaS layer.

What does Carrier Grade mean?
Learning from the Weather Chanel experience during the Sandy super-storm

The Weather Channel’s experience during Sandy is an excellent example of the need for carrier grade services. Below is a list of some of the key statistics during Hurricane Sandy:

• 1000% - The Weather Company’s traffic increase during Hurricane Sandy
• 110 GB- The amount of data, served every second during Sandy
• 170,000 - Peak number of simultaneous streams of video served during Sandy
• 1 - The amount of data centers that went bust during the storm

To address this demand during the storm, the Weather Chanel was running from 13 Data Centers managed by Verizon across North America, all with load balancing between them. During the storm Verizon increased their bandwidth capacity to meet the peak demand.

This sort of increased traffic behavior wasn't unique to the weather channel, as noted below:

So what can be learned from this process? What makes a service Carrier Grade?

Learning from the Weather Channel experience we can define a Carrier Grade service as a service with the following attributes:

• Critical to the business function
• Designed for massive scale
• Designed to deal with major usage spikes
• Location sensitive
• Designed to provide deterministic response during extreme condition

This is obviously a fairly simplistic definition, but for the sake of this discussion I think it will suffice.

What Makes a Cloud/PaaS Carrier Grade?

There are various attributes that makes a Cloud/PaaS carrier grade, as I noted earlier. The two most important attributes IMO are the network and multi-site deployments. Let me explain why:

The Network

One of the main elements that is extremely important in in a Carrier Grade environment is the ability to assert control over the network.

That include the control over:

• Isolation
• Bandwidth
• Latency

Cross-Cloud/Data Center Deployments:

Another critical element of a successful Carrier environment is the multi-site deployment. As seen with the Weather Channel’s use of 13 sites, multi-site deployment is important to address continues availability and scaling. Optimizing the latency by surveying the content closer to the location of the end user also helps to deal with challenges of data delivery.

So how are things done today?

The current Carrier backbone runs on physical appliances which basically maps to lots of irons. In this environment scaling capacity means buying more appliances. While this model works, it has two main drawbacks:

1. cost (infrastructure/operation)
2. lack of agility (i.e. it takes months and sometimes years to launch new service in this environment).

Alcatel CloudBand -- Carrier Grade IaaS/PaaS

Alcatel CloudBand is a new platform  that let Telco apps easily leverage the carrier cloud services.

It is comprised of a few main elements.

Multi node/site IaaS -- a multi-site/Cloud infrastructure. The CloudBand infrastructure is essentially a policy based management on large numbers of cloud nodes. Each cloud node can run either an OpenStack or CloudStack-based infrastructure. These nodes can live in many disparate data centers. Alcatel CloudBand glues all of the disparate nodes together into a single big cloud that is accessible through an OpenStack API.

CPaaS -- Stands for Carrier Grade PaaS, which is essentially the framework enabling the on-boarding of the carrier services into the CloudBand infrastructure via a simple click and run user interface. Cloudify is integrated into this as an integral part of the CloudBand offering.

CloudBand's Unique Approach: Putting Network and Application Together

One of the unique aspects of the CloudBand architecture is its holistic approach to Network and Application. Standard cloud infrastructures tend to look at the two pieces as separate black boxes that run one on top of the other.

What does this new approach to Network and Applications really means?

Two example scenarios that I often use to describe the value of putting network and application together is in the areas of Disaster Recovery and Cloud Bursting. In today’s cloud, DR involves lots of wiring in which i need to explicitly point a segment of the application into a particular cloud zone and the other to another zone. Beyond the complexity of setting these zones up, it also means that there is a good degree of manual intervention required to handle a recovery or a scaling process in this environment.

Taking an automated SLA-driven approach to IaaS

Instead of identifying explicitly the zones in our cloud, with automated SLA we can simply ask the cloud to figure out the right zone for the job based on our application SLA. For example, a user could simply say something like "deploy RingTone service" where continuous availability=true redundancy=3 and distance between sites=100km". Most of that information is known to the CloudBand management at the time of deployment and it can therefore allocate machine instances not solely on image ID and zone ID, but also based on those SLA requirements.

Integrating the PaaS with the network

Many of the current PaaS solutions were designed to work with a simple cloud infrastructure.

If we design our PaaS solutions to work on top of a more intelligent infrastructure, like CloudBand, that can accept SLA-driven calls to coordinate infrastructure management, a revolution will happen. We can start looking at  offloading some of the responsibility for allocating the right machine instance to a particular application tier to the infrastructure. The infrastructure could be made aware that we're deploying a data service and would therefore ensure that the nodes of that database don't reside on the same physical machine or even the same data center. Another area where the responsibility could be delegated to the infrastructure is the network isolation. Instead of dealing with security groups, the system can attach a particular network for a given application or a tier within that application and the infrastructure will make sure that any machine that is allocated for this service would be attached to this network.

Final Words

For years there has been discussion on the missing piece in the cloud puzzle - the network. Today we're at a point where this gap is starting to be filled up by projects like Quantum in OpenStack. In addition to OpenStack the Telco industry is also launching a new initiative titled NFV which stand for Network Function Virtualization. NFV was born in October of 2012 when AT&T, BT, China Mobile, Deutsche Telekom and many other Telcos introduced the NFV Call to Action document. It basically aims to combine new network API with virtualization and thus provide a standard model for a virtulized Carrier Cloud.

While it seems that the industry is moving in the right direction toward the Virtualization of the backbone systems, most of the effort seem to be focused on standardisation at the lower level of stack. Very little has been done to draw the real end game i.e. how would an end to end Carrier backbone would look like given that new virtualized infrastructure in place. More importantly we haven't yet began to think of what would be the implication of that infrastructure change on the application and services ontop of it.

This is what excites me in the CloudBand project. CloudBand doesn't just end up with yet another fancy infrastructure piece that we don't necessarily know how and what do with it. It actually takes the holistic approach and maps those fancy features into a real end to end solution which at the most basic level maps to the fact that setting up data and network clusters, disaster recovery or cloud bursting scenarios can now be fully automated in a much simpler fashion than in most of the current cloud infrastructure environment.

At a more strategic level that means that Carrier can now rely on the the cloud as an infrastructure that could manage their backbone services and thus leverage the cloud economics to meet their cost and business challenges.

References

• Carrier Grade PaaS presentation by Uri Cohen
• Network Function Virtualization or NFV Explained
• Alcatel & Gigaspaces Joint partnership
• GigaSpaces Cloudify
• Alcatel Carrier Cloud

What is Cloudify? an End User Perspective

I just finished a great call with Joseph Weitekamp

Joe got introduced to Cloudify by one of my friends and shared with me his perspective on what is Cloudify. I realy liked the way he was able to put this togather and thought that the rest of you would be intrested in his perspective as well, so here it is..

What is Cloudify:

Cloudify is a software runtime lifecycle management system for cloud-hosted services.

-          It manages in an automated fashion:

• The provisioning of cloud infrastructure
• The installation and configuration of services on that infrastructure
• Monitors those services for health and parameters.  Respond to SLA violations.
• The uninstallation of service software and deprovisioning of cloud infrastructure
• Provide customized ways to interact with the running system that hide its cloudy deployment.

Summary

-          Cloud-provider agnostic through abstraction layer.   Public or private, ie, Amazon, Rackspace…

-          Recipe-based definition of deployments: end-state and means for achieving it

• Recipes are parameters + app lifecycle event handlers written in Groovy, a Python-like JVM-hosted language.
• Recipes for complex suites of services can be declared, which manages the coordinated configuration of these services.
• Service interdependencies declarable and respected.  Services don’t start until those they depend one are up.
• Recipe parameterization and inheritance (generalization / specialization) allow re-use and standardization

-          Custom Commands can be defined in Groovy and run interactively in a shell or web browser

-          Probes can be defined to surface telemetry on the services.  Standard probes for standard telemetry are provided.  Custom probe can be coded in Groovy or Java

-          Web Service Dashboard (meters and time-series graphs) can be declared to monitor service health/performance

-          Automated SLA Assurance can be declared to assure required number of service instances are always running.  Additional service instances are provisioned, installed and started as necessary

-          Auto-scaling based on telemetry can similar drive the provisioning, installation and startup / shutdown of service instances.

 

 

A Groovy Kind of Java

Today's application stack is a much more complex environment than the traditional JEE one. Moving from a Spring/JEE background to the cloud can be a complicated transition.

In talk given at the San Francisco Java Meetup Group i tried to outline how Groovy can help you map and translate an old JEE evironment to a new cloud-based one using  Cloudify in a way that anyone can adapt to their own needs. You can find the slides from the talk here.

Living with cloud outages as a way of life...and being ok with it

Learning from the experience of others has always been a great source for many of the posts in this blog.

I happened to have had a meeting with Ron Zavner, Applications Architect at GigaSpaces when he reviewed his planned presentation for the AWS Meetup this week in London.

I thought that the information that Ron gathered for this purpose could be extremely beneficial for everyone who is either running, or plans to run their application in the cloud.

In this post I tried to capture in words, the content of Ron’s presentation on the lessons from recent cloud outages.

Recap - History of Cloud Outages

• 21 April 2011 - Some parts of Amazon Web Services suffered a major outage. A portion of volumes utilizing the Elastic Block Store (EBS) service became "stuck" and were unable to fulfill read/write requests. It took at least two days for service to be fully restored. Reddit, one of the better-known sites to go down due to the error, said it has 700 EBS volumes with Amazon. Sites like Quora and Reddit were able to come back online in "read-only" mode, but users couldn't post new content for many hours.
• 29 June 2012 - Several websites that rely on Amazon Web Services were taken offline due to a severe storm of historic proportions in the Northern Virginia area where Amazon's largest data center is located.
• 22 October 2012 - A major outage occurred, affecting many sites, again  such as Reddit, Foursquare, Pinterest, and others. The cause was a latent bug in an operational data collection agent.
• Christmas Eve 2012 - Amazon AWS again suffered an outage, causing websites such as Netflix instant video to be unavailable for some customers, particularly in the Northeastern US. Amazon later issued a statement detailing the issues with the Elastic Load Balancing service that led up to the outage.

Cloud outages are not the sole property of Amazon – they’re everywhere

While most of the more notable failure events happened to be related to Amazon AWS, failures tend be in direct proportion to the usage of the infrastructure, and right now Amazon is probably running the biggest workloads on their cloud, and is growing pretty fast. With these statistics, it’s very likely that their failure would be more notable than others simply because they simply have a wider impact.

As we have experienced, failure has dropped a visit to other cloud infrastructure providers as well.

Microsoft Azure outage

• 28 December 2012 -  some owners of Microsoft's XBox 360 gaming console were unable to access some of their cloud-based storage files.
• 26 July 2012 - Service for Microsoft’s Windows Azure Europe region went down for more than two hours
• 29 February 2012 - The ultimate result was service impacts of 8-10 hours for users of Azure data centers in Dublin, Ireland, Chicago, and San Antonio.

Main take way

Looking at all of these failures, it becomes apparent that they don't quite follow a common pattern.

Failure tends to happens when and where you least expect it.

Rather than relying on the infrastructure to prevent such failure from happening, we need to learn how to cope with them as a way of living.

What does 99% availability mean anyway?

Quite often when we talk about availability, we’re referring to % of uptime.

In this context, for most people 99% uptime sounds good enough. Is it?

Let's examine what that means in days:

• 99% - 3.65 days downtime
• 99.9% - 8.76 hours downtime
• 99.99% - 53 minutes downtime
• 99.999% - 5.26 minutes downtime

99% downtime means that we need to be ready to tolerate almost 4 days of downtime, and no one can ensure how those 4 days will be spread across the year.

The impact of cloud outages

Although AWS went offline for a few hours only, the downtime experience did have an impact on customers’ businesses. There is no known data for the number of people affected by a cloud computing service outage. It is estimated that the travel service provider Amadeus loses $89,000 per hour during any cloud computing outage, while Paypal loses around $225,000 per hour.

How to survive cloud outages - (Lessons from RightScale & Netflix)

A good source for learning how to survive cloud outages is Netflix and RightScale who have had a good track record for surviving many of the previously mentioned cloud outages.

Below is a summary of the main takeaways.

• Make sure to have a dedicated expert to manage your disaster recovery (DR) architecture, processes and testing.
• Define what your target recovery time and recovery point is.
• Be pessimistic and design for failures – (assume everything will fail and design a solution that is capable of handling it).
• Avoid single points of failures – all parts of your app should be highly available (different AZ / regions / cloud) – load balancers, app servers, web servers, message bus, database.
• Use monitoring and alerts for failover processes and for every change in state.
• Document your DR operational processes and automations.
• Try to “break” different parts in your application. From unplugging the network, to turning off machines…then try it again.

Netflix didn't just provided its share of advice but has started to open source many of the tools it uses internally. The first one of them is  ”Chaos Monkey.”  This tool was designed to purposely cause failure in order to increase the resiliency of an application in Amazon Web Services (AWS.)

Designing your application to survive failure – It’s all about redundancy

Netflix provides an excellent toolset for surviving outages on Amazon on the operational level.

In this section, I wanted to zoom-in more on the design implications on our application.

The core principle for surviving failure is actually fairly simple, and in fact, applies to any system - not just cloud -  whether they happen to be airplanes, missiles, cars. At the end of the day, it’s all about redundancy. The degree of tolerance is often determined by how many alternate systems or parts of the system we have in our design, and how separate they are from one another. The degree of tolerance is also determined by how fast we can detect the broken part in our system and make the switch.

In software terms, the common parts that comprise our system are built out of two main groups - the business logic and the data.

Making a redundant software application that can survive failure is often based on setting up clones for two of those  parts of our system.

Cloning your data

There are two models for creating redundant data systems.

     1. Use database replication

     2. Use generic replication as in the case of CDN

For database replication - each database tends to have a different replication scheme.  Amazon RDS is based on a read replica that can take over when the master node fails. More modern databases, such as Cassandra and MongoDB, tend to have more flexibility and control for setting up the database replication. So the first choice that you need to make for setting up your data redundancy, is choosing the right database.

Quite often, using database replication tends to be good enough within a certain geography but can be too fragile for geo-redundancy. A model that has proven itself for replicating data across WAN is CDN.

What’s more, CDN is not tied to a specific data source, and therefore can use a generic service for replicating data from multiple sources that doesn't necessarily reside within a certain database.

Having said that, CDN is fit mostly to read data and doesn't ensure the consistency of the data. For that purpose, we need to use a generic replication service that could plug into any data source and replicate it to other locations in a way that will allow us to control the replication route, the latency, and consistency.

Cloning your application

To clone our application business logic, we need to be able to ensure that all parts of our system run the exact same version of all of our software components. That includes not just the binaries, but also the configuration, the scripts that run our application, and more importantly that all our post-deployment procedures such as failover, scaling and monitoring are also kept consistent.

Quite often the things that makes the cloning of our business logic complex, are due to the fact that the information on how to run our application is often scattered across many different sources such as scripts, as well as the mind of the people that runs these apps.

To make the job of cloning our application much simpler, and thus more consistent we need to be able to capture all parts of the information for running our apps in the same place.

Configuration management tools such as Chef, Puppet and in the case of Amazon CloudFormation can help in this regard.

Making it simpler through Cloudify

To make the work of setting all this up simpler, we tried to bake all those patterns into ready-made tools scripted into out of the box recipes.

Cloudify recipes include:

• Database cluster recipes with support for MySQL, MongoDB, Cassandra, Postgress...
• Integration with Chef and Puppet
• Automation of failover, scaling and continuous maintenance of our application.
• Application recipes that allow you to capture all the aspect of running your application, including the post-deployment aspects such as failover, scaling and monitoring.

This is a good example for setting a redundant web application between sites using Cloudify.

Final words

Cloud brings lots of promise for making our business more agile.

Cloud has also become a huge shared infrastructure in which every failure has a much more significant impact on our business worldwide.

The experience in the past year has taught us that even a robust cloud infrastructure such as Amazon can fail.

Through this experience we've learned that rather than relying on the infrastructure for preventing failure we need to design our system to cope with failure and get used to failure as away of life.

Having said that, the investment required to build a robust application can be fairly large and not something that everyone can afford.

Using tools like Cloudify, Chef, Puppet and if you’re a pure Amazon shop Netflix  could help to greatly reduce this effort by pre-baking a lot of these patterns into recipes.

 

References

• Cloudifying High Availability
• AWS Outage - Moving from Multi-Availability-Zone to Multi-Cloud
• Lessons From The Heroku Amazon Outage

 

 

Follow Nati on Twitter!

Notes from DevOpsCon Israel

The first DevOps conference in Israel turned out to be a great success.

With a room filled to capacity, and even people sitting on the floor just to take part - the energy level just hit the roof.

A great testimony to that is the number of comments on the Meetup page and tweets in the #devopscon twitter thread.

First thing first - in this post I'm going to focus only things that were covered in the main hall, as I  wasn't able to attend all tracks.

Main take away (In 140 Characters)

• Ben Kepes - Keepers of the Keys to the Kingdom - IT as manufacturing pipeline, undeployed code as wasted goods in inventory
• Ran Tavory - DevOps a Jungle of Tools - Infra management and application deployment are two distinct things
• Uri Cohen - One Does Not Simply Walk Into DevOps: There are three parts to DevOps. Culture & Processes, Infrastructure as Code, Application as Services.
• Clive Foley - We Tried to Build a DevOps Team and All We Got was this Lousy T-shirt - Having a DevOps team that is responsible for DevOps, is the wrong way to run DevOps
• Yosef Dinerstein - Continuous Deployment with Node.js over Azure - Microsoft Azure is not a .Net shop anymore

Common takeaways (Some, in slightly more than 140 Characters)

• NoOps? What's the role of IT in a DevOps world? The role of IT is to build tools and infrastructure that will allow developers to run their stuff straight into production.
• DevOps is about culture not tools.
• Applications need to be designed as loosely coupled services, otherwise it becomes impossible to manage continuous changes.
• It’s about accountability - Developers need to be accountable for running their code in production.

Panel discussion:

• Why do most applications in the cloud run outside of PaaS <- most people build their own kind of *private* PaaS
• Will (public) PaaS makes ops obsolete? <- heated debate - my personal view, probably in 3-5 years if nothing gets in the way <- Avishai you can relax, you have plenty of time before you'll need to find a new Job :)
• How to convince your management that DevOps is the right thing? – It’s about culture. You need to get the buy-in from top management. A useful way to get that buy-in is through the "build-it and they will come" approach. Once people "see the light" it’s much easier to get their buy-in for an idea.

Quotes of the day:

If it moves graph it, if it matters alert it (Allspaw)

You build it, You run it (Werner Vogels)

Done means Released (Jez Humble)

"Code that has been written but not yet deployed is very similar to inventory, you’ve paid the cost to develop the software but are not yet getting any of the benefit from it”- Mark Imbriaco, GitHub

Final Words - DevOps as a Movement

The are three elements that makes a movement IMO -  Frustration->Disruption->Passion

DevOps certainly fits into all three.

• Frustration (Pain) - many of us have been frustrated by not being able to do our Job, but rather get stacked by someone else’s job.
• Disruption (Cure)- DevOps leads to a complete disruption on many fronts - the way we design our apps (Services), the way our organization works (breaking up silos),  the way we run our business (agile, continuous delivery)
• Passion (Tell the world) – as a human being when we go through such pain and we feel that we’ve found a cure for that pain, we get passionate about it and we want to share it with others who share the same pain. DevOps seems to be that cure.

To put it in Seth Godin’s words

Leaders make movements, movements make change.

Now it’s your turn to become a leader or a messenger, and there are now many tools to make your job easier - you can start by proposing meetups, sessions, blogging, tweeting, and of course RT... or if you’re too shy for all that, just click Like or simply RSVP to the next event..

Where do we go from here?

There are currently two scheduled (local) events that are worth noting in this context:

- Check out the next Carburetor podcast - Ran Tavory, Ori Lahav and Myself will be sharing thoughts from the event on DevOps

- Reversim Summit 2013

If you have a great story that you want to share suggest a new DevOps meetup on the IGT Meetup page and we’ll take care of the rest..

Reference

• DevOps Conference Israel take-aways (Shahar Kedar)
• My talk about continuous delivery at SAP IT (Alon Aizenberg's)
• Impressions from DevOps Con 2013 (Sharon Revah Zitzman)
• DevOpscon website

DevOps Con Israel - Behind the Scenes

I'm happy to announce the first DevOps conference in Israel that will take place on the 28th of January.

In this post I wanted to share with you a bit of the stories behind the scene on how it all started.

How it all started - The story behind the scenes

For those who are not familiar with the topic of DevOps, it can basically be categorized as a movement more than a technology per se.

As with any movement, it comes with a disruptive message that aims to change many of the existing dogmas for how to run Development and IT, in order to improve agility and shorten time to market for launching new features and products. Cloud is not directly related to DevOps, but was a major accelerator, since with Cloud we’re no longer dependent on our IT to do the basic stuff such as setting up machines and networks. Which presented the opportunity to think about doing things differently, and finally break the wall between Development and Operations.

Netflix has been one of the main advocates behind this philosophy, and has served as a great inspiration for many of us who felt that the current setup between devs and ops just isn’t working, but didn't necessarily know how to deal with this. (A good reference on Netflix can be found in Adrian Cockcroft's post - NoOps)

We got involved in DevOps through our work with the Cloudify project which was highly inspired by that same Netflix experience. Through this work we took part in many DevOps related meetups and came to learn and appreciate the people behind the movement both here in Israel and abroad. There was something common that we could all identify with at all of those DevOps meetups - everyone that got involved felt very passionate about it. While this was the case, we still felt that most of the industry events just don’t cover the topic well enough, and we throughout that the community, would be better served if we’d set up a conference that would be dedicated solely to DevOps.

The Team Behind the Vision

Uri Cohen -  Uri was the first to come with the idea for setting up a DevOps event in Israel and was and still is, the leading force behind it.

Ran Tavory - Not sure that I have enough words to express how much has Ran contributed to the event. I'm sure that everyone that knows Ran, and how completely he dedicates himself to a cause he believes in,  would know what I mean. I simply can't imagine a DevOps event without Ran.

Sharone Revah Zitzman - Sharone took on the challenge of setting up the event, and has been extremely passionate about it. Everything that you see starting from the website,  through the t-shirts are her making.

Avner Algom and IGTCloud who helped us deal with many of the logistics such as setting the venues and getting the word out.

I also had the honor to be a part of this great team as Uri's and Sharone right arm.

Challenges

Skepticism:

As this was the first DevOps event in Israel we didn't really know how big the community actually is in Israel. During all the initial phases we had lots of doubts even amongst ourselves on whether we can justify a full day event dedicated only to this topic. That has been probably the biggest challenge we faced in setting this all up.

It's all about the content: 

We knew early on that if we really want to get the DevOps community involved we must and cannot compromise on the content, and that we needed to get the best presenters on the topics. With Ran’s help we got a quick list of the top speakers in Israel. To give the event more of a global perspective it was also important for us to bring internationally renowned speakers. This is an area where we faced lots of difficulties, and still thinks that we have lots of lessons to learn in this regard, for the next event. It was only until I met Ben Kepes during HP Discover event in Frankfurt that I felt that we found the right speaker in this category. Luckily enough, Ben has some Jewish and Israeli roots, so convincing him to come wasn't as difficult as I thought.

Free or paid?

We were also debating whether the event should be kept free or paid - a paid event would have helped us to funds lots of the costs involved in bringing speakers, setting up venues, lunch and more, and would have contributed immensely to doing so without the pressure on the content that you often comes with sponsorship. On the other hand, we knew that many of the people involved in DevOps are individuals who wouldn't be able to come if this would have been a paid event. This led us to keep the event free for everyone but we knew that we’d have to maintain a tight budget in order not to be  too dependent on sponsorships. This is where the IGT has been very helpful with getting Microsoft onboard. Microsoft, and specifically Noga Tal, were extremely generous and offered their venue, as well as covering all of the food & refreshment costs for the event. In this way, we were able to move most of the budgetary pressure out of the way and focus on the content. This helped us focus first on setting up the right content for the event, and only once the majority of this bulk was completed, we brought the other sponsors onboard. We also kept a tight editorial review led by Uri an Ran who made sure that the content was aligned with the purpose and vision.

 

Looking Good!

Luckily today we face quite the opposite challenge than when we started.

Our challenge today is not whether or not there would be enough people interested in DevOps went but whether we will actually have enough space for everyone, and the waiting list keeps growing.

 

How can you help?

We only just set up the platform, and it is now up to you to take advantage of it by participating, speaking and spreading the word.

 

Special thanks

There are few other who deserve special thanks for their help in setting up the event.

JFrog, Baruch and Shlomi - who shared their experience and lessons in setting similar events, and for being active participants in both speaking and sponsoring the event.

Ben Kepes who is making his way all the way from New Zealand to Israel to share his international perspective with our local crowd.

Microsoft – Who have been extremely supportive in offering the venue and large bulk of the sponsorship.

Speakers - A big thanks to all the speakers who agreed to present and take part in the event - without you there would be no event!

Sponsors – And of course a big thanks to all the sponsors, without you none of this would have been made possible: Battery Ventures, GigaSpaces Technologies,  IGTCloud, Lightspeed Venture Partners, Wix (check out their job openings here), Wonga.com, and last but not least, Yahoo!.

In Memory Computing (Data Grid) for Big Data

The Drive for In Memory Computing?

Memory-based databases and caching product has been available for over a decade. However, so far they have been used in a fairly small niche in the data management solution market.

There have been multiple advances in the industry in both hardware and software architecture, which makes memory-based computing more relevant today than in the past, as outlined in the diagram below.

In a nutshell, the availability of new classes of hardware with the support of 64bit CPU can now support 2TB on a single device. In addition, the advances in software architecture and solutions toward distributed architecture and cloud make it easier to utilize these new hardware capabilities.

In-Memory Computing

In many ways, In-Memory Computing is a close relative of In-Memory Databases. As with many databases, it was designed to enable all the data management aspects that are often expected from traditional databases, such as queries and transactions, with the difference that the data is managed on RAM devices and not disks and thus comes with potentially x1000 better performance and latency according to various benchmarks.

The main differences between the traditional in-memory databases and in-memory-based-computing are that In-Memory-Computing is:

1. Designed for distributed and elastic environments

2. Designed for In-Memory data processing

Executing the code where the data is:

The fact that we can store our data in the same address space as our application is the biggest gain.

Unlike disk and even flash disk devices, we can access our data by reference and thus perform complex data manipulation without any serialisation/de-seralization overhead. With the new class of dynamic languages such as Java, JavaScript, JRuby, and Scala, it is also significantly easier to pass complex logic over the wire and execute it on a remote device.

In-Memory Computing  relies heavily on that capability, and exposes a new class of complex data processing capabilities that fits well with the distributed nature of the data through real-time map/reduce and stream-based processing as a core element of its architecture.

The Big Data Context

According to our recent survey more than 70 percent of the responders said their business requires real-time processing of big data -- either in large volumes, at high velocity, or both.

Interestingly enough another survey by Ventana Research indicated that one of the biggest technical challenges in Big Data is the lack of real-time capabilities (67%). The report also indicated that many of the organizations are planning to use in-memory databases (40%) as part of their Big Data stack. This places the In-Memory Database as a second choice, before specialised DBMS (33%) and Hadoop (32%).  One of the conclusions this survey leads to is that organizations see Data Warehouse Appliances and In-Memory Databases as one of their first choices to deal with the lack of real-time capabilities.

 

No One-Size-Fits-All Solution

While In-Memory Databases fit well in the planned Big Data stack, it is clear that  theres no one-size-fits-all solution. The Big Data stack is going to be based on a blend of various technologies, each covering different aspects of the challenges of Big Data, from batch to real-time, from vertical to horizontal solutions, etc.

The question is: How do we integrate them all, without adding even more complexity to an already complex system?

In this post I will focus specifically on one of the approaches the we used for combining In-Memory Computing together with other Big Data solutions, such as Hadoop and Cassandra.

Putting In-Memory Computing Togther with a NoSQL DB

One of the main motivations to integrate in-memory-based solutions with a NoSQL DB is to reduce the cost per GB of data.

Putting our entire data purely in-memory can be too costly especially for data that were not going to access frequently.

There are various approaches to doing this -- the approach we found most useful is a two-tier approach.

With the two-tier approach the In-Memory Computing systems run separately from the NoSQL database, which acts as the long-term storage.

The Challenge

The main challenge with this approach is the complexity that is associated with synchronising two separate data systems. Specifically, how to ensure that data that is written into the front end In Memory Computing engine gets populated into the NoSQL database reliably, and vice versa.

The Solution

To deal with this challenge we used a similar approach to the one that we used before with RDBMS. Have an implicit plug-in that gets called whenever new data is written and populate it into the underlying database. The plug-in also deals with pre-loading of the data when the system starts. In the RDBMS world we used frameworks like Hibernate to deal with the implicit mapping of the data between the in-memory front end and the underlying database.

Working with Dynamic Data Structure (a.k.a Document Model)

When we tried to apply the same approach with NoSQL databases we could no longer rely on Hibernate as the default framework for mapping the data between the two data systems, as NoSQL databases like Cassandra tend to be fundamentally different from traditional RDBMSs. The main difference is the use of dynamic data structures, a.k.a the Document Model.

To deal with dynamic data structure we added the following hooks:

• Introducing new documents and objects: Users can choose to write or load data in various forms -- Document for non-structured data or Objects or POJOs for structured or semi-structured data.

• Introducing and loading new meta data: To map the data to and from the NoSQL database we also added the ability to introduce new meta data and load the meta data of the object before the actual data is loaded.

• Introducing new indexes: In NoSQL databases you cannot effectively access data that is not indexed. For that purpose we included the ability to introduce indexes on the fly.

You can see how this works specifically with Cassandra in this post -- From The Memory Grid to Cassandra

The Benefit -- Best of both worlds

The main benefit of this two-tier approach is that it allows us to take the best of the memory and file-based approaches without adding too much complexity. The two tiers behave and work as one data system from an end-user perspective. The bits and pieces of how the two system get synchronized is carved out of the system.

Furthermore, the two-tier approach opens up a new degree of flexibility in how we design our Big Data system.

New Degree of Flexibility

If we look at the entire data flow from the point in which a user interacts with our system (this is where we expect low latency and high degree of consistency to our analytics systems; where we record those actions and analyze them, latency is less of an issue and we can also relax some of the consistency constraints), we can see that each stage in our data processing has different  onsistency, latency and performance requirements.

With the two-tier approach there is more flexibility in dealing with those different requirements and still keep everything working as if it were one big system from a usability perspective. Here are a few examples of how this setup can work:

• Consistent data flow from Real-Time to Batch: The integration enables us to handle real-time data processing at in-memory speed and deal with more long-term data processing through the underlying database.
• Performance & latency: The In-Memory Computing system can handle the event processing before the data gets into the database. Or, another approach is to keep the last day (or days) in-memory and the rest of the data in the NoSQL database.
• Mixed consistency model: The In-Memory Computing system is often built for extreme consistency where NoSQL databases often work best with eventual consistency. Usually, the consistency requirements are more relevant at the front end of the system, and becomes less relevant as the data gets older. The combined approach enables us to set our front end for extreme consistency and back end for eventual consistency.
• Deterministic behavior: In many cases, we must ensure that a given set of data can be served under constant performance. Many databases use an LRU based cache to optimise the data access. The limitation of this approach is that the speed at which we can access our data becomes non-deterministic as we often do not control which data is served through the database cache; thus, in some cases we will get a fast response time if we hit the cache and in other cases the same operation can be 10 times longer if we miss the cache. By splitting our in-memory data from our file-based storage we get more explicit control over which data is served at in-memory speed and which data is not, thus ensuring consistent access to that data.
• Faster ETL: By front-ending our Big Data storage with In Memory Computing we can also speed up the time it takes to pre-process and load data to our long-term data system. In this context, we can push the filtering, validation, compression and other aspect of our data processing into memory before it goes into our long-term databases.

Final Words

Big Data systems are complex beast and it is clear that the one-size-fits-all approach doesn't work.

On the other hand, having too many data systems increases the complexity of managing our Big Data system almost exponentially; and our ability to ensure consistent behaviour, data integrity and reliable synchronization across the varous systems becomes an almost un-manageable task if done manually.

Adding real-time capabilities to our Big Data system is a classic area where the kind of integration described in this article is needed. The integration between In-Memory and File-Based approaches as two separate tiers also introduces additioanl areas of flexibility in how we can handle often contradictory requirements such as consistency, scale, latency, and cost. Instead of trying to come up with a least common denominator, we can optimize each tier to the area it fits best.

References:

• From The Memory Grid to Cassandra
• GigaSpaces Big Data Survey
• The Challenges of Big Data Survey (Ventana Research)
• In Memory Computing (Wikipedia)
• http://www.ventanaresearch.com/uploadedFiles/Content/Landing_Pages/Ventana_Research_Big_Data_Benchmark_Research_Presentation.pdf

 

2013 The year of a major cloud world shakeout

Cloud has been around for couple of years now. Following the technology lifecycle, you would expect that we should now be entering the maturity lifecycle where we can expect to see less disruption and more consolidation in the market.

The reality shows that quite the contrary is happening, as we enter the land grabbing stage where everyone is competing for market share, and the incentive for major disruption is higher than ever before.

As a result of this, I expect during 2013 to see some major disruptions and shakeouts that will have a dramatic effect on the cloud world as we know it.

I've listed the main three below:

1. The IaaS Shakeout - So far, the IaaS world has been dominated by Amazon who is by far the leader in the market, where Rackspace its closest competitor comes in a far second. In 2013, we’re expecting to see Google entering the game, and Microsoft – who is not new to this game – to now finally start making the right moves, and with these is expected grab more market share from its existing customer base.  The other big disruption is the open source coalition behind OpenStack which was lacking the maturity needed to present a true viable alternative. With Rackspace, HP GA-ing its OpenStack-based cloud offering, and IBM  expected to follow suit as well, and with real production references such as Cisco’s Webex, I expect that we will now be seeing a far greater adoption of OpenStack, which to me currently looks like the main viable alternative to Amazon. The war on who rules the cloud is not going to be just a pricing war as some predict. One of the areas where there is huge potential for disruption is the network, as well as the software-based data center, which is a completely uncharted territory, with lots of potential for new innovation with how applications utilize the infrastructure.

2. The PaaS Shakeout - I've seen lots of predictions that talk about 2013 as the year of PaaS. They miss one major thing though. The definition of PaaS is undergoing a major disruption as well. PaaS was built initially to serve simple applications over the web. We’re now seeing a completely new category of PaaS coming mostly from open source initiatives, such as Cloud Foundry, which makes it possible to run PaaS in your local data center or even desktop. I'm also expecting us to see more convergence between PaaS and configuration management frameworks such as Chef, as in the case of Cloudify. If we follow Amazon, one can also note that the line between IaaS and PaaS is blurring, as Amazon has started to move up the stack. In addition to this, open source IaaS frameworks such as OpenStack and CloudStack make it possible to break down the wall between these two layers, and share a large part of the infrastructure between the IaaS and PaaS stack, blurring the boundaries between the two layers more than ever, as I pointed out in this post (PaaS as Infrastructure).

3. Big Data Shakeout - The Big Data world has primarily been dominated by Hadoop. The Hadoop project has since gone through major fragmentation with many different and incompatible versions of the Hadoop distribution such as the one coming from MapR, Horton Works, EMC, IBM, and more.  This gives a rise to some of the NoSQL vendors such as DataStax, the company behind Cassandra, and 10Gen the company behind MongoDB to come out with their own alternatives in this space. In addition, the move to real time makes the use of Map/Reduce less popular than it used to be. Frameworks like Twitter’s Storm and other stream processing frameworks are gaining popularity. With all this going on, the future of Haddoop as we know it remains fairly uncertain IMO, and while the Big Data market is expected to grow quite substantially over the next coming years, it is still unclear which technology will be the one to drive this change. One thing that has becomes quite clear though, is that it doesn’t appear as if there is going to be one solution that covers them all, making the challenge of integration and management of the various framework more critical than before.

So all in all, I think that 2013 is going to hold a lot of surprises and disruptions, and we are still pretty far from the point where we’re going to see consolidation.

OpenStack Israel II

I'm very excited to announce our second OpenStack Israel event which will take place on Tuesday this week, November 27th.

At this event you'll be able learn about the latest developments in OpenStack from Jonathan Bryce Executive Director at the OpenStack Foundation.

 

One of the other highlights of the event is a local success story by Koby Holzer System & Storage Engineer Group Leader at LivePerson who will talk about their experience with running OpenStack in production across multiple data centers. We will also have several talks focused on one of the hottest areas in OpenStack development - Network/ Quantum and SDM delivered by Yaron Haviv from Mellanox. Vachnis Avi from Alcatel will present their choice of OpenStack APIs as the de-facto standard for Cloud APIs. 

Those who are also looking to get their hands dirty will have the chance to learn how to setup a simple OpenStack environment on your desktop from Ori Tzoran, DevOps Senior Engineer @ "Tikal Knowledge".

You can see the full agenda at the event page.

As with previous times, we’ve reserved slots for lightning talks -  so, if you have an interesting story you'll have the chance to share it during the event.

I look forward to seeing you all at the event.

Please make sure to RSVP to secure your place at the event as the seats are limited.

Moving Enterprise Workloads to the Cloud on a Massive Scale - The Cloudify Way

In my previous post Moving Enterprise Workloads to the Cloud on a Massive Scale, I pointed out the reasons for moving Enterprise workloads to the cloud.

I also argued that the current approach doesn't work, as we need to think of moving apps not as one-off projects, or as an afterthought, but at a large-scale. Moving apps at a large scale requires a completely different thinking.

I laid out what I believe should be the core principles behind this approach:

• No Code Change - Any change, even the slightest one, to our existing apps will slow down our migration efforts substantially. Furthermore, many of the apps are third-party apps, where we often times don't even have control over the application code, so changing the app is likely not an option. There are many things that we can benefit from by moving our apps to the cloud, such as automation, ease of change, and configuration. We can obviously benefit more if we change our apps to take better advantage of the cloud infrastructure, but the right order would be to do this after we’ve already migrated the app to the cloud in the first place. There, of course, will still be many cases where just automating the way we run our apps and make them available on-demand would be sufficient enough.
• Plug-in to the Current Way of Running Apps - The common way for running apps today is either through custom shell scripts in the simple case, or through configuration management tools such as Chef or Puppet in the more advanced case. If we want to migrate apps at a massive scale, we need to be able to plug-in to the most common environments, and make it possible to import them almost as is. In this way, we could leverage the knowledge and know-how that has already been invested into these systems and make the migration process vastly shorter.
• Use Shared Infrastructure Across the Stack Regardless of the Language - Using different frameworks for each element of our application stack based on the language and environment in not going to scale for obvious reasons, such as complexity, a large number of moving parts, inconsistency, among other reasons.
• Add Cloud Properties from the Outside - There are many cloud properties that we can apply to our existing apps without forcing any changes. For example, we can automate the scaling from small to bigger machines, we can also automate the configuration, and setup. We can significantly simplify the way we run the app by making it available on-demand. All these cloud properties can be plugged into almost any existing app without forcing a real change, as noted in the MySQL example.
• Use the Baby Steps Approach - One of the main inhibitors to migrating apps was the all or nothing approach. There are many risks involved in migrating apps that go well beyond the technology challenge. The easiest way to mitigate such risks is by taking them one at a time. For example, we can first make our apps cloud ready, in which case we will abstract the automation part from the target infrastructure. By doing so, the decision where to run our app becomes tactical and less strategic. We can choose to run our production in our private data center, and run only testing and demos on the cloud. In this way, we can also make the migration gradual and smoother.
• Incorporate Bare-Metal into our Cloud - One of the main challenges that I’ve seen for moving true enterprise workloads to the cloud wasn't necessarily related to the cloud, but to the virtualization overhead that is often used as the underlying infrastructure of many clouds. Today it is possible, to gain access to bare metal clouds  which make it possible to run almost any workload including I/O intensive ones without any performance overhead, as I noted in my post about the emergence of bare-metal clouds.

 

In this post I will be referring to the way we've implemented this approach with Cloudify.

General approach

The least common denominator with all applications, regardless of their size and language, is that they all map to a group of processes with some degree of interdependency between them. Cloudify manages applications at the process level, and as such uses the same underlying architecture for any application regardless of the language or the technology stack that comprises the application. Having said that, working at the process level is often not enough as not all processes were made the same. For example, databases behave quite differently from web containers and load-balancers. In order for us to still get in-depth knowledge about the managed application’s processes, Cloudify uses a recipe-based approach. The recipe-based approach enables us to describe the elements that are specific to that individual process such as, the configuration element, the dependency on other processes, the specific key performance indicators that tell if that process’ behavior is aligned with its SLA, and such.

Working on the process level makes it possible to plug-in to a large variety of infrastructure environments whether they happen to be in public, private or bare-metal environments. Cloudify uses an abstraction layer known as the Cloud Driver that allows it to plug-in to a large variety of cloud and private data center environments through simple configuration.

Here is how Cloudify maps to the specific principles that I laid out above:

• No Code Change -  Because Cloudify works on the process level, and uses a recipe-based approach it doesn't require any code change to get Cloudified.
• Plug-in to the Current Way of Running Apps - The Cloudify recipes was designed to plug-in to diverse application environment configurations and setups. This currently includes basic shell/batch files in the simple scenario, as well as configuration management with tools such as Chef or Puppet in more advanced scenarios.
• Use a Shared Infrastructure Across the Stack Regardless of the Language - Cloudify uses the same infrastructure to automate the deployment, fail-over and scaling, or in other words, a large variety of workloads - starting with simple web apps to Big Data apps. All this is possible due to the fact that Cloudify uses a process + recipe-based approach as I outlined earlier. In addition, the Cloudify abstraction of the underlying infrastructure details makes it possible to use the exact same application recipe on different target platforms whether they happen to be on public, private and bare-metal environments.
• Add Cloud Properties from the Outside - Cloudify automates the deployment and configuration of the application even in the case of complex multi-tier applications, making it possible to launch the application on-demand on any given environment. In addition, Cloudify can plug-in to the application-specific metrics which makes it possible to detect application-specific SLA breaches, and thus automate the fail-over process as well. On top of that, Cloudify comes with a scaling rules plug-in that makes it extremely simple to add more instances to the application for scale-out purposes, or simply to enable it to move from smaller to bigger instances, in the case of a scale-up model. (See the MySQL use case as an example).
• Use the Baby Steps Approach - With Cloudify all you need to get started is a start and stop script. All the other elements such as custom metrics, auto-scaling and more can be added at a later stage, making it possible to grow gradually with the value, and add more cloud properties as we grow. Additionally, the Cloud portability enables to first automate our apps on our existing environment, and then move to the cloud at a later stage. A good example for this would be to keep our production on our existing environment, and testing on the public environment. A more advanced incarnation of this could be to combine the two into a complete hybrid-cloud deployment.
• Incorporate Bare-Metal into our Cloud - Cloudify was designed to work also on regular ol’ non-virtualized environments. In such a case, it takes a list of IP addresses as a fixed pool and manages the workloads elastically within the boundary of that pool as noted in this post bare-metal clouds.

Live Demo:

Below is two references that shows how Cloudify is used  in the context of migrating enterprise Workloads to the cloud.

The first example uses IBM Infosphere BigInsights which an enterprise grade Hadoop distribution. 

 

 

 The Second example shows IBM WebSphere application deployment:

 

 

Final notes

Cloudify was designed, to a large degree, on the lessons of our earlier generation PaaS solution. In our earlier generation solution, we took a cautious decision to bind it specifically to Amazon and chose Java as the primary language of choice. A lot of that had to do with the maturity of the cloud market at the time, and the availability of tools and frameworks. Since then, there has been a very rapid advancement on almost every front which changed the entire landscape. That brought us to develop Cloudify.

When we came up with Cloudify the first thing that was clear to us was that we could take a vastly different approach to the way most PaaS solutions were designed to to run, and set the ambitious goal for running any app, on any cloud without compromising anything in terms of the control factor. I didn't expect that within a period of a year we’d see major enterprises and ISVs use Cloudify to run apps that, even I,  never imagined could run on the cloud--such as network-based services and legacy apps that were written ten years ago when cloud was only an item in our weather report.

Today, I'm even more surprised to see how independent users have become with the platform. For example, one of our partners is using Cloudify for continuous delivery and integration. Another partner is using it to speed up the project delivery time for his customers, and use that as a major differentiator from their primary competitor. A major enterprise is using Cloudify in combination with Chef where they build an entire framework to automate the deployment of applications that were described in Chef, and auto-generate the Cloudify recipe on top of that. A major ISV is using Cloudify as part of its own cloud solution to offer their existing apps as a service, as well as on-board customer apps to their cloud platform. These are only few examples that represent some of the more interesting use cases that I came across recently. There are many projects that are coming along specifically around our recent integration with Chef and the work that we've done on the big data front.

References 

• Moving Enterprise Workloads to the Cloud on a Massive Scale
• Putting DevOps and PaaS together
• Bare Metal PaaS
• MySQL Example
• PaaS as an Infrastructure
• The Elephant in the Cloud -- Putting Hadoop on any Cloud

Follow Nati on Twitter!

Moving Enterprise Workloads to the Cloud on a Massive Scale

Cloud has quickly become a disruptive technology, with the main disruption being a revolution in the way in which organizations run their IT.

The poster child for these cloud discussions was often the new Web 2.0 companies such as Google and Amazon who have been true pioneers in this new and emerging world. Quite often enterprises were used as the reference for how bad IT can get, and quickly became the representation of the antithesis for IT in these cloud discussions and conferences.

Two weeks ago I was at the OpenStack conference and other meetups that took place around the same time. The thing that was pretty apparent is that the large majority of the audiences in each room were from the enterprise world. What’s more, a quick look at the OpenStack foundation shows that those leading one some of the most trendy technologies in the cloud space are run by Enterprise HW and Software companies such as HP, IBM, Dell, Red Hat, among others.

Could this mark the beginning of a new era for enterprise and the cloud?

- A series of recent surveys on the subject seems to indicate that this is indeed the case:

Research conducted by HP found that the majority of businesses in the EMEA region are planning to move their mission-critical apps to the cloud. Of the 940 respondents, 80 percent revealed plans to move mission-critical apps at some point over the next two to five years.

A more recent survey, by research firm MeriTalk and sponsored by VMware and EMC (NYSE:EMC), showed that one-third of respondents say they plan to move some mission-critical applications to the cloud in the next year. Within two years, the IT managers said they will move 26 percent of their mission-critical apps to the cloud, and in five years, they expect 44 percent of their mission-critical apps to run in the cloud.

This shouldn't come as a surprise as  most enterprises are using the cloud today but don't necessarily know they are doing so, and getting huge benefits from it (SalesForce.com, ServiceNow)

The Cloud needs Enterprises just as Enterprises need the Cloud?

As I was sharing this observation with a few of the folks during the conference, I heard an interesting comment by one HP Executive Thomas Ryan who made an interesting statement: "Linux wouldn't be where it is today if enterprises wouldn't have adopted it". Quite often we look at the cloud (OpenStack in particular) as the new Linux. If that's true, then it looks like we’re reaching the point where for cloud to remain a viable technology it needs enterprise adoption just as enterprises needs the benefits the cloud brings.

The Renaissance of Enterprises in the Cloud

So are we seeing the renaissance of enterprises in the cloud? It's a bit too early to say, but there are clear indicators that this is where the market is heading. There are a couple of reasons for this:

1. IT is becoming a new manufacturing pipeline for enterprises in the new information era

It's clear today, that in this data driven world, the ability of any business to survive is determined by how agile they can be, how quickly they can respond to change, and react accordingly. In this context, it becomes more apparent for many enterprises that their ability to reach this level of agility is dependent on how well they are running their IT. At the same it becomes clear that cloud is just a better way of running their IT, and in this context it’s pretty much a no brainer.

2. Cloud technology is maturing

Many of the technology gaps that prevented enterprises from adopting cloud technology such as security, complexity, and more, have been addressed or are in the process of being addressed. There are many options today including private cloud, hybrid cloud, with different degrees of SLAs and cost that make it easier to run enterprise workloads in the cloud.

3. Strong business incentive

Enterprise applications tend to be one of the more costly workloads today in terms of IT infrastructure and resources, due to both the size and the complexity involved with running these systems. Any savings with running these systems would yield a significant impact to an organization’s bottom line, as indicated in MeriTalk research:

Enterprises can save $16.6B, or 21% if they moved just three mission-critical apps to the cloud.

The Challenge - How to bring Hundreds of Enterprise Apps to the Cloud

All that, of course, is great, but the reality is that cloud economics only start making sense when there are true workloads that utilize the cloud infrastructure.

If the large majority of your apps fall outside of this category, then you’re not going to benefit much from the cloud. In fact, you’re probably going to lose money, rather than save money. 

The Current Approach

• Focus on building IaaS - Current cloud strategies of many enterprises has been centered on making the infrastructure cloud ready. This basically means ensuring that they are able to spawn machines more easily than they were before. A quick look at many initiatives of this nature shows that there is still only a small portion of enterprises whose applications run on such new systems.
• Build a new PaaS - PaaS has been taught as the answer to run apps on the cloud. The reality however, is that most of the existing PaaS solutions only cater to new apps and quite often the small, and “non” mission-critical share of our enterprise applications, which still leaves the majority of our enterprise workload outside of our cloud infrastructure.
• App Migration as a One Off Project - The other approach for migrating applications to the cloud has been to select a small group of applications, and then migrate these one by one to the cloud. Quite often the thought behind this approach has been that application migration is a one-off project. The reality is that applications are more of a living organism – things fail, are moved, or need to be added and removed over time. Therefore it’s not enough to move apps to the cloud using some sort of virtualization technique, it’s critical that the way they’re run and maintained will also fit the dynamic nature of the cloud.

Why is This not Going to Work?

Simple math shows that if you apply this model to the rest of your apps, it’s probably going to take years of effort to migrate all your apps to the cloud. The cost of doing so is going to be extremely high, not to mention the time to market issue which can be even an even greater risk in the end, as it will reflect on cost of operation, profit margins and even the ability to survive in this an extremely competitive market, if it is too long.

What's missing?

What we’re missing is a simple and systematic way to brings all these hundreds and thousands of apps to the cloud.

Moving Enterprise Workloads to the Cloud at a Massive Scale

Instead of thinking of cloud migration as a one-off thing, we need to think of cloud migration on a massive scale.

Thinking in such terms drives a fairly different approach.

I've listed the principles, that IMO, are the crux to such a strategy, below:

• No Code Change - Any change, even the slightest one, to our existing apps will slow down our migration efforts substantially. Furthermore, many of the apps are third-party apps, where we often times don't even have control over the application code, so changing the app is likely not an option. There are many things that we can benefit from by moving our apps to the cloud, such as automation, ease of change, and configuration. We can obviously benefit more if we change our apps to take better advantage of the cloud infrastructure, but the right order would be to do this after we’ve already migrated the app to the cloud in the first place. There, of course, will still be many cases where just automating the way we run our apps and make them available on-demand would be sufficient enough.
• Plug-in to the Current Way of Running Apps - The common way for running apps today is either through custom shell scripts in the simple case, or through configuration management tools such as Chef or Puppet in the more advanced case. If we want to migrate apps at a massive scale, we need to be able to plug-in to the most common environments, and make it possible to import them almost as is. In this way, we could leverage the knowledge and know-how that has already been invested into these systems and make the migration process vastly shorter.
• Use Shared Infrastructure Across the Stack Regardless of the Language - Using different frameworks for each element of our application stack based on the language and environment in not going to scale for obvious reasons, such as complexity, a large number of moving parts, inconsistency, among other reasons.
• Add Cloud Properties from the Outside - There are many cloud properties that we can apply to our existing apps without forcing any changes. For example, we can automate the scaling from small to bigger machines, we can also automate the configuration, and setup. We can significantly simplify the way we run the app by making it available on-demand. All these cloud properties can be plugged into almost any existing app without forcing a real change, as noted in the MySQL example.
• Use the Baby Steps Approach - One of the main inhibitors to migrating apps was the all or nothing approach. There are many risks involved in migrating apps that go well beyond the technology challenge. The easiest way to mitigate such risks is by taking them one at a time. For example, we can first make our apps cloud ready, in which case we will abstract the automation part from the target infrastructure. By doing so, the decision where to run our app becomes tactical and less strategic. We can choose to run our production in our private data center, and run only testing and demos on the cloud. In this way, we can also make the migration gradual and smoother.
• Incorporate Bare-Metal into our Cloud - One of the main challenges that I’ve seen for moving true enterprise workloads to the cloud wasn't necessarily related to the cloud, but to the virtualization overhead that is often used as the underlying infrastructure of many clouds. Today it is possible, to gain access to bare metal clouds  which make it possible to run almost any workload including I/O intensive ones without any performance overhead, as I noted in my post about the emergence of bare-metal clouds.

Final Notes

I often like to compare the revolution that we’re experiencing right now in the IT industry to a similar revolution that happened in the automotive industry few decades ago.

You probably know what I’m referring to. The automotive industry experienced a similar revolution when Henry Ford came up with his Model-T car. The change wasn't so much in the engine or other parts of the car, but rather in the scale and cost in which cars could now be produced. As I noted earlier, in many aspects cloud is the IT industry manufacturing assembly line.

As with cars, it doesn't make sense to run only 10% of our cars on the new manufacturing assembly line, and build the rest in the traditional way. So is the case with our IT. Now the question becomes how to make the move.

In this post I outlined what I believe should be the principles to migrate apps at a massive scale.The second post describes more specifically the Cloudify way of doing this.

PaaS as Infrastructure

The blurring line between PaaS and IaaS 

The core of IaaS services is centered around  Compute, Storage Network however over the past years the infrastructure moved up the stack and includes more application services such as Database as a Service, Hadoop Map/Reduce, and application lifecycle services. 

On the other hand, PaaS services offer similar services through the PaaS platform itself with the main difference being that quite often those services were designed to fit only the application that runs on that particular PaaS.

So rather than the traditional view, where we draw a clear line between the IaaS and PaaS layer we need to view them differently, where the PaaS and IaaS layers overlap each other on the software services layer.

 

IaaS and PaaS share a similar technology stack

A close look into the OpenStack infrastructure shows that even on the infrastructure level running the compute, storage and network services relies on messaging services such as RabbitMQ, databases such as Postgress, a load balancer such as Nginx and other software services. Running these software services for our infrastructure is not much different than running our apps.  With both the infrastructure and application we would need to address the scalability and high availabilityof these services and manage them through our infrastructure.

If that is the case wouldn't it make sense to share the same underlying technology to run our infrastructure and our apps?

Sharing the software services between your IaaS and PaaS

A typical public PaaS platform is built out of two main parts - a self-service portal, and a backend system that runs our apps.

With many of the public PaaS solutions such as GAE, Heroku the backend that runs our PaaS is treated as a blackbox.

Amazon Elastic Beanstalk does things differently in that respect - as it took a bottom-up approach. It uses the same AWS infrastructure to run Elastic Beanstalk and the infrastructure - this provides a lot of flexibility to its users, and equally as important, consistency between the way they run and manage their app and infrastructure. VMware took even a bigger step in this regard with CloudFoundry and BOSH and made their entire PaaS backend open source which gives users even greater flexibility. With Cloudify we took this idea a step further by introducing the concept of recipes into the core of the Cloudify infrastructure, and recently introduced our integration with Chef in a way that will enable users to run their Chef cookbooks as part of Cloudify. Chef has already gained popularity as a configuration and automation framework for setting up many of the OpenStack cloud deployments. Having Chef tightly integrated into the PaaS layer, simply put, makes perfect sense.

Example - Building your own RDS with Cloudify

To illustrate the idea of how we can use share the same deployment framework between our IaaS and PaaS layer I'll pick one of these software services to demonstrate the idea with. In this particular case, I've chosen to use a database as a service. Database as a service is often a good reference for a service that fits in both our infrastructure and application. Here is how we can easily set a database as a service with Cloudify:

1. Directly through the use of Cloudify recipes

If you allready developed your scripts for setting your database you can use cloudify recipe to wrap those script into a service. 

 Here a snippet of how that would look like:

service {
	name "mysql"
	icon "mysql.png"
	type "DATABASE"
	elastic true
	numInstances 1
	minAllowedInstances 1
	maxAllowedInstances 3	
	compute {
		template "SMALL_LINUX"
	}	
	lifecycle{
 
		install "mysql_install.groovy"
		start "mysql_start.groovy"
		startDetectionTimeoutSecs 900
		startDetection "mysql_startDetection.groovy"
		stopDetection {	
			!ServiceUtils.isPortOccupied(jdbcPort)
		}
        }	

You can refer to the full MySQL recipe here.

2. Through the use of Chef cookbooks

If you are already using Chef you can plug-in to the Chef MySQL cookbook by extending the Cloudify recipe to use Chef, as you can see in the snippet below:

service {
    extend "../../../services/chef"
    name "mysql-vanilla"
    type "DATABASE"
    numInstances 1
…
}

In addition to that you could attach Chef attributes and run parameters into the Cloudify recipe that will tell Chef which cookbook and recipe to run:

runParams = [
  "mysql": [ "bind_address": "0.0.0.0",
             "tunable": ["query_cache_limit": "2M"]  
  ],
  "run_list": ["recipe[mysql::server]"]]

You can read the full details on how to run Chef with Cloudify in our documentation.

Running through bare metal environment

Even in the case where we share the same software services through our infrastructure and application, we tend to run them differently.

Quite often infrastructure services will run on bare-metal environments, as we don't want to create dependencies between our virtualiation layer and the underlying infrastructure that runs it.  

To enable this level of flexibility, we need to abstract the underlying compute layer from our service orchestration layer. In this way, we can use the same recipe and deploy it on either a bare-metal environment or a virtaulization layer. (Read: Configuring a Traditional Data Center [BYON])

Final notes

Up until recently, PaaS and IaaS were treated as two separate and closed entities and the integration between the two was done pretty much in a blackbox approach. The fact that both the IaaS and PaaS underlying infrastructure were closed was the primary reason behind this.

Now with the growing adoption of open source IaaS such as OpenStack and CloudStack, and similarly with PaaS such CloudFoundry and Cloudify we have the opportunity to change this paradigm. There isn't much of a reason why we shouldn't be able to share the same orchestration, and configuration layers between our IaaS and PaaS layers. Amazon Elastic Beanstalk is a good reference why this makes a lot of sense.

Special thanks

Special thanks to Boris Renski who was the main inspiration behind this post.

References:

• Presentation on the Chef Meetup Group
• Online presentation 

 

 

The Elephant in the Cloud -- Putting Hadoop on any Cloud

In one of my earlier posts I discussed  in general terms why it makes a lot of sense to put  Big Data on the cloud.

The first step that we took in this regard with Cloudify was to make NoSQL databases, such as Cassandra and MongoDB, run on any the cloud through Cloudify recipes. Uri Cohen gave an excellent talk on this during the Cassandra Summit where he provided a insights into this work.

In the past couple of weeks we've been working on the next phase of this project: Putting Hadoop on the cloud.

As there are various solutions that aim to do something similar, I thought that a good start would be to first outline where we fit in this ecosystem.

Cloudifying Hadoop -- What Does That Actually Mean?

Yet another Hadoop distribution?

There are multiple distributions of the Hadoop framework today. All come with strong management and sets of tools and do a fairly good job. With that in mind, it was clear to us that our goal wouldn't be to come up with yet another Hadoop distribution but rather to integrate with the ones that are already out there. We picked IBM BigInsights and the Cloudera distribution as the first targets for the project.

When we realized that this was the path we wanted to pursue, the main question that immediately came up was:

What value can we add on top of IBM BigInsights and the Cloudera distribution?

Big Data systems tend to be complex to manage and operate. BigInsights and Cloudera provide good tools to make the process of configuring and deploying Hadoop significantly simpler. That's good for the Hadoop part of the story. Big Data systems and applications often include other services such as relational databases, other NoSQL databases such as Cassandra or MongoDB, stream processing such as Twitter-Storm and GigaSpaces XAP, web front ends such as Tomcat, Play framework and Node.js. Each framework comes with its own management, installation, configuration, and scaling solutions, as shown in the diagram below:

  

Managing each component of your Big Data system seperately is an operational nightmare. That complexity grows exponentially as the system gets bigger, and in Big Data that's just to be expected. 

We realized that one of the areas in which we can reduce this operational complexity is through Consistent Management. By Consistent Management, I'm referring specifically to consistent deployment, configuration, and management across the stack. Consistent management applies not only to the deployment phase, but also to post-deployment, including fail-over, scaling, and upgrades. In addition, Big Data systems tend to consume a lot of infrastructure resources that can easily pile up to thousands of nodes. We realized that we can optimize the infrastructure cost for running the Big Data system through Cloud Enablement and Cloud Portability. Cloud Portabiltiy enables you to choose the right cloud for the job. For example, you could choose a bare-metal cloud for I/O intensive workloads or a virtualized/public cloud for more sporadic workloads. Below is a more detailed description on how we implemeted these two properties:

1. Consistent Management 

With consistent management, we wanted to make the experience of managing each of the tiers and services in the Big Data System consistent throughout the entire stack. This is where Cloudify plugs in.

Cloudify already plugs into a variety of web containers, databases, and through the integration with Chef also into hundreds of services available through the Chef Cookbook.

The process for achieving consistent management for BigInsights and Cloudera Hadoop distribution basically maps to the creation of a Cloudify recipe. The purpose of the recipe in this specific case was to map all the capabilities that come with the IBM BigInsights and Cloudera distribution in a way that would be later consistent with other services in the stack. 

For this, we needed to come up with the following:

• Deployment automation -- A Cloudify recipe enables us to automate the installation, configuration, and deployment of a given service. In the context of a Hadoop distribution, which comes with its own scripts for automating these phases, the process of creating a Cloudify recipe basically meant mapping the specific Hadoop distribution scripts into the Cloudify convention, as well as using the Cloudify discovery, global context, and other services to dynamically inject configuration values that are driven on-the-fly through the deployment process. In the case of BigInsights, this basically mapped into a simple recipe that would launch the machines, set up the SSH environment, and update the BigInsights configuration file with the relevant information. We then called BigInsights to install the NameNode, DataNode, Management, Hive and other services that come with the BigInsights distribution. 
• Automation of post-deployment operations -- In the Hadoop scenario, post-deployment operations mapped into the abilities to add a node, rebalance, and test the environment. To automate these processes, we used Cloudify custom commands. Custom commands enable us to give an alias to those maintenance scripts, and then it enables the users to call those scripts using a simple naming convention without knowing the physical location of Hadoop deployment. 
• SLA-based monitoring and auto-scaling -- SLA-based monitoring enables measure how the Hadoop distribution behaves after it's been deployed, and also to map specific actions when a given SLA is breached. For example, you can monitor a situation where a node fails and then spawns a new machine to take over that fail-node. You can also use the monitors to trigger any of the actions defined in the custom commands, which basically means that you can automate not only a fail-over process but also the ability to scale, by adding more capacity. To do this with Cloudify, use Custom Metrics to monitor the specific metrics that are of interest. One great power of custom metrics IMO is that you're not bound to metrics that are provided by the Hadoop distribution, and you can easily generate compound metrics that can be generated out of correlating metrics from multiple sources, such as such as network statistics or a cloud management system. You can then use the Cloudify Scaling Rules to attach rules to each of those metrics and trigger a fail-over or scaling event.

2. Cloud Enablement and Portability 

The Cloud is basically a more efficient way to run IT. Today, there are growing numbers of cloud offerings and types of clouds, each providign different SLAs and pricing models. There are the public clouds such as Amazon, Rackspace, and also HP, IBM and Dell who's coming out with their own cloud offering. Microsoft and Google are also coming out with new offerings that will undoubtedly change the dynamic in this space. There are also the bare-metal clouds, the private cloud, the OpenSource cloud, etc., and obviously the VMware cloud, which are also an important part of the mix.

Enabling your Big Data systems for all these environments means you can leverage all the agility, efficiency, and flexibiltiy of the cloud, while also allowing you to choose the right cloud for the job.

Given the rapid dynamic of the cloud world, you also want to keep your options open so that you can leverage any future developments and services as they become available.

Controlling Your Cloud Infrastructure is a Critical Part of Controlling Your Big Data Costs

There are various ways to control Big Data costs through controlling  infrastructure. Here are a few examples:

• Bare-Metal -- Reduce the number of machine instances for high I/O workloads.
• Private-Cloud --Optimize performance for your specific workload through specialized hardware (Inifiband, dedicated network, specialized hard drives, etc.) and also have better control over our data secrurity.
• Hybrid-Cloud -- Offload some of the work into the public cloud and optimize costs through more elastic computing.

Take Zynga for example. Zynga recently launched their own private cloud offering, zCloud, and is now using a hybrid cloud strategy in which they move some of the heavy workload from the Amazon cloud into their private cloud environment. With this approach Zynga was able to increase utilization by 3x, which means that they will need 1/3 of the servers that they would need from Amazon for the same workload, as noted by CTO Allan Leinwand on Zynga’s engineering blog:

For social games specifically, zCloud offers 3x the efficiency of standard public cloud infrastructure. For example, where our games in the public cloud would require three physical servers, zCloud only uses one. We worked on provisioning and automation tools to make zCloud even faster and easier to set up. We’ve optimized storage operations and networking throughput for social gaming. Systems that took us days to set up instead took minutes. zCloud became a sports car that’s finely tuned for games.

How Does Cloudify Add Cloud Portability to Your Hadoop Distribution?

Cloudify recipes uses a feature called machine templates. Machine templates are an abstraction of the underlying compute resource. The mapping of the machine template into the particular cloud infrastructure is done through the Cloud Driver.

Cloudify comes with a large and continuously growing list of Cloud Drivers  available for HP, Rackspace, Amazon, Azure, and CloudStack, as well as for completely non-virtualized environments, which is basically just a bunch of bare-metal machines with an IP and network. The Cloud Driver also plugs in with JClouds and as such can plug into any cloud that is supported through the JClouds framework. 

With all this, running your Hadoop deployment on any cloud becomes just a matter of a simple configuration of the target cloud end-point.

Current Status:

We've made the entire project available on  GitHub. The technical description of that work is provided here. These days, we're working closely with our IBM partners to harden and optimize our BigInsights integration, and will be coming out with more updates shortly. Obviously, this is meant to be an ongoing process, so I'd really appreciate any feedback or comments or contributions.

In the next post we will outline more specifics about the work with IBM BigInsights. 

Final Notes

Adding consistent management and cloud portability to your Big Data system allows you to reduce the operational and infrastructure cost of running Big Data systems. It's clear to me that this is just the begining for what we can achieve through this work. The flexibilty that we can add through the integration of Cloudify with IBM BigInsights and the Cloudera distribution will allow us to easily plug into other services and cloud infrastructures. It will also allow us to deploy and manage even the most complex systems through a single-command... So stay tuned.

References

• Big Data in the cloud with Cloudify
• Cloudify and IBM InfoSphere BigInsights
• The Rise of Bare metal Clouds
• Lessons from Zynga & Sony on moving from Amazon AWS
• Putting BigData/ Cassandra on the Cloud (Online session from the Cassandra Summit 2012)
• Cloudify Hadoop Recipies peoject on GitHub

 

 

Lessons from the Heroku/Amazon Outage

Earlier this week we experienced the third significant AWS outage in the past 14 months, as reported by Justin Lee in his report Heroku, Pinterest Among Sites Knocked Offline in Amazon Data Center Outage on theWhirr magazine:

An Amazon data center in Ashburn, Virginia suffered a power outage at 9:45 p.m. PDT on Thursday, causing some websites using AWS cloud technology to go offline. High-profile websites like Heroku, Pinterest, Quora and HootSuite saw downtime, as well as many smaller sites.

In this post I'd like to briefly address the lessons from this experience, and more importantly, focus on the lessons from this sort of failure and their effect on public PaaS offerings, such as Heroku.

Lesson from the Heroku Outage: Choose the Right PaaS for the Job

One of the promises of PaaS is productivity. PaaS providers like Heroku claim to increase productivity by abstracting the user from the details of the underlying infrastructure, and even go so far as to claim that PaaS makes application operations redundant.

Lesson 1: Choose the Right PaaS for the Job

The main lesson from this outage is that relying on the PaaS provider to carry all your operations isn't always a safe bet. When we move to PaaS we still need to understand how they run their disaster recovery, high-availability, and scaling procedures. Heroku-like PaaS also forces you to a lowest common denominator approach to dealing with continuous availability and scalability. In reality, however, there are many tradeoffs between scalability, performance, and high availability. The best fit between those tradeoffs tends to be application specific, so compromising on a lowest common denominator could be less productive and more costly at the end of the day.

Which brings me to the last point in this section -- PaaS was meant to provide a higher productivity for running our apps on the cloud by abstracting the details of how we run our application (the operation) from the application developer. The black-box approach of many of the public PaaS offerings, such as Heroku, is often an extreme measure in this regard. There is often a close coupling between what the application does and the way we run it. A new class of Open PaaS platforms such as Cloudify, CloudFoundry, and OpenShift offer a different open source alternative that gives you more control of the underlying PaaS platform. Cloudify takes it even further, providing an open recipe model that integrates with the likes of Chef, enabling you to easily customize and control your operations without affecting developer productivity.

Lesson 2: Database Availability Must Address Datacenter Failure

The other area that Heroku, and to be honest, most other PaaS offering don't adequately address is database high-availability, which is obviously a tough area. Specificaly, in the event of data center failure or availability zone failure, as in the present case. To deal with database availability, it is necessary to ensure real-time synchronization of the database across sites. The example at the bottom of this post refers to a specific way this can be done, between two mySql instances running on Amazon and Rackspace with Cloudify.

General Lessons:

Lesson 3: Coping with Failure, Avoiding a Single Point of Failure

The general lesson from this and previous failures is actually not new. To be fair, this lesson is not specific to AWS or to any cloud service. Failures are inevitable, and often happen when and where we least expect them to. Instead of trying to prevent failure from happening we should design our systems to cope with failure. 
The method of dealing with failures is also not that new -- use redundancy, don't rely on a single point failure (including a data center or even a data center provider). Automate the fail-over process, etc...

Haven't Learned from Past Lessons?

The question that comes out of this experience IMO is not necessarily how to deal with failures (those lessons are as old as the mainframe or even older), but rather -- why are we failing to implement the lessons? Assuming that the people running these systems are among the best in the industry makes this question even more intresting. Here is my take on it:

• We give up responsibility when we move to the cloud: When we move our operations to the cloud, we often assume that we're out-sourcing our data center operation completely, including our disaster recovery procedures. The truth is that when we move to the cloud we're only outsourcing the infrastructure, not our operations, and the responsibility of how to use this infrastructure remain ours. 
• Complexity: The current DR processes and tools were designed for a pre-cloud world, and do not work well in a dynamic environment, such as the cloud. Many of the tools that are provided by the cloud vendors (Amazon in this specific case) are still fairly complex to use.

Implementing Past Lessons in a Cloud World

The first point is is easy -- we need to assume full responsibility for our applications' disaster recovery procedures, in the cloud world just as if we were running our own data center. The hard part in the cloud world is that we often have less visibility, control, and knowledge of the infrastructure, which affects our ability to protect our applications -- and each sub-component of our application -- from failure. On the other hand, the cloud enables us to spawn new instances easily on various data center locations, a.k.a Availability Zones. 

And so, most failures can be addressed by moving from the failed system to a completely different system regardless of the root cause of the failure. Therefore, the first lesson is that in the cloud world it is easier to implement disaster recovery plans, by moving our application traffic to a completely different redundant system in a snap, rather than trying to protect every component of our application from a failure. If we're willing to tolerate a short window of downtime, we can even use an on-demand backup site rather than pay the consistent cost and overhead of maintaining a hot backup site.

Which brings me to the next point: What do we need to build such a solution?

A consistent redundant environment that is ready to take over in case of failure needs to include the following elements:

• Workload Migration: Specifically, the ability to clone your application environment and configuration in a consistent way accorss sites, and on demand.
• Data Synchronization: The ability to maintain a real-time copy of the data between two sites. 
• Network Connectivity: Enabling the flow of network traffic between two sites.

Which leads to the second challenge: Complexity. Here, I would use an example of a simple web-app and show how we can easily create two sites on demand. I would even go so far as to set this environment on two separate clouds to show how we can ensure an even higher degree of redundancy by running our application across two different cloud providers.

A Step by Step Example: Fail-Over from AWS to Rackspace

In this example, we picked Amazon and Rackspace as the two target sites. The same solution would also work between two availability zones in Amazon or data centers. We've also tried the same example with a combination of HP Cloud Services and a flavor of a private cloud.

The example demonstrates a very simple web application with global load-balancer (Rackspace), and a Web application (Pet Clinic) based on Tomcat as the Web front end and MySQL as the database.

On both ends we used GigaSpaces XAP Transactional WAN replication as a replication channel between the two instances of MySQL and Cloudify to handle the workload migration between the sites.

The Goal: Fail-over with no change to the target application

The goals that we set for ourselves were seamlessness and no change to the target application or database. We achieved this by plugging the replication service into the existing instances of MySQL. The replicating service listened to the MySQL events and replicated every change to its peer MySQL instance. Cloudify enabled us to clone the same application in both Amazon and Racksapce while maintaining a consistent configuration setup as well as consistent scaling and fail-over SLAs. Cloudify does this by abstracting all the information through portable recipe definitions. Cloudify wraps the application instances with its management and control services based on the definition provided in the recipe. This enabled us to clone the environment as well as add elastic scaling without changing the target application (Pet Clinic in this case).

You can read the full details, including code references on Github, in Dotan Horvits' blog post: AWS Outage Thoughts on Disaster Recovery Policies.

Join GigaSpaces for our upcoming events to learn more about this topic:

• Transactional Cross-Site Data Replication webinar this Wednesday June 20th (9:00 AM PST) that will teach you how to achieve consistent data replication across sites in just 10 minutes, enabling disastery recovery, cloud bursting, and more. Register here.
• The DevOps PaaS Infusion Meetup this Tuesday June 19th at 6:30 PM - NetFlix and OpsCode will be joining GigaSpaces at this meetup in NY where one of the highlights will be a panel on lesson learning from the recent AWS outage.  Register here.

 

References:

• Heroku, Pinterest Among Sites Knocked Offline in Amazon Data Center Outage
• AWS Outage Thoughts on Disaster Recovery Policies
• Mapping the Cloud Stack
• Making Cloud Bursting a Practical Reality

Making Hadoop Run Faster

One of the challenges in processing data is that the speed at which we can input data is quite often much faster than the speed at which we can process it. This problem becomes even more pronounced in the context of Big Data, where the volume of data keeps on growing, along with a corresponding need for more insights, and thus the need for more complex processing also increases.

Batch Processing to the Rescue

Hadoop was designed to deal with this challenge in the following ways:

1. Use a distributed file system: This enables us to spread the load and grow our system as needed.

2. Optimize for write speed: To enable fast writes the Hadoop architecture was designed so that writes are first logged, and then processed. This enables fairly fast write speeds.

3. Use batch processing (Map/Reduce) to balance the speed for the data feeds with the processing speed.

Batch Processing Challenges

The challenge with batch-processing is that it assumes that the feeds come in bursts. If our data feeds come in on a continuous basis, the entire assumption and architecture behind batch processing starts to break down.

If we increase the batch window, the result is higher latency between the time the data comes in until the time we actually get it into our reports and insights. Moreover, the number of hours is finite -- in many systems the batch window is done on a daily basis. Often, the assumption is that most of the processing can be done during off-peak hours. But as the volume gets bigger, the time it takes to process the data gets longer, until it reaches the limit of the hours in a day and then we face dealing with a continuously growing backlog. In addition, if we experience a failure during the processing we might not have enough time to re-process.

Speed Things Up Through Stream-Based Processing

The concept of stream-based processing is fairly simple. Instead of logging the data first and then processing it, we can process it as it comes in. 

A good analogy to explain the difference is a manufacturing pipeline. Think about a car manufacturing pipeline: Compare the process of first putting all the parts together and then assembling them piece by piece, versus a process in which you package each unit at the manufacturer and only send the pre-packaged parts to the assembly line. Which method is faster?

Data processing is just like any pipeline. Putting stream-based processing at the front is analogous to pre-packaging our parts before  they get to the assembly line, which is in our case is the Hadoop batch processing system.

As in manufacturing, even if we pre-package the parts at the manufacturer we still need an assembly line to put all the parts together. In the same way, stream-based processing is not meant to replace our Hadoop system, but rather to reduce the amount of work that the system needs to deal with, and to make the work that does go into the Hadoop process easier, and thus faster, to process.

In-memory stream processing can make a good stream processing system, as Curt Monash’s points out on his research traditional databases will eventually end up in RAM. An example of how this can work in the context of real-time analytics for Big Data is provided in this case study, where we demonstrate the processing of Twitter feeds using stream-based processing that then feeds a Big Data database for the serving providing the historical agregated view as described in the diagram below.

Faster Processing the Google Way: Using Stream-Based Processing Instead of Map/Reduce

Due to a lack of alternatives at the time, in many Big Data systems today Map/Reduce is used in areas where it wasn't a very good fit in the first place.  A good example is using Map/Reduce for maintaining a global search index. With Map/Reduce, we basically rebuild the index, where it would actually make more sense to update it with changes as they come in.

Google moved large part of its index processing from Map/Reduce into a more real-time processing model, as noted in this recent post:

So, how does Google manage to make its search results increasingly real-time? By displacing GMR in favor of an incremental processing engine called Percolator. By dealing only with new, modified, or deleted documents and using secondary indices to efficiently catalog and query the resulting output, Google was able to dramatically decrease the time to value. As the authors of the Percolator paper write, ”[C]onverting the indexing system to an incremental system … reduced the average document processing latency by a factor of 100.” This means that new content on the Web could be indexed 100 times faster than possible using the MapReduce system!

..Some datasets simply never stop growing ..it is why trigger-based processing is now available in HBase, and it is a primary reason that Twitter Storm is gaining momentum for real-time processing of stream data.

Final Notes

We can make our Hadoop system run faster by pre-processing some of the work before it gets into our Hadoop system. We can also move the types of workload for which batch processing isn't a good fit out of the Hadoop Map/Reduce system and use Stream Processing, as Google did.

Interestingly enough, I recently found out that Twitter Storm came up with an option to integrate an in-memory data store into Storm through the Trident-State project. The combination of the two makes lots of sense and something were currently looking at right now so stay tuned.

 

References

• Build Your Own Twitter Like Real Time Analytics - A Step By Step Guide (Blog Post)
• Build Your Own Twitter Like Real Time Analytics - A Step By Step Guide (Online Tutorial)
• Real Time analytics for Big Data: Facebook's New Realtime Analytics System
• Beyond hadoop: fast queries from big data
• Traditional databases will eventually wind up in RAM
• Why the days are numbered for Hadoop as we know it

Related articles

Elephants and Eyeballs: Real-Time Big Data in the Real World

Cassandra on ACID

GigaOm: 'Hadoop's days are numbered...' Are they?

Hadoop Officially a Big Deal for Big Data