Nati Shalom's Blog

The Elephant in the Cloud -- Putting Hadoop on any Cloud

In one of my earlier posts I discussed  in general terms why it makes a lot of sense to put  Big Data on the cloud.

The first step that we took in this regard with Cloudify was to make NoSQL databases, such as Cassandra and MongoDB, run on any the cloud through Cloudify recipes. Uri Cohen gave an excellent talk on this during the Cassandra Summit where he provided a insights into this work.

In the past couple of weeks we've been working on the next phase of this project: Putting Hadoop on the cloud.

As there are various solutions that aim to do something similar, I thought that a good start would be to first outline where we fit in this ecosystem.

Cloudifying Hadoop -- What Does That Actually Mean?

Yet another Hadoop distribution?

There are multiple distributions of the Hadoop framework today. All come with strong management and sets of tools and do a fairly good job. With that in mind, it was clear to us that our goal wouldn't be to come up with yet another Hadoop distribution but rather to integrate with the ones that are already out there. We picked IBM BigInsights and the Cloudera distribution as the first targets for the project.

When we realized that this was the path we wanted to pursue, the main question that immediately came up was:

What value can we add on top of IBM BigInsights and the Cloudera distribution?

Big Data systems tend to be complex to manage and operate. BigInsights and Cloudera provide good tools to make the process of configuring and deploying Hadoop significantly simpler. That's good for the Hadoop part of the story. Big Data systems and applications often include other services such as relational databases, other NoSQL databases such as Cassandra or MongoDB, stream processing such as Twitter-Storm and GigaSpaces XAP, web front ends such as Tomcat, Play framework and Node.js. Each framework comes with its own management, installation, configuration, and scaling solutions, as shown in the diagram below:

  

Managing each component of your Big Data system seperately is an operational nightmare. That complexity grows exponentially as the system gets bigger, and in Big Data that's just to be expected. 

We realized that one of the areas in which we can reduce this operational complexity is through Consistent Management. By Consistent Management, I'm referring specifically to consistent deployment, configuration, and management across the stack. Consistent management applies not only to the deployment phase, but also to post-deployment, including fail-over, scaling, and upgrades. In addition, Big Data systems tend to consume a lot of infrastructure resources that can easily pile up to thousands of nodes. We realized that we can optimize the infrastructure cost for running the Big Data system through Cloud Enablement and Cloud Portability. Cloud Portabiltiy enables you to choose the right cloud for the job. For example, you could choose a bare-metal cloud for I/O intensive workloads or a virtualized/public cloud for more sporadic workloads. Below is a more detailed description on how we implemeted these two properties:

1. Consistent Management 

With consistent management, we wanted to make the experience of managing each of the tiers and services in the Big Data System consistent throughout the entire stack. This is where Cloudify plugs in.

Cloudify already plugs into a variety of web containers, databases, and through the integration with Chef also into hundreds of services available through the Chef Cookbook.

The process for achieving consistent management for BigInsights and Cloudera Hadoop distribution basically maps to the creation of a Cloudify recipe. The purpose of the recipe in this specific case was to map all the capabilities that come with the IBM BigInsights and Cloudera distribution in a way that would be later consistent with other services in the stack. 

For this, we needed to come up with the following:

• Deployment automation -- A Cloudify recipe enables us to automate the installation, configuration, and deployment of a given service. In the context of a Hadoop distribution, which comes with its own scripts for automating these phases, the process of creating a Cloudify recipe basically meant mapping the specific Hadoop distribution scripts into the Cloudify convention, as well as using the Cloudify discovery, global context, and other services to dynamically inject configuration values that are driven on-the-fly through the deployment process. In the case of BigInsights, this basically mapped into a simple recipe that would launch the machines, set up the SSH environment, and update the BigInsights configuration file with the relevant information. We then called BigInsights to install the NameNode, DataNode, Management, Hive and other services that come with the BigInsights distribution. 
• Automation of post-deployment operations -- In the Hadoop scenario, post-deployment operations mapped into the abilities to add a node, rebalance, and test the environment. To automate these processes, we used Cloudify custom commands. Custom commands enable us to give an alias to those maintenance scripts, and then it enables the users to call those scripts using a simple naming convention without knowing the physical location of Hadoop deployment. 
• SLA-based monitoring and auto-scaling -- SLA-based monitoring enables measure how the Hadoop distribution behaves after it's been deployed, and also to map specific actions when a given SLA is breached. For example, you can monitor a situation where a node fails and then spawns a new machine to take over that fail-node. You can also use the monitors to trigger any of the actions defined in the custom commands, which basically means that you can automate not only a fail-over process but also the ability to scale, by adding more capacity. To do this with Cloudify, use Custom Metrics to monitor the specific metrics that are of interest. One great power of custom metrics IMO is that you're not bound to metrics that are provided by the Hadoop distribution, and you can easily generate compound metrics that can be generated out of correlating metrics from multiple sources, such as such as network statistics or a cloud management system. You can then use the Cloudify Scaling Rules to attach rules to each of those metrics and trigger a fail-over or scaling event.

2. Cloud Enablement and Portability 

The Cloud is basically a more efficient way to run IT. Today, there are growing numbers of cloud offerings and types of clouds, each providign different SLAs and pricing models. There are the public clouds such as Amazon, Rackspace, and also HP, IBM and Dell who's coming out with their own cloud offering. Microsoft and Google are also coming out with new offerings that will undoubtedly change the dynamic in this space. There are also the bare-metal clouds, the private cloud, the OpenSource cloud, etc., and obviously the VMware cloud, which are also an important part of the mix.

Enabling your Big Data systems for all these environments means you can leverage all the agility, efficiency, and flexibiltiy of the cloud, while also allowing you to choose the right cloud for the job.

Given the rapid dynamic of the cloud world, you also want to keep your options open so that you can leverage any future developments and services as they become available.

Controlling Your Cloud Infrastructure is a Critical Part of Controlling Your Big Data Costs

There are various ways to control Big Data costs through controlling  infrastructure. Here are a few examples:

• Bare-Metal -- Reduce the number of machine instances for high I/O workloads.
• Private-Cloud --Optimize performance for your specific workload through specialized hardware (Inifiband, dedicated network, specialized hard drives, etc.) and also have better control over our data secrurity.
• Hybrid-Cloud -- Offload some of the work into the public cloud and optimize costs through more elastic computing.

Take Zynga for example. Zynga recently launched their own private cloud offering, zCloud, and is now using a hybrid cloud strategy in which they move some of the heavy workload from the Amazon cloud into their private cloud environment. With this approach Zynga was able to increase utilization by 3x, which means that they will need 1/3 of the servers that they would need from Amazon for the same workload, as noted by CTO Allan Leinwand on Zynga’s engineering blog:

For social games specifically, zCloud offers 3x the efficiency of standard public cloud infrastructure. For example, where our games in the public cloud would require three physical servers, zCloud only uses one. We worked on provisioning and automation tools to make zCloud even faster and easier to set up. We’ve optimized storage operations and networking throughput for social gaming. Systems that took us days to set up instead took minutes. zCloud became a sports car that’s finely tuned for games.

How Does Cloudify Add Cloud Portability to Your Hadoop Distribution?

Cloudify recipes uses a feature called machine templates. Machine templates are an abstraction of the underlying compute resource. The mapping of the machine template into the particular cloud infrastructure is done through the Cloud Driver.

Cloudify comes with a large and continuously growing list of Cloud Drivers  available for HP, Rackspace, Amazon, Azure, and CloudStack, as well as for completely non-virtualized environments, which is basically just a bunch of bare-metal machines with an IP and network. The Cloud Driver also plugs in with JClouds and as such can plug into any cloud that is supported through the JClouds framework. 

With all this, running your Hadoop deployment on any cloud becomes just a matter of a simple configuration of the target cloud end-point.

Current Status:

We've made the entire project available on  GitHub. The technical description of that work is provided here. These days, we're working closely with our IBM partners to harden and optimize our BigInsights integration, and will be coming out with more updates shortly. Obviously, this is meant to be an ongoing process, so I'd really appreciate any feedback or comments or contributions.

In the next post we will outline more specifics about the work with IBM BigInsights. 

Final Notes

Adding consistent management and cloud portability to your Big Data system allows you to reduce the operational and infrastructure cost of running Big Data systems. It's clear to me that this is just the begining for what we can achieve through this work. The flexibilty that we can add through the integration of Cloudify with IBM BigInsights and the Cloudera distribution will allow us to easily plug into other services and cloud infrastructures. It will also allow us to deploy and manage even the most complex systems through a single-command... So stay tuned.

References

• Big Data in the cloud with Cloudify
• Cloudify and IBM InfoSphere BigInsights
• The Rise of Bare metal Clouds
• Lessons from Zynga & Sony on moving from Amazon AWS
• Putting BigData/ Cassandra on the Cloud (Online session from the Cassandra Summit 2012)
• Cloudify Hadoop Recipies peoject on GitHub

 

 

Lessons from the Heroku/Amazon Outage

Earlier this week we experienced the third significant AWS outage in the past 14 months, as reported by Justin Lee in his report Heroku, Pinterest Among Sites Knocked Offline in Amazon Data Center Outage on theWhirr magazine:

An Amazon data center in Ashburn, Virginia suffered a power outage at 9:45 p.m. PDT on Thursday, causing some websites using AWS cloud technology to go offline. High-profile websites like Heroku, Pinterest, Quora and HootSuite saw downtime, as well as many smaller sites.

In this post I'd like to briefly address the lessons from this experience, and more importantly, focus on the lessons from this sort of failure and their effect on public PaaS offerings, such as Heroku.

Lesson from the Heroku Outage: Choose the Right PaaS for the Job

One of the promises of PaaS is productivity. PaaS providers like Heroku claim to increase productivity by abstracting the user from the details of the underlying infrastructure, and even go so far as to claim that PaaS makes application operations redundant.

Lesson 1: Choose the Right PaaS for the Job

The main lesson from this outage is that relying on the PaaS provider to carry all your operations isn't always a safe bet. When we move to PaaS we still need to understand how they run their disaster recovery, high-availability, and scaling procedures. Heroku-like PaaS also forces you to a lowest common denominator approach to dealing with continuous availability and scalability. In reality, however, there are many tradeoffs between scalability, performance, and high availability. The best fit between those tradeoffs tends to be application specific, so compromising on a lowest common denominator could be less productive and more costly at the end of the day.

Which brings me to the last point in this section -- PaaS was meant to provide a higher productivity for running our apps on the cloud by abstracting the details of how we run our application (the operation) from the application developer. The black-box approach of many of the public PaaS offerings, such as Heroku, is often an extreme measure in this regard. There is often a close coupling between what the application does and the way we run it. A new class of Open PaaS platforms such as Cloudify, CloudFoundry, and OpenShift offer a different open source alternative that gives you more control of the underlying PaaS platform. Cloudify takes it even further, providing an open recipe model that integrates with the likes of Chef, enabling you to easily customize and control your operations without affecting developer productivity.

Lesson 2: Database Availability Must Address Datacenter Failure

The other area that Heroku, and to be honest, most other PaaS offering don't adequately address is database high-availability, which is obviously a tough area. Specificaly, in the event of data center failure or availability zone failure, as in the present case. To deal with database availability, it is necessary to ensure real-time synchronization of the database across sites. The example at the bottom of this post refers to a specific way this can be done, between two mySql instances running on Amazon and Rackspace with Cloudify.

General Lessons:

Lesson 3: Coping with Failure, Avoiding a Single Point of Failure

The general lesson from this and previous failures is actually not new. To be fair, this lesson is not specific to AWS or to any cloud service. Failures are inevitable, and often happen when and where we least expect them to. Instead of trying to prevent failure from happening we should design our systems to cope with failure. 
The method of dealing with failures is also not that new -- use redundancy, don't rely on a single point failure (including a data center or even a data center provider). Automate the fail-over process, etc...

Haven't Learned from Past Lessons?

The question that comes out of this experience IMO is not necessarily how to deal with failures (those lessons are as old as the mainframe or even older), but rather -- why are we failing to implement the lessons? Assuming that the people running these systems are among the best in the industry makes this question even more intresting. Here is my take on it:

• We give up responsibility when we move to the cloud: When we move our operations to the cloud, we often assume that we're out-sourcing our data center operation completely, including our disaster recovery procedures. The truth is that when we move to the cloud we're only outsourcing the infrastructure, not our operations, and the responsibility of how to use this infrastructure remain ours. 
• Complexity: The current DR processes and tools were designed for a pre-cloud world, and do not work well in a dynamic environment, such as the cloud. Many of the tools that are provided by the cloud vendors (Amazon in this specific case) are still fairly complex to use.

Implementing Past Lessons in a Cloud World

The first point is is easy -- we need to assume full responsibility for our applications' disaster recovery procedures, in the cloud world just as if we were running our own data center. The hard part in the cloud world is that we often have less visibility, control, and knowledge of the infrastructure, which affects our ability to protect our applications -- and each sub-component of our application -- from failure. On the other hand, the cloud enables us to spawn new instances easily on various data center locations, a.k.a Availability Zones. 

And so, most failures can be addressed by moving from the failed system to a completely different system regardless of the root cause of the failure. Therefore, the first lesson is that in the cloud world it is easier to implement disaster recovery plans, by moving our application traffic to a completely different redundant system in a snap, rather than trying to protect every component of our application from a failure. If we're willing to tolerate a short window of downtime, we can even use an on-demand backup site rather than pay the consistent cost and overhead of maintaining a hot backup site.

Which brings me to the next point: What do we need to build such a solution?

A consistent redundant environment that is ready to take over in case of failure needs to include the following elements:

• Workload Migration: Specifically, the ability to clone your application environment and configuration in a consistent way accorss sites, and on demand.
• Data Synchronization: The ability to maintain a real-time copy of the data between two sites. 
• Network Connectivity: Enabling the flow of network traffic between two sites.

Which leads to the second challenge: Complexity. Here, I would use an example of a simple web-app and show how we can easily create two sites on demand. I would even go so far as to set this environment on two separate clouds to show how we can ensure an even higher degree of redundancy by running our application across two different cloud providers.

A Step by Step Example: Fail-Over from AWS to Rackspace

In this example, we picked Amazon and Rackspace as the two target sites. The same solution would also work between two availability zones in Amazon or data centers. We've also tried the same example with a combination of HP Cloud Services and a flavor of a private cloud.

The example demonstrates a very simple web application with global load-balancer (Rackspace), and a Web application (Pet Clinic) based on Tomcat as the Web front end and MySQL as the database.

On both ends we used GigaSpaces XAP Transactional WAN replication as a replication channel between the two instances of MySQL and Cloudify to handle the workload migration between the sites.

The Goal: Fail-over with no change to the target application

The goals that we set for ourselves were seamlessness and no change to the target application or database. We achieved this by plugging the replication service into the existing instances of MySQL. The replicating service listened to the MySQL events and replicated every change to its peer MySQL instance. Cloudify enabled us to clone the same application in both Amazon and Racksapce while maintaining a consistent configuration setup as well as consistent scaling and fail-over SLAs. Cloudify does this by abstracting all the information through portable recipe definitions. Cloudify wraps the application instances with its management and control services based on the definition provided in the recipe. This enabled us to clone the environment as well as add elastic scaling without changing the target application (Pet Clinic in this case).

You can read the full details, including code references on Github, in Dotan Horvits' blog post: AWS Outage Thoughts on Disaster Recovery Policies.

Join GigaSpaces for our upcoming events to learn more about this topic:

• Transactional Cross-Site Data Replication webinar this Wednesday June 20th (9:00 AM PST) that will teach you how to achieve consistent data replication across sites in just 10 minutes, enabling disastery recovery, cloud bursting, and more. Register here.
• The DevOps PaaS Infusion Meetup this Tuesday June 19th at 6:30 PM - NetFlix and OpsCode will be joining GigaSpaces at this meetup in NY where one of the highlights will be a panel on lesson learning from the recent AWS outage.  Register here.

 

References:

• Heroku, Pinterest Among Sites Knocked Offline in Amazon Data Center Outage
• AWS Outage Thoughts on Disaster Recovery Policies
• Mapping the Cloud Stack
• Making Cloud Bursting a Practical Reality

Making Hadoop Run Faster

One of the challenges in processing data is that the speed at which we can input data is quite often much faster than the speed at which we can process it. This problem becomes even more pronounced in the context of Big Data, where the volume of data keeps on growing, along with a corresponding need for more insights, and thus the need for more complex processing also increases.

Batch Processing to the Rescue

Hadoop was designed to deal with this challenge in the following ways:

1. Use a distributed file system: This enables us to spread the load and grow our system as needed.

2. Optimize for write speed: To enable fast writes the Hadoop architecture was designed so that writes are first logged, and then processed. This enables fairly fast write speeds.

3. Use batch processing (Map/Reduce) to balance the speed for the data feeds with the processing speed.

Batch Processing Challenges

The challenge with batch-processing is that it assumes that the feeds come in bursts. If our data feeds come in on a continuous basis, the entire assumption and architecture behind batch processing starts to break down.

If we increase the batch window, the result is higher latency between the time the data comes in until the time we actually get it into our reports and insights. Moreover, the number of hours is finite -- in many systems the batch window is done on a daily basis. Often, the assumption is that most of the processing can be done during off-peak hours. But as the volume gets bigger, the time it takes to process the data gets longer, until it reaches the limit of the hours in a day and then we face dealing with a continuously growing backlog. In addition, if we experience a failure during the processing we might not have enough time to re-process.

Speed Things Up Through Stream-Based Processing

The concept of stream-based processing is fairly simple. Instead of logging the data first and then processing it, we can process it as it comes in. 

A good analogy to explain the difference is a manufacturing pipeline. Think about a car manufacturing pipeline: Compare the process of first putting all the parts together and then assembling them piece by piece, versus a process in which you package each unit at the manufacturer and only send the pre-packaged parts to the assembly line. Which method is faster?

Data processing is just like any pipeline. Putting stream-based processing at the front is analogous to pre-packaging our parts before  they get to the assembly line, which is in our case is the Hadoop batch processing system.

As in manufacturing, even if we pre-package the parts at the manufacturer we still need an assembly line to put all the parts together. In the same way, stream-based processing is not meant to replace our Hadoop system, but rather to reduce the amount of work that the system needs to deal with, and to make the work that does go into the Hadoop process easier, and thus faster, to process.

In-memory stream processing can make a good stream processing system, as Curt Monash’s points out on his research traditional databases will eventually end up in RAM. An example of how this can work in the context of real-time analytics for Big Data is provided in this case study, where we demonstrate the processing of Twitter feeds using stream-based processing that then feeds a Big Data database for the serving providing the historical agregated view as described in the diagram below.

Faster Processing the Google Way: Using Stream-Based Processing Instead of Map/Reduce

Due to a lack of alternatives at the time, in many Big Data systems today Map/Reduce is used in areas where it wasn't a very good fit in the first place.  A good example is using Map/Reduce for maintaining a global search index. With Map/Reduce, we basically rebuild the index, where it would actually make more sense to update it with changes as they come in.

Google moved large part of its index processing from Map/Reduce into a more real-time processing model, as noted in this recent post:

So, how does Google manage to make its search results increasingly real-time? By displacing GMR in favor of an incremental processing engine called Percolator. By dealing only with new, modified, or deleted documents and using secondary indices to efficiently catalog and query the resulting output, Google was able to dramatically decrease the time to value. As the authors of the Percolator paper write, ”[C]onverting the indexing system to an incremental system … reduced the average document processing latency by a factor of 100.” This means that new content on the Web could be indexed 100 times faster than possible using the MapReduce system!

..Some datasets simply never stop growing ..it is why trigger-based processing is now available in HBase, and it is a primary reason that Twitter Storm is gaining momentum for real-time processing of stream data.

Final Notes

We can make our Hadoop system run faster by pre-processing some of the work before it gets into our Hadoop system. We can also move the types of workload for which batch processing isn't a good fit out of the Hadoop Map/Reduce system and use Stream Processing, as Google did.

Interestingly enough, I recently found out that Twitter Storm came up with an option to integrate an in-memory data store into Storm through the Trident-State project. The combination of the two makes lots of sense and something were currently looking at right now so stay tuned.

 

References

• Build Your Own Twitter Like Real Time Analytics - A Step By Step Guide (Blog Post)
• Build Your Own Twitter Like Real Time Analytics - A Step By Step Guide (Online Tutorial)
• Real Time analytics for Big Data: Facebook's New Realtime Analytics System
• Beyond hadoop: fast queries from big data
• Traditional databases will eventually wind up in RAM
• Why the days are numbered for Hadoop as we know it

Related articles

Elephants and Eyeballs: Real-Time Big Data in the Real World

Cassandra on ACID

GigaOm: 'Hadoop's days are numbered...' Are they?

Hadoop Officially a Big Deal for Big Data

Woz is Worried About “Everything Going to the Cloud” — the Real Issue is Giving Up Control

 

In a recent article, Steve Wozniak, who co-founded Apple with the late Steve Jobs, predicted "horrible problems" in the coming years as cloud-based computing takes hold. 

 

"I really worry about everything going to the cloud,".. "I think it's going to be horrendous. I think there are going to be a lot of horrible problems in the  next five years. ...."...with the cloud, you don’t own anything. You already signed it away.”

 

When I first read the title I thought, Wozniak sounds like Larry Ellison two years ago, when he pitched the Cloud is hype, before he made a 180-degree turn to acknowledge Oracle wished to be a cloud vendor too. 

 

Reading it more carefully, I realized the title is instead just misleading. Wozniak actually touches on something that I hear more often, as the cloud hype cycle is moves from a Peak of Inflated Expectations into through the Trough of Disillusionment.

 

Wozniak echos an important lesson, that IMO, is major part of the reason many of the companies that moved to cloud have experienced lots of outages during the past months. I addressed several of these aspects in in a recent blog post: Lessons from the Heroku/Amazon Outage.

 

When we move our operations to the cloud, we often assume that we're out-sourcing our data center operation completely, including our disaster recovery procedures. The truth is that when we move to the cloud we're only outsourcing the infrastructure, not our operations, and the responsibility of how to use this infrastructure remain ours. 

 

Choosing better tradeoffs between producivity and control

 

For companies today, the main reason we chose to move to the cloud in the first place was to gain better agility and productivity. But in starting this cloud journey, we found that we had to give up some measure of control to achieve the agility and productivity.

 

The good news is that as the industry mature there are more choices that provides better tradeoffs between producivity and control:

• Open source cloud such as OpenStack and CloudStack 
• Private cloud offering 
• DevOps and automation tools such as Chef and Puppet
• OpenSource PaaS such as Cloudify, OpenShift and CloudFoundry
• DevOps and PaaS combined such Cloudify

As businesses look at cloud strategy today, there isn't a need to give up control over productivity. With technologies like Cloudify, businesses can get the best out of both worlds.

 

References

• Apple Co-Founder Steve Wozniak Distrusts the Cloud: Is He Right?
• Mapping the Cloud/PaaS Stack
• Why Larry Don't Get It
• Lessons from Zynga & Sony on moving from Amazon AWS
• Public vs Private clouds (Again!)- it's not about the cost

Putting DevOps and PaaS together with Cloudify

 

Putting DevOps and PaaS together with Cloudify

DevOps and PaaS represent two different paradigms for delivering applications to the cloud.

• DevOps - DevOps takes an automation approach - with DevOps we basically script the process of installation, configuration and deployment of the application stack. 
• PaaS - PaaS takes an abstraction approach by abstracting the details of the cloud infrastructure from the developers. 

DevOps tends to cater to operations guys that want to maintain the flexibility and control over the infrastructure and PaaS caters more to developers who just want to write code and deliver it without worrying too much about the details of the underlying infrastructure.

 

For a while DevOps and PaaS had been considered two competing paradigms as noted in Tom Mornini’s (Engine Yard) post DevOps is DOA:

"DevOps essentially requires everyone involved in the development, deployment, and management of software to change the way they work, to cross-train for multiple functions, and to collaborate with people they’ve all too often viewed as adversaries. It creates new processes and methodologies and, in many cases, uses new tools to get the job done. In addition, it often requires a small army of consultants to implement and manage the transition. That is why the “promise” and “potential” of DevOps has yet to be realized by more than a handful of real-world companies.

...The appeal of the DevOps model fades even further when you consider the fact that there is a better option! You can achieve the intended benefits of DevOps without any of the turmoil by taking advantage of Platform-as-a-Service offerings, or PaaS."

Putting DevOps and PaaS together

The truth is that it's hard to choose between the two paradigms. DevOps provides a high degree of control and flexibility but this flexibility comes at the cost of complexity and reduced productivity and often requires a culture change as Tom Mornini pointed out. PaaS on the other hand, provides an extremely simple deployment environment but it also forces you to fit into the PaaS environment, including the language of choice, middleware stack and cloud environment etc.

The best of both worlds- The control of DevOps and Productivity of PaaS

Do we really need to choose between the two? We can use DevOps as the foundation of the PaaS infrastructure. In this way, the operations guys can choose their stack of choice, package their specific blueprint, pick their cloud of choice and package all that as a pre-baked environment to the developers who in turn can be relieved of all of the details as they would be with any other PaaS. In this way we get the control of DevOps and productivity of PaaS.

At this point I'll refer to Cloudify as an implementation reference of that concept to illustrate my point in this regard.

Putting DevOps and PaaS together with Cloudify

Cloudify is an Open PaaS Stack that brings DevOps and PaaS together. Here is how:

The control of DevOps 

1. Application Blueprint - Cloudify allows you to add your own application blueprint into the platform through recipes. Recipes are at the heart of Cloudify architecture. A recipe in Cloudify is a Groovy-based DSL. The recipe is where you define your application stack, including the installation, configuration, monitoring, scaling and fail-over and tell Cloudify what, how and where to deploy your app.
2. Integration with DevOps tools such as Chef - Cloudify allows you to call Chef cookbooks from within a Cloudify recipe. This gives Cloudify users a way to leverage the wide variety of existing Chef cookbooks and community and decorate it with a Cloudify recipe if needed, to add scaling, fail-over automation etc.
3. Operational Task Automation - Custom Commands  allow you to attach common operation tasks such as update-app, such as in the case of a web app, run-snapshot in the case of a database and more, as part of a service recipe, as well as automate the post-deployment operational procedures of an application alongside its lifecycle.

The productivity of PaaS

Developers can run one of the existing recipes as they would with any other PaaS without going through any of the steps above. Unlike other Platforms-as-a-Service, they can also customize the existing recipes or pick custom recipes that cover their specific environment and application blueprint. All this without losing that same productivity. Developers can also get better visibility into the underlying infrastructure, and have a better experience when troubleshooting. They can also use Cloudify as an automation tool as part of their continuous deployment environment.

Final words

Adrian Cockcroft a highly regarded Cloud Architect at Netflix provides an excellent insight on how DevOps and PaaS work at Netflix in his post Ops, DevOps and PaaS (NoOps) at Netflix:

“We have a small number of DevOps engineers embedded in the development organization who are building and extending automation for our PaaS, and we have hundreds of developers using NoOps to get their code and datastores deployed in our PaaS.”

An interesting lesson from the Netflix experience is that an important part of running an online service is the ability to resolve any friction for developing and managing such a service. Moving a large part of the operational responsibility to the developers makes it even more important that the development platform (PaaS) come with operations tools that are pre-baked into the developer environment.  At the same time, that same environment needs to also cater to a dedicated (but a significantly smaller) operational team.  This makes the idea of putting DevOps and PaaS together even more critical to reach the business agility and operational efficiency. Netflix built their own PaaS environment to address such a need. Cloudify provides a similar solution for those who want to build their own Netflix environment but without the hassle involved with doing it themselves.

References

• PaaS, Does It Really Mean No-Ops?
• Ops, DevOps and PaaS (NoOps) at Netflix
• Mapping the Cloud/PaaS Stack
• GigaSpaces Cloudify & VMware Cloud Foundry the new PaaS Jailbreaker
• DevOps is DOA
• Productivity vs. Control tradeoffs in PaaS
• DevOps and PaaS - Friend or Foe?
• PaaS shouldn’t be built in Silos