Nati Shalom's Blog

The Breakup: Why OpenStack Needs a Native Alternative to CloudFoundry and OpenShift

 

Alex Freedland, Co-Founder and Chairman at Mirantis, wrote a controversial post in which he basically marks the breakup between some existing PaaS frameworks and OpenStack - CloudFoundry specifically.

Personally, I think that Freedland highlights a growing sentiment in the OpenStack community. The fact that OpenStack is open source should change the way we think of the PaaS and IaaS layers, as I noted in one of my earlier posts, The Blurring Line Between the IaaS and PaaS Layers.

In the case of CloudFoundry and OpenShift specifically, PaaS was built with the assumption that the IaaS layer provides basic compute, storage and network services and all the rest should be handled at the layers above. The reality, though, is that the IaaS layer is continuously moving up the stack in a very similar way to which Amazon has grown its stack.

This shift up the stack changes many of the built-in design assumptions of many existing PaaS products, CloudFoundry and OpenShift specifically. For example, the way we handle user-security, metering and scheduling is often in a completely parallel stack that exists in both the IaaS and PaaS layers. This leads to a lot of unnecessary redundancy and therefore complexity and inconsistency. On top of that, the middleware stack (Ruby vs Python) also leads to another layer of redundancy and complexity, especially for organizations looking to deploy the end-to-end stack. For example, why would we need two different models to handle database availability? Why do we need different agent systems to monitor our instances? If we already have a model to detect and monitor device failure, why do we need to add another failure detection mechanism on top to detect the same failure? In many of these cases, we already built solutions at the IaaS layer and we only need to make the right use of them.

Now there is nothing wrong with CloudFoundry or OpenShift; they do exactly what they were designed to do. That being said, on OpenStack we can do things differently than we can on other closed IaaSs, creating a great opportunity to not just simplify things, but do do things that are much harder otherwise. Let me explain.

The Need for a Native Open PaaS

A native open PaaS is a PaaS that comes with a much thinner stack and relies more heavily on a common set of services provided by the IaaS layer for user management, metering, etc.

Its main focus is to provide a higher level abstraction that would simplify the deployment and management of applications on OpenStack.

Another area that a native open PaaS differs from its existing predecessors is the native DevOps support. Many of the existing PaaS platforms were built with the assumption that developers only worry about their code. However, in real life, continuous delivery or deployment can expose changes at all layers, including the configuration of the application and PaaS stack, and should therefore be consistent with the way updates and configuration management takes place in the IaaS layer.

A native open PaaS should leverage the same DevOps tools used by the IaaS layers to push updates to the application layer.

The Missing Piece in the Puzzle - Open DevOps Automation

Earlier this year, Amazon launched a new piece in their stack - OpsWorks, which basically filled a gap between their existing IaaS and PaaS layers. They refer to it as DevOps automation.

AWS Stack

 

AWS OpsWorks provides an orchestration and workflow service that currently runs on top of Chef and provides users with the ability to build their own custom stacks on top of AWS, as I noted in my earlier post DevOps, PaaS and Everything in Between.

The OpenStack Heat project fills the AWS CloudFormation box in the above image, providing a basic templating model for orchestrating and automating the deployment of software stacks on OpenStack.

I believe that OpenStack also needs  to fill in the OpsWorks box as higher level abstraction for application deployment. Given the open nature of OpenStack and the private cloud focus I think that this piece should be more open and support not just Chef, but also other configuration management frameworks such as Puppet, Ansible and Salt through Heat. There should also be a much stronger emphasis on the application orchestration piece to enable automation of complex application related processes, such as continuous delivery, deployment automation, recovery and scaling and for that we need a more advanced rule engine to support those advanced flows.

If done right, tools like OpsWorks in conjunction with Heat can serve as a great foundation for PaaS. At the same time, I can see many cases in which users would simply use something like OpsWorks directly without needing another layer of abstraction on top of it, hence why it makes sense to keep those boxes also as independent services.

Consolidating the Efforts

There are currently many projects and initiatives in the OpenStack community, each trying to address different aspects of the application deployment and management complexity. Let me name a few:

Adrian Otto, Principal Architect at Rackspace, announced Project Solum support from Rackspace, eBay, Red Hat, Ubuntu / Canonical, dotCloud as well as GigaSpaces.

Some of the guys from my team at GigaSpaces are also going to give a talk during the OpenStack Hong Kong event about delivering the equivalent of Amazon OpsWorks on OpenStack - See the details here.

There are many other related projects and initiatives that I haven’t listed here and many others that are probably being worked on as we speak - it is beyond the scope of this post to list them all.

Looking Into the Future

I think that this is a great and an exciting opportunity for the OpenStack community to deliver on its promise and really push the open source cloud forward. It can do so by breaking the traditional barriers between IaaS and PaaS, and go above and beyond what we've seen normally in closed source clouds. There are many areas in which such an approach can lead to an even greater degree of innovation. For example, think about bringing the network and the application together, making multi-site deployment a breeze.

I look forward to meeting everyone on the Solum team next week during OpenStack Hong Kong and to being a part of this next wave of the revolution.

References

• Project Solum announcement

• Deployment and Management of Your Application on OpenStack using an All Encompassing DSL based on TOSCA (OpenStack session in HK)

• Mirantis To PaaS Vendors: Stop Fighting OpenStack Gravity

• Rackspace jumps on bandwagon with open source PaaS project

• The blurring Line between the IaaS and PaaS layers

• DevOps, PaaS and Everything in Between

 

An Application-Centric Approach to DevOps

 

Matt Jaynes, Founder of DevOps University, wrote a great post on Hacker News, What is Continuous Deployment, in which he points out one of challenges that many people gloss over when they embark on their DevOps journey. In a nutshell, Matt was saying that, "You should get your house in order before aspiring to do automation or CD." In other words, if your testing is not fully automated or is taking too long to process, then having continuous deployment may not be right for you.

The thing that gets me so interested in this entire DevOps movement is that it forces you to think holistically about software development, business cycles and organizational culture. Some refer to this transition as the industrialization of IT. Indeed, there are many parallels between the shift from making cars by hand into a fully automated manufacturing pipeline and the DevOps movement.

With regards to Jaynes’ post, even if you can build a car faster, lack of proper QA may lead to shipping defective parts, and you will still be shipping slowly. In both options, your fast pipeline would end up producing diminishing return, as you may end up paying more for the warehouse or for handling "recall" or fixing defects, let alone the potential loss of reputation or even the entire business.

In short, you need to get your house in order before launching continuous deployment right into production.

Another area of complexity in building continuous delivery and deployment is that it often involves fairly complex processes for handling updates. I noticed that in many cases, people need to write a lot of custom code to implement these processes. That custom code can be fairly hard to maintain and track.

In this post, I would like to share a different approach to handling continuous delivery based on the experience of a Cloudify user - PaddyPower.

Paddy Power are a betting and entertainment company headquartered in Dublin Ireland.  Innovation has been the key to their success and from day one they sought to differentiate themselves from the chasing pack.  Like many organisations, they faced the need to speed up their development and deployment cycle. They went through a complete cycle of moving into agile development as well as transforming their infrastructure into a cloud-based infrastructure based on CloudStack.

They use Cloudify and Chef to run their DevOps automation and continuous delivery.

The way they approached DevOps in general, and continuous delivery in particular, is quite interesting because they took an application-centric approach to DevOps, as opposed to an infrastructure-centric approach as used in some other cases. Let me explain...

What is an Application-Centric Approach to DevOps?

In many DevOps discussions we often tend to jump into low level discussions about tools, and fairly quickly we start to lose sight of what we are actually trying to achieve.

Quite naturally, many start with one of the first steps in automation, which is configuration management - i.e. our ability to consistently configure, setup and update machines with the right stack. In this model, the world looks like a bunch of machines at the core onto which we then tie our software parts. I would refer to that approach as an infrastructure-centric approach.

An application-centric approach is looking at the same problem, but from a different perspective in which we start with a consistent definition of our application stack and blue-print (Load-Balancer, Web Container, Database, etc) and SLAs (failure detection, scaling rules, upgrade processes, etc). Configuration management comes as a natural complementary step for handling the installation and setup of our environment.

In this model, there is a greater emphasis on the orchestration of our application and its associated processes. A good example of such a process is continuous delivery. The execution of a continuous delivery process involves many steps that go far beyond a one-time execution of a given script. We often need to determine in which environment the script is going to be executed, which pieces of our stack are going to be included in each particular deployment, which configuration change is going to be included as well as code updates, and if there will be a database update- then which part will be affected by the change. Another example is continuous availability, automating the recovery process of our application in the event of failure. Such processes can become quite complex as we start to handle multiple availability zones, regions, as well as multiple environments for dev, test and prod.

With an infrastructure-centric approach, we often need to implement those processes into the code or scripts. Quite often, those scripts become very hard to maintain.

With an application-centric approach, we use a set of rules and events to handle the automation of those processes in a more generic fashion. With this approach, many of the policies and processes are handled more implicitly. For example, we can have a policy that if we fail during an update, the process will roll-back, or we can have a policy for creating an exact clone of our system for QAT, and only when it passes the full test may we start rolling it out, gradually moving traffic into those new processes.

Another difference between an application vs infrastructure-centric approach is in the level of abstraction that we expose to our developers. With an infrastructure-centric approach, each developer is often exposed to every part of our system. A continuous delivery model, where developers continuously interact with the system exposing the low level details of the infrastructure to every developer, can lead to complexity that is often unnecessary when all that is needed is to push code to an existing deployment. It can also lead to high vulnerability as a simple mistake can very easily bring an entire system down.

I thought that the following slide by John Turner, Software Development Manager at PaddyPower, does a good job of explaining how an application-driven approach through PaaS abstraction helps to address this challenge:

In summary, PaaS fits nicely into PaddyPower’s DevOps and continuous delivery strategy in the following ways:

• PaaS strives to abstract provisioning and deployment, thus reducing complexity.

• PaaS constraints applications to use features provided by the platform.

• PaaS is development focused.

In the case of Cloudify, the integration with Chef provides a great foundation for more open PaaS, one that fits nicely into DevOps environment. Chef cookbooks and recipes can be incorporated as part of the Cloudify definition of the application stack. It also fits well with the configuration and operational management of the platform.

Join me in our upcoming podcast with Ben Kepes of Clouderati and Diversity.Net, John Turner of PaddyPower and Andy Hawkins from Opscode as we discuss PaddyPower’s journey to app-centric DevOps. Watch it live via Google+ Hangout on Air | October 30th, 6:00 PM GMT, 10:00 AM PST.

What are the Specific Features in Cloudify that Makes it DevOps Oriented?

I will use Matt Jaynes’ definition from his book Taste Test:

• Directive & Directives Script -  Cloudify currently uses Groovy-based recipes with extended DSL as its directives. The Cloudify recipe allows you to combine Chef or Puppet directives as part of the Cloudify recipie. Our future release will include support for Yaml/TOSCA directives and may also include support for Ansible.

• Master Node & Children Nodes - Cloudify automatically bootstraps Master and Child nodes. It has a cloud plug-in that will allow you to automate this entire process and avoid any manual host/IP configuration. The Cloudify integration with Chef takes care of provisioning of Chef Master and Child nodes as well.

• Remote Execution - Cloudify uses custom-commands as a way to route calls to a group of individual sets of service instances. The integration with Chef includes built-in custom commands for updating Chef Coookbooks as well as for executing Chef-Knife and other Chef commands as described here.

In addition, Cloudify provides built-in services to query the state of the cluster through the ServiceContext API and share states across services in the cluster through the Attribute Store API.

Executing a Typical DevOps Task with Cloudify

1. Cloning a service or a complete stack - One of the features of Cloudify is the abstraction of its recipe (blue print) from the underlying infrastructure through a Cloud Driver. A recipe can point to a type of compute or storage node. The actual binding to the physical compute or storage resources takes place at runtime during the deployment process, based on the target environment. That means that you can easily clone a complete or only part of a given application stack across DEV, QA, PROD in a consistent way that will ensure that you use the exact setup between environments. The CloudDriver abstraction allows you to also adapt the size of the resource in each stage without touching the actual recipe - so for example a SMALL-LINUX machine in QA may point to a machine with smaller size and SLA than a SMALL_LINUX in production.

2. Updating a service with a new version - Custom-commands allow you to push updates to a particular instance or group of instances. Quite often, custom command calls will be triggered through a build system, such as Jenkins. Each command has access to the state of the cluster through the Attribute Store and ServiceContext and can, therefore, coordinate the update processes with the state of other services.

3. Orchestration flow - You can use the Cloudify Rest-API or Command Line to write custom flows. Certain flows, such as fail-over and auto-scaling, are built-in and in those cases you will only need to provide custom code to trigger for the flow.

Looking Into the Future

One of the things that we are working on right now is enhancing our orchestration capabilities, allowing users to choose more built-in flows or define their own custom flows that will be fully managed by the Cloudify orchestration engine.

We hope you join us for our live podcast on October 30th at 6:00 PM GMT / 10:00 AM PST on Google+ Hangout on Air.

References

• DevOps PaaS and everything in between.

• Its the apps stupid

• Application Centric Devops

• GigaSpaces Cloudify & VMware Cloud Foundry the new PaaS Jailbreaker

• Putting DevOps and PaaS together with Cloudify

 

Facebook’s vs Twitter’s Approach to Real-Time Analytics

Last year, Twitter and Facebook have released new versions of their real-time analytics systems.

 In both cases, the motivation was relatively similar — they wanted to provide their customers with better insights on the performance and effectiveness of their marketing activities. Facebook’s measurement includes “likes” and “comments” to monitor interactions. For Twitter, the measurement is based on the effectiveness of a given tweet – typically called “Reach” – basically a measure of the number of followers that were exposed to the tweet. Beyond the initial exposure, you often want to measure the number of clicks on that tweet, which indicate the number of users who saw the tweet and also looked into its content.

Facebook’s vs Twitter’s Approach to real-time Analytics 

Facebook Real-Time Analytics Architecture – Logging-Centric Approach:

  Relies on Apache Hadoop framework for real-time and batch (map/reduce) processing. Using the same underlying system simplifies the maintenance of that system.

• Limited real-time processing — the logging-centric approach basically delegates most of the heavy lifting to the backend system. Performing even a fairly simple correlation beyond simple counting isn’t a trivial task.

• real-time is often measured in tens of seconds. In many analytics system, this order of magnitude is often more than enough to express a real-time view of what is going on in your system.

• It is suitable for simple processing. Because of the logging nature of the Facebook architecture, most of the heavy lifting of processing cannot be done in real-time and is often pushed into the backend system.

• Low parallelization — Hadoop systems do not give you ways to ensure ordering and consistency based on the data. Because of that, Facebook came up with their Puma service that collects and inputs data into a centralized service, thus making it easier to processes events in order.

• Facebook collects user click streams from your Facebook wall through an Ajax listener which, then, sends those events back into the Facebook data centers. The info is stored on Hadoop File System via Scribe and collected by PTail.

• Puma aggregates logs in-memory, batching them in windows of 1.5 seconds and stores the information in Hbase. 

• The Facebook approach puts a huge limit as to the volume of events that the system can handle and have significant implications over the utilization of the overall system.

 

Twitter Real-Time Analytics Architecture – Event-Driven Approach:

• Unlike Facebook, Twitter uses Hadoop for batch processing and Storm for real-time processing. Storm was designed to perform fairly complex aggregation of the data that comes through the stream as it flows into the system, before it is sent back to the batch system for further analysis.

• Real-time can be measured in milliseconds. While having second or millisecond latency is not crucial to the end user — it does have a significant effect on the overall processing time and the level of analysis that we can produce and push through the system. As many of those analyses involve thousands of operations to get to the actual result.

• It is suitable for complex processing. With Storm, it is possible to perform a large range of complex aggregation while the data flows through the system. This has a significant impact on the complexity of the processing. A good example is calculating trending words. With the event-driven approach, we can assume that we have the current state and just make the change to that state to update the list of trending words. In contrast, a batch system will have to read the entire set of words, re-calculate, and re-order the words for every update. This is why those operations are often done in long batches.

• Extremely parallel – Asynchronous events are, by definition, easier to parallelize. Storm was designed for extreme parallelization. Ultimately, it determines the speed level of utilization that we can get per machine in our system. Looking at the bigger picture, this quite substantially adds to the cost of our system and to our ability to perform complex analyses.

Final Words

 Quite often, we get caught in the technical details of these discussions and lose sight of what this all really means.

If all you are looking for is to collect data streams and simply update counters, then both approaches would work. The main difference between the two is felt in the level and complexity of processing that you would like to process in real-time. If you want to continuously update a different form of sorted lists or indexes, you’ll find that doing so in an event-driven approach, as is the case of Twitter, can be exponentially faster and more efficient than the logging-centric approach. To put some numbers behind that, Twitter reported that calculating the reach without Storm took 2 hours whereas Storm could do the same in less than a second.

Such a difference in speed and utilization have a direct correlation with the business bottom line, as it determines the level and depth of intelligence that it can run against its data. It also determines the cost of running the analytics systems and, in some cases, the availability of those systems. When the processing is slower there would be larger number of scenarios that could saturate the system.

References:

• Slides: real-time Big Data with Storm, Cassandra and In Memory Computing
• Video: Real-time Analytics for Big Data: A Facebook Case Study
• Video: C* Summit 2013: Real-Time Big Data with Storm, Cassandra, and In Memory Computing
• Nathan Marz (Twitter) on Storm